A model inference method based on intelligent resource optimization and related equipment

Abstract

This invention relates to the field of artificial intelligence model inference technology, specifically disclosing a model inference method and related equipment based on intelligent resource optimization. This invention utilizes a resource manager instance to perform real-time monitoring and multi-dimensional scoring scheduling of heterogeneous device resources, establishes a dynamic management mode for pre-allocated memory pools, and achieves zero-copy data transmission through file mapping. Simultaneously, it employs an asynchronous parallel pipeline processing method for multi-model serial tasks, overcoming the limitations of existing inference frameworks in cross-platform device intelligent selection, fragmented memory allocation, data migration overhead between hosts and devices, and serial execution of complex inference processes. This improves the utilization rate of heterogeneous computing resources and the concurrent processing capability of inference tasks, reduces end-to-end inference latency and memory allocation overhead, and enhances the throughput and overall execution efficiency of multi-model collaborative inference.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model reasoning technology, and in particular to a model reasoning method and related equipment based on intelligent resource optimization. Background Technology

[0002] In the field of deep learning model deployment, performance optimization of model inference frameworks has always been a key focus of technological research and development. Existing mainstream inference frameworks mainly include TensorRT, OpenVINO, and TVM. Although these frameworks perform well in specific scenarios, they still have significant limitations when handling large-scale models and complex inference tasks.

[0003] TensorRT, a high-performance inference optimization library from NVIDIA, optimizes model inference performance through techniques such as layer fusion and precision calibration. However, TensorRT is primarily optimized for NVIDIA GPU hardware, lacking cross-platform compatibility and offering limited support for multi-device collaborative processing, making it difficult to adapt to the needs of heterogeneous computing environments. OpenVINO, an inference toolkit developed by Intel, supports various Intel hardware platforms such as CPUs, GPUs, and VPUs. It uses a model optimizer to convert trained models into intermediate representations, which are then executed by the inference engine. While OpenVINO boasts good cross-Intel hardware compatibility, there is still room for improvement in heterogeneous device management and resource scheduling, failing to fully leverage the performance advantages of hybrid computing architectures. TVM is an end-to-end deep learning compilation stack that supports various deep learning frameworks and hardware backends. It generates efficient inference code through automatic scheduling and optimization techniques, offering good flexibility and scalability. However, its learning curve is steep, and significant manual tuning is required during actual deployment, increasing the complexity of engineering implementation.

[0004] Currently, the technologies closest to this invention mainly include NVIDIA's Multi-Process Service (MPS) and Google's TensorFlow Serving. NVIDIA MPS allows multiple CUDA processes to share the same GPU context, reducing context switching overhead through centralized scheduling and resource sharing mechanisms, thereby improving GPU utilization and concurrent processing capabilities. However, MPS is limited to the NVIDIA GPU platform, and its load balancing mechanism in multi-GPU systems is not intelligent enough to dynamically adjust resource allocation according to actual task requirements. TensorFlow Serving, developed by Google, is a machine learning model service system focused on model deployment in production environments, supporting model version management, batch processing optimization, and A / B testing, making it suitable for large-scale distributed deployment scenarios. However, TensorFlow Serving is relatively simple in its underlying hardware resource management, lacking fine-grained device selection and memory management mechanisms, making it difficult to meet the high resource optimization requirements of complex inference tasks.

[0005] In addition, some research has proposed relevant solutions. For example, Microsoft's NNFusion optimizes the execution efficiency of neural networks through unified intermediate representations and cross-platform code generation technology; Alibaba's Pangu improves the performance of large-scale model training and inference through distributed scheduling and elastic scaling mechanisms. However, these solutions mainly focus on single-dimensional optimization and still have shortcomings in comprehensive resource management and performance optimization.

[0006] In summary, existing technologies still have significant shortcomings in areas such as intelligent allocation of device resources, efficient memory management, data transmission optimization, and complex inference process orchestration. A more intelligent and efficient model inference method is urgently needed to solve these problems. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a model reasoning method and related equipment based on intelligent resource optimization.

[0008] In a first aspect, the present invention provides a model inference method based on intelligent resource optimization, the technical solution of which is as follows: Initialize the inference environment, load the model file in the inference environment, and create a resource manager instance; The resource manager instance is used to monitor available device resources in real time, and to obtain the memory usage, number of running tasks and device utilization of each available device. Based on the multi-dimensional device scoring algorithm, the memory usage rate, the number of running tasks, and the device utilization rate of each available device are weighted and summed according to preset weights to obtain a comprehensive score, and the available device with the best comprehensive score is selected as the target inference device; The target inference device maintains a list of free memory blocks in a pre-allocated memory pool. When a memory allocation request is received, a suitable free memory block is first searched in the list of free memory blocks. If no suitable free memory block is found, a new memory block is allocated. The memory pool is automatically expanded when the memory pool capacity is insufficient. The system receives input data to be inferred, maps the input data to memory space using file mapping, identifies the memory location of the input data, directly calls the input data to execute model inference when the input data is located in device memory, and allocates device memory through the memory pool and migrates the input data to device memory when the input data is located in host memory. The model inference calculation is performed based on the input data in the target inference device to obtain the inference result; When the inference task is a multi-model concatenation task, the multi-model concatenation task is divided into multiple stages, with each stage corresponding to one model. The inference operations of each stage on their respective batches of data are executed in parallel through asynchronous processing, and the output results of each stage are merged into the final inference result.

[0009] The beneficial effects of the model inference method based on intelligent resource optimization of the present invention are as follows: The method of this invention performs real-time monitoring and multi-dimensional scoring scheduling of heterogeneous device resources by deploying resource manager instances, establishes a dynamic management mode of pre-allocated memory pools and achieves zero-copy data transmission with file mapping, and adopts an asynchronous parallel pipeline processing method for multi-model serial tasks. This solves the limitations of existing inference frameworks in cross-platform device intelligent selection, fragmented memory allocation, data migration overhead between hosts and devices, and serial execution of complex inference processes. It improves the utilization of heterogeneous computing resources and the concurrent processing capability of inference tasks, reduces end-to-end inference latency and memory allocation overhead, and improves the throughput and overall execution efficiency of multi-model collaborative inference.

[0010] Based on the above scheme, the model reasoning method based on intelligent resource optimization of the present invention can be further improved as follows.

[0011] In one alternative approach, the steps of initializing the inference environment, loading the model file in the inference environment, and creating a resource manager instance include: Call the device count acquisition interface to determine the total number of available devices, and construct a device information mapping table based on the device identifier information of each available device; The resource manager instance is created based on the device information mapping table. During the creation of the resource manager instance, the memory pool mapping table is initialized to establish the association between each available device and the corresponding memory pool. The model file is loaded through the environment manager in the inference environment, and the storage path of the model file is associated with the model context and cached.

[0012] The advantages of adopting the above optional method are as follows: by further constructing a device information mapping table and an initialization memory pool mapping table, the association between available devices and the memory pool is established, and the model file storage path is associated with the model context for caching, thereby improving the completeness of the inference environment initialization and the response speed of subsequent resource scheduling.

[0013] In one alternative approach, the step of monitoring available device resources in real time through the resource manager instance to obtain the memory usage, number of running tasks, and device utilization of each available device includes: A monitoring thread is started through the resource manager instance, and the monitoring thread calls the resource monitoring function at preset time intervals; The resource monitoring function iterates through all available devices in the device information mapping table, queries the current used memory capacity and total memory capacity for each available device and calculates the memory utilization rate, counts the number of inference tasks being executed on each available device to obtain the number of running tasks, and collects the activity time ratio of the corresponding computing unit in each available device to obtain the device utilization rate. Update the memory usage, number of running tasks, and device utilization of each available device to the device status record maintained by the resource manager instance.

[0014] The advantages of adopting the above optional approach are as follows: by periodically calling the resource monitoring function through the monitoring thread, all available devices are traversed and memory usage, number of running tasks and device utilization are collected, and the collected data is updated to the device status record, which enhances the real-time nature of device status awareness and the data support capability for resource scheduling decisions.

[0015] In one optional approach, the step of obtaining a comprehensive score by weighting and summing the memory usage, number of running tasks, and device utilization of each available device according to a multi-dimensional device scoring algorithm, and selecting the available device with the best comprehensive score as the target inference device, includes: Read the current memory usage, number of running tasks, and device utilization of each available device from the device status record; For each available device, the memory utilization rate is multiplied by a first preset weight, the number of running tasks is multiplied by a second preset weight, and the device utilization rate is multiplied by a third preset weight, and then summed to obtain a comprehensive score for each available device, wherein the sum of the first preset weight, the second preset weight, and the third preset weight is a fixed value; The overall scores of all available devices are compared, and the available device with the lowest overall score is determined as the target inference device.

[0016] The beneficial effects of adopting the above optional method are as follows: by further calculating the comprehensive score of each available device by weighting and summing the memory usage rate, the number of running tasks and the device utilization rate according to the preset weights, the available device with the smallest comprehensive score value is determined as the target inference device, thereby optimizing the judgment accuracy of heterogeneous device load balancing and the rationality of target inference device selection.

[0017] In one alternative approach, the step of maintaining a list of free memory blocks in a pre-allocated memory pool in the target inference device, and when a memory allocation request is received, first searching for a suitable free memory block in the list of free memory blocks; if no suitable free memory block is found, allocating a new memory block, and automatically expanding the memory pool when its capacity is insufficient, includes: During the initialization phase of the target inference device, a fixed-size device memory block is pre-allocated to the memory pool corresponding to the target inference device, and a list of free memory blocks is established to record information on reusable free memory blocks. When the resource manager instance receives a memory allocation request, it traverses the list of free memory blocks to find a free memory block with a capacity not less than the requested capacity. If a free memory block with a capacity not less than the requested capacity is found, the found free memory block is marked as used and returned. If no free memory block with a capacity not less than the requested capacity is found in the list of free memory blocks, the memory allocation interface is called to allocate a new memory block to the memory pool, and the allocated new memory block is marked as used. If the memory pool capacity has reached the preset upper limit and still cannot meet the memory allocation request during the allocation of new memory blocks, the upper limit of the memory pool capacity will be automatically expanded before allocating new memory blocks.

[0018] The advantages of adopting the above optional method are as follows: by pre-allocating fixed-size device memory blocks and establishing a list of free memory blocks, when a memory allocation request is received, a suitable free memory block is searched first. If no suitable block is found, a new memory block is allocated, and the memory pool is automatically expanded when the capacity is insufficient, thereby reducing the degree of memory fragmentation and the overhead of repeated allocation.

[0019] In one alternative approach, the steps of receiving input data to be inferred, mapping the input data to memory space using file mapping, identifying the memory location of the input data, directly calling the input data to perform model inference when the input data is located in device memory, and allocating device memory through the memory pool and migrating the input data to device memory when the input data is located in host memory include: Receive input data to be inferred, wherein the input data includes file path information; The file reading component maps the file content corresponding to the file path information to the memory space, and uses the input data mapped to the memory space as the input data to be processed. The input data to be processed is preprocessed to convert it into the input format required for model inference; Before creating the model input dataset, check the memory location of the preprocessed input data. If the memory location is the device memory of the target inference device, call the input data in the device memory to perform model inference. When the memory location is host memory, the resource manager instance requests and allocates device memory from the memory pool corresponding to the target inference device, and then migrates the pre-processed input data from the host memory to the allocated device memory before performing model inference.

[0020] The advantages of using the above optional method are as follows: the input data is further loaded into the memory space and its location is identified through file mapping. When the input data is in the device memory, the inference is executed directly. When the input data is in the host memory, the device memory is allocated through the memory pool and the data is migrated before the inference is executed. This eliminates unnecessary data copying operations and shortens the processing latency of the data preparation stage.

[0021] In one alternative approach, the step of performing model inference calculations based on the input data in the target inference device to obtain an inference result includes: Obtain the model context corresponding to the model file through the environment manager, and load the model context on the target inference device; Based on the loaded model context, a model input dataset and a model output buffer are created, and the pre-processed input data is filled into the model input dataset. The model inference execution interface is invoked to input the model input dataset into the target inference device for computation, thereby obtaining the model output data. The model output data is read from the model output buffer and returned as the inference result.

[0022] The advantages of using the above optional method are as follows: the model context is further obtained through the environment manager and the input dataset and output buffer are created. The pre-processed input data is filled into the model input dataset, the model inference execution interface is called to perform calculations and the results are read from the output buffer, thus ensuring the standardization of the inference calculation process and the integrity of the inference result output.

[0023] Secondly, the present invention provides a model inference system based on intelligent resource optimization, the technical solution of which is as follows: Create a module to initialize the inference environment, load the model file in the inference environment, and create a resource manager instance; The acquisition module is used to monitor available device resources in real time through the resource manager instance, and to acquire the memory usage, number of running tasks and device utilization of each available device. The selection module is used to calculate a comprehensive score by weighting the memory usage rate, the number of running tasks, and the device utilization rate of each available device according to a multi-dimensional device scoring algorithm, and then selecting the available device with the best comprehensive score as the target inference device. The processing module is used to maintain a list of free memory blocks in the pre-allocated memory pool in the target inference device. When a memory allocation request is received, it first searches for a suitable free memory block in the list of free memory blocks. If no suitable free memory block is found, a new memory block is allocated. The memory pool is automatically expanded when the memory pool capacity is insufficient. The running module is used to receive input data to be inferred, map the input data to memory space using file mapping, identify the memory location of the input data, directly call the input data to execute model inference when the input data is located in device memory, and allocate device memory through the memory pool and migrate the input data to device memory when the input data is located in host memory. The inference module is used to perform model inference calculations based on the input data in the target inference device to obtain inference results; The merging module is used to divide the multi-model concatenation task into multiple stages when the inference task is a multi-model concatenation task, with each stage corresponding to one model. The inference operations of each stage on its respective batch of data are executed in parallel through asynchronous processing, and the output results of each stage are merged into the final inference result.

[0024] The beneficial effects of the model inference system based on intelligent resource optimization of the present invention are as follows: The system of this invention performs real-time monitoring and multi-dimensional scoring scheduling of heterogeneous device resources by deploying resource manager instances, establishes a dynamic management mode of pre-allocated memory pools and achieves zero-copy data transmission with file mapping, and adopts an asynchronous parallel pipeline processing method for multi-model serial tasks. This solves the limitations of existing inference frameworks in cross-platform device intelligent selection, fragmented memory allocation, data migration overhead between hosts and devices, and serial execution of complex inference processes. It improves the utilization of heterogeneous computing resources and the concurrent processing capability of inference tasks, reduces end-to-end inference latency and memory allocation overhead, and improves the throughput and overall execution efficiency of multi-model collaborative inference.

[0025] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the model inference method based on intelligent resource optimization as described in this invention.

[0026] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the model inference method based on intelligent resource optimization of the present invention.

[0027] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0028] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of a model reasoning method based on intelligent resource optimization according to the present invention. Figure 2 This is a schematic diagram of the overall architecture of the model inference framework; Figure 3 This is a schematic diagram of the intelligent resource management process; Figure 4 This is a schematic diagram of the memory pool management process; Figure 5 This is a schematic diagram of the zero-copy data transfer process; Figure 6 This is a schematic diagram of the pipeline execution process; Figure 7This is a schematic diagram of the structure of an embodiment of a model inference system based on intelligent resource optimization according to the present invention; Figure 8 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0029] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0030] Figure 1 This diagram illustrates a flowchart of an embodiment of a model inference method based on intelligent resource optimization provided by the present invention. This model inference method can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the model inference method based on intelligent resource optimization by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps: S1. Initialize the inference environment, load the model file in the inference environment and create a resource manager instance.

[0031] The inference environment refers to the software runtime context used to perform model inference tasks, including loaded hardware drivers, model runtime libraries, and system configuration parameters. For example, in a computer equipped with both a central processing unit (CPU) and a neural network processor, initializing the inference environment includes loading the driver and inference library for processor A, preparing the basic runtime conditions for subsequent model loading and resource scheduling. The model file refers to the data file storing the deep learning model structure and weight parameters, which can be directly used for inference computation after offline training or conversion. For example, a file named "model_face.om" stores the network layer definitions and trained weight values ​​of a face recognition model; loading this file into the inference environment allows for feature extraction from face images. The resource manager instance refers to a singleton object created in the inference environment, responsible for uniformly managing information from multiple available devices, monitoring device status, and allocating the optimal device for inference requests. For example, during initialization, the resource manager constructor is called to generate a globally unique manager instance. This instance internally maintains a device information mapping table and a memory pool mapping table; all subsequent device selection and memory operations are coordinated through this instance.

[0032] S2. Monitor available device resources in real time through the resource manager instance to obtain the memory usage, number of running tasks and device utilization of each available device.

[0033] Available device resources refer to the set of hardware devices in the inference environment that can perform computational tasks, including various processors and associated memory resources. For example, processor A, processor B and the host CPU installed in a computer together constitute available device resources. Each processor can independently execute model inference and has its own dedicated memory.

[0034] Memory utilization refers to the percentage of memory currently used on a specific device relative to its total memory capacity, measuring the activity level of the device's memory resources. For example, if processor A has 8 GB of dedicated memory, and 3 GB is currently used for model input data and intermediate feature maps, the memory utilization rate is calculated as 3 divided by 8, which equals 37.5%. The number of running tasks refers to the number of inference tasks currently being executed concurrently on a specific device, reflecting the instantaneous level of the device's computational load. For example, if two face recognition inference tasks and one attribute analysis inference task are running simultaneously on processor A, the number of running tasks is 3. Device utilization refers to the proportion of time that computing units on a specific device are active within the sampling period, reflecting the intensity of the device's computing resource utilization. For example, in the most recent 1-second monitoring period, processor A's AI computing cores were in operation for 600 ms, and the device utilization rate is calculated as 600 divided by 1000, which equals 60%.

[0035] S3. Based on the multi-dimensional device scoring algorithm, the memory usage rate, the number of running tasks, and the device utilization rate of each available device are weighted and summed according to preset weights to obtain a comprehensive score, and the available device with the best comprehensive score is selected as the target inference device.

[0036] The overall score refers to the weighted sum of memory usage, number of running tasks, and device utilization based on a multi-dimensional device scoring algorithm. A lower score indicates a lighter current device load. For example, processor A has a memory usage of 37.5%, 3 running tasks, and a device utilization of 60%. Calculated using a first preset weight of 0.4, a second preset weight of 0.3, and a third preset weight of 0.3, the overall score is 0.375 × 0.4 + 3 × 0.3 + 0.6 × 0.3 = 1.23. The target inference device is the available device with the best overall score selected after multi-dimensional scoring comparison. This device will carry out the execution of the inference task. For example, processor B has a memory usage of 20%, 1 running task, and a device utilization of 30%. Its overall score is 0.2 × 0.4 + 1 × 0.3 + 0.3 × 0.3 = 0.47, lower than processor A's 1.23. Therefore, processor B is identified as the target inference device.

[0037] S4. Maintain a list of free memory blocks in the pre-allocated memory pool in the target inference device. When a memory allocation request is received, first search for a suitable free memory block in the list of free memory blocks. If no suitable free memory block is found, allocate a new memory block. Automatically expand the memory pool when the memory pool capacity is insufficient.

[0038] The free memory block list refers to a data structure maintained in the memory pool that records all unused and available memory blocks for reallocation. For example, when pre-allocating a memory pool for a target inference device, a vector container is created, recording free memory blocks with capacities of 64 MB, 128 MB, and 256 MB, along with their starting addresses. A suitable free memory block is a free memory block found in the free memory block list whose capacity is greater than or equal to the memory allocation request. For example, when an inference task needs to allocate 96 MB of memory to store input feature maps, traversing the free memory block list reveals a 128 MB free memory block that meets the request; this 128 MB free memory block is the suitable free memory block.

[0039] The memory pool capacity refers to the maximum amount of memory that the memory pool pre-allocated to the target inference device can currently provide. For example, the memory pool of the target inference device is initially set to a preset upper limit of 4 GB. After two automatic expansions, the upper limit is adjusted to 8 GB, and 8 GB is the current memory pool capacity.

[0040] S5. Receive the input data to be inferred, map the input data to the memory space using file mapping, identify the memory location of the input data, directly call the input data to perform model inference when the input data is located in the device memory, and allocate device memory through the memory pool and migrate the input data to the device memory when the input data is located in the host memory.

[0041] Input data refers to the raw data to be fed into the model for inference calculations. This can be external data such as images, text, and speech that have not been processed by the inference framework. For example, a frontal photo of a person taken by a camera and stored on a disk in JPEG format, with the filename "face.jpg", is input data. Memory space refers to the address range in a computer system used to store data and programs, including host-side memory space and device-side memory space. For example, after mapping the contents of the "face.jpg" file to the virtual address space of a process through file mapping operations, this virtual address range constitutes memory space for subsequent preprocessing and inference access.

[0042] In this context, "memory location" refers to the physical storage area where the input data currently resides. It's used to determine if the data is already in the device's memory to avoid redundant copying. For example, if the pre-processed input data's address is found to be within the host's dynamic random access memory (DRAM) range, then the memory location is determined to be host memory. "Device memory" refers to dedicated physical storage space on the target inference device, characterized by high bandwidth and low latency, specifically used to accelerate data access during computation. For example, the 8 GB high-bandwidth memory integrated on processor B is device memory and can be directly accessed at high speed by the AI ​​core within processor B.

[0043] Model inference refers to the process of feeding input data into a loaded model for computation and outputting prediction results. For example, preprocessed facial image data is fed into a facial recognition model, which outputs a 512-dimensional feature vector after a series of convolution and fully connected operations. This complete computation process is model inference. Host memory refers to the system's main memory, directly managed by the central processing unit (CPU), and is separate from device memory. Devices access host memory through a data bus. For example, the 32 GB Double Data Rate Synchronous Dynamic Random Access Memory (DRAM) installed on a computer constitutes host memory, and the input data after file mapping is initially stored in host memory.

[0044] S6. Perform model inference calculations based on the input data in the target inference device to obtain the inference result.

[0045] The inference result refers to the output data generated after the model completes its inference calculation. It can be in the form of classification labels, detection box coordinates, feature vectors, or text sequences. For example, after the face recognition model infers from the input photo, it outputs a 512-dimensional floating-point array as a face feature vector, which is the inference result.

[0046] S7. When the inference task is a multi-model concatenation task, the multi-model concatenation task is divided into multiple stages, with each stage corresponding to a model. The inference operations of each stage on their respective batches of data are executed in parallel through asynchronous processing, and the output results of each stage are merged into the final inference result.

[0047] Multi-model concatenation tasks refer to composite inference tasks where multiple models are executed sequentially according to their dependencies, with the output of one model serving as the input of the next. For example, a complete face analysis process first uses a face detection model to locate face regions in a photo, then a face recognition model to extract identity features, and finally an attribute analysis model to determine age and gender. These three models are concatenated to form a multi-model concatenation task. Asynchronous processing refers to executing multiple operations in parallel through non-blocking calls and callback mechanisms, allowing computational and data transmission tasks to overlap and improve overall throughput. For example, while the first batch of face detections is being processed in stage one of the pipeline execution, stage two has already begun extracting features from the previous batch of detection results. Each stage achieves batch parallelism through asynchronous flow management.

[0048] The final inference result refers to the complete inference conclusion obtained by merging the output results of each stage in the multi-model chain task according to a predetermined logic. For example, the face detection model outputs the coordinates of the face bounding box, the face recognition model outputs a 512-dimensional feature vector, and the attribute analysis model outputs the age of 25 and the gender label of female. Merging the three outputs into a structured data object constitutes the final inference result.

[0049] The technical solution of this embodiment performs real-time monitoring and multi-dimensional scoring scheduling of heterogeneous device resources by deploying resource manager instances, establishes a dynamic management mode of pre-allocated memory pools and achieves zero-copy data transmission with file mapping, and adopts an asynchronous parallel pipeline processing method for multi-model serial tasks. This solves the limitations of existing inference frameworks in cross-platform device intelligent selection, memory fragmentation allocation, data migration overhead between hosts and devices, and serial execution of complex inference processes. It improves the utilization rate of heterogeneous computing resources and the concurrent processing capability of inference tasks, reduces end-to-end inference latency and memory allocation overhead, and improves the throughput and overall execution efficiency of multi-model collaborative inference.

[0050] In one alternative approach, S1 specifically includes: The device quantity acquisition interface is called to determine the total number of available devices, and a device information mapping table is constructed based on the device identifier information of each available device.

[0051] The device quantity acquisition interface refers to the application programming interface function provided by the inference framework for querying the number of available computing devices in the current system. For example, calling the `aclrtGetDeviceCount` function returns the number of processors detected in the computer; a return value of 2 indicates that there are two available processors in the system. Device identification information refers to the number or name used to uniquely identify available devices. For example, the device identification information for processor A is the integer 0, and the device identification information for processor B is the integer 1. When traversing all available devices, 0 and 1 are obtained sequentially as identifiers.

[0052] The device information mapping table refers to a data structure with device identification information as the key and device details as the value, used to quickly query the attributes and status of each device. For example, the mapping table is constructed to map device identifier 0 to the device information structure of processor A, and device identifier 1 to the device information structure of processor B.

[0053] The resource manager instance is created based on the device information mapping table. During the creation of the resource manager instance, the memory pool mapping table is initialized to establish the association between each available device and the corresponding memory pool.

[0054] The memory pool mapping table refers to a data structure that establishes the association between available devices and corresponding memory pool instances, used to quickly locate memory pools based on device identifiers. For example, in the resource manager instance, a mapping table is maintained where a key of 0 points to the memory pool object of processor A, and a key of 1 points to the memory pool object of processor B.

[0055] The model file is loaded through the environment manager in the inference environment, and the storage path of the model file is associated with the model context and cached.

[0056] The storage path refers to the complete location string of the model file in the file system, used to locate and load the model; for example, the model file "model_face.om" is stored in the " / home / AI / models / " directory on the computer, with the storage path being " / home / AI / models / model_face.om". The model context refers to the runtime object generated on the target inference device after the model file is loaded, containing the model computation graph, a memory copy of the weights, and the environment handle required for execution; for example, after loading the storage path " / home / AI / models / model_face.om" through the environment manager, a model context is created on processor B, which encapsulates all the execution elements of the face recognition model.

[0057] In the above optional methods, the association between available devices and memory pools is further established by constructing a device information mapping table and an initialization memory pool mapping table, and the model file storage path is associated with the model context for caching, which improves the completeness of the inference environment initialization and the response speed of subsequent resource scheduling.

[0058] In one alternative approach, S2 specifically includes: A monitoring thread is started through the resource manager instance, and the monitoring thread calls the resource monitoring function at preset time intervals.

[0059] The monitoring thread refers to a background thread started by the resource manager instance, responsible for periodically executing resource status collection tasks. For example, the resource manager instance starts a thread named "ResourceMonitor," which is woken up every 100ms to call the resource monitoring function. The preset time interval refers to the fixed waiting time between two executions of the resource monitoring function by the monitoring thread. For example, if the preset time interval is configured to 100ms, the monitoring thread will sleep for 100ms after each execution of the resource monitoring function before entering the next loop.

[0060] Among them, the resource monitoring function refers to the specific functional function called by the monitoring thread, which is used to traverse all available devices and collect memory usage, number of running tasks and device utilization. For example, the resource monitoring function traverses the device identifiers 0 and 1 in the device information mapping table in a loop, and calls the query interface for each device to obtain memory usage, task count and computing core active time.

[0061] The resource monitoring function iterates through all available devices in the device information mapping table, queries the current used memory capacity and total memory capacity for each available device, calculates the memory utilization rate, counts the number of inference tasks being executed on each available device to obtain the number of running tasks, and collects the activity time ratio of the corresponding computing unit in each available device to obtain the device utilization rate.

[0062] The currently used memory capacity refers to the amount of memory allocated and used by a specific available device at the time of this monitoring; for example, on processor A, the resource monitoring function query memory status interface returns 3072 MB of used memory. The total memory capacity refers to the total physical amount of memory for a specific available device; for example, processor A has a total memory capacity of 8192 MB.

[0063] The number of inference tasks currently being executed refers to the number of inference requests that are running but not yet completed on a specific available device. For example, the resource monitoring function determines, by counting the context queue length of processor A, that two face recognition tasks and one attribute analysis task are currently being executed, resulting in a total of 3 inference tasks. The activity time ratio of the computing unit refers to the percentage of time the AI ​​computing cores in the device are busy within the sampling period. For example, on processor A, the resource monitoring function calls the performance acquisition interface to obtain data showing that the AI ​​cores have been busy for 60 ms in the last 100 ms, indicating that the activity time ratio of the computing unit is 60%.

[0064] Update the memory usage, number of running tasks, and device utilization of each available device to the device status record maintained by the resource manager instance.

[0065] In the above-mentioned optional methods, the resource monitoring function is further called periodically by the monitoring thread to traverse all available devices and collect memory usage, number of running tasks and device utilization. The collected data is then updated to the device status record, which enhances the real-time nature of device status awareness and the data support capability for resource scheduling decisions.

[0066] In one alternative approach, S3 specifically includes: Read the current memory usage, number of running tasks, and device utilization of each available device from the device status record.

[0067] For each available device, the memory utilization rate is multiplied by a first preset weight, the number of running tasks is multiplied by a second preset weight, and the device utilization rate is multiplied by a third preset weight, and then summed to obtain a comprehensive score for each available device, wherein the sum of the first preset weight, the second preset weight, and the third preset weight is a fixed value.

[0068] The first preset weight refers to the weighting coefficient assigned to memory usage in the multi-dimensional device scoring algorithm. For example, setting the first preset weight to 0.4 in the comprehensive scoring formula means that memory usage accounts for 40% of the device selection decision. The second preset weight refers to the weighting coefficient assigned to the number of running tasks in the multi-dimensional device scoring algorithm. For example, setting the second preset weight to 0.3 in the comprehensive scoring formula means that the number of running tasks accounts for 30% of the device selection decision. The third preset weight refers to the weighting coefficient assigned to device utilization in the multi-dimensional device scoring algorithm. For example, setting the third preset weight to 0.3 in the comprehensive scoring formula means that device utilization accounts for 30% of the device selection decision.

[0069] It should be noted that each preset weight can be dynamically adjusted based on the inference task type or historical load data; for example, for memory-intensive inference tasks, the value of the first preset weight can be increased; for computationally intensive inference tasks, the value of the third preset weight can be increased, thereby matching the device selection strategy with the task characteristics.

[0070] The overall scores of all available devices are compared, and the available device with the lowest overall score is determined as the target inference device.

[0071] In the above-mentioned optional methods, the comprehensive score of each available device is calculated by weighting and summing memory usage, number of running tasks and device utilization according to preset weights, and the available device with the smallest comprehensive score is determined as the target inference device, thereby optimizing the judgment accuracy of heterogeneous device load balancing and the rationality of target inference device selection.

[0072] In one alternative approach, S4 specifically includes: During the initialization phase of the target inference device, a fixed-size device memory block is pre-allocated to the memory pool corresponding to the target inference device, and a list of free memory blocks is established to record information on free memory blocks that can be reused.

[0073] In this context, a device memory block refers to a contiguous storage area allocated within device memory, serving as the basic unit for memory pool management. For example, a 256 MB device memory block is pre-allocated to the memory pool of the target inference device, with a starting address of 0x800000000. Free memory block information refers to the attribute data recorded in the free memory block list for each free memory block, including at least the block size and starting address. For instance, an entry in the free memory block list might record a size of 128 MB and a starting address of 0x900000000; this information constitutes the descriptive data for the free memory block.

[0074] When the resource manager instance receives a memory allocation request, it traverses the list of free memory blocks to find a free memory block with a capacity not less than the requested capacity. If a free memory block with a capacity not less than the requested capacity is found, the found free memory block is marked as used and returned.

[0075] Among them, a memory allocation request refers to a request for memory issued to the resource manager instance during the inference process. The request content includes the required memory capacity. For example, when creating the model input dataset, the framework initiates a memory allocation request to the resource manager instance, requesting a capacity of 96 MB to store the preprocessed input image data.

[0076] If no free memory block with a capacity not less than the requested capacity is found in the list of free memory blocks, the memory allocation interface is called to allocate a new memory block to the memory pool, and the allocated new memory block is marked as used.

[0077] The memory allocation interface refers to the application programming interface functions that interact with the underlying hardware driver to actually allocate memory; for example, when there is no matching block in the list of free memory blocks, the aclrtMalloc function is called to request the allocation of new memory of size 128 MB from the driver.

[0078] If the memory pool capacity has reached the preset upper limit and still cannot meet the memory allocation request during the allocation of new memory blocks, the upper limit of the memory pool capacity will be automatically expanded before allocating new memory blocks.

[0079] The upper limit of the memory pool capacity can be dynamically calculated based on the total physical memory of the target inference device and the current amount of free memory in the system, so as to avoid frequent expansion due to setting the upper limit too low or waste of memory resources due to setting the upper limit too high.

[0080] In the above optional methods, by pre-allocating fixed-size device memory blocks and establishing a list of free memory blocks, when a memory allocation request is received, a suitable free memory block is searched first. If no suitable block is found, a new memory block is allocated, and the memory pool is automatically expanded when the capacity is insufficient, thereby reducing the degree of memory fragmentation and the overhead of repeated allocation.

[0081] In one alternative approach, S5 specifically includes: Receive input data to be inferred, the input data including file path information.

[0082] The file path information refers to the complete path string used to locate the file when the input data exists in the form of a file; for example, the received input data to be inferred contains the file path information " / data / images / face.jpg", indicating the specific location of the original image in the file system.

[0083] The file reading component maps the file content corresponding to the file path information to the memory space, and uses the input data mapped to the memory space as the input data to be processed.

[0084] The file reading component refers to a functional unit in the inference framework specifically responsible for reading data from the file system, supporting efficient reading methods such as memory mapping. For example, the file reading singleton object provides the mmap file mapping function, which can directly map the contents of the file " / data / images / face.jpg" to the process's virtual address space.

[0085] The input data to be processed is preprocessed to convert it into the input format required for model inference.

[0086] The input format refers to the organization of the input data required for model inference and computation, including data dimensions, data types, and memory layout. For example, a face recognition model requires the input format to be a batch, three-channel, 112-pixel-high, 112-pixel-wide single-precision floating-point tensor. The format preprocessing step resizes the JPEG-decoded image and converts it into a tensor that meets the requirements.

[0087] Before creating the model input dataset, check the current memory location of the preprocessed input data. If the memory location is the device memory of the target inference device, call the input data in the device memory to perform model inference.

[0088] When the memory location is host memory, the resource manager instance requests and allocates device memory from the memory pool corresponding to the target inference device, and then migrates the pre-processed input data from the host memory to the allocated device memory before performing model inference.

[0089] In the above optional methods, the input data is further loaded into the memory space through file mapping and its location is identified. When the input data is in the device memory, the inference is executed directly. When the input data is in the host memory, the device memory is allocated through the memory pool and the data is migrated before the inference is executed. This eliminates unnecessary data copying operations and shortens the processing latency of the data preparation stage.

[0090] In one alternative approach, S6 specifically includes: Obtain the model context corresponding to the model file through the environment manager, and load the model context on the target inference device.

[0091] Based on the loaded model context, a model input dataset and a model output buffer are created, and the pre-processed input data is filled into the model input dataset.

[0092] The model output buffer refers to a pre-allocated memory area in memory used to store model output data. For example, after creating a model context, the total size of the output data is obtained by calling the get output size interface and a memory block of the corresponding size is allocated as the model output buffer. After inference is completed, the output feature vector is written to this buffer.

[0093] The model inference execution interface is invoked to pass the model input dataset to the target inference device for computation, thereby obtaining the model output data.

[0094] The model inference execution interface refers to the application programming interface function that initiates the actual computation process of the model, sends the input dataset to the device for execution, and triggers inference. For example, calling the almdlExecuteAsync function and passing in the model context handle, input dataset, and output buffer allows the interface to asynchronously start the forward computation of the face recognition model on the target inference device. Model output data refers to the computational result data generated in memory or the output buffer after the model inference computation is completed. For example, after the face recognition model finishes execution, a 512-dimensional single-precision floating-point array is obtained in the output buffer; this array is the model output data.

[0095] The model output data is read from the model output buffer and returned as the inference result.

[0096] In the above optional methods, the model context is further obtained through the environment manager and an input dataset and output buffer are created. The pre-processed input data is filled into the model input dataset, the model inference execution interface is called to perform calculations and the results are read from the output buffer, thus ensuring the standardization of the inference calculation process and the integrity of the inference result output.

[0097] like Figure 2 As shown, the overall framework architecture comprises an application layer and an environment manager. Below the application layer is a resource manager, which in turn manages devices and memory pools. Three core functional modules—context management, pipeline execution, and file reading—work collaboratively. Context management is responsible for maintaining the model context lifecycle; pipeline execution supports multi-stage inference process orchestration; and file reading achieves efficient data loading through mmap file mapping. Meanwhile, sub-modules such as model management, stream management, and mmap file mapping provide underlying support capabilities. The resource manager, as the core scheduling unit, coordinates device selection, memory allocation, and context caching, forming a complete inference chain from the upper-layer application to the underlying hardware.

[0098] The intelligent resource manager is responsible for the unified management and intelligent allocation of device resources. Its main innovations include a multi-dimensional device scoring algorithm and dynamic resource monitoring. In the multi-dimensional device scoring algorithm, memory usage assessment monitors device memory usage to avoid memory overload, running task statistics balance the task load across devices, and device utilization calculation comprehensively considers the efficiency of device computing resource utilization. The comprehensive scoring mechanism calculates a comprehensive score for each device based on the aforementioned three dimensions and selects the optimal device. Dynamic resource monitoring monitors the resource usage of each device in real time, dynamically adjusts resource allocation strategies, and provides resource usage statistics.

[0099] The intelligent resource management process is implemented through a resource manager class, employing a multi-dimensional device scoring algorithm for device selection. For example... Figure 3 As shown, the process begins with device initialization. It retrieves the number of available devices by calling the device count acquisition interface, iterates through all devices, and initializes the device information structure. Next, it initializes the resource manager, creating a singleton instance and initializing the device information mapping table and memory pool mapping table. During the device scanning and monitoring phase, a monitoring thread periodically calls the resource monitoring method to continuously update device utilization information. In the inference request processing phase, when the environment manager receives an inference request, it checks for the existence of a cached context. The context cache check searches the context mapping table using the model file path. If a cached context exists, it is used directly; otherwise, a device selection request is initiated, calling the optimal device acquisition method and passing in the required memory size parameter. The multi-dimensional device scoring step calculates a comprehensive score for each device using the formula: Score = 0.4 × memory utilization + 0.3 × number of running tasks + 0.3 × device utilization. After sorting the comprehensive scores, the device with the lowest score (i.e., the lightest load) is selected as the optimal device. Finally, the optimal device identifier is returned for creating a new context instance.

[0100] Memory pool optimization technology improves memory management efficiency through pre-allocation, memory block reuse, automatic expansion, and intelligent cleanup mechanisms. The pre-allocation mechanism pre-allocates large blocks of memory during system initialization, reducing runtime memory allocation overhead. Memory block reuse maintains a list of free memory blocks, prioritizing the use of already released memory blocks to avoid frequent memory allocation and deallocation. The automatic expansion mechanism automatically expands the memory pool when memory is insufficient, dynamically adjusting its size based on usage. The intelligent cleanup mechanism periodically cleans up long-unused memory blocks, releasing system resources and preventing memory leaks.

[0101] The memory pool management process is implemented through a memory pool class, employing pre-allocation and block reuse mechanisms. For example... Figure 4 As shown, the process begins with memory pool initialization. A memory pool instance is created for each device in the resource manager instance, pre-allocating a memory pool of a specified size. During system initialization, large blocks of device memory are pre-allocated through the memory allocation interface to reduce runtime allocation overhead. Subsequently, a list of free blocks is maintained, using a free block vector to record released but not destroyed memory blocks. When memory allocation is needed, the memory allocation request is processed first, searching for a block of suitable size in the free block list. The free block search and marking step traverses the free block list, finding a block of matching size and marking it as used. If no suitable free block is found, a new memory block is created, and the new block creation method is called to allocate the new memory block. When memory is insufficient, the memory pool capacity is automatically expanded, with a maximum expansion limit of eight gigabytes. The free block cleanup step periodically cleans up long-unused free blocks, releasing system resources.

[0102] Zero-copy data transfer optimization reduces copy operations through device memory persistence, mmap file mapping, and memory location control. Device memory persistence allows data to be processed directly in device memory, avoiding data copying between the host and device. mmap file mapping uses memory mapping technology to achieve fast file reading, directly mapping the file to memory space, avoiding multiple copies required for traditional file reading. Memory location control provides three memory location control modes: device memory, host memory, and automatic, selecting the optimal memory location based on actual needs to maximize data processing efficiency.

[0103] Zero-copy data transfer is implemented using a file reader and a model processing class. For example... Figure 5 As shown, the process begins with input data preparation, which can be in the form of file paths or memory data. Then, a file reader reads the data using a singleton instance, supporting multiple data types. Next, mmap file mapping is performed, directly mapping the file to memory space via a memory mapping system call, avoiding multiple copies. The data preprocessing step preprocesses the mapped file data, converting it to the format required for model inference. The model processing class calls the "Create Input" method to create the model input dataset. A memory location check step determines whether the input data is already in device memory, avoiding unnecessary data copying. In zero-copy optimization, if the data is already in device memory, it is used directly without copying. If the data is in host memory, device memory is allocated through the resource manager's memory pool, and the data is copied.

[0104] A unified context management mechanism enables efficient inference execution through context sharing and device management. The context sharing mechanism uses a context cache based on model file paths, allowing the same model to share the same context instance, reducing the overhead of context creation and initialization. Device management integrates the resource manager's intelligent device selection, ensuring correct device context settings and supporting device operations in multi-threaded environments.

[0105] Pipeline execution optimization supports the orchestration of complex inference processes through a pipeline mechanism, including stage management, parallel execution, and asynchronous processing. Stage management decomposes complex inference processes into multiple stages and supports the management of dependencies between stages. Parallel execution supports parallel processing of multiple stages, improving overall inference throughput. Asynchronous processing provides an asynchronous inference interface, supporting non-blocking inference execution.

[0106] The execution steps of a pipeline are implemented through pipeline classes and pipeline manager classes. For example... Figure 6As shown, the pipeline is first initialized by creating a pipeline instance and initializing the flow manager for asynchronous processing. The stage management step adds pipeline stages using the add stage method, with each stage corresponding to a model. When adding a pipeline stage, the model path and context are set for each stage. The inference request processing receives input data and starts the pipeline execution flow. Pipeline execution executes each stage sequentially, supporting both synchronous and asynchronous execution modes. In asynchronous mode, stages one, two, and three can process different batches of data in parallel; asynchronous processing uses streams and callback mechanisms to achieve non-blocking inference execution. The result merging step combines the outputs of each stage into the final inference result and returns it.

[0107] Figure 7 A schematic diagram of an embodiment of a model inference system 200 based on intelligent resource optimization provided by the present invention is shown. Figure 7 As shown, the model inference system 200 based on intelligent resource optimization includes: Create module 201 to initialize the inference environment, load model files in the inference environment and create a resource manager instance; The acquisition module 202 is used to monitor available device resources in real time through the resource manager instance and acquire the memory usage, number of running tasks and device utilization of each available device. The selection module 203 is used to obtain a comprehensive score by weighting and summing the memory usage rate, the number of running tasks and the device utilization rate of each available device according to a multi-dimensional device scoring algorithm, and selecting the available device with the best comprehensive score as the target inference device. Processing module 204 is used to maintain a list of free memory blocks in the pre-allocated memory pool in the target inference device. When a memory allocation request is received, it first searches for a suitable free memory block in the list of free memory blocks. If no suitable free memory block is found, a new memory block is allocated. The memory pool is automatically expanded when the memory pool capacity is insufficient. The running module 205 is used to receive the input data to be inferred, map the input data to the memory space using file mapping, identify the memory location of the input data, directly call the input data to execute model inference when the input data is located in the device memory, and allocate device memory through the memory pool and migrate the input data to the device memory when the input data is located in the host memory. Inference module 206 is used to perform model inference calculations based on the input data in the target inference device to obtain inference results; The merging module 207 is used to divide the multi-model concatenation task into multiple stages when the inference task is a multi-model concatenation task, with each stage corresponding to a model, and to execute the inference operations of each stage on its respective batch of data in parallel through asynchronous processing, and merge the output results of each stage into the final inference result.

[0108] In an alternative embodiment, the creation module 201 is specifically used for: Call the device count acquisition interface to determine the total number of available devices, and construct a device information mapping table based on the device identifier information of each available device; The resource manager instance is created based on the device information mapping table. During the creation of the resource manager instance, the memory pool mapping table is initialized to establish the association between each available device and the corresponding memory pool. The model file is loaded through the environment manager in the inference environment, and the storage path of the model file is associated with the model context and cached.

[0109] In an alternative embodiment, the acquisition module 202 is specifically used for: A monitoring thread is started through the resource manager instance, and the monitoring thread calls the resource monitoring function at preset time intervals; The resource monitoring function iterates through all available devices in the device information mapping table, queries the current used memory capacity and total memory capacity for each available device and calculates the memory utilization rate, counts the number of inference tasks being executed on each available device to obtain the number of running tasks, and collects the activity time ratio of the corresponding computing unit in each available device to obtain the device utilization rate. Update the memory usage, number of running tasks, and device utilization of each available device to the device status record maintained by the resource manager instance.

[0110] In an alternative embodiment, the selection module 203 is specifically used for: Read the current memory usage, number of running tasks, and device utilization of each available device from the device status record; For each available device, the memory utilization rate is multiplied by a first preset weight, the number of running tasks is multiplied by a second preset weight, and the device utilization rate is multiplied by a third preset weight, and then summed to obtain a comprehensive score for each available device, wherein the sum of the first preset weight, the second preset weight, and the third preset weight is a fixed value; The overall scores of all available devices are compared, and the available device with the lowest overall score is determined as the target inference device.

[0111] In an alternative embodiment, the processing module 204 is specifically used for: During the initialization phase of the target inference device, a fixed-size device memory block is pre-allocated to the memory pool corresponding to the target inference device, and a list of free memory blocks is established to record information on reusable free memory blocks. When the resource manager instance receives a memory allocation request, it traverses the list of free memory blocks to find a free memory block with a capacity not less than the requested capacity. If a free memory block with a capacity not less than the requested capacity is found, the found free memory block is marked as used and returned. If no free memory block with a capacity not less than the requested capacity is found in the list of free memory blocks, the memory allocation interface is called to allocate a new memory block to the memory pool, and the allocated new memory block is marked as used. If the memory pool capacity has reached the preset upper limit and still cannot meet the memory allocation request during the allocation of new memory blocks, the upper limit of the memory pool capacity will be automatically expanded before allocating new memory blocks.

[0112] In an alternative embodiment, the operating module 205 is specifically used for: Receive input data to be inferred, wherein the input data includes file path information; The file reading component maps the file content corresponding to the file path information to the memory space, and uses the input data mapped to the memory space as the input data to be processed. The input data to be processed is preprocessed to convert it into the input format required for model inference; Before creating the model input dataset, check the memory location of the preprocessed input data. If the memory location is the device memory of the target inference device, call the input data in the device memory to perform model inference. When the memory location is host memory, the resource manager instance requests and allocates device memory from the memory pool corresponding to the target inference device, and then migrates the pre-processed input data from the host memory to the allocated device memory before performing model inference.

[0113] In an alternative embodiment, the inference module 206 is specifically used for: Obtain the model context corresponding to the model file through the environment manager, and load the model context on the target inference device; Based on the loaded model context, a model input dataset and a model output buffer are created, and the pre-processed input data is filled into the model input dataset. The model inference execution interface is invoked to input the model input dataset into the target inference device for computation, thereby obtaining the model output data. The model output data is read from the model output buffer and returned as the inference result.

[0114] It should be noted that the beneficial effects of the intelligent resource optimization-based model inference system 200 provided in the above embodiments are the same as those of the intelligent resource optimization-based model inference method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0115] The intelligent resource optimization-based model inference system 200 of the present invention can be a computer program (including program code) running on a computer device. For example, the intelligent resource optimization-based model inference system 200 of the present invention is an application software that can be used to execute the corresponding steps in the intelligent resource optimization-based model inference method of the present invention.

[0116] In some embodiments, the intelligent resource optimization-based model inference system 200 of the present invention can be implemented in a combination of hardware and software. As an example, the intelligent resource optimization-based model inference system 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the intelligent resource optimization-based model inference method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0117] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0118] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned model inference methods based on intelligent resource optimization. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the model inference method based on intelligent resource optimization shown in any embodiment of the present invention by calling the computer program.

[0119] In one alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0120] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0121] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0122] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0123] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0124] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0125] It should be noted that, Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0126] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the aforementioned model inference methods based on intelligent resource optimization.

[0127] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0128] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned model inference method based on intelligent resource optimization.

[0129] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0133] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0134] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0135] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0136] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A model reasoning method based on intelligent resource optimization, characterized in that, include: Initialize the inference environment, load the model file in the inference environment, and create a resource manager instance; The resource manager instance is used to monitor available device resources in real time, and to obtain the memory usage, number of running tasks and device utilization of each available device. Based on the multi-dimensional device scoring algorithm, the memory usage rate, the number of running tasks, and the device utilization rate of each available device are weighted and summed according to preset weights to obtain a comprehensive score, and the available device with the best comprehensive score is selected as the target inference device; The target inference device maintains a list of free memory blocks in a pre-allocated memory pool. When a memory allocation request is received, a suitable free memory block is first searched in the list of free memory blocks. If no suitable free memory block is found, a new memory block is allocated. The memory pool is automatically expanded when the memory pool capacity is insufficient. The system receives input data to be inferred, maps the input data to memory space using file mapping, identifies the memory location of the input data, directly calls the input data to execute model inference when the input data is located in device memory, and allocates device memory through the memory pool and migrates the input data to device memory when the input data is located in host memory. The model inference calculation is performed based on the input data in the target inference device to obtain the inference result; When the inference task is a multi-model concatenation task, the multi-model concatenation task is divided into multiple stages, with each stage corresponding to one model. The inference operations of each stage on their respective batches of data are executed in parallel through asynchronous processing, and the output results of each stage are merged into the final inference result.

2. The model inference method based on intelligent resource optimization according to claim 1, characterized in that, The steps of initializing the inference environment, loading the model file in the inference environment, and creating a resource manager instance include: Call the device count acquisition interface to determine the total number of available devices, and construct a device information mapping table based on the device identifier information of each available device; The resource manager instance is created based on the device information mapping table. During the creation of the resource manager instance, the memory pool mapping table is initialized to establish the association between each available device and the corresponding memory pool. The model file is loaded through the environment manager in the inference environment, and the storage path of the model file is associated with the model context and cached.

3. The model inference method based on intelligent resource optimization according to claim 2, characterized in that, The steps of real-time monitoring of available device resources through the resource manager instance to obtain the memory usage, number of running tasks, and device utilization of each available device include: A monitoring thread is started through the resource manager instance, and the monitoring thread calls the resource monitoring function at preset time intervals; The resource monitoring function iterates through all available devices in the device information mapping table, queries the current used memory capacity and total memory capacity for each available device and calculates the memory utilization rate, counts the number of inference tasks being executed on each available device to obtain the number of running tasks, and collects the activity time ratio of the corresponding computing unit in each available device to obtain the device utilization rate. Update the memory usage, number of running tasks, and device utilization of each available device to the device status record maintained by the resource manager instance.

4. The model inference method based on intelligent resource optimization according to claim 3, characterized in that, Based on a multi-dimensional device scoring algorithm, the steps of weighting and summing the memory usage, number of running tasks, and device utilization of each available device according to preset weights to obtain a comprehensive score, and selecting the available device with the best comprehensive score as the target inference device, include: Read the current memory usage, number of running tasks, and device utilization of each available device from the device status record; For each available device, the memory utilization rate is multiplied by a first preset weight, the number of running tasks is multiplied by a second preset weight, and the device utilization rate is multiplied by a third preset weight, and then summed to obtain a comprehensive score for each available device, wherein the sum of the first preset weight, the second preset weight, and the third preset weight is a fixed value; The overall scores of all available devices are compared, and the available device with the lowest overall score is determined as the target inference device.

5. The model inference method based on intelligent resource optimization according to claim 4, characterized in that, The steps of maintaining a list of free memory blocks in a pre-allocated memory pool in the target inference device, and when a memory allocation request is received, first searching for a suitable free memory block in the list of free memory blocks; if no suitable free memory block is found, allocating a new memory block, and automatically expanding the memory pool when its capacity is insufficient, include: During the initialization phase of the target inference device, a fixed-size device memory block is pre-allocated to the memory pool corresponding to the target inference device, and a list of free memory blocks is established to record information on reusable free memory blocks. When the resource manager instance receives a memory allocation request, it traverses the list of free memory blocks to find a free memory block with a capacity not less than the requested capacity. If a free memory block with a capacity not less than the requested capacity is found, the found free memory block is marked as used and returned. If no free memory block with a capacity not less than the requested capacity is found in the list of free memory blocks, the memory allocation interface is called to allocate a new memory block to the memory pool, and the allocated new memory block is marked as used. If the memory pool capacity has reached the preset upper limit and still cannot meet the memory allocation request during the allocation of new memory blocks, the upper limit of the memory pool capacity will be automatically expanded before allocating new memory blocks.

6. The model inference method based on intelligent resource optimization according to claim 5, characterized in that, The steps of receiving input data to be inferred, mapping the input data to memory space using file mapping, identifying the memory location of the input data, directly calling the input data to perform model inference when the input data is located in device memory, and allocating device memory through the memory pool and migrating the input data to device memory when the input data is located in host memory include: Receive input data to be inferred, wherein the input data includes file path information; The file reading component maps the file content corresponding to the file path information to the memory space, and uses the input data mapped to the memory space as the input data to be processed. The input data to be processed is preprocessed to convert it into the input format required for model inference; Before creating the model input dataset, check the memory location of the preprocessed input data. If the memory location is the device memory of the target inference device, call the input data in the device memory to perform model inference. When the memory location is host memory, the resource manager instance requests and allocates device memory from the memory pool corresponding to the target inference device, and then migrates the pre-processed input data from the host memory to the allocated device memory before performing model inference.

7. The model inference method based on intelligent resource optimization according to claim 6, characterized in that, The step of performing model inference calculations based on the input data in the target inference device to obtain inference results includes: Obtain the model context corresponding to the model file through the environment manager, and load the model context on the target inference device; Based on the loaded model context, a model input dataset and a model output buffer are created, and the pre-processed input data is filled into the model input dataset. The model inference execution interface is invoked to input the model input dataset into the target inference device for computation, thereby obtaining the model output data. The model output data is read from the model output buffer and returned as the inference result.

8. A model inference system based on intelligent resource optimization, characterized in that, include: Create a module to initialize the inference environment, load the model file in the inference environment, and create a resource manager instance; The acquisition module is used to monitor available device resources in real time through the resource manager instance, and to acquire the memory usage, number of running tasks and device utilization of each available device. The selection module is used to calculate a comprehensive score by weighting the memory usage rate, the number of running tasks, and the device utilization rate of each available device according to a multi-dimensional device scoring algorithm, and then selecting the available device with the best comprehensive score as the target inference device. The processing module is used to maintain a list of free memory blocks in the pre-allocated memory pool in the target inference device. When a memory allocation request is received, it first searches for a suitable free memory block in the list of free memory blocks. If no suitable free memory block is found, a new memory block is allocated. The memory pool is automatically expanded when the memory pool capacity is insufficient. The running module is used to receive input data to be inferred, map the input data to memory space using file mapping, identify the memory location of the input data, directly call the input data to execute model inference when the input data is located in device memory, and allocate device memory through the memory pool and migrate the input data to device memory when the input data is located in host memory. The inference module is used to perform model inference calculations based on the input data in the target inference device to obtain inference results; The merging module is used to divide the multi-model concatenation task into multiple stages when the inference task is a multi-model concatenation task, with each stage corresponding to one model. The inference operations of each stage on its respective batch of data are executed in parallel through asynchronous processing, and the output results of each stage are merged into the final inference result.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the model inference method based on intelligent resource optimization as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, implements the model inference method based on intelligent resource optimization as described in any one of claims 1 to 7.

Images