Artificial intelligence computing power allocation method, system, device, storage medium and program product
By dynamically matching AI computing resources, the problem of low utilization of computing resources under static allocation is solved, realizing flexible allocation and efficient utilization of resources, reducing latency and maintenance costs, and is applicable to computing platforms such as vehicle cockpit platforms.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI ANQINZHIXING AUTOMOTIVE ELECTRONICS CO LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, AI computing power platforms use a static allocation method to allocate AI computing power resources, resulting in low utilization of computing power resources and high task inference latency. They are unable to dynamically respond to fluctuations in computing power load, and the system maintenance is complex and costly.
By acquiring the business requirements of the target inference business and the current usage status of heterogeneous computing resources in real time, the target computing resources are dynamically matched to achieve dynamic allocation of AI computing resources, avoiding overload or idleness, improving resource utilization and reducing task latency.
It enables flexible allocation of AI computing resources, improves resource utilization, reduces task inference latency, enhances data processing capabilities, increases adaptability, and reduces system maintenance costs.
Smart Images
Figure CN122111655A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to an AI computing power allocation method, system, device, storage medium and program product. Background Technology
[0002] With the rapid development of intelligent vehicle technology, artificial intelligence (AI) computing platforms (such as in-vehicle cockpit platforms) are gradually integrating various AI algorithm models to achieve functions such as intelligent interaction, environmental perception, speech recognition, and image processing. For example, in-vehicle systems need to run algorithms such as object detection (e.g., YOLO algorithm), facial recognition (e.g., InsightFace algorithm), and voice command parsing simultaneously to support scenarios such as driver monitoring, passenger behavior analysis, and navigation voice interaction.
[0003] In related technologies, for AI computing power platforms that include various computing power resources (such as central processing units (CPUs), graphics processing units (GPUs), and neural processing units (NPUs)) and different AI algorithm models, a static allocation method is usually adopted to allocate AI computing power resources. That is, the computing power resources are assigned to AI algorithm models. Specifically, a specific AI algorithm model is fixedly bound to a certain computing power resource, and tasks are distributed through hardware drivers or middleware.
[0004] However, the above-mentioned method of allocating AI computing resources using static allocation has the problem of low utilization rate of computing resources. Summary of the Invention
[0005] This application provides an AI computing power allocation method, system, device, storage medium, and program product to solve the problem of low computing power resource utilization caused by static allocation of AI computing power resources in related technologies.
[0006] Firstly, this application provides an AI computing power allocation method, comprising: acquiring the current frame data of the target inference service and the current usage status of each heterogeneous computing power resource, wherein each heterogeneous computing power resource supports loading multiple AI algorithm models of different types; determining the target AI algorithm model corresponding to the business requirements of the target inference service; dynamically matching the target computing power resource for loading the target AI algorithm model based on the processing requirements of multiple AI algorithm models, the preset adaptability rules of each heterogeneous computing power resource, and the current usage status of each heterogeneous computing power resource; and inputting the current frame data into the target AI algorithm model for business inference to obtain the inference result output by the target AI algorithm model.
[0007] In one possible implementation, based on the processing requirements of multiple AI algorithm models and the preset adaptability rules of each heterogeneous computing resource, as well as the current usage status of each heterogeneous computing resource, the target computing resource for loading the target AI algorithm model is dynamically matched. This includes: matching candidate computing resources that are compatible with the target AI algorithm model among the heterogeneous computing resources according to the processing requirements of multiple AI algorithm models and the preset adaptability rules of each heterogeneous computing resource; and determining the target computing resource based on the current usage status of the candidate computing resources.
[0008] In one possible implementation, the current usage status includes the current utilization rate. Determining the target computing resource based on the current usage status of the candidate computing resources includes: in response to the existence of a dedicated AI computing resource among the candidate computing resources, determining whether the current utilization rate of the dedicated AI computing resource is less than a first preset threshold; if the current utilization rate of the dedicated AI computing resource is less than the first preset threshold, then determining the dedicated AI computing resource as the target computing resource; if the current utilization rate of the dedicated AI computing resource is greater than or equal to the first preset threshold, then determining the target computing resource based on the current utilization rate of the candidate computing resources.
[0009] In one possible implementation, determining the target computing power resource based on the current utilization rate of candidate computing power resources includes: determining whether the current utilization rate of other candidate computing power resources (excluding dedicated AI computing power resources) is less than or equal to a first preset threshold; if at least one other candidate computing power resource has a current utilization rate less than the first preset threshold, then, according to a preset computing power resource priority order, determining the candidate computing power resource with the highest priority among the at least one other candidate computing power resource as the target computing power resource; if the current utilization rate of all other candidate computing power resources is greater than or equal to the first preset threshold, then determining whether the current utilization rate of the dedicated AI computing power resource is less than a second preset threshold; if the current utilization rate of the dedicated AI computing power resource is less than the second preset threshold, then determining the dedicated AI computing power resource as the target computing power resource; if the current utilization rate of the dedicated AI computing power resource is greater than or equal to the second preset threshold, then determining the target computing power resource based on the current utilization rate of other candidate computing power resources.
[0010] In one possible implementation, before inputting the current frame data into the target AI algorithm model for business inference, the method further includes: decoding the current frame data according to the data format requirements of the target AI algorithm model to generate target data corresponding to the current frame data, wherein the data format of the target data is adapted to the target AI algorithm model.
[0011] In one possible implementation, after decoding the current frame data according to the data format requirements of the target AI algorithm model to generate the target data corresponding to the current frame data, the method further includes: standardizing the target data based on the data input requirements of the target AI algorithm model to obtain standardized data.
[0012] Secondly, this application provides an AI computing power allocation system, comprising:
[0013] The computing resource monitoring module is used to obtain the current usage status of each heterogeneous computing resource;
[0014] The data processing module is used to receive the current frame data of the target inference service;
[0015] The computing power resource initialization module is used to initialize various types of AI algorithm models for each heterogeneous computing power resource, so that each heterogeneous computing power resource can support loading multiple AI algorithm models.
[0016] The computing power resource allocation module is used to determine the target AI algorithm model corresponding to the business needs of the target inference business; and dynamically match the target computing power resources for loading the target AI algorithm model based on the processing needs of multiple AI algorithm models, the preset adaptability rules of each heterogeneous computing power resource, and the current usage status of each heterogeneous computing power resource.
[0017] The algorithm model inference module is used to input the current frame data into the target AI algorithm model for business inference and obtain the inference result output by the target AI algorithm model.
[0018] In one possible implementation, the computing power resource allocation module is further configured to: in response to the existence of dedicated AI computing power resources among the candidate computing power resources, determine whether the current utilization rate of the dedicated AI computing power resources is less than a first preset threshold; if the current utilization rate of the dedicated AI computing power resources is less than the first preset threshold, determine the dedicated AI computing power resources as target computing power resources; if the current utilization rate of the dedicated AI computing power resources is greater than or equal to the first preset threshold, determine the target computing power resources based on the current utilization rate of the candidate computing power resources.
[0019] In one possible implementation, the computing power resource allocation module is further configured to: determine whether the current utilization rate of other candidate computing power resources besides dedicated AI computing power resources is less than or equal to a first preset threshold; if the current utilization rate of at least one other candidate computing power resource is less than the first preset threshold, then, according to the preset computing power resource priority order, determine the candidate computing power resource with the highest priority among at least one other candidate computing power resource as the target computing power resource; if the current utilization rate of all other candidate computing power resources is greater than or equal to the first preset threshold, then determine whether the current utilization rate of dedicated AI computing power resources is less than a second preset threshold; if the current utilization rate of dedicated AI computing power resources is less than the second preset threshold, then determine the dedicated AI computing power resource as the target computing power resource; if the current utilization rate of dedicated AI computing power resources is greater than or equal to the second preset threshold, then determine the target computing power resource based on the current utilization rate of other candidate computing power resources.
[0020] In one possible implementation, before inputting the current frame data into the target AI algorithm model for business inference, the data processing module is further used to: decode the current frame data according to the data format requirements of the target AI algorithm model, and generate target data corresponding to the current frame data, wherein the data format of the target data is adapted to the target AI algorithm model.
[0021] In one possible implementation, after decoding the current frame data according to the data format requirements of the target AI algorithm model to generate the target data corresponding to the current frame data, the data processing module is also used to: standardize the target data based on the data input requirements of the target AI algorithm model to obtain standardized data.
[0022] Thirdly, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the AI computing power allocation method provided in the first aspect above.
[0023] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the AI computing power allocation method provided in the first aspect above.
[0024] Fifthly, this application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the AI computing power allocation method provided in the first aspect above.
[0025] The AI computing power allocation method, system, device, storage medium, and program product provided in this application acquire the current frame data of the target inference service and the current usage status of each heterogeneous computing power resource. Each heterogeneous computing power resource supports loading various types of AI algorithm models. Based on the business requirements of the target inference service, the method determines the target AI algorithm model corresponding to the business requirements. Furthermore, based on the processing requirements of various AI algorithm models and the preset adaptation rules of each heterogeneous computing power resource, and the current usage status of each heterogeneous computing power resource, the method dynamically matches the target computing power resource for loading the target AI algorithm model. Then, the method inputs the current frame data into the target AI algorithm model for business inference and obtains the inference result output by the target AI algorithm model. This application determines the corresponding target AI algorithm model based on the real-time business requirements of the target inference business. Then, it combines the processing requirements of multiple AI algorithm models with the preset adaptability rules of various heterogeneous computing resources and the current usage status of each heterogeneous computing resource to dynamically match the target computing resources used to load the target AI algorithm model. This achieves dynamic allocation of AI computing resources, avoids overload or idle computing resources, improves the utilization rate of computing resources, reduces task inference latency, and thus improves data processing capabilities. Attached Figure Description
[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0027] Figure 1 This is a schematic diagram of the structure of the AI computing power allocation system provided in the embodiments of this application;
[0028] Figure 2 A flowchart illustrating the AI computing power allocation method provided in this application embodiment. Figure 1 ;
[0029] Figure 3 A flowchart illustrating the AI computing power allocation method provided in this application embodiment. Figure 2 ;
[0030] Figure 4 A flowchart illustrating the AI computing power allocation method provided in this application embodiment. Figure 3 ;
[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0032] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0034] Currently, the allocation of AI computing resources mainly tends to switch between homogeneous computing resources. For example, in a GPU server cluster, computing load can only be allocated between computing cards of the same model. Their computing units have the same architecture and instruction set, so no additional adaptation is required. However, for heterogeneous computing resources with significant differences in architecture (such as between CPU, GPU, and NPU), flexible computing power switching and task migration cannot be achieved.
[0035] In related technologies, AI computing power platforms that include various computing resources (such as CPU, GPU, NPU) and different AI algorithm models typically employ a static allocation method to allocate AI computing power resources. This involves assigning computing power resources to specific AI algorithm models, binding a particular AI algorithm model to a specific computing power resource, and distributing tasks through hardware drivers or middleware. However, this static allocation method has several drawbacks. First, when a particular computing power resource is overloaded, it cannot dynamically migrate inference tasks to other idle computing power resources, nor can it dynamically respond to fluctuations in computing load. This leads to an imbalance where some computing power resources are idle while others are overloaded, resulting in low utilization and high inference latency. Second, when switching computing power resources is necessary, the system requires specific modifications and adaptations to match the target computing power resource's algorithm runtime environment, leading to complex system maintenance processes and high maintenance costs.
[0036] Based on the problems existing in related technologies, the embodiments of this application determine the corresponding target AI algorithm model in real time based on the business requirements of the target inference business. Then, by combining the processing requirements of multiple AI algorithm models with the preset adaptability rules of each heterogeneous computing power resource and the current usage status of each heterogeneous computing power resource, the target computing power resource used to load the target AI algorithm model is dynamically matched to achieve dynamic allocation of AI computing power resources, avoid overload or idle computing power resources, improve the utilization rate of computing power resources, reduce task inference latency, and thus improve data processing capabilities.
[0037] The application scenarios of the embodiments of this application will be described below first.
[0038] The AI computing power allocation method provided in this application is applicable to multi-algorithm concurrent processing scenarios in AI computing power platforms (such as vehicle cockpit platforms). Specifically, it can be applied to business scenarios that require the simultaneous execution of multiple different types of AI algorithm models, such as performing inference tasks such as autonomous driving perception, cockpit voice interaction, and image recognition in a vehicle scenario.
[0039] Figure 1 This is a schematic diagram of the AI computing power allocation system provided in an embodiment of this application. Figure 1 As shown, the AI computing power allocation system includes a computing power resource monitoring module, a data processing module, a computing power resource initialization module, a computing power resource allocation module, and an algorithm model inference module.
[0040] The computing resource monitoring module is used to obtain the current usage status of each heterogeneous computing resource.
[0041] For example, the current usage status of each heterogeneous computing resource includes, but is not limited to, one or more of the following: current utilization rate, power consumption, and response latency of each heterogeneous computing resource.
[0042] In one possible implementation, during the operation of the AI computing power allocation system, a computing power resource monitoring service is activated to monitor and obtain key usage status parameters such as the current utilization rate, power consumption, and response latency of each heterogeneous computing power resource in real time, providing data reference for matching suitable computing power resources for the target inference task.
[0043] The data processing module is used to receive the current frame data of the target inference service.
[0044] For example, after the AI computing power allocation system is initialized, the data processing module is used to receive the current frame data of the target inference business from the business side.
[0045] For example, in one possible implementation, the data processing module is also used to perform decoding and standardization processing on the current frame data.
[0046] The computing resource initialization module is used to initialize various types of AI algorithm models for each heterogeneous computing resource, so that each heterogeneous computing resource can support loading multiple AI algorithm models.
[0047] In one possible implementation, in response to the startup of the AI computing power allocation system, the various AI algorithm models of different types contained in the AI computing power allocation system are initialized in each heterogeneous computing power resource, so that each heterogeneous computing power resource can support loading multiple AI algorithm models.
[0048] The computing power resource allocation module is used to determine the target AI algorithm model corresponding to the business requirements of the target inference business; and dynamically match the target computing power resources for loading the target AI algorithm model based on the processing requirements of various AI algorithm models, the preset adaptability rules of each heterogeneous computing power resource, and the current usage status of each heterogeneous computing power resource.
[0049] The algorithm model inference module is used to input the current frame data into the target AI algorithm model for business inference and obtain the inference result output by the target AI algorithm model.
[0050] For example, the AI computing power allocation system provided in this application embodiment can be integrated into AI computing power platforms with limited power energy supply and strict requirements for computing power energy efficiency ratio, such as vehicle platforms (e.g., vehicle cockpit platforms). Such platforms need to support the concurrent execution of multiple types of AI inference tasks under limited power consumption. The dynamic AI computing power allocation mechanism of the AI computing power allocation system provided in this application embodiment can effectively improve the utilization rate of computing power resources and reduce ineffective energy consumption.
[0051] The following is based on the above. Figure 1 The AI computing power allocation system shown is the execution entity, and the specific implementation of the AI computing power allocation method provided in this application embodiment will be described in detail with reference to specific embodiments.
[0052] Figure 2 A flowchart illustrating the AI computing power allocation method provided in this application embodiment. Figure 1 .like Figure 2 As shown, a specific implementation of this AI computing power allocation method may include the following steps:
[0053] S201, obtain the current frame data of the target inference service and the current usage status of each heterogeneous computing resource. Each heterogeneous computing resource supports loading various AI algorithm models of different types.
[0054] For example, target inference services can be AI inference tasks such as target detection, face recognition, and speech recognition.
[0055] For example, the current frame data can be raw video stream data, image frame data, or audio frame data collected in real time by sensors (such as cameras, LiDAR, millimeter-wave radar, and microphones).
[0056] It is understandable that different types of target inference services correspond to matching current frame data from different sources.
[0057] For example, heterogeneous computing resources may include GPUs, CPUs, NPUs, and AI Processing Units (APUs).
[0058] For example, the current usage status of each heterogeneous computing resource may include one or more of the following: current utilization rate, energy consumption, and response latency.
[0059] For example, different types of AI algorithm models may include the YOLO algorithm for object detection, the InsightFace algorithm for feature calculation, and voice command parsing algorithms.
[0060] In this step, one possible implementation is as follows: during the operation of the AI computing power allocation system, the data processing module receives the current frame data of the target inference service sent by the business side in real time, and the computing power resource monitoring module monitors and obtains the current usage status of each heterogeneous computing power resource in real time.
[0061] S202, Based on the business requirements of the target inference business, determine the target AI algorithm model corresponding to the business requirements.
[0062] For example, the business requirements for target inference services can be target detection business requirements, face recognition business requirements, and speech recognition business requirements, etc.
[0063] One possible implementation of this step is as follows: Based on the business identification information carried in the current frame data, identify the business requirements of the target inference business, and then, based on the business requirements of the target inference business, determine the target AI algorithm model corresponding to the business requirements from the various AI algorithm models of different types pre-configured in the AI computing power allocation system.
[0064] S203 dynamically matches the target computing resources used to load the target AI algorithm model based on the processing requirements of various AI algorithm models, the preset adaptability rules of various heterogeneous computing resources, and the current usage status of various heterogeneous computing resources.
[0065] For example, the processing requirements of various AI algorithm models can be the characteristics of the computing resources required by each AI algorithm model during the execution of AI inference tasks, including but not limited to accuracy requirements (such as the YOLO algorithm for object detection requiring NPU support for INT8 quantization calculation, and the InsightFace algorithm for face recognition requiring GPU support for FP32 precision calculation, etc.) and data format compatibility requirements.
[0066] For example, the pre-defined compatibility rules between the processing requirements of various AI algorithm models and heterogeneous computing resources can be the matching logic between each AI algorithm model and heterogeneous computing resources, including but not limited to the energy efficiency ratio, inference task priority, and data format compatibility of heterogeneous computing resources.
[0067] For example, the priority of the preset adaptability rules, from high to low, can be inference task priority, accuracy requirement adaptability, and energy efficiency ratio. For instance, high-precision inference tasks are preferentially allocated to GPUs, and low-precision inference tasks are preferentially allocated to NPUs or APUs. At the same time, the current usage status of each heterogeneous computing resource is considered. If the computing resource with the highest priority is overloaded, the system will switch to the next lower priority computing resource.
[0068] S204: Input the current frame data into the target AI algorithm model for business reasoning, and obtain the reasoning result output by the target AI algorithm model.
[0069] For example, when the target AI algorithm model is the YOLO algorithm, the current frame data is input into the target AI algorithm model for target detection, and the target detection result output by the target AI algorithm model is obtained.
[0070] In this embodiment, the target AI algorithm model is determined in real time based on the business requirements of the target inference business. Then, the processing requirements of multiple AI algorithm models are combined with the preset adaptability rules of each heterogeneous computing power resource and the current usage status of each heterogeneous computing power resource. The target computing power resource used to load the target AI algorithm model is dynamically matched to achieve dynamic allocation of AI computing power resources, avoid overload or idle computing power resources, improve the utilization rate of computing power resources, reduce task inference latency, and thus improve data processing capabilities.
[0071] It is understood that the AI computing power allocation method provided in this application, targeting AI computing power platforms equipped with various heterogeneous computing power resources and different types of AI algorithm models, can adaptively match each AI algorithm model to a suitable computing power resource based on the real-time usage status of each heterogeneous computing power resource, thereby achieving computing power load balancing and effectively improving the utilization rate of computing power resources in the AI computing power allocation system. Furthermore, compared to related technologies where AI computing power resource allocation primarily focuses on switching between homogeneous computing power resources, the AI computing power allocation method provided in this application adapts to switching between different heterogeneous computing power resources, improving the flexibility of computing power resource allocation and further enhancing the adaptability of the AI computing power allocation system to computing power demands.
[0072] The following is combined Figure 3 The following is a detailed explanation of a specific implementation method for step S203, which dynamically matches the target computing resources used to load the target AI algorithm model based on the processing requirements of various AI algorithm models, the preset adaptability rules of each heterogeneous computing resource, and the current usage status of each heterogeneous computing resource.
[0073] Figure 3 A flowchart illustrating the AI computing power allocation method provided in this application embodiment. Figure 2 .like Figure 3As shown, in this AI computing power allocation method, a specific implementation of dynamically matching the target computing power resource for loading the target AI algorithm model based on the processing requirements of various AI algorithm models, the preset adaptability rules of each heterogeneous computing power resource, and the current usage status of each heterogeneous computing power resource may include the following steps:
[0074] S301, based on the processing requirements of various AI algorithm models and the preset adaptability rules of various heterogeneous computing resources, matches candidate computing resources that are compatible with the target AI algorithm model among various heterogeneous computing resources.
[0075] The preset adaptation rules are similar to those described above, and will not be repeated here.
[0076] One possible implementation of this step is to determine at least one heterogeneous computing resource that is compatible with the target AI algorithm model as a candidate computing resource based on the processing requirements of various AI algorithm models and the preset adaptability rules of each heterogeneous computing resource.
[0077] S302, determine the target computing resources based on the current usage status of the candidate computing resources.
[0078] For example, the current usage status of a candidate computing resource can be its current utilization rate.
[0079] Optionally, this step may include the following steps:
[0080] S3021, in response to the existence of dedicated AI computing resources among the candidate computing resources, determine whether the current utilization rate of the dedicated AI computing resources is less than a first preset threshold.
[0081] For example, dedicated AI computing resources refer to computing resources specifically designed for AI inference task architectures, such as NPU and APU, whose instruction sets and computing units are customized to adapt to the parallel computing characteristics of AI algorithm models.
[0082] For example, the first preset threshold can be 60%. This application does not limit the size of the first preset threshold; it can be determined based on actual application requirements.
[0083] For example, when the current utilization rate of computing resources is less than a first preset threshold, it indicates that the computing resources are currently in an idle state; when the current utilization rate of computing resources is greater than or equal to the first preset threshold, it indicates that the computing resources are currently in a normal operating state or an overload state.
[0084] In this step, it is determined whether the current utilization rate of the dedicated AI computing power resources is less than the first preset threshold. If the current utilization rate of the dedicated AI computing power resources is less than the first preset threshold, then step S3022 is executed; if the current utilization rate of the dedicated AI computing power resources is greater than or equal to the first preset threshold, then step S3023 is executed.
[0085] S3022, Determine dedicated AI computing resources as the target computing resources.
[0086] It is understood that in the AI computing power allocation algorithm provided in this application embodiment, there are dedicated AI computing power resources among the candidate computing power resources adapted to the target AI algorithm model, and the current utilization rate of the dedicated AI computing power resources is less than a first preset threshold. That is, when the dedicated AI computing power resources are currently idle, the dedicated AI computing power resources are used first to achieve flexible allocation of computing power resources.
[0087] S3023, determine the target computing resources based on the current utilization rate of the candidate computing resources.
[0088] For example, in one possible implementation, the target computing resources are determined based on the current utilization rate of the dedicated AI computing resources and the current utilization rate of other candidate computing resources besides the dedicated AI computing resources.
[0089] In this embodiment, by matching candidate computing resources that are compatible with the target AI algorithm model among various heterogeneous computing resources according to the processing requirements of multiple AI algorithm models and the preset adaptability rules of each heterogeneous computing resource, and then combining the current usage status of the candidate computing resources, the target computing resource is determined, thereby realizing flexible allocation of computing resources and improving the utilization rate of computing resources.
[0090] The following is combined Figure 4 The following is a detailed explanation of a specific implementation method for determining the target computing power resource based on the current utilization rate of the candidate computing power resources in step S3023.
[0091] Figure 4 A flowchart illustrating the AI computing power allocation method provided in this application embodiment. Figure 3 .like Figure 4 As shown, in this AI computing power allocation method, a specific implementation of determining the target computing power resource based on the current utilization rate of candidate computing power resources may include the following steps:
[0092] S401, determine whether the current utilization rate of other candidate computing resources besides dedicated AI computing resources is less than or equal to a first preset threshold.
[0093] The first preset threshold is similar to that described above, and will not be repeated here.
[0094] In this step, it is determined whether the current utilization rate of other candidate computing resources besides dedicated AI computing resources is less than or equal to a first preset threshold. If at least one other candidate computing resource has a current utilization rate less than the first preset threshold, then step S402 is executed; if the current utilization rate of all other candidate computing resources is greater than or equal to the first preset threshold, then step S403 is executed.
[0095] It is understandable that when there is only one computing resource among the other candidate computing resources whose current utilization rate is less than the first preset threshold, that computing resource is identified as the target computing resource.
[0096] S402, according to the preset priority order of computing resources, determine the highest priority candidate computing resource as the target computing resource from at least one other candidate computing resources.
[0097] For example, the preset priority order of computing resources can be determined based on the data processing efficiency of each computing resource. The priority order of computing resources is positively correlated with the data processing efficiency of each computing resource, that is, the higher the data processing efficiency of the computing resource, the higher its priority.
[0098] S403, determine whether the current utilization rate of dedicated AI computing resources is less than the second preset threshold.
[0099] For example, the value of the second preset threshold can be 90%. This application embodiment does not limit the size of the second preset threshold; it can be determined according to actual application requirements.
[0100] For example, when the current utilization rate of computing resources is less than a first preset threshold, it indicates that the computing resources are currently in an idle state; when the current utilization rate of computing resources is greater than or equal to the first preset threshold and less than a second preset threshold, it indicates that the computing resources are currently in a normal operating state; when the current utilization rate of computing resources is greater than or equal to the second preset threshold, it indicates that the computing resources are currently in an overload state.
[0101] In this step, it is determined whether the current utilization rate of the dedicated AI computing power resources is less than the second preset threshold. If the current utilization rate of the dedicated AI computing power resources is less than the second preset threshold, then step S404 is executed; if the current utilization rate of the dedicated AI computing power resources is greater than or equal to the second preset threshold, then step S405 is executed.
[0102] S404, identify dedicated AI computing resources as the target computing resources.
[0103] S405: Determine the target computing resources based on the current utilization rate of other candidate computing resources.
[0104] For example, one possible implementation is as follows: determine whether the current utilization rate of other candidate computing resources is less than a second preset threshold. If at least one other candidate computing resource has a current utilization rate less than the second preset threshold, then determine the candidate computing resource with the highest priority as the target computing resource according to the preset computing resource priority order. If the current utilization rate of other candidate computing resources is greater than the second preset threshold, then determine the computing resource with the lowest current utilization rate among the candidate computing resources as the target computing resource, or trigger the inference task queuing mechanism.
[0105] It is understandable that when there is only one other candidate computing resource whose current utilization rate is less than the second preset threshold, that candidate computing resource is determined as the target computing resource.
[0106] In this embodiment, when the current utilization rate of at least one other candidate computing resource is less than a first preset threshold, the highest priority candidate computing resource among the at least one other candidate computing resource is determined as the target computing resource according to the preset computing resource priority order; when the current utilization rate of all other candidate computing resources is greater than or equal to the first preset threshold, and the current utilization rate of the dedicated AI computing resource is less than a second preset threshold, the dedicated AI computing resource is determined as the target computing resource; when the current utilization rate of the dedicated AI computing resource is greater than or equal to the second preset threshold, the target computing resource is determined based on the current utilization rate of other candidate computing resources, thereby achieving flexible allocation of computing resources and improving the utilization rate of computing resources.
[0107] Optionally, in response to the absence of dedicated AI computing resources among the candidate computing resources, one possible implementation for determining the target computing resource based on the current usage status of the candidate computing resources is as follows: determine whether the current utilization rate of the candidate computing resources is less than a first preset threshold; if the current utilization rate of one candidate computing resource is less than the first preset threshold, then that computing resource is determined as the target computing resource; if the current utilization rate of multiple candidate computing resources is less than the first preset threshold, then, according to the preset computing resource priority order, the candidate computing resource with the highest priority among the multiple candidate computing resources is determined as the target computing resource; if the current utilization rate of all candidate computing resources is greater than the first preset threshold... If the current utilization rate of a candidate computing resource is equal to or greater than the first preset threshold, then it is determined whether the current utilization rate of the candidate computing resource is less than the second preset threshold. If there is a candidate computing resource whose current utilization rate is less than the second preset threshold, then that computing resource is determined as the target computing resource. If there are multiple candidate computing resources whose current utilization rate is less than the second preset threshold, then the candidate computing resource with the highest priority is determined as the target computing resource according to the preset computing resource priority order. If the current utilization rate of all candidate computing resources is greater than or equal to the second preset threshold, then the computing resource with the lowest current utilization rate among the candidate computing resources is determined as the target computing resource, or the inference task queuing mechanism is triggered.
[0108] It is understood that the AI computing power allocation method provided in this application embodiment can maximize the utilization potential of heterogeneous AI computing power resources in scenarios with relatively limited computing resources (such as vehicle platforms), thereby improving the AI computing power and data processing throughput on the edge. When a certain computing power resource reaches its throughput limit, the system can automatically schedule and enable other available computing power resources by monitoring the usage status of computing power resources in real time, and perform parallel computing on the backlog of data to be processed, thereby improving the overall data throughput efficiency and the operational stability of the software system.
[0109] Optionally, the AI computing power allocation method provided in this application embodiment further includes, before inputting the current frame data into the target AI algorithm model for business inference, decoding the current frame data according to the data format requirements of the target AI algorithm model to generate target data corresponding to the current frame data, wherein the data format of the target data is adapted to the target AI algorithm model.
[0110] For example, when the target AI algorithm model is the YOLO algorithm model, its corresponding data format requirement can be YUV format data, and when the target AI algorithm model is the Insightface algorithm model, its corresponding data format requirement can be RGB format data, etc.
[0111] For example, in one possible implementation, the data processing module in the AI computing power allocation system decodes the current frame data according to the data format requirements of the target AI algorithm model to generate the target data corresponding to the current frame data.
[0112] Accordingly, one possible implementation of inputting the current frame data into the target AI algorithm model for business reasoning and obtaining the reasoning result output by the target AI algorithm model is as follows: decode the current frame data to obtain the target data corresponding to the current frame data, and input the target data into the target AI algorithm model for business reasoning to obtain the reasoning result output by the target AI algorithm model.
[0113] Optionally, the AI computing power allocation method provided in this application embodiment, after decoding the current frame data according to the data format requirements of the target AI algorithm model to generate the target data corresponding to the current frame data, further includes: standardizing the target data based on the data input requirements of the target AI algorithm model to obtain standardized data.
[0114] For example, the data input requirements of the target AI algorithm model may include the size of the data input, etc.
[0115] For example, the transformation processing includes one or more of the following: cropping, scaling, or normalization. This application does not limit this; the specific processing should be determined according to actual application requirements.
[0116] Accordingly, one possible implementation of inputting the current frame data into the target AI algorithm model for business reasoning and obtaining the reasoning result output by the target AI algorithm model is as follows: convert the current frame data into standardized data compatible with the target AI algorithm model, perform serialization and sorting processing on the standardized data according to the timestamps carried in the standardized data, and then input it into the target AI algorithm model to perform business reasoning and obtain the reasoning result output by the target AI algorithm model.
[0117] It is understandable that before inputting the current frame data into the target AI algorithm model for business inference, the current frame data is decoded into target data whose data format is adapted to the target AI algorithm model, or the target data is further standardized and converted into standardized data that meets the data input requirements of the target AI algorithm model. The target data or labeled data is then input into the target AI algorithm model to perform business inference, thereby improving the efficiency of business inference.
[0118] In summary, the AI computing power allocation method provided in this application has the following beneficial effects:
[0119] 1) By determining the corresponding target AI algorithm model based on the real-time business requirements of the target inference business, and then combining the processing requirements of multiple AI algorithm models with the preset adaptability rules of various heterogeneous computing resources and the current usage status of each heterogeneous computing resource, the target computing resources used to load the target AI algorithm model are dynamically matched to achieve dynamic allocation of AI computing resources, avoid overload or idle computing resources, improve the utilization rate of computing resources, reduce task inference latency, and thus improve data processing capabilities; at the same time, it can improve the adaptability of system resources to computing power requirements.
[0120] 2) The AI computing power allocation method provided in this application embodiment can be directly implemented based on the hardware management interface and algorithm running environment of the AI computing power platform, without relying on third-party custom plugins or middleware, effectively reducing system deployment and maintenance costs;
[0121] 3) The AI computing power allocation method provided in this application differs from the load compatibility processing method in GPU multi-card server clusters in related technologies. Based on the limited power energy characteristics of the vehicle platform, it is designed for heterogeneous computing power resources (such as CPU, GPU, NPU, APU) with significant architectural differences. This AI computing power allocation method can fully perceive the real-time computing power resource usage status of the vehicle platform and prioritize the scheduling of dedicated AI computing power resources to execute adapted inference tasks, thereby achieving flexible allocation and efficient utilization of computing power resources. At the same time, in terms of power energy consumption, it can significantly reduce energy consumption in the AI computing process, avoid a single computing power resource from being in a high-load operation state for a long time, and extend the stable operation time of the vehicle computing power platform.
[0122] 4) The AI computing power allocation method provided in this application embodiment can efficiently perform big data processing in the AI computing process of the vehicle platform, especially in scenarios where real-time frame data needs to be processed at the terminal, and significantly improve data processing efficiency.
[0123] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0124] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the mobile terminal 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.
[0125] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.
[0126] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0127] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0128] The memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk storage device.
[0129] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0130] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0131] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0132] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0133] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0134] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0136] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0137] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0138] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0139] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. An AI computing power allocation method, characterized in that, include: Obtain the current frame data of the target inference service and the current usage status of each heterogeneous computing resource, wherein each heterogeneous computing resource supports loading multiple AI algorithm models of different types; Based on the business requirements of the target inference business, determine the target AI algorithm model corresponding to the business requirements; Based on the processing requirements of the various AI algorithm models and the preset adaptability rules of the heterogeneous computing resources, as well as the current usage status of the heterogeneous computing resources, the target computing resources for loading the target AI algorithm model are dynamically matched. The current frame data is input into the target AI algorithm model for business reasoning, and the reasoning result output by the target AI algorithm model is obtained.
2. The AI computing power allocation method according to claim 1, characterized in that, The step of dynamically matching target computing resources for loading the target AI algorithm model based on the processing requirements of the various AI algorithm models, the preset adaptability rules of the heterogeneous computing resources, and the current usage status of the heterogeneous computing resources includes: Based on the processing requirements of the various AI algorithm models and the preset adaptability rules of the heterogeneous computing resources, candidate computing resources that are compatible with the target AI algorithm model are matched among the heterogeneous computing resources. The target computing resource is determined based on the current usage status of the candidate computing resources.
3. The AI computing power allocation method according to claim 2, characterized in that, The current usage status includes the current utilization rate. Determining the target computing power resource based on the current usage status of the candidate computing power resources includes: In response to the presence of dedicated AI computing resources among the candidate computing resources, it is determined whether the current utilization rate of the dedicated AI computing resources is less than a first preset threshold. If the current utilization rate of the dedicated AI computing power resource is less than the first preset threshold, then the dedicated AI computing power resource is determined to be the target computing power resource. If the current utilization rate of the dedicated AI computing power resource is greater than or equal to the first preset threshold, then the target computing power resource is determined based on the current utilization rate of the candidate computing power resources.
4. The AI computing power allocation method according to claim 3, characterized in that, Determining the target computing power resource based on the current utilization rate of the candidate computing power resources includes: Determine whether the current utilization rate of other candidate computing resources besides the dedicated AI computing resources is less than or equal to the first preset threshold; If the current utilization rate of at least one other candidate computing power resource is less than the first preset threshold, then according to the preset computing power resource priority order, the candidate computing power resource with the highest priority among at least one other candidate computing power resource is determined as the target computing power resource. If the current utilization rate of the other candidate computing resources is greater than or equal to the first preset threshold, then determine whether the current utilization rate of the dedicated AI computing resources is less than the second preset threshold. If the current utilization rate of the dedicated AI computing power resource is less than the second preset threshold, then the dedicated AI computing power resource is determined to be the target computing power resource. If the current utilization rate of the dedicated AI computing power resource is greater than or equal to the second preset threshold, then the target computing power resource is determined based on the current utilization rate of other candidate computing power resources.
5. The AI computing power allocation method according to any one of claims 1 to 3, characterized in that, Before inputting the current frame data into the target AI algorithm model for business reasoning, the method further includes: According to the data format requirements of the target AI algorithm model, the current frame data is decoded to generate target data corresponding to the current frame data, and the data format of the target data is adapted to the target AI algorithm model.
6. The AI computing power allocation method according to claim 5, characterized in that, After decoding the current frame data according to the data format requirements of the target AI algorithm model to generate the target data corresponding to the current frame data, the method further includes: Based on the data input requirements of the target AI algorithm model, the target data is standardized to obtain standardized data.
7. An AI computing power allocation system, characterized in that, include: The computing resource monitoring module is used to obtain the current usage status of each heterogeneous computing resource; The data processing module is used to receive the current frame data of the target inference service; The computing power resource initialization module is used to initialize various AI algorithm models of different types for each heterogeneous computing power resource, so that each heterogeneous computing power resource can support loading the various AI algorithm models. The computing power resource allocation module is used to determine the target AI algorithm model corresponding to the business requirements of the target inference business; and dynamically match the target computing power resources for loading the target AI algorithm model according to the processing requirements of the multiple AI algorithm models, the preset adaptability rules of each heterogeneous computing power resource, and the current usage status of each heterogeneous computing power resource. The algorithm model inference module is used to input the current frame data into the target AI algorithm model for business inference and obtain the inference result output by the target AI algorithm model.
8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory to implement the AI computing power allocation method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the AI computing power allocation method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, include: A computer program that, when executed by a processor, implements the AI computing power allocation method as described in any one of claims 1 to 6.