A method and apparatus for dynamically allocating AI inference tasks

Abstract

Embodiments of this disclosure provide a method and apparatus for dynamically allocating AI inference tasks. The method includes: real-time monitoring of the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in a cloud-edge collaborative system, wherein the computing nodes include local edge devices, edge servers, and cloud platforms; dynamically allocating tasks to different computing nodes based on task latency requirements, computing needs, data privacy requirements, and the real-time status of each computing node; adjusting the complexity of the AI ​​model used to execute the task based on the resource usage of the computing nodes; and optimizing the task allocation strategy based on historical performance data and real-time network conditions during task execution.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence computing, and more specifically, to a method and apparatus for dynamically allocating AI inference tasks. Background Technology

[0002] In many AI applications, especially in autonomous driving, smart cities, industrial automation, and IoT devices, real-time response is crucial. While traditional cloud-based AI processing boasts powerful computing capabilities, its reliance on network transmission introduces significant latency and bandwidth limitations. Edge processing, while reducing latency, is typically constrained by computing power, memory, and energy consumption. Therefore, dynamically scheduling AI inference workloads between edge computing and cloud computing to achieve higher performance and faster response times has become key to improving the effectiveness of AI applications. Summary of the Invention

[0003] To optimize the performance of AI applications, the embodiments described herein provide a method, apparatus, and computer-readable storage medium storing computer programs for dynamically allocating AI inference tasks. By considering multiple factors such as real-time network conditions, load, energy availability, and required response time, AI inference tasks are dynamically allocated among edge devices, edge servers, and the cloud to achieve flexible workload scheduling and meet the real-time and computing needs of different application scenarios.

[0004] According to a first aspect of this disclosure, a method for dynamically allocating AI inference tasks is provided, comprising: real-time monitoring of the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in a cloud-edge collaborative system, wherein the computing nodes include local edge devices, edge servers, and cloud platforms; dynamically allocating tasks to different computing nodes based on the task's latency requirements, computing needs, data privacy requirements, and the real-time status of each computing node; adjusting the complexity of the AI ​​model used to execute the task based on the resource usage of the computing nodes; and optimizing the task allocation strategy based on historical performance data and real-time network conditions during task execution.

[0005] In some embodiments of this disclosure, real-time monitoring of the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in a cloud-edge collaborative system, where the computing nodes include local edge devices, edge servers, and cloud platforms, includes: using system monitoring tools to track the CPU, GPU, and thread utilization, number of tasks processed, and execution time of the computing nodes to assess the load level of each node; checking the memory usage of each computing node and monitoring the total and remaining memory of each node; running network performance tests to assess latency, upload and download bandwidth, and packet loss rate between different computing nodes; real-time monitoring of the energy consumption of each computing node; and transmitting monitoring data from each computing node to a centralized data platform, using a distributed storage system to store historical monitoring data, and displaying the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in real time.

[0006] In some embodiments of this disclosure, dynamically allocating tasks to different computing nodes based on task latency requirements, computational needs, data privacy requirements, and the real-time status of each computing node includes: collecting data from computing nodes through sensors, analyzing task requirements based on the collected data, determining whether the task has strict latency requirements, clarifying the computational resources required by the task, and whether it involves sensitive data; classifying AI inference tasks, and allocating AI inference tasks to different computing nodes through a hierarchical computing architecture.

[0007] In some embodiments of this disclosure, AI inference tasks are classified and allocated to different computing nodes through a hierarchical computing architecture, including: prioritizing the allocation of low-latency tasks, lightweight inference tasks, preliminary data processing, and AI tasks involving sensitive data to local edge devices; prioritizing the allocation of high-latency-tolerant tasks and tasks with high computing requirements to the cloud platform; and dynamically selecting the optimal node based on the current load of each computing node.

[0008] In some embodiments of this disclosure, adjusting the complexity of the AI ​​model used to perform the task based on the resource usage of the computing node includes: for resource-constrained computing nodes, reducing the number of layers or neurons in the AI ​​model through quantization, pruning, or model segmentation; for resource-sufficient computing nodes, increasing the number of layers or neurons in the AI ​​model.

[0009] In some embodiments of this disclosure, optimizing the task allocation strategy based on historical performance data and real-time network conditions during task execution includes: tracking and recording the processing power, latency, load, and computing resource usage of each computing node in real time; monitoring network bandwidth, latency, and stability in real time to determine whether the current network environment is suitable for task execution or whether task allocation needs to be adjusted; continuously monitoring the execution time, resource consumption, and task completion quality of inference tasks; and dynamically adjusting the task allocation strategy through reinforcement learning algorithms.

[0010] In some embodiments of this disclosure, dynamically adjusting the task allocation strategy using a reinforcement learning algorithm includes: defining a state space and an action space, wherein the state space includes historical performance data, current network status, and task complexity information, and the action space includes decisions on allocating tasks to different computing nodes; calculating rewards based on the performance of the task execution, wherein a positive reward is given if the task is successfully completed within a predetermined time and computing resources are used properly; and a negative reward is given if the task execution fails or resources are used improperly.

[0011] In some embodiments of this disclosure, the method further includes: compressing and encrypting the data transmitted between different computing nodes during data transmission.

[0012] According to a second aspect of this disclosure, a dynamic allocation device for AI inference tasks is provided. The device includes at least one processor and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the device causes the following: real-time monitoring of the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in a cloud-edge collaborative system, including local edge devices, edge servers, and cloud platforms; dynamically allocating tasks to different computing nodes based on task latency requirements, computational demands, data privacy requirements, and the real-time status of each computing node; adjusting the complexity of the AI ​​model used to execute the task based on the resource usage of the computing nodes; and optimizing the task allocation strategy based on historical performance data and real-time network conditions during task execution.

[0013] According to a third aspect of this disclosure, a computer-readable storage medium is provided storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described according to a first aspect of this disclosure.

[0014] The AI ​​inference task dynamic allocation method and apparatus according to embodiments of this disclosure dynamically adjusts the complexity of the AI ​​model for task execution by real-time monitoring of the resource usage of computing nodes. This effectively addresses the differences in computing capabilities among different nodes and avoids excessive resource consumption. In a multi-layered computing environment encompassing local edge devices, edge servers, and cloud platforms, task execution can adjust the complexity of the computing model based on the current state of the nodes, thereby achieving more efficient resource utilization. The task allocation strategy can be automatically adjusted according to the needs of different tasks, environmental changes, and resource status. With the accumulation of task executions, continuous learning and optimization ultimately form a more intelligent and efficient task allocation model. Therefore, this solution can significantly improve the execution efficiency and overall performance of AI inference tasks in cloud-edge collaborative systems. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein:

[0016] Figure 1 This is an exemplary flowchart of a method for dynamically allocating AI inference tasks according to embodiments of the present disclosure;

[0017] Figure 2 This is a schematic block diagram of a dynamic allocation device for AI inference tasks according to embodiments of the present disclosure.

[0018] It should be noted that the elements in the attached diagram are schematic and not drawn to scale. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0020] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having meanings consistent with their meanings in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. Furthermore, terms such as “first” and “second” are used only to distinguish one component (or part of a component) from another component (or another part of a component).

[0021] To improve the processing efficiency and response speed of AI inference tasks, this disclosure provides a dynamic allocation method for AI inference tasks, which enables AI inference tasks to be optimally allocated and executed in a cloud-edge collaborative environment. By combining real-time resource monitoring, intelligent task scheduling, and adaptive adjustment of AI models, the overall performance of the system can be significantly improved and resource utilization efficiency optimized.

[0022] Figure 1 An exemplary flowchart illustrating a method for dynamically allocating AI inference tasks according to embodiments of the present disclosure is shown. (Refer to...) Figure 1 As shown, in block S102 of the dynamic allocation method 100 for AI inference tasks, the workload, memory availability, network latency, bandwidth and energy consumption of each computing node in the cloud-edge collaborative system are monitored in real time. The computing nodes include local edge devices, edge servers and cloud platforms.

[0023] According to one embodiment of this disclosure, the cloud-edge collaborative system includes multi-layered computing nodes, consisting of local edge devices, edge servers, and a cloud platform. Local edge devices (such as smartphones, cameras, and embedded AI chips) handle the initial processing of tasks, particularly suitable for low-latency and high-frequency processing needs, and can execute relatively simple AI models or tasks, such as image preprocessing and preliminary analysis of sensor data. Edge servers are located closer to the user, typically in facilities such as 5G base stations or micro data centers, and can perform more in-depth analysis and computation. The cloud platform is suitable for tasks requiring substantial computing resources, data storage, and large-scale model training. Monitoring agents are deployed on each computing node (local edge device, edge server, and cloud platform). Each agent is responsible for collecting local resource usage information and transmitting the data to a centralized monitoring platform.

[0024] System monitoring tools (such as top, htop, and Prometheus) can be used to track the CPU, GPU, and thread utilization, number of tasks processed, and execution time of compute nodes, assessing the load level of each node. Tools (such as free, vmstat, and Prometheus) can be used to collect and analyze memory usage, checking the memory usage of each compute node and monitoring the total and remaining memory of each node. Network performance tests (such as ping or traceroute) can be run to evaluate latency, upload and download bandwidth usage, and packet loss rate between different compute nodes. Energy monitoring hardware can be used to monitor the energy consumption of each compute node.

[0025] Finally, the monitoring data is transmitted from each computing node to a centralized data platform, where a distributed storage system is used to store historical monitoring data and to display the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in real time.

[0026] Subsequently, in box S104, tasks are dynamically allocated to different computing nodes based on the task's latency requirements, computational needs, data privacy requirements, and the real-time status of each computing node.

[0027] Edge computing performs initial, low-resource-consumption tasks closer to the data source, reducing data transmission time and bandwidth requirements. Cloud computing can utilize massive computing resources, storage space, and powerful computing capabilities for high-precision analysis and task completion, but network latency can impact response speed. Distributing computing tasks to the cloud can handle large-scale training tasks, but edge devices are more suitable for inference tasks requiring low latency. The system can dynamically select edge devices or the cloud for inference processing based on different task requirements and resource availability.

[0028] According to one embodiment of this disclosure, a hierarchical computing architecture for multi-layered AI processing nodes can efficiently schedule tasks based on the needs of different tasks and the availability of computing resources, thereby achieving efficient allocation and execution of AI inference tasks. Edge devices, such as smartphones, cameras, and embedded AI chips, are primarily responsible for performing lightweight inference tasks or preliminary data processing. For example, preliminary analysis and detection of images or basic preprocessing of speech signals. Furthermore, AI tasks involving sensitive data are processed on local nodes to ensure data privacy and security. Edge servers, such as 5G base stations and micro data centers, can handle tasks requiring more computing resources or medium-complexity inference tasks transmitted from edge devices. For example, partial inference of deep learning models, data aggregation, and preliminary data analysis. Cloud computing is suitable for AI tasks requiring substantial computing resources, such as deep learning training, large-scale data processing, and high-precision inference.

[0029] Data from computing nodes can be collected via sensors, and task requirements can be analyzed based on this data. For example, cameras, microphones, and IoT devices can be used to collect real-time data from computing nodes, including sensor information such as images, sound, temperature, and humidity, to assess the current environment and task requirements. After data preprocessing, task requirements are analyzed to determine if the task has strict latency requirements, identify the required computing resources (such as CPU, GPU, and memory), and whether it involves sensitive data.

[0030] AI inference tasks are categorized and allocated to different computing nodes using a hierarchical computing architecture. For example, low-latency tasks, lightweight inference tasks, preliminary data processing, and AI tasks involving sensitive data are prioritized for allocation to local edge devices. Low-latency tasks are those with high real-time requirements, such as speech recognition and real-time video processing. Lightweight inference tasks have low computational complexity and are suitable for rapid processing and real-time feedback. Preliminary data processing tasks involve preprocessing data and feature extraction, which typically do not require large-scale computing resources. High-latency-tolerant tasks and tasks with high computational demands are prioritized for allocation to the cloud platform; for example, deep learning model training and large-scale data analysis are prioritized for processing on the cloud platform. The optimal node is dynamically selected based on the current load of each computing node. For example, if a node is heavily loaded and resource-constrained, the task can be allocated to a less loaded node. During task execution, if the resource usage of a node changes (e.g., excessive load, increased latency), the task can be dynamically migrated to other nodes to ensure system efficiency and stability. By comprehensively analyzing the task's latency requirements, computational demands, data privacy requirements, and the real-time status of each computing node, optimal task allocation can be achieved.

[0031] In artificial intelligence inference tasks, data exchange between computing nodes is unavoidable. In this environment, data security and transmission efficiency are particularly important. In one embodiment of this disclosure, during data transmission between different computing nodes, data compression and encryption are performed on the data transmitted across nodes to ensure the security and efficiency of data transmission to the greatest extent possible.

[0032] Next, in box S106, the complexity of the AI ​​model used to perform the task is adjusted according to the resource usage of the computing node.

[0033] According to one embodiment of this disclosure, for resource-constrained computing nodes, the number of layers or neurons in an AI model can be reduced through quantization, pruning, or model partitioning. Quantization reduces the floating-point precision of the AI ​​model to a lower precision (e.g., using low-bit-width integers instead of floating-point numbers), thereby reducing the model's storage and computational requirements. For example, quantizing a 32-bit floating-point model into an 8-bit integer model can significantly reduce memory usage and computational load. Pruning reduces the computational complexity of the model by removing redundant neurons or connections from the neural network. For example, structured or unstructured pruning techniques can be used to remove parameters that have little impact on model performance. Model partitioning divides a large-scale AI model into multiple sub-models, each responsible for only a portion of the model's functionality. This allows the model to be distributed across multiple computing nodes, reducing the computational burden on individual nodes. The partitioned model can run a simplified portion on a local device, while the more complex portion is pushed to a more powerful edge server or the cloud for processing.

[0034] For computing nodes with ample resources, increase the number of layers or neurons in the AI ​​model. Increasing the number of layers in a neural network improves the model's learning and expressive capabilities; more layers typically mean the model can learn more complex features, making it suitable for handling high-precision or complex tasks. Increasing the number of neurons in each layer increases the model's computational power, enabling it to handle more complex features and tasks. This usually increases computational demands, but in resource-rich environments, it can effectively improve model performance.

[0035] During task execution, the resource usage of each computing node (such as memory, computing load, bandwidth, etc.) is continuously monitored, and the need to adjust the model complexity is assessed in real time. If resource stress is detected (e.g., insufficient memory, excessive CPU load, etc.), the model size and accuracy can be automatically adjusted to ensure the continuous execution of the task without resource overload.

[0036] Finally, in box S108, the task allocation strategy is optimized based on historical performance data and real-time network conditions during task execution.

[0037] By tracking and recording metrics such as processing capacity, latency, load, and computing resource usage of each node in real time, we can understand the performance of different nodes when handling different tasks. We monitor network bandwidth, latency, and stability in real time to determine whether the current network environment is suitable for task execution or whether task allocation needs adjustment. We continuously monitor the execution time, resource consumption, and task completion quality of inference tasks. We dynamically adjust task allocation strategies using reinforcement learning algorithms.

[0038] In some embodiments of this disclosure, the specific process of the feedback-based optimization mechanism is as follows: Initial workload allocation is performed based on current network conditions, historical performance data, and the basic characteristics of the task (such as task type, priority, and resource requirements). During task execution, the execution status of the task is continuously monitored, including node load, task progress, and network stability. The workload allocation strategy is dynamically adjusted using a reinforcement learning algorithm to optimize task execution. Strategy updates may include adjusting task allocation priorities, selecting more suitable computing nodes, or adjusting the parameters of the allocation strategy. The reinforcement learning algorithm includes defining a state space and an action space. The state space includes information such as historical performance data, current network conditions, and task complexity. The action space includes decisions on allocating tasks to different computing nodes, such as edge devices, edge servers, or cloud computing nodes. Rewards are calculated based on the performance of the task. For example, if the task is successfully completed within a predetermined time and computing resources are used appropriately, the system will provide a positive reward; if the task fails or resources are used improperly, a negative reward will be given. Through the feedback mechanism, the system can determine which tasks should be prioritized for allocation to edge devices, which need to be processed by edge servers, and which need to be pushed to the cloud. This ensures fast processing of low-latency tasks without wasting computing resources on edge nodes for high-complexity tasks.

[0039] Figure 2 This is a schematic block diagram of a dynamic allocation apparatus for AI inference tasks according to embodiments of the present disclosure. Figure 2 As shown, the device 200 may include a processor 210 and a memory 220 storing a computer program. When the computer program is executed by the processor 210, the device 200 is made capable of performing actions such as... Figure 1 The steps of method 100 are shown. In one example, device 200 may be a computer device or a cloud computing node. Device 200 can monitor in real time the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in the cloud-edge collaborative system, including local edge devices, edge servers, and cloud platforms; dynamically allocate tasks to different computing nodes based on task latency requirements, computing needs, data privacy requirements, and the real-time status of each computing node; adjust the complexity of the AI ​​model used to execute the task based on the resource usage of the computing nodes; and optimize the task allocation strategy based on historical performance data and real-time network conditions during task execution.

[0040] In some embodiments of this disclosure, the apparatus 200 can use system monitoring tools to track the CPU, GPU, and thread utilization, number of tasks processed, and execution time of computing nodes, and assess the load level of each node; check the memory usage of each computing node, and monitor the total memory and remaining memory of each node; run network performance tests to evaluate the latency, upload and download bandwidth, and packet loss rate between different computing nodes; monitor the energy consumption of each computing node in real time; and transmit monitoring data from each computing node to a centralized data platform, use a distributed storage system to store historical monitoring data, and display the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in real time.

[0041] In some embodiments of this disclosure, the device 200 can collect data from computing nodes through sensors, analyze task requirements based on the collected data, determine whether the task has strict requirements on latency, clarify the computing resources required by the task and whether it involves sensitive data; classify AI inference tasks, and allocate AI inference tasks to different computing nodes through a hierarchical computing architecture.

[0042] In some embodiments of this disclosure, the device 200 may prioritize the allocation of low-latency tasks, lightweight inference tasks, preliminary data processing, and AI tasks involving sensitive data to local edge devices; prioritize the allocation of high-latency-tolerant tasks and computationally demanding tasks to the cloud platform; and dynamically select the optimal node based on the current load of each computing node.

[0043] In some embodiments of this disclosure, the device 200 can reduce the number of layers or neurons in the AI ​​model for resource-constrained computing nodes through quantization, pruning, or model segmentation; and increase the number of layers or neurons in the AI ​​model for resource-sufficient computing nodes.

[0044] In some embodiments of this disclosure, the device 200 can track and record the processing power, latency, load, and computing resource usage of each computing node in real time; monitor network bandwidth, latency, and stability in real time to determine whether the current network environment is suitable for executing tasks or whether task allocation needs to be adjusted; continuously monitor the execution time, resource consumption, and task completion quality of inference tasks; and dynamically adjust the task allocation strategy through reinforcement learning algorithms.

[0045] In some embodiments of this disclosure, the device 200 may define a state space and an action space. The state space includes historical performance data, current network status, and task complexity information. The action space includes decisions on assigning tasks to different computing nodes. Rewards are calculated based on the performance of the task. If the task is completed successfully within a predetermined time and computing resources are used properly, a positive reward is given. If the task fails or resources are used improperly, a negative reward is given.

[0046] In some embodiments of this disclosure, the apparatus 200 can compress and encrypt data transmitted between different computing nodes during data transmission.

[0047] In embodiments of this disclosure, processor 210 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. Memory 220 may be any type of memory implemented using data storage technologies, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk storage, etc.

[0048] Furthermore, in embodiments of this disclosure, device 200 may also include input device 230, such as a keyboard, mouse, etc. Additionally, device 200 may also include output device 240, such as a display, etc.

[0049] In other embodiments of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program, when executed by a processor, is capable of performing the following functions: Figure 1 The steps of the method shown.

[0050] In summary, the AI ​​inference task dynamic allocation method and apparatus according to the embodiments of this disclosure effectively address the differences in computing capabilities among different nodes and avoid excessive resource consumption by dynamically adjusting the complexity of the AI ​​model for task execution through real-time monitoring of the resource usage of computing nodes. In a multi-layered computing environment encompassing local edge devices, edge servers, and cloud platforms, task execution can adjust the complexity of the computing model based on the current state of the nodes, thereby achieving more efficient resource utilization. The task allocation strategy can be automatically adjusted according to the needs of different tasks, environmental changes, and resource status. With the accumulation of task execution experience, continuous learning and optimization ultimately lead to a more intelligent and efficient task allocation model. Therefore, this solution can significantly improve the execution efficiency and overall performance of AI inference tasks in cloud-edge collaborative systems.

[0051] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses and methods according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0052] Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the terms “comprising” and “including” shall be interpreted as including rather than exclusively. Likewise, the terms “including” and “or” shall be interpreted as including unless such interpretation is expressly prohibited herein. Where the term “example” is used herein, particularly when it follows a set of terms, the “example” is merely exemplary and illustrative and should not be considered exclusive or extensive.

[0053] Further aspects and scope of adaptation become apparent from the description provided herein. It should be understood that various aspects of this application may be implemented individually or in combination with one or more other aspects. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0054] Several embodiments of this disclosure have been described in detail above. However, it is obvious that those skilled in the art can make various modifications and variations to the embodiments of this disclosure without departing from the spirit and scope of this disclosure. The scope of protection of this disclosure is defined by the appended claims.

Claims

1. A method for dynamically allocating AI inference tasks, characterized in that, include: Real-time monitoring of the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node in the cloud-edge collaborative system, including local edge devices, edge servers, and cloud platforms; Based on the task's latency requirements, computational demands, data privacy requirements, and the real-time status of each computing node, tasks are dynamically allocated to different computing nodes. This includes: collecting data from computing nodes via sensors, analyzing the collected data to determine task requirements, whether the task has strict latency requirements, clarifying the required computing resources, and whether it involves sensitive data; classifying AI inference tasks and allocating them to different computing nodes through a layered computing architecture; prioritizing the allocation of low-latency tasks, lightweight inference tasks, preliminary data processing, and AI tasks involving sensitive data to local edge devices; prioritizing the allocation of high-latency-tolerant tasks and tasks with high computational demands to the cloud platform; and dynamically selecting the optimal node based on the current load of each computing node. Based on the resource usage of computing nodes, the complexity of the AI ​​model used to perform the task is adjusted, including: for resource-constrained computing nodes, reducing the number of layers or neurons in the AI ​​model through quantization, pruning, or model partitioning; for resource-sufficient computing nodes, increasing the number of layers or neurons in the AI ​​model; and Based on historical performance data and real-time network conditions during task execution, the task allocation strategy is optimized. This includes: real-time tracking and recording of the processing capacity, latency, load, and computing resource usage of each computing node; real-time monitoring of network bandwidth, latency, and stability to determine whether the current network environment is suitable for task execution or whether task allocation needs to be adjusted; continuous monitoring of the execution time, resource consumption, and task completion quality of inference tasks; and dynamic adjustment of the task allocation strategy through reinforcement learning algorithms, including: calculating rewards based on the performance of task execution; providing positive rewards if the task is successfully completed within a predetermined time and computing resources are used appropriately; and providing negative rewards if the task fails or resources are used improperly.

2. The dynamic allocation method for AI inference tasks according to claim 1, characterized in that, The real-time monitoring system for cloud-edge collaboration tracks the workload, memory availability, network latency, bandwidth, and energy consumption of each computing node. These computing nodes include local edge devices, edge servers, and the cloud platform. Use system monitoring tools to track the CPU, GPU, and thread utilization, number of tasks processed, and execution time of compute nodes to assess the load level of each node. Check the memory usage of each compute node, and monitor the total memory and remaining memory of each node; Run network performance tests to evaluate latency, upload and download bandwidth, and packet loss rate between different computing nodes; Real-time monitoring of energy consumption for each computing node; and The monitoring data is transmitted from each computing node to a centralized data platform, and a distributed storage system is used to store historical monitoring data. The workload, memory availability, network latency, bandwidth and energy consumption of each computing node are displayed in real time.

3. The dynamic allocation method for AI inference tasks according to claim 1, characterized in that, The strategy for dynamically adjusting task allocation through reinforcement learning algorithms includes: Define a state space and an action space. The state space includes historical performance data, current network status, and task complexity information. The action space includes decisions on assigning tasks to different computing nodes.

4. The dynamic allocation method for AI inference tasks according to claim 1, characterized in that, The method further includes: compressing and encrypting the data transmitted between different computing nodes during the data transmission process.

5. A dynamic allocation device for AI inference tasks, characterized in that, The device includes: At least one processor; and At least one memory storing a computer program; When the computer program is executed by the at least one processor, the apparatus performs the steps of the method according to any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 4.

Images