AI-based large-scale computer room computing power resource dynamic scheduling operation and maintenance system
By using AI-driven kernel status monitoring and synchronous tick scheduling, precise computing power allocation for critical path tasks in large-scale data centers is achieved, solving the problems of resource fragmentation and system throughput reduction in existing technologies and improving the overall performance of computing clusters.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHAANXI KERIDI ELECTRONIC TECH CO LTD
- Filing Date
- 2026-03-10
- Publication Date
- 2026-06-09
Smart Images
Figure CN122173238A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of cloud computing resource management technology, and in particular relates to an AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources. Background Technology
[0002] Currently, large-scale data center computing resource management relies on a distributed orchestration framework, which distributes virtualized computing power quotas based on the physical load status of computing nodes. When processing directed acyclic graph computing tasks with strong synchronization constraints, resource consumption exhibits nonlinear step characteristics, and the computing phase switching is on the order of microseconds, requiring the underlying resource orchestration mechanism to complete the computing power reset within an extremely short time window. However, existing cloud computing software adjusts resource quotas through the operating system kernel interface, and cross-kernel system calls generate millisecond-level latency, causing a physical misalignment between the timing of resource injection and the computing task rhythm. This misalignment leads to thread blocking and system bus congestion, reducing the overall throughput efficiency of the cluster.
[0003] To address these challenges, the industry has attempted to smooth computing power fluctuations by shortening the sampling period or introducing hysteresis intervals. However, this linear improvement approach based on physical clocks cannot reconcile the inherent contradiction between scheduling accuracy and system stability, leading to fragmented aggregation of computing power in clusters under heavy load conditions. Besides linear allocation strategies at the overall resource level, existing technologies attempt to introduce state machine models at the software control flow level to optimize task delivery logic. For example, Chinese invention patent application CN121579165A discloses a cloud-based CAE workflow control subsystem and method that converts CAE workflow configuration files that may contain circular dependencies into directed acyclic graph state machines through strongly connected component analysis, dynamically deriving and... Task groups are scheduled to run in the cloud. However, existing technologies implicitly rely on idealized premises, namely that the task is completed, the topology is decoupled, and the task is distributed. The underlying heterogeneous computing nodes can seamlessly and instantly take over the computing power demand. The control logic only stays at the overall business application layer and flows through the directed acyclic graph. When the task is not processed by a large-scale cluster with strong synchronization constraints, the underlying microsecond-level computing phase switching and network protocol stack synchronization block the objective physical reality. The mechanism that emphasizes overall logic decoupling but neglects the surface physical perception cannot penetrate to the kernel state to capture system call trajectories and communication waiting windows. Under actual high-frequency bursts and strong synchronization conditions, resource scheduling instructions are constrained by the millisecond-level latency barrier of cross-kernel calls of the operating system, causing system bus congestion and high-frequency context switching overhead, resulting in a precipitous drop in cluster throughput.
[0004] Therefore, the technical problem to be solved by this invention is how to establish a scheduling mechanism that breaks through the kernel call latency barrier and achieves microsecond-level precise resource injection for computing tasks with strong synchronization dependencies in a large-scale heterogeneous computing environment. Summary of the Invention
[0005] This invention provides an AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources, comprising: The kernel status monitoring module is used to obtain the system call trajectory of the task to be scheduled in the operating system kernel mode and the synchronous blocking state of the network protocol stack, so as to extract runtime characteristic data that characterizes the execution phase of the task to be scheduled. The scheduling priority quantification module is used to calculate and generate a scheduling urgency index that reflects the level of computing resource allocation based on the topological weight of the task to be scheduled in the directed acyclic graph of tasks and the running characteristic data. The resource allocation retrieval module is used to convert scheduling urgency indicators into retrieval vectors according to preset vector conversion rules, and match the corresponding target control group quotas from the preset resource allocation blueprint library. The synchronous tick scheduling execution module is used to monitor the synchronous blocking state. When the synchronous blocking state indicates that the task to be scheduled has entered the communication waiting window, the target control group quota is written to the resource limit parameter file of the task to be scheduled through the kernel control group interface. The resource quota update action is locked within the communication waiting window so as to complete the computing resource scheduling within the logical rest period of the task to be scheduled.
[0006] Preferably, the resource allocation retrieval module includes a strategy preloading unit and a retrieval unit; the strategy preloading unit is used to generate a discretized resource allocation blueprint in advance based on the task topology characteristics; the retrieval unit is used to map the scheduling urgency index to the corresponding target control group limit through a hash index, and to update the computing power addressing boundary value in the resource limit parameter file of the task to be scheduled in situ using virtual file system overwrite technology.
[0007] Preferably, the kernel status monitoring module includes a call counting unit and a network status acquisition unit; the call counting unit is used to count the frequency of kernel function calls of the task to be scheduled within a preset period; the network status acquisition unit is used to identify the communication blocking period generated by the task to be scheduled in the parallel computing cluster by capturing the state of synchronization primitives in the kernel protocol stack, and to identify the communication blocking period as a communication waiting window.
[0008] Preferably, the scheduling priority quantification module is also used to identify the critical path attribute of the task to be scheduled in the directed acyclic graph of tasks; if the task to be scheduled belongs to a critical path node, the synchronous tick scheduling execution module is guided to allocate computing power quota to the task to be scheduled by increasing the calculation weight of the scheduling urgency index.
[0009] Preferably, the scheduling priority quantification module performs the following calculation logic when generating the scheduling urgency index: , where λ is the scheduling urgency index; The critical path weights of the tasks to be scheduled in the directed acyclic graph of tasks; This refers to the average synchronization wait time of the scheduled task within the communication wait window, collected by the kernel status monitoring module. This represents the fragmentation rate of computing resources on the computing node where the task to be scheduled resides.
[0010] Preferably, the target control group limit includes memory bandwidth limit parameters and disk I / O priority parameters; when updating the resource limit parameter file, the synchronous tick scheduling execution module synchronously adjusts the memory bandwidth limit parameters and disk I / O priority parameters.
[0011] Preferably, the system also includes a prediction calibration module, which is used to model the occurrence period of the communication waiting window of the task to be scheduled based on the long short-term memory network model, and output pre-trigger control instructions to the synchronous tick scheduling execution module.
[0012] Preferably, before updating the execution parameters, the synchronous clock scheduling execution module also verifies whether the remaining duration of the communication waiting window is greater than the execution delay threshold required for the resource limit parameter file to take effect; if the remaining duration is less than the execution delay threshold, the current update action is postponed to the next communication blocking cycle.
[0013] Preferably, the kernel status monitoring module is also used to monitor the global utilization rate of the computing resource pool in real time. When the global utilization rate exceeds 90%, the resource configuration retrieval module is triggered to execute the resource preemption logic.
[0014] Preferably, the system also includes a dynamic topology maintenance module, which is used to capture dynamic dependency changes between tasks to be scheduled and update the directed acyclic graph of tasks to correct the input topology parameters of the scheduling priority quantization module.
[0015] Compared to existing technologies, the AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources of this invention has the following advantages: 1. In the dynamic scheduling of large-scale data center computing resources for AI, the consumption feature acquisition module and the urgency calculation module work together to extract the first-order consumption gradient of the task and couple it with the global topology synchronization barrier weight to establish a resource evaluation mechanism that can perceive the logical dependencies of distributed tasks. This enables the scheduling center to identify task nodes in the cluster that are on the critical path and face computing power bottlenecks. By accurately injecting computing power quotas into high-weight synchronization nodes, the global computing thread suspension state caused by local computing power shortage is broken. This logical evolution from single-point load balancing to global synchronization and collaboration eliminates the physical basis for the generation of long-tail tasks and ensures the high consistency of large-scale parallel computing tasks in terms of logical rhythm.
[0016] 2. The computing power topology map computer-aided design unit and the high-speed retrieval unit work together to pre-generate and pre-load spatial logical allocation blueprints for nodes with different discrete urgency levels based on the characteristics of the directed acyclic graph. When the resource urgency index changes in real time, the target allocation map is hit through feature vector hash retrieval, and the addressing boundary of the task is updated in situ using bypass memory mapping technology. This scheme transforms the complex real-time scheduling instruction calculation into a map template retrieval with extremely low latency, bypassing the latency barrier of the operating system kernel mode call, reducing the resource response cycle to the microsecond level, and avoiding the loss of synchronization window due to instruction delay.
[0017] 3. The phase-locked scheduling module uses probes in the kernel network protocol stack to sense the synchronous blocking state of the second computing task, locking the reallocation of computing resources within the network communication waiting period of the task itself. By capturing the physical rest time window during the task's operation, and completing the reduction and increase of the quota of the underlying control group within the window, the system interruption cost generated by computing scheduling coincides with the inherent communication delay of the task. This lossless stripping logic at the surface phase eliminates the additional context switching overhead caused by high-frequency resource preemption, and solves the system bus congestion problem that is easily caused when a large number of requests are scheduled in a short period of time in a large-scale data center. Attached Figure Description
[0018] Figure 1 This is a functional module and architecture diagram of the AI large-scale data center computing power resource dynamic scheduling system of the present invention; Figure 2 This is a schematic diagram of the logical flow of task execution phase perception and accurate update of computing resource quota in this invention. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0020] It should be noted that all directional and positional terms used in this invention, such as: up, down, left, right, front, back, vertical, horizontal, inner, outer, top, bottom, transverse, longitudinal, center, etc., are only used to explain the relative positional relationship and connection between components in a specific state (as shown in the accompanying drawings). They are only for the convenience of describing this invention and do not require that this invention be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention. In addition, the descriptions of "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.
[0021] In the description of this invention, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections; they can refer to direct connections or indirect connections through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in this invention can be understood in conjunction with the specific circumstances.
[0022] In the description of this specification, references to the terms "an embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example, and the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0023] A large-scale data center computing resource dynamic scheduling and operation and maintenance system based on AI, comprising: The kernel status monitoring module is used to obtain the system call trajectory of the task to be scheduled in the operating system kernel mode and the synchronous blocking state of the network protocol stack, so as to extract runtime characteristic data that characterizes the execution phase of the task to be scheduled. The scheduling priority quantification module is used to calculate and generate a scheduling urgency index that reflects the level of computing resource allocation based on the topological weight of the task to be scheduled in the directed acyclic graph of tasks and the running characteristic data. The resource allocation retrieval module is used to convert scheduling urgency indicators into retrieval vectors according to preset vector conversion rules, and match the corresponding target control group quotas from the preset resource allocation blueprint library. The synchronous tick scheduling execution module is used to monitor the synchronous blocking state. When the synchronous blocking state indicates that the task to be scheduled has entered the communication waiting window, the target control group quota is written to the resource limit parameter file of the task to be scheduled through the kernel control group interface. The resource quota update action is locked within the communication waiting window so as to complete the computing resource scheduling within the logical rest period of the task to be scheduled.
[0024] Preferably, the resource allocation retrieval module includes a strategy preloading unit and a retrieval unit; the strategy preloading unit is used to generate a discretized resource allocation blueprint in advance based on the task topology characteristics; the retrieval unit is used to map the scheduling urgency index to the corresponding target control group limit through a hash index, and to update the computing power addressing boundary value in the resource limit parameter file of the task to be scheduled in situ using virtual file system overwrite technology.
[0025] Preferably, the kernel status monitoring module includes a call counting unit and a network status acquisition unit; the call counting unit is used to count the frequency of kernel function calls of the task to be scheduled within a preset period; the network status acquisition unit is used to identify the communication blocking period generated by the task to be scheduled in the parallel computing cluster by capturing the state of synchronization primitives in the kernel protocol stack, and to identify the communication blocking period as a communication waiting window.
[0026] Preferably, the scheduling priority quantification module is also used to identify the critical path attribute of the task to be scheduled in the directed acyclic graph of tasks; if the task to be scheduled belongs to a critical path node, the synchronous tick scheduling execution module is guided to allocate computing power quota to the task to be scheduled by increasing the calculation weight of the scheduling urgency index.
[0027] Preferably, the scheduling priority quantification module performs the following calculation logic when generating the scheduling urgency index: , where λ is the scheduling urgency index; The critical path weights of the tasks to be scheduled in the directed acyclic graph of tasks; This refers to the average synchronization wait time of the scheduled task within the communication wait window, collected by the kernel status monitoring module. This represents the fragmentation rate of computing resources on the computing node where the task to be scheduled resides.
[0028] Preferably, the target control group limit includes memory bandwidth limit parameters and disk I / O priority parameters; when updating the resource limit parameter file, the synchronous tick scheduling execution module synchronously adjusts the memory bandwidth limit parameters and disk I / O priority parameters.
[0029] Preferably, the system also includes a prediction calibration module, which is used to model the occurrence period of the communication waiting window of the task to be scheduled based on the long short-term memory network model, and output pre-trigger control instructions to the synchronous tick scheduling execution module.
[0030] Preferably, before updating the execution parameters, the synchronous clock scheduling execution module also verifies whether the remaining duration of the communication waiting window is greater than the execution delay threshold required for the resource limit parameter file to take effect; if the remaining duration is less than the execution delay threshold, the current update action is postponed to the next communication blocking cycle.
[0031] Preferably, the kernel status monitoring module is also used to monitor the global utilization rate of the computing resource pool in real time. When the global utilization rate exceeds 90%, the resource configuration retrieval module is triggered to execute the resource preemption logic.
[0032] Preferably, the system also includes a dynamic topology maintenance module, which is used to capture dynamic dependency changes between tasks to be scheduled and update the directed acyclic graph of tasks to correct the input topology parameters of the scheduling priority quantization module.
[0033] Example 1: In a heterogeneous computing cluster environment that supports massively parallel computing and high-frequency bursty AI inference tasks, the computing tasks exhibit discrete step fluctuation characteristics during operation. Since the computation phase switching occurs on the order of microseconds, the underlying resource orchestration mechanism needs to complete the computational power reset within an extremely short time. Under this condition, the kernel status monitoring module acquires the system call trajectory of the task to be scheduled in the operating system kernel state and the synchronous blocking state of the network protocol stack, extracting runtime characteristic data representing the execution phase of the task to be scheduled. The scheduling priority quantization module calculates the scheduling urgency index λ, reflecting the level of computing resource allocation, based on the topological weight of the task in the directed acyclic graph and the runtime characteristic data. The scheduling priority quantization module executes the following calculation logic: , where λ is the scheduling urgency index; The critical path weights of the tasks to be scheduled in the directed acyclic graph of tasks; This represents the average synchronization wait time of the task to be scheduled within the communication waiting window. This represents the fragmentation rate of computing resources on the computing node where the task to be scheduled resides.
[0034] The resource allocation retrieval module uses preset vector transformation rules to convert scheduling urgency indicators. The process involves converting the data into a retrieval vector and matching the target control group quota from a pre-defined resource allocation blueprint library. The synchronous tick scheduling execution module monitors the synchronous blocking state and, when the synchronous blocking state indicates that the task to be scheduled has entered a communication waiting window, writes the target control group quota into the resource limit parameter file of the task to be scheduled via the kernel control group interface. This step locks the resource quota update action within the logical rest period of the task to be scheduled, utilizing the inherent communication latency gaps of the task to complete the adjustment of the underlying kernel control group quota. The target control group quota includes memory bandwidth limit parameters and disk I / O priority parameters. Before the synchronous tick scheduling execution module performs the resource quota overwrite, the system obtains the system timestamp through a kernel-mode high-precision timer. The communication window for the prediction calibration module outputs the expected end timestamp. Determine the remaining duration of the window Read the execution latency calibration table pre-stored in the memory-mapped area to obtain the average latency of resource limit parameter file taking effect under the hardware environment. With respect to the system scheduling jitter threshold σ; When the condition is met, the synchronous tick scheduling execution module pre-establishes file descriptor handles and offset parameters, and uses kernel bypass I / O technology to write the target control group quota data into the virtual file system memory buffer. Specifically, during system initialization, the control group configuration file is linked to the user-mode virtual address space via a memory mapping interface. Based on the file system physical page alignment standard, the absolute addressing offset of the resource limit parameters within the file is determined to be 1024 bytes. During updates, a 64-bit wide atomic storage instruction is executed directly at this offset address to overwrite the data. This operation avoids the system delay of more than 2ms caused by inode lock contention and log synchronization processes involved in standard file system calls, ensuring that parameter injection is completed within 500ns without triggering kernel-mode context switching, and avoiding the disk index refresh overhead in the standard file system synchronization process. If the condition is not met, the current scheduling instruction is pushed into the processing queue and postponed to the next communication blocking cycle. The average effective delay is... By performing 1000 zero-load write tests during the system initialization phase and determining the execution time at the 95th percentile, the scheduling action is ensured to be absolutely phase-aligned with the task's logical rest period at the physical clock level. The system monitors the global utilization of the computing resource pool in real time. When the global utilization exceeds 90%, the resource configuration retrieval module is triggered to execute the resource preemption logic. This scheduling method makes the system interruption cost caused by computing power scheduling coincide with the inherent communication latency of the task, eliminates the additional context switching overhead caused by high-frequency resource preemption, and improves the overall throughput efficiency of the cluster and the global turnover efficiency of the resource pool.
[0035] Example 2: In a distributed computing cluster containing 64 heterogeneous computing nodes, the AI inference business trajectory stream from the production environment was captured as input data. This data included 1000 directed acyclic graphs of tasks with topological dependencies. In the experimental environment, 50Hz power frequency electromagnetic interference noise and network background jitter with a 5% random packet loss rate were injected via a signal generator. The physical server executing the monitoring task possessed kernel probing capabilities, achieving a measurement resolution of 1μs. The sampling period was... The determining logic lies in balancing the capture accuracy of kernel monitoring with the processor cycle overhead caused by the monitoring actions. When the system call frequency of the monitored task is at a high frequency of over 10,000 times per second, the sampling period is adjusted to capture phase switching points on the order of microseconds. The critical path weights required for the scheduling urgency index λ are set to 100μs. The synchronization wait time is calculated based on the topology of the directed acyclic graph of the task. Resource fragmentation rate is obtained in real time during task execution logic via kernel protocol stack probes. The ratio of the idle computing power page table under the control group to the total computing power quota is used to determine the allocation. Control group 1 adopts a static computing power allocation method, while control group 2 adopts a dynamic scheduling scheme without a phase-locked loop (PLL) mechanism. The present invention's sample group executes PLL scheduling through a synchronous tick scheduling execution module. Under conditions of superimposed background interference, the average task turnaround time for control group 1 is 125.6 ms. Control group 2 experiences frequent system call retries due to misalignment between scheduling instructions and computation phases. The tail latency rose to 185.3ms. The prototype of this invention locks the resource write operation within a communication waiting window. The tail latency stabilized at 45.2ms, and the system throughput increased from 320 tasks per second in control group 2 to 850 tasks per second, indicating that using the task logic rest period for quota overwriting reduced the performance loss caused by scheduling.
[0036] In a gradient scenario with low, medium, and high task synchronization densities, when the synchronization density changes abruptly from low to high, the computing power utilization of control group 2 shows a decreasing trend. In the high synchronization density scenario, its overall cluster throughput efficiency drops to below 30% of the initial value. This invention's sample group, due to its scheduling urgency index... Synchronization wait time Dynamic compensation is achieved through the increase of [something], so that the cluster's throughput performance under high-density synchronization pressure is maintained at more than 92% of the initial value. The formula for calculating the scheduling urgency index λ is as follows: , where λ is the scheduling urgency index; The critical path weights of the tasks to be scheduled in the directed acyclic graph of tasks; This refers to the average synchronization wait time of the tasks to be scheduled within the communication waiting window. The fragmentation rate of computing resources on the computing node where the task to be scheduled resides is shown in the verification results for the parameter boundaries, which indicate that during the sampling period... After exceeding the 500μs threshold, the success rate of the system in capturing sub-millisecond communication waiting windows decreased, and the number of measured context switches increased by 4.5 times compared to the optimal range, leading to a deterioration in system performance. This confirms the stability of the parameter range in ensuring the physical phase-locking process between scheduling instructions and the calculated phase.
[0037] Example 3: In a heterogeneous computing cluster carrying out large language model inference tasks, the directed acyclic graph of tasks exhibits multi-level logical dependencies. Computing nodes experience asymmetric load fluctuations when processing long text sequences. The kernel state monitoring module sets the sampling frequency of the kernel detector to 10kHz for the logical kernels in the computing nodes and establishes a circular buffer of kernel-state events with a capacity of 4096 entries. The scheduling priority quantization module extracts the execution dependency chain of the inference task and determines the critical path weights by performing Laplace matrix eigenvalue decomposition on the directed acyclic graph of tasks. , Based on the ratio of out-degree to in-degree of task nodes, the resource allocation retrieval module uses a dimensionality reduction method based on locality-sensitive hashing to project the vector composed of running feature data and weight information onto a 128-dimensional feature space to generate a retrieval vector. Discrete computing power quota templates are pre-stored in the resource allocation blueprint library. Each quota template corresponds to a specific computing power addressing boundary value, which is the central processing unit's periodic quota. The resource allocation retrieval module normalizes the scheduling priority quantization module output by the feature extraction operator, the scheduling urgency index λ, the out-degree to in-degree ratio of the directed acyclic graph node to which the task to be scheduled belongs, and the current free page table size of the computing node. These are then concatenated in a preset dimension order to generate a 128-dimensional retrieval vector V. The locality-sensitive hashing algorithm is used to perform a nearest neighbor search in the preset resource allocation blueprint library space to calculate the cosine similarity between the retrieval vector V and the existing feature vectors in the library, and to match the discretized resource allocation template with the minimum Euclidean distance.
[0038] The template is pre-generated by exhaustively enumerating the system call latency distribution under different load gradients and establishing hash mapping relationships during the offline benchmark testing phase. The system maintains a high-speed cache index table with a capacity of 4096 entries in kernel mode to realize the mapping between the retrieval vector V and the target control group limit with constant-time complexity, compressing the scheduling decision delay to within 500 nanoseconds, and ensuring that the resource injection action takes effect before the computation phase switches to the active state. The retrieval unit matches the target control group limit with the smallest Euclidean distance from the resource allocation blueprint library according to the cosine similarity criterion. After the synchronous tick scheduling execution module obtains the target control group limit, it establishes a pointer to the resource limit parameter file in the virtual file system. The system calls the file descriptor handle and execute system calls based on preset offset parameters when the network status acquisition unit returns a synchronization primitive release signal. This operation writes the target control group quota data content to the resource limit parameter file of the task to be scheduled in place through the file system inode. This operation does not trigger the file system cache refresh action, and the physical latency of a single scheduling instruction is less than 500ns. This aligns the quota adjustment action of computing resources with the logical rest period of computing tasks, offsetting the execution loss in the cross-node synchronization process. Under the condition that the concurrent request volume of the computing cluster increases by 200%, the fluctuation range of response latency is within 3%.
[0039] Example 4: When the system faces the initial deployment of a heterogeneous computing power cluster, the construction process of the resource allocation blueprint library is implemented through an offline benchmark test program. Parallel computing tasks containing multiple sets of independent variables are started within the computing nodes. The kernel status monitoring module collects system call trajectories under different computing power quotas. The scheduling priority quantization module determines performance indicators based on the measured response latency and throughput data in the system call trajectories. The resource configuration retrieval module uses a mapping function to convert the performance indicators into resource templates. The retrieval unit converts the resource templates into 128-dimensional feature vectors using a hash index and persists them in the feature space of the resource allocation blueprint library.
[0040] When the computing cluster contains heterogeneous computing nodes with different instruction set architectures, the system initiates a pre-deployment calibration procedure before task loading, and the network status acquisition unit sends synchronization primitives to each computing node to measure the baseline communication latency. The synchronous clock scheduling execution module is based on the baseline communication delay. Synchronization wait time with real-time acquisition The ratio determines the scheduling urgency index. The corrected weights; where λ is the scheduling urgency index, As a reference communication delay, To synchronize waiting time, the system monitors the response frequency of the virtual file system's inodes to calibrate the offset parameters required for in-situ overwrite, thereby locking the update timing of the resource limit parameter file at the beginning phase of the logical rest period, so that the computing resource pool can maintain the consistency of task execution rhythm in a cross-architecture scheduling environment.
[0041] Example 5: In a large-scale data center computing resource dynamic scheduling and operation system with a heterogeneous instruction set architecture, the system performs benchmark performance sampling for computing nodes, loads a standardized test load containing matrix multiplication and fast Fourier transform logic into the kernel status monitoring module, captures the system call latency distribution of the central processing unit under a load gradient of 10% to 90%, and extracts the benchmark communication latency from the scheduling priority quantization module. The collected computing power consumption curves are logically mapped to the physical addressing boundary values of the resource pool, generating a resource allocation blueprint library containing 1024 discrete sampling points. Each data entry is located using a hash index, and a correspondence between the scheduling urgency index λ and the target control group limit is established within a 128-dimensional feature space. λ represents the baseline communication delay, and λ represents the scheduling urgency index.
[0042] When a new computing node is added to the computing cluster, the system starts a threshold calibration program. The kernel status monitoring module determines the average duration of the communication waiting window by monitoring the release frequency of synchronization primitives in the network protocol stack. Based on this, it determines the judgment coefficient of the scheduling urgency index λ. The synchronous tick scheduling execution module performs an empty write test at the beginning phase of the logical rest period to measure the physical latency residual from the initiation of the kernel state instruction to the effective date of the virtual file system parameters. Based on the physical latency residual, it corrects the preset offset parameters of the system call so that the overwriting of the resource limit parameter file is completed within the communication waiting window. The computing resource allocation status of the scheduled task maintains consistent tick under different hardware architecture environments.
[0043] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit of this application and the scope of protection of this invention, and all of these forms are within the protection scope of this application.
Claims
1. A large-scale data center computing resource dynamic scheduling and operation and maintenance system based on AI, characterized in that, include: The kernel status monitoring module is used to obtain the system call trajectory of the task to be scheduled in the operating system kernel mode and the synchronous blocking state of the network protocol stack, so as to extract runtime characteristic data that characterizes the execution phase of the task to be scheduled. The scheduling priority quantification module is used to calculate and generate a scheduling urgency index that reflects the level of computing resource allocation based on the topological weight of the task to be scheduled in the directed acyclic graph of tasks and the running characteristic data. The resource allocation retrieval module is used to convert scheduling urgency indicators into retrieval vectors according to preset vector conversion rules, and match the corresponding target control group quotas from the preset resource allocation blueprint library. The synchronous tick scheduling execution module is used to monitor the synchronous blocking state. When the synchronous blocking state indicates that the task to be scheduled has entered the communication waiting window, the target control group quota is written to the resource limit parameter file of the task to be scheduled through the kernel control group interface. The resource quota update action is locked within the communication waiting window so as to complete the computing resource scheduling within the logical rest period of the task to be scheduled.
2. The AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources according to claim 1, characterized in that, The resource allocation retrieval module includes a policy preloading unit and a retrieval unit. The policy preloading unit is used to generate a discretized resource allocation blueprint in advance based on the task topology characteristics. The retrieval unit is used to map the scheduling urgency index to the corresponding target control group limit through a hash index, and to update the computing power addressing boundary value in the resource limit parameter file of the task to be scheduled in situ using virtual file system overwrite technology.
3. The AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources according to claim 1, characterized in that, The kernel status monitoring module includes a call counting unit and a network status acquisition unit. The call counting unit is used to count the frequency of kernel function calls of the scheduled task within a preset period. The network status acquisition unit is used to identify the communication blocking period generated by the scheduled task in the parallel computing cluster by capturing the state of synchronization primitives in the kernel protocol stack, and to identify the communication blocking period as a communication waiting window.
4. The AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources according to claim 1, characterized in that, The scheduling priority quantization module is also used to identify the critical path attributes of the tasks to be scheduled in the directed acyclic graph of tasks; if the task to be scheduled belongs to a critical path node, the synchronous tick scheduling execution module is guided to allocate computing power quota to the task to be scheduled by increasing the calculation weight of the scheduling urgency index.
5. The AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources according to claim 1, characterized in that, When generating the scheduling urgency index, the scheduling priority quantization module performs the following calculation logic: , where λ is the scheduling urgency index; The critical path weights of the tasks to be scheduled in the directed acyclic graph of tasks; This refers to the average synchronization wait time of the scheduled task within the communication wait window, collected by the kernel status monitoring module. This represents the fragmentation rate of computing resources on the computing node where the task to be scheduled resides.
6. The AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources according to claim 1, characterized in that, The target control group limits include memory bandwidth limit parameters and disk I / O priority parameters; When updating the resource limit parameter file, the synchronous clock scheduling execution module simultaneously adjusts the memory bandwidth limit parameters and disk I / O priority parameters.
7. The AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources according to claim 1, characterized in that, The system also includes a prediction calibration module, which is used to model the occurrence period of the communication waiting window of the task to be scheduled based on the long short-term memory network model, and output pre-trigger control instructions to the synchronous tick scheduling execution module.
8. The AI-based dynamic scheduling and operation and maintenance system for large-scale data center computing resources according to claim 1, characterized in that, Before updating the execution parameters, the synchronous clock scheduling execution module also checks whether the remaining duration of the communication waiting window is greater than the execution delay threshold required for the resource limit parameter file to take effect; if the remaining duration is less than the execution delay threshold, the current update action will be postponed to the next communication blocking cycle.
9. A large-scale data center computing resource dynamic scheduling and operation and maintenance system based on AI according to claim 1, characterized in that, The kernel status monitoring module is also used to monitor the global utilization of the computing resource pool in real time. When the global utilization exceeds 90%, the resource configuration retrieval module is triggered to execute the resource preemption logic.
10. A large-scale data center computing resource dynamic scheduling and operation and maintenance system based on AI according to claim 1, characterized in that, The system also includes a dynamic topology maintenance module, which captures dynamic dependency changes between tasks to be scheduled and updates the directed acyclic graph of tasks to correct the input topology parameters of the scheduling priority quantization module.