A Multi-Source Data Task Scheduling Optimization Method Based on Kunpeng Computing Power Intelligent Agent

Abstract

This invention relates to the field of computer task scheduling and computing resource allocation technology, and particularly to a multi-source data task scheduling optimization method driven by a Kunpeng computing power agent. The method includes: collecting task requests, Kunpeng computing node running status information, task running logs, and queue scheduling information; generating a Kunpeng scheduling status dataset by timestamping and periodic alignment; extracting task source identifiers, data formats, data volumes, priorities, and data input addresses to generate task feature data; and combining this with the Kunpeng scheduling status dataset to generate a Kunpeng computing node resource profile; further performing adaptation structure difference analysis, load fluctuation analysis, data reading path processing, and dependent task status processing to generate a matching degree evaluation result, thereby forming a pre-scheduling result, task deployment instructions, and resource allocation instructions. This invention can improve the accuracy of task and node matching and enhance the continuity and adaptability of the scheduling link.

Description

Technical Field

[0001] This invention relates to the field of computer task scheduling and computing resource allocation technology, and in particular to a multi-source data task scheduling optimization method driven by Kunpeng computing power intelligent agents. Background Technology

[0002] In the field of computer task scheduling and computing resource allocation technology, existing solutions for Kunpeng computing nodes to carry multi-source data tasks typically collect task requests, Kunpeng computing node running status information, task running logs, and queue scheduling information. They then classify tasks, allocate resources, and deploy tasks based on priority, queue scheduling information, and node running status. However, these solutions suffer from limitations such as insufficient correlation between task characteristic data and Kunpeng computing node resource profiles, insufficient alignment between task execution cycles and node status cycles, and a lack of data reading path processing and dependency task status processing in the matching degree evaluation results. Existing methods often rely on the current resource status, fixed queue rules, or general scheduling strategies to generate task deployment and resource allocation instructions.

[0003] In scenarios where multi-source data tasks have different sources, data formats, data volumes, priorities, and data input addresses, existing solutions are prone to problems such as unclear correspondence between task requests and Kunpeng computing node running status information, difficulty in participating in pre-scheduling sorting of task running logs, and disconnection between queue scheduling information and load fluctuation analysis. These issues make it difficult to meet the requirements for the stable formation of matching degree evaluation results, pre-scheduling results, task deployment instructions, and resource allocation instructions.

[0004] For the joint processing of task feature data, Kunpeng computing node resource profiles, matching degree evaluation results, and feedback data, existing technologies generally lack a consistent process from collection, alignment, evaluation, pre-scheduling, instruction generation to feedback updates. This makes it difficult to establish continuous records between task requests, node running status, task running logs, queue scheduling information, and scheduling policy libraries in the task scheduling scenario of Kunpeng computing nodes, resulting in insufficient update links between matching degree evaluation results and feedback data. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a multi-source data task scheduling optimization method driven by a Kunpeng computing power intelligent agent, comprising:

[0006] S100: Collect task requests, Kunpeng computing node running status information, task running logs and queue scheduling information, add timestamps and align them with the task execution cycle and node status cycle to generate Kunpeng scheduling status dataset;

[0007] S200. Based on the Kunpeng scheduling status dataset, the task source identifier, data format, data volume, priority, and data input address are extracted by the lightweight intelligent agent module to generate task feature data.

[0008] S300. Generate a resource profile of the Kunpeng computing node based on the task feature data and the Kunpeng scheduling status dataset;

[0009] S400. Based on the task feature data and the Kunpeng computing node resource profile, perform adaptation structure difference analysis, load fluctuation analysis, data reading path processing and dependent task status processing to generate a matching degree evaluation result.

[0010] S500: Based on the matching degree evaluation results, perform task classification, pre-scheduling sorting, and resource pre-allocation to generate pre-scheduling results;

[0011] S600. Generate task deployment instructions and resource allocation instructions based on the pre-scheduling results, monitor the execution status of the instructions to generate feedback data, and update the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library.

[0012] Furthermore, in S100, the task request is collected by the task submission interface; the Kunpeng computing node running status information is collected by the node resource monitoring system; the task running log is collected by the task running log unit; and the queue scheduling information is collected by the queue scheduling management system and the scheduler global control component.

[0013] Furthermore, the time alignment includes writing the task submission time, task enqueue time, node status sampling time, and task execution time into the same scheduling window to generate a scheduling mapping relationship between the task and the Kunpeng computing node.

[0014] Furthermore, in S200, the task characteristic data includes computational complexity characteristics, real-time requirement characteristics, resource dependency characteristics, data format conversion requirements, data locality, input / output pressure, and resource consumption of similar historical tasks.

[0015] Furthermore, in S300, the Kunpeng computing node resource profile includes node static attributes, node real-time attributes, node historical attributes, and node adaptation attributes; the node static attributes include Kunpeng computing node identifier, processor core structure, memory capacity, storage channel, network interface, and operating environment; the node real-time attributes include processor core status, memory status, storage read / write status, network bandwidth and latency, and task queue length; the node historical attributes include node historical resource snapshots, node historical load rate sequences, and historical executed task data; the node adaptation attributes include task success rate and historical records of similar task executions.

[0016] Furthermore, in S400, the adaptation structure difference analysis includes: comparing the resource requirement description in the task feature data with the node scheduling resource status in the Kunpeng computing node resource profile to generate a list of tasks with good structure adaptation, tasks with unbalanced structure adaptation, and candidate nodes.

[0017] Further, in S400, the data reading path processing includes: generating a data reading path based on the data input address and the data locality; the dependent task status processing includes: generating a dependent task execution status based on the resource dependency characteristics and the task running log; and writing the data reading path and the dependent task execution status into the matching degree evaluation result.

[0018] Furthermore, in S500, task classification includes writing the task feature data into the execution queue; pre-scheduling sorting includes arranging the candidate node list according to the matching degree evaluation result, task queue length and execution window; resource pre-allocation includes generating target Kunpeng computing nodes, alternative Kunpeng computing nodes, resource allocation requests and execution windows.

[0019] Furthermore, in S600, the task deployment instruction includes the target Kunpeng computing node, the execution queue, the execution window, and the data reading path; the resource allocation instruction includes computing resources, storage resources, network resources, and runtime environment configuration; when the migration triggering condition matches the instruction execution status, a migration instruction is generated.

[0020] Furthermore, in step S600, the feedback data includes resource utilization rate, task success rate, deviation rate between actual resource consumption and estimated value, task queue length, migration execution status, and resource release information; and updates the resource requirement description in the task feature data, the node adaptation attributes in the Kunpeng computing node resource profile, and the candidate node screening rules, resource pre-allocation rules, and migration triggering conditions in the scheduling strategy library based on the feedback data.

[0021] The key innovations of this invention include:

[0022] (1) Establish a matching degree evaluation link between task feature data and Kunpeng computing node resource profile. Based on the task feature data and the Kunpeng computing node resource profile, perform adaptation structure difference analysis, load fluctuation analysis, data reading path processing and dependent task status processing to generate matching degree evaluation results.

[0023] (2) The matching degree evaluation result is used as the input for task classification, pre-scheduling sorting and resource pre-allocation to generate pre-scheduling result, so that the task deployment instruction and resource allocation instruction come from the same pre-scheduling result.

[0024] (3) Generate task deployment instructions and resource allocation instructions based on the pre-scheduling results, and generate feedback data by monitoring the execution status of the instructions. Use the feedback data to update the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library.

[0025] The following are its main beneficial effects:

[0026] (1) To address the problem of insufficient correlation between task feature data and Kunpeng computing node resource profile in the existing scheme, by adapting structural difference analysis, load fluctuation analysis, data reading path processing and dependent task status processing, the matching degree evaluation result includes both task-side information and node-side information, reducing the correspondence deviation caused by generating scheduling results based solely on the current resource status.

[0027] (2) To address the problem of insufficient connection between task classification and resource allocation in the existing scheme, the matching degree evaluation results are input into task classification, pre-scheduling sorting and resource pre-allocation, so that the pre-scheduling results include task feature data, candidate node list, data reading path and dependent task execution status, which facilitates the subsequent generation of task deployment instructions and resource allocation instructions.

[0028] (3) To address the issue that feedback data in the existing scheme is only used for adjusting a single scheduling strategy, feedback data is generated by monitoring the execution status of instructions, and the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library are updated, so that task requests, node running status, task running logs, and queue scheduling information are continuously recorded in subsequent scheduling processes.

[0029] (4) To address the problem of scattered data input addresses in multi-source data tasks, the data reading path is written into the matching degree evaluation result and transmitted to the task deployment instruction by the pre-scheduling result, so that there is a clear correspondence between the task deployment instruction and the data input address.

[0030] (5) To address the problem that the status of dependent tasks is difficult to participate in scheduling judgment, the status of dependent tasks is processed according to the task running log and written into the matching degree evaluation result, so that the pre-scheduling sorting and resource pre-allocation call the status of dependent tasks in the same scheduling link. Attached Figure Description

[0031] Figure 1 A flowchart illustrating the multi-source data task scheduling optimization method driven by Kunpeng computing power intelligent agent provided in this application embodiment;

[0032] Figure 2 The structural block diagram of the multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent provided in the embodiments of this application is shown. Detailed Implementation

[0033] Example 1: Refer to Figure 1 This is a flowchart illustrating a multi-source data task scheduling optimization method driven by a Kunpeng computing power intelligent agent, provided in an embodiment of the present invention. The process may include at least steps S100-S600:

[0034] S100: Collect task requests, Kunpeng computing node running status information, task running logs and queue scheduling information, add timestamps and align them with the task execution cycle and node status cycle to generate Kunpeng scheduling status dataset;

[0035] S200. Based on the Kunpeng scheduling status dataset, the task source identifier, data format, data volume, priority, and data input address are extracted by the lightweight intelligent agent module to generate task feature data.

[0036] S300. Generate a resource profile of the Kunpeng computing node based on the task feature data and the Kunpeng scheduling status dataset;

[0037] S400. Based on the task feature data and the Kunpeng computing node resource profile, perform adaptation structure difference analysis, load fluctuation analysis, data reading path processing and dependent task status processing to generate a matching degree evaluation result.

[0038] S500: Based on the matching degree evaluation results, perform task classification, pre-scheduling sorting, and resource pre-allocation to generate pre-scheduling results;

[0039] S600. Generate task deployment instructions and resource allocation instructions based on the pre-scheduling results, monitor the execution status of the instructions to generate feedback data, and update the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library.

[0040] S100: Collect task requests, Kunpeng computing node running status information, task running logs and queue scheduling information, add timestamps and align them with the task execution cycle and node status cycle to generate Kunpeng scheduling status dataset;

[0041] In this step, the task submission interface, node resource monitoring system, task execution log unit, queue scheduling management system, and scheduler global control component together constitute the data collection source. The task submission interface receives task requests from the business system, which include task resource request information, data format, data volume, priority, and data input address. The node resource monitoring system collects Kunpeng computing node running status information, including processor core status, memory status, storage read / write status, network bandwidth, latency, and task queue length. The task execution log unit reads task execution logs, which include task submission time, task enqueue time, task execution time, historical task execution data, actual resource consumption, and task success rate. The queue scheduling management system and scheduler global control component collect queue scheduling information, including the execution queue, task queue length, node status sampling time, and scheduling environment status information.

[0042] Specifically, when the task submission interface receives a task request, it writes the task request into the task submission record and writes the task submission time as a timestamp into the task submission record. The node resource monitoring system reads the processor core status, memory status, storage read / write status, network bandwidth, and latency of the Kunpeng computing node according to the node status cycle, and writes the node status sampling time into the node status record. The task execution log unit reads the task execution log when a task enters the execution queue, starts running, ends running, migrates to another execution queue, and releases resources, and writes the task execution time into the task execution record. The queue scheduling management system reads queue scheduling information when a task is enqueued, dequeued, the execution queue changes, or the task queue length changes, and writes the task enqueuing time into the queue scheduling record. The scheduler global control component reads the scheduling environment status information and writes the scheduling environment status information into the queue scheduling record.

[0043] Specifically, timestamp processing is performed by the scheduler's global control component. This component writes the task submission time, task enqueue time, node status sampling time, and task execution time into the same scheduling window. The scheduling window is a shared area for task requests, Kunpeng computing node running status information, task execution logs, and queue scheduling information. The task execution period is the time range from when a task enters the execution queue after submission to when the task execution record ends. The node status period is the sampling range within which the node resource monitoring system continuously collects Kunpeng computing node running status information. The scheduler's global control component aligns the task execution period with the node status period to generate a scheduling mapping relationship between tasks and Kunpeng computing nodes. This mapping relationship records the Kunpeng computing node corresponding to the task request, the execution queue, the task queue length, the node status sampling time, and the task execution time.

[0044] Understandably, in scenarios where multiple data processing task requests concurrently enter the Kunpeng computing platform, different business systems submit task requests at different times, and the node resource monitoring system collects Kunpeng computing node running status information at different intervals. The scheduler's global control component maps fields of task submission records, node status records, task execution records, and queue scheduling records within the same scheduling window. This field mapping includes associating the data format, data volume, priority, and data input address in the task request with the task submission time; associating the Kunpeng computing node running status information with the node status sampling time; associating the task execution log with the task execution time; and associating the queue scheduling information with the task enqueue time. Through this field mapping, the Kunpeng computing node running status information corresponding to the same task request during submission, enqueueing, execution, and release is written into the same scheduling mapping relationship.

[0045] In one implementation, the Kunpeng computing platform receives task requests from the data processing business system. The task requests enter the queue scheduling management system via the task submission interface. The node resource monitoring system synchronously collects the processor core status, memory status, storage read / write status, and network bandwidth of each Kunpeng computing node. The task execution log unit reads historical task execution data and task execution logs. The scheduler's global control component performs timestamp verification based on the task submission time, task enqueue time, node status sampling time, and task execution time. When a task request lacks a task submission time, a node status record lacks a node status sampling time, or a task execution log lacks a task execution time, the scheduler's global control component writes the corresponding record to the exception record area and retains the original record. Records with completed timestamps re-enter the scheduling window. For records where timestamps cannot be completed, the scheduler's global control component only writes the original record and an exception identifier, without writing it to the Kunpeng scheduling status dataset.

[0046] In another implementation, when the queue scheduling management system detects a change in task queue length, the scheduler's global control component triggers a scheduling window update. This update includes rereading task submission records, node status records, task execution records, and queue scheduling records, and regenerating the scheduling mapping relationship. When the node resource monitoring system detects a change in the running status information of the Kunpeng computing nodes, it writes the new node status sampling time into the node status record. The scheduler's global control component updates the running status information of the Kunpeng computing nodes within the scheduling window according to the new node status sampling time. This update process retains the scheduling mapping relationship from the previous version and writes the current version's scheduling mapping relationship into the Kunpeng scheduling status dataset. The version record includes the scheduling window, timestamp, task request, Kunpeng computing node running status information, task execution log, and queue scheduling information.

[0047] Specifically, the Kunpeng scheduling status dataset consists of a task request field, a node status field, a log field, a queue field, a time field, and a scheduling mapping relationship. The task request field carries information about the task request, task resource request, data format, data volume, priority, and data input address. The node status field carries information about the Kunpeng computing node's running status, processor core status, memory status, storage read / write status, network bandwidth, latency, and task queue length. The log field carries task running logs, historical task execution data, actual resource consumption, and task success rate. The queue field carries queue scheduling information, the execution queue, task queue length, and scheduling environment status information. The time field carries the task submission time, task enqueue time, node status sampling time, and task execution time. The scheduling mapping relationship carries the correspondence between task requests and Kunpeng computing nodes. The Kunpeng scheduling status dataset serves as input to S200, is invoked by the lightweight intelligent agent module within S200, and is used by S200 to extract task source identifiers, data formats, data volumes, priorities, and data input addresses. The node status fields, log fields, and queue fields in the Kunpeng scheduling status dataset also serve as input to S300 for generating a Kunpeng computing node resource profile. The scheduling mapping relationship is used by S400 for adaptation structure difference analysis, load fluctuation analysis, data reading path processing, and dependent task status processing.

[0048] The technical effect of this step can be summarized as follows: the Kunpeng scheduling status dataset integrates task requests, Kunpeng computing node running status information, task running logs, and queue scheduling information into the same scheduling window. The scheduling mapping relationship preserves the temporal correspondence between task requests and Kunpeng computing nodes. The Kunpeng scheduling status dataset provides common input for S200, S300, and S400.

[0049] S200. Based on the Kunpeng scheduling status dataset, the task source identifier, data format, data volume, priority, and data input address are extracted by the lightweight intelligent agent module to generate task feature data.

[0050] In this step, the Kunpeng scheduling status dataset serves as the input source. This dataset, generated by S100, includes task request fields, node status fields, log fields, queue fields, time fields, and scheduling mapping relationships. After receiving the Kunpeng scheduling status dataset, the lightweight intelligent agent module first reads the task request fields, then the time fields and scheduling mapping relationships, and writes the task resource request information, data format, data volume, priority, and data input address from the task request fields into the same task record. The lightweight intelligent agent module is a processing module deployed within the scheduler's global control component. Internally, it runs a task classification algorithm, data field mapping configuration, and resource requirement description configuration. The task classification algorithm identifies the task source identifier and priority from the task request. The data field mapping configuration maps field names submitted by different business systems to unified fields. The resource requirement description configuration converts data format, data volume, and data input address into resource requirement descriptions.

[0051] Specifically, the task source identifier is the source field of the business system to which the task request belongs, extracted from the task submission record written by the task submission interface. When the same task request contains multiple source fields, the lightweight intelligent agent module reads the source field that first enters the execution queue according to the task submission time and writes it into the task source identifier. The data format refers to the structure type and file type of the data to be processed in the task request. The lightweight intelligent agent module reads the data format field from the task request fields and maps it to the historical execution records of similar tasks in the task execution log. The data volume is the capacity field of the data to be processed in the task request. The lightweight intelligent agent module reads the capacity field and converts records with inconsistent units to the same capacity unit. The priority is the sorting field when the task request enters the execution queue. The lightweight intelligent agent module reads the priority from the queue fields and writes it into the task record in combination with the task queue length. The data input address is the storage location of the data to be processed pointed to by the task request. The lightweight intelligent agent module reads the data input address from the task request fields and establishes an association with the Kunpeng computing node in the scheduling mapping relationship.

[0052] Furthermore, the lightweight intelligent agent module performs field validation on the task source identifier, data format, data volume, priority, and data input address. The field validation includes field integrity validation, timestamp validation, and scheduling window validation. Field integrity validation checks whether the task source identifier, data format, data volume, priority, and data input address are written to the same task record. Timestamp validation checks whether the task submission time, task enqueue time, and node status sampling time belong to the same scheduling window. Scheduling window validation checks whether a corresponding scheduling mapping relationship exists for the task request. If the field validation fails, the lightweight intelligent agent module writes the corresponding task record to the exception record area, retaining the task request field, time field, and exception identifier. If the field validation passes, the lightweight intelligent agent module writes the corresponding task record to the feature generation area and triggers key task feature extraction processing.

[0053] Specifically, the key task feature extraction and processing includes the generation process of computational complexity features, real-time requirement features, resource dependency features, data format conversion requirements, data locality, input / output pressure, and resource consumption of similar historical tasks. The computational complexity feature is generated jointly by data format, data volume, and resource consumption of similar historical tasks. The lightweight intelligent agent module reads historical task execution data from the task execution log, identifies task records with the same data format and similar data volume as historical similar task execution records, and reads the actual resource consumption in these historical similar task execution records to generate the computational complexity feature. The real-time requirement feature is generated jointly by priority and task queue length. The lightweight intelligent agent module reads the priority and task queue length from the queue field, writes them into the same task record, and generates the real-time requirement feature. The resource dependency feature is generated jointly by task resource request information and scheduling mapping relationships. The lightweight intelligent agent module reads the computing resources, storage resources, and network resources from the task resource request information and combines them with the Kunpeng computing nodes in the scheduling mapping relationship to generate resource dependency features.

[0054] Furthermore, the data format conversion requirement is generated from the data format and historical execution records of similar tasks. The lightweight intelligent agent module compares the data format in the task request with the data format in the historical execution records of similar tasks. When the current data format is inconsistent with the data format already recorded in the execution queue, the lightweight intelligent agent module writes the data format conversion requirement into the task record. The data locality is generated from the data input address and the scheduling mapping relationship. The lightweight intelligent agent module reads the storage channel corresponding to the data input address and associates it with the Kunpeng computing node in the scheduling mapping relationship to generate data locality. The input / output pressure is generated from the data volume, data input address, result output method, and storage read / write status. The lightweight intelligent agent module reads the storage read / write status in the node status field and writes it, along with the data volume and data input address, into the same task record to generate input / output pressure. The resource consumption of historical similar tasks is generated from the actual resource consumption in the task execution log and historical executed task data, and is written into the same task record along with the task source identifier, data format, and data volume.

[0055] In one engineering embodiment, the Kunpeng computing platform receives task requests from a data processing business system. The task request carries data format, data volume, priority, and data input address. The Kunpeng scheduling status dataset generated by S100 has recorded the task request, task submission time, task enqueue time, node status sampling time, and the scheduling mapping relationship between the task and the Kunpeng computing node. When a task request enters the execution queue, the lightweight intelligent agent module automatically reads the Kunpeng scheduling status dataset and extracts the task source identifier, data format, data volume, priority, and data input address from the task request fields. Subsequently, the lightweight intelligent agent module reads historical execution records of similar tasks from the task execution log and writes the actual resource consumption from these historical records to the current task record. Next, the lightweight intelligent agent module reads the storage read / write status, network bandwidth, and latency from the node status field and writes these parameters to the current task record. Thus, the current task record forms characteristics of computational complexity, real-time requirements, resource dependencies, data format conversion requirements, data locality, input / output pressure, and resource consumption of historical similar tasks.

[0056] Understandably, the triggering conditions for the lightweight intelligent agent module include: a task request being written to the task submission record; a task enqueue time being written to the queue scheduling record; a change in the task queue length; the appearance of historical execution records of similar tasks in the task execution log; and the completion of updates to the scheduling mapping relationship. When any of these triggering conditions occurs, the lightweight intelligent agent module reads the current version of the Kunpeng scheduling status dataset and generates the current version's task feature data in the feature generation area. After the current version's task feature data is generated, the lightweight intelligent agent module writes the version record to the task record. The version record includes the task submission time, task enqueue time, scheduling window, and task source identifier. The task feature data of the previous version is retained in the task execution log. The task feature data of the current version is used in subsequent steps.

[0057] Specifically, the task feature data consists of basic fields, resource requirement descriptions, and feature fields. The basic fields include task source identifier, data format, data volume, priority, and data input address. The resource requirement description includes computing resources, storage resources, and network resources. The feature fields include computational complexity features, real-time requirement features, resource dependency features, data format conversion requirements, data locality, input / output pressure, and historical resource consumption of similar tasks. This task feature data serves as input to S300, working together with the Kunpeng scheduling status dataset to generate a resource profile of the Kunpeng computing node. The task feature data also serves as input to S400, enabling S400 to perform adaptation structure difference analysis, load fluctuation analysis, data read path processing, and dependent task status processing. The data input address and data locality in the task feature data are used by S400 to generate data read paths. The resource dependency features and task execution logs in the task feature data are used by S400 to generate the execution status of dependent tasks.

[0058] The technical effect of this step can be summarized as follows: the task feature data writes the basic fields from the task request and the resource consumption of similar historical tasks from the task execution log into the same task record. The lightweight intelligent agent module completes the extraction of key task features and data field mapping within the scheduling window. The task feature data provides task-side input for the generation of Kunpeng computing node resource profiles for S300 and the generation of matching degree evaluation results for S400.

[0059] S300. Generate a resource profile of the Kunpeng computing node based on the task feature data and the Kunpeng scheduling status dataset;

[0060] In this step, the task feature data and the Kunpeng scheduling status dataset serve as input sources. The task feature data, generated by S200, includes task source identifier, data format, data volume, priority, data input address, computational complexity characteristics, real-time requirement characteristics, resource dependency characteristics, data format conversion requirements, data locality, input / output pressure, and historical resource consumption of similar tasks. The Kunpeng scheduling status dataset, generated by S100, includes task request fields, node status fields, log fields, queue fields, time fields, and scheduling mapping relationships. After reading the task feature data, the lightweight intelligent agent module then reads the node status fields, log fields, queue fields, and scheduling mapping relationships from the Kunpeng scheduling status dataset, associating task-side records and node-side records within the same scheduling window to generate a Kunpeng computing node resource profile. The Kunpeng computing node resource profile is a record of the node scheduling resource status for each Kunpeng computing node, including node static attributes, node real-time attributes, node historical attributes, and node adaptation attributes.

[0061] Specifically, the lightweight agent module first reads the scheduling mapping relationship. This mapping relationship records the correspondence between task requests and Kunpeng computing nodes, and records the task submission time, task enqueue time, node status sampling time, and task execution time. Based on the scheduling mapping relationship, the lightweight agent module selects the node status field, log field, and queue field from the Kunpeng scheduling status dataset that are within the same scheduling window as the current task's feature data. The node status field provides processor core status, memory status, storage read / write status, network bandwidth, latency, and task queue length. The log field provides a snapshot of the node's historical resources, a sequence of historical node load rates, historical task execution data, and task success rate. The queue field provides the execution queue, task queue length, and scheduling environment status information. The lightweight agent module writes these fields into the same node record according to the Kunpeng computing node identifier, forming a basic node resource record.

[0062] Furthermore, the node's static attributes consist of a Kunpeng computing node identifier, processor core structure, memory capacity, storage channel, network interface, and operating environment. The Kunpeng computing node identifier is the node number written by the node resource monitoring system. The processor core structure includes the number of processor cores, core grouping relationships, and the currently callable core range. The memory capacity is the range of memory resources that the node can access through the scheduler's global control component. The storage channel is the local or remote storage access channel corresponding to the data input address. The network interface is the communication interface for transmitting task deployment instructions and resource allocation instructions. The operating environment is the currently available task execution configuration for the node. The lightweight intelligent agent module reads the above fields from the node status field and scheduling environment status information in the Kunpeng scheduling status dataset and writes them into the node's static attributes.

[0063] Specifically, the real-time node attributes consist of processor core status, memory status, storage read / write status, network bandwidth, latency, and task queue length. The processor core status is the core occupancy and idle records read by the node resource monitoring system at the node status sampling time. The memory status is the occupied and allocable memory records read by the node resource monitoring system. The storage read / write status is the read / write record of the storage channel corresponding to the data input address at the node status sampling time. The network bandwidth and latency are read by the node resource monitoring system from the network interface. The task queue length is written by the queue scheduling management system between the task enqueue time and the node status sampling time. The lightweight intelligent agent module correlates the real-time node attributes with the data volume, data input address, and input / output pressure in the task feature data, and writes the correlation result into the current Kunpeng computing node resource profile.

[0064] Furthermore, the node historical attributes consist of a node historical resource snapshot, a node historical load rate sequence, and historical task execution data. The node historical resource snapshot is the processor core state, memory state, storage read / write state, network bandwidth, and latency recorded by the task execution log unit during historical task execution. The node historical load rate sequence is the load change record written by the node resource monitoring system within a continuous node state cycle. The historical task execution data consists of the historical task submission time, historical task execution time, actual resource consumption, and resource release information recorded by the task execution log unit. After reading the historical task execution data, the lightweight intelligent agent module identifies records with the same task source identifier, the same data format, or similar data volume as historical similar task execution records and writes these historical similar task execution records into the node historical attributes.

[0065] Understandably, the node adaptation attribute consists of the task success rate and historical execution records of similar tasks. The task success rate is written by the task execution log unit based on historical task execution logs. The historical execution records of similar tasks are generated jointly by the task feature data and the historical executed task data. The lightweight intelligent agent module maps computational complexity features, real-time requirement features, resource dependency features, data format conversion requirements, data locality, and input / output pressure to historical executed task data. After the field mapping is completed, the lightweight intelligent agent module writes the actual resource consumption, task success rate, and resource release information of the Kunpeng computing node in similar tasks into the node adaptation attribute. When the current task feature data lacks historical execution records of similar tasks, the lightweight intelligent agent module writes the task source identifier, data format, data volume, and data input address into the current node record, and leaves the task success rate field as null, waiting for S600 feedback data updates.

[0066] In one engineering embodiment, the Kunpeng computing platform includes multiple Kunpeng computing nodes. After receiving a task request from the data processing business system, the task submission interface generates a Kunpeng scheduling status dataset in step S100 and task feature data in step S200. The lightweight intelligent agent module automatically triggers this step after the task feature data is generated. The lightweight intelligent agent module first reads the scheduling mapping relationship based on the task source identifier and data input address, and then reads the node status field of the corresponding Kunpeng computing node based on the scheduling mapping relationship. Subsequently, the lightweight intelligent agent module reads historical task execution data from the task execution log and filters historical similar task execution records according to data format and data volume. Next, the lightweight intelligent agent module reads the execution queue and task queue length from the queue scheduling information and writes the execution queue and task queue length into the node's real-time attributes. Finally, the lightweight intelligent agent module writes the node's static attributes, node's real-time attributes, node's historical attributes, and node's adaptation attributes into the same Kunpeng computing node resource profile.

[0067] Specifically, when the node resource monitoring system detects changes in processor core status, memory status, storage read / write status, network bandwidth, latency, or task queue length, the lightweight intelligent agent module rereads the Kunpeng scheduling status dataset and generates the current version of the Kunpeng computing node resource profile. When the task execution log unit writes new actual resource consumption, task success rate, or resource release information, the lightweight intelligent agent module updates the node's historical attributes and node adaptation attributes. The current version of the Kunpeng computing node resource profile is retained in the task execution log along with the previous version. The current version is written to the scheduling strategy library call record. When a node status field, log field, or scheduling mapping relationship is missing, the lightweight intelligent agent module writes the corresponding Kunpeng computing node record to the exception record area, retaining the read node status field and time field. Exception records do not participate in the generation of the matching degree evaluation result for the current scheduling window.

[0068] Furthermore, the basic fields in the Kunpeng computing node resource profile include the Kunpeng computing node identifier, processor core status, memory status, storage read / write status, network bandwidth, latency, and task queue length. These basic fields serve as inputs for S400 to perform adaptation structure difference analysis and load fluctuation analysis. The associated fields in the Kunpeng computing node resource profile include the operating environment, node historical resource snapshots, node historical load rate sequence, historical executed task data, task success rate, and historical execution records of similar tasks. These associated fields serve as inputs for S400 to perform data read path processing and dependent task status processing. The Kunpeng computing node resource profile is sent to S400 along with the task feature data, allowing S400 to compare the resource requirement description in the task feature data with the node scheduling resource status in the Kunpeng computing node resource profile and generate a matching degree evaluation result.

[0069] The technical effects of this step can be summarized as follows: the Kunpeng computing node resource profile writes the node's static attributes, real-time attributes, historical attributes, and adaptation attributes into the same node record. The Kunpeng computing node resource profile retains the scheduling mapping relationship between task feature data and the Kunpeng scheduling status dataset. The Kunpeng computing node resource profile provides node-side input for S400 adaptation structure difference analysis, load fluctuation analysis, data read path processing, and dependent task status processing.

[0070] S400. Based on the task feature data and the Kunpeng computing node resource profile, perform adaptation structure difference analysis, load fluctuation analysis, data reading path processing and dependent task status processing to generate a matching degree evaluation result.

[0071] In this step, the task feature data and the Kunpeng computing node resource profile serve as input sources. The task feature data is generated by S200 and includes computational complexity features, real-time requirement features, resource dependency features, data format conversion requirements, data locality, input / output pressure, historical resource consumption of similar tasks, and resource requirement descriptions. The Kunpeng computing node resource profile is generated by S300 and includes node static attributes, node real-time attributes, node historical attributes, and node adaptation attributes. After receiving the task feature data, the lightweight intelligent agent module reads the resource requirement description and then reads the node scheduling resource status in the Kunpeng computing node resource profile, writing the task-side record and the node-side record into the same matching processing area. The matching processing area saves processing records according to the scheduling window and inherits the scheduling mapping relationship generated by S100.

[0072] Specifically, the adaptation structure difference analysis is performed by the lightweight intelligent agent module. This analysis involves comparing fields between resource requirement descriptions and node scheduling resource states. The resource requirement descriptions include computing resources, storage resources, network resources, runtime environment configuration, data format conversion requirements, and input / output pressure. The node scheduling resource states include processor core state, memory state, storage read / write state, network bandwidth, latency, runtime environment, and task queue length. The lightweight intelligent agent module compares computing resources with processor core state, storage resources with memory state and storage read / write state, network resources with network bandwidth and latency, runtime environment configuration with runtime environment, and input / output pressure with storage read / write state. After comparison, the lightweight intelligent agent module writes a structure adaptation record to the matching processing area. The structure adaptation record includes well-adapted tasks, poorly adapted tasks, and a candidate node list.

[0073] Specifically, the well-fitted tasks are task records where the resource requirement description and node scheduling resource status both have corresponding fields in terms of computing resources, storage resources, network resources, and runtime environment configuration. The unbalanced tasks are task records where the resource requirement description and node scheduling resource status have missing fields, conflicting fields, or uncallable fields. The candidate node list consists of the Kunpeng computing node identifiers corresponding to the well-fitted tasks. When generating the candidate node list, the lightweight agent module retains the corresponding processor core status, memory status, storage read / write status, network bandwidth, latency, runtime environment, and task queue length. For unbalanced tasks, the lightweight agent module writes an imbalance reason record and removes the corresponding Kunpeng computing node from the current candidate node list. The imbalance reason record is saved with the scheduling window for S600 feedback data to update the scheduling strategy library.

[0074] Furthermore, the load fluctuation analysis is performed by the lightweight intelligent agent module based on the node's historical load rate sequence, node's historical resource snapshot, node's real-time attributes, and task queue length. The node's historical load rate sequence is a record of load changes within a continuous node state cycle. The node's historical resource snapshot includes the processor core state, memory state, storage read / write state, network bandwidth, and latency during historical task execution. The lightweight intelligent agent module reads the node's real-time attributes within the current scheduling window and compares them with the node's historical resource snapshot from the previous scheduling window to obtain a load fluctuation record. This load fluctuation record is written to the candidate node list and participates in the candidate node list arrangement. When the task queue length of the current Kunpeng computing node changes, the queue scheduling management system writes the new queue scheduling information to the Kunpeng scheduling status dataset, and the lightweight intelligent agent module rereads the corresponding node's real-time attributes and updates the load fluctuation record.

[0075] Furthermore, the data read path processing is executed by the lightweight intelligent agent module based on the data input address and data locality. The data input address is the storage location of the data to be processed, as indicated by the task request. Data locality refers to the storage channel association status between the data input address and the Kunpeng computing node. The lightweight intelligent agent module reads the data input address and queries the corresponding storage channel, network interface, network bandwidth, and latency in the Kunpeng computing node resource profile. If the storage channel corresponding to the data input address has a local storage association with a Kunpeng computing node in the candidate node list, the lightweight intelligent agent module records the Kunpeng computing node as a local read node. If the storage channel corresponding to the data input address requires access via a network interface, the lightweight intelligent agent module records the Kunpeng computing node as a remote read node and writes the network bandwidth and latency. The local read node, remote read node, storage channel, network interface, network bandwidth, and latency together form the data read path.

[0076] Specifically, the dependent task status processing is performed by the lightweight intelligent agent module based on resource dependency features and task execution logs. The resource dependency features are records in the task feature data that characterize the relationship between task resource request information and the execution order of other tasks. The task execution logs include historical task execution data, task execution time, actual resource consumption, task success rate, and resource release information. After reading the resource dependency features, the lightweight intelligent agent module queries the task execution time and success rate corresponding to the dependent task in the task execution logs. If the dependent task has already been written with resource release information, the lightweight intelligent agent module records the dependent task as released. If the dependent task is still in the execution queue, the lightweight intelligent agent module records the dependent task as queued. If the dependent task has a migration execution state, the lightweight intelligent agent module writes the migration execution state into the dependent task execution state. The dependent task execution state is then written to the matching processing area and participates in the generation of the matching degree evaluation result along with the data reading path.

[0077] In one engineering embodiment, the data processing business system submits a task request. Task feature data generated by S200 includes data format, data volume, data input address, resource requirement description, and resource dependency characteristics. The Kunpeng computing node resource profile generated by S300 includes the processor core status, memory status, storage read / write status, network bandwidth, latency, task queue length, and historical execution records of similar tasks for multiple Kunpeng computing nodes. The lightweight intelligent agent module first compares the resource requirement description with the node scheduling resource status of each Kunpeng computing node and writes it into a structure adaptation record. Then, the lightweight intelligent agent module reads the node's historical load rate sequence and task queue length and writes them into a load fluctuation record. Next, the lightweight intelligent agent module queries the storage channel based on the data input address and generates a data read path. Finally, the lightweight intelligent agent module reads the task execution log based on the resource dependency characteristics and generates the execution status of dependent tasks. All of the above records are written into the same matching degree evaluation result.

[0078] Understandably, the matching degree evaluation result consists of a structure adaptation record, a load fluctuation record, a data read path, the execution status of dependent tasks, and a candidate node list. The structure adaptation record carries tasks with good structure adaptation and tasks with unbalanced structure adaptation. The load fluctuation record carries a comparison record between the node's historical load rate sequence, historical resource snapshots of the node, real-time node attributes, and task queue length. The data read path carries the data input address, data locality, storage channel, network interface, network bandwidth, and latency. The dependent task execution status carries resource dependency characteristics, task running logs, task execution time, migration execution status, and resource release information. The candidate node list carries the candidate Kunpeng computing node identifier, operating environment, task queue length, and node adaptation attributes. The matching degree evaluation result serves as input to the S500, allowing the S500 to perform task classification, pre-scheduling sorting, and resource pre-allocation; wherein, the candidate node list allows the S500 to arrange target Kunpeng computing nodes and alternative Kunpeng computing nodes, and the task queue length and data read path allow the S500 to generate execution windows and resource allocation requests.

[0079] Furthermore, the lightweight agent proxy module retains version records when generating matching degree evaluation results. These version records include the scheduling window, task source identifier, Kunpeng computing node identifier, structure adaptation record, load fluctuation record, data reading path, and dependent task execution status. When the Kunpeng computing node resource profile is updated in S300, the lightweight agent proxy module re-executes the adaptation structure difference analysis and load fluctuation analysis. When the task feature data is updated in S200, the lightweight agent proxy module re-executes the data reading path processing and dependent task status processing. The matching degree evaluation results of the previous version are retained in the task execution log. The matching degree evaluation results of the current version are written to the scheduling strategy library call record. If the task feature data lacks a data input address, the lightweight agent proxy module writes the data reading path to the abnormal status. If the task execution log lacks dependent task records, the lightweight agent proxy module writes the dependent task execution status to the pending query status. Abnormal status and pending query status do not participate in the current candidate node list arrangement.

[0080] The technical effect of this step can be summarized as follows: the matching degree evaluation result completes the correspondence between task feature data and Kunpeng computing node resource profiles within the same scheduling window. The matching degree evaluation result also records adaptation structure differences, load fluctuations, data read paths, and dependent task execution status. The matching degree evaluation result provides a candidate node list and task-side constraints for S500 task classification, pre-scheduling sorting, and resource pre-allocation.

[0081] S500: Based on the matching degree evaluation results, perform task classification, pre-scheduling sorting, and resource pre-allocation to generate pre-scheduling results;

[0082] In this step, the matching degree evaluation result serves as the input source. This matching degree evaluation result is generated by S400 and includes structural adaptation records, load fluctuation records, data reading paths, dependent task execution status, and a candidate node list. After reading the matching degree evaluation result, the lightweight intelligent agent module writes the structural adaptation record into the task hierarchical processing area, and writes the load fluctuation record, data reading path, dependent task execution status, and candidate node list into the pre-scheduling processing area. It then calls the execution queue and task queue length in the queue scheduling management system. Task hierarchical processing involves writing tasks into corresponding execution queues based on task characteristic data. Pre-scheduling sorting involves arranging the candidate node list according to the matching degree evaluation result, task queue length, and execution window. Resource pre-allocation involves generating the target Kunpeng computing node, alternative Kunpeng computing nodes, resource allocation requests, and execution windows before the task is officially executed.

[0083] Specifically, the lightweight agent module first reads the well-fitted and unfitted tasks from the structure adaptation record. For well-fitted tasks, the lightweight agent module reads the priority, real-time requirements, computational complexity, resource dependencies, input / output pressure, and data format conversion requirements from the task feature data, and writes the task feature data into the execution queue. For unfitted tasks, the lightweight agent module reads the imbalance cause record and writes the corresponding task into the exception record area. The exception record area stores the task source identifier, data format, data volume, priority, data input address, structure adaptation record, and scheduling window. Tasks in the exception record area are not included in the current resource pre-allocation. When the current scheduling window is updated, the lightweight agent module rereads the matching degree evaluation result and re-executes the task classification.

[0084] Furthermore, the execution queue is maintained by a queue scheduling management system. The execution queue includes records of tasks waiting to be executed within the current scheduling window. When the lightweight intelligent agent module writes task feature data into the execution queue, it writes the features according to priority and real-time requirements, and then writes the execution status of dependent tasks according to resource dependency features. The computational complexity feature is used to record the computational resource requests corresponding to the task. The input / output pressure is used to record the storage resource requests and network resource requests corresponding to the task. The data format conversion requirements are used to record the runtime environment configuration. After a task is written into the execution queue, the queue scheduling management system updates the task queue length and writes the updated task queue length into the pre-scheduled processing area.

[0085] Specifically, the pre-scheduling sorting is performed by the lightweight intelligent agent module based on the candidate node list. The candidate node list carries the candidate Kunpeng computing node identifier, operating environment, task queue length, and node adaptation attributes. The lightweight intelligent agent module reads load fluctuation records and writes the node's historical load rate sequence, real-time node attributes, and task queue length into the candidate node list. The lightweight intelligent agent module reads the data reading path and arranges local reading nodes before remote reading nodes. The lightweight intelligent agent module reads the execution status of dependent tasks, retains candidate nodes corresponding to released states in the candidate node list, and writes the queue status and pending query status into the candidate node's remarks field. After the candidate node list is sorted, a node sorting record is formed. The node sorting record is written to the pre-scheduling processing area.

[0086] Furthermore, the execution window is generated jointly by the task submission time, task enqueue time, node state sampling time, task execution time, and task queue length. The lightweight intelligent agent module reads the time field from the Kunpeng scheduling state dataset and the task queue length from the matching degree evaluation result. The lightweight intelligent agent module writes the task submission time and task enqueue time into the window start record, the node state sampling time into the window state record, and the task execution time into the window end record. The window start record, window state record, and window end record together form the execution window. The execution window is written into the candidate node list and participates in the selection of the target Kunpeng computing node and the alternative Kunpeng computing node.

[0087] Specifically, the resource pre-allocation is jointly executed by a lightweight intelligent agent module and a queue scheduling management system. The lightweight intelligent agent module reads the node sorting record and writes the top-ranked Kunpeng computing node whose operating environment matches the resource requirement description into the target Kunpeng computing node field. The lightweight intelligent agent module continues to read subsequent candidate nodes and writes Kunpeng computing nodes whose operating environment, network bandwidth, latency, and task queue length are adjacent to the target Kunpeng computing node into the alternative Kunpeng computing node field. The resource allocation request consists of computing resources, storage resources, network resources, and operating environment configuration. The computing resources are generated from computational complexity characteristics and processor core states. The storage resources are generated from data volume, input / output pressure, memory status, and storage read / write status. The network resources are generated from data read paths, network bandwidth, and latency. The operating environment configuration is generated from data format conversion requirements and the operating environment.

[0088] In one engineering embodiment, the data processing tasks submitted by the data processing business system have already generated task feature data in S200 and a matching degree evaluation result in S400. After reading the matching degree evaluation result, the lightweight intelligent agent module first writes the structurally well-fitted tasks into the execution queue. Subsequently, the lightweight intelligent agent module reads the candidate node list and arranges the candidate Kunpeng computing nodes according to the load fluctuation record, data reading path, and dependent task execution status. If there is a local reading node in the candidate node list, and the task queuing length of the local reading node does not exceed the current execution queue record, then the lightweight intelligent agent module writes the local reading node into the target Kunpeng computing node field. If the task queuing length of the local reading node exceeds the current execution queue record, then the lightweight intelligent agent module reads the remote reading node and writes the remote reading node whose network bandwidth and latency meet the current data reading path into the candidate Kunpeng computing node field. Subsequently, the lightweight intelligent agent module generates a resource allocation request and an execution window, and writes the pre-scheduling result.

[0089] Understandably, when the queue scheduling management system detects a change in task queue length, the lightweight intelligent agent module rereads the matching degree evaluation results and updates the pre-scheduling order. When the current Kunpeng computing node resource profile is updated, the lightweight intelligent agent module rereads the candidate node list and updates the target Kunpeng computing node and alternative Kunpeng computing nodes. When the current task feature data is updated, the lightweight intelligent agent module regenerates the resource allocation request. The pre-scheduling result is written to the version record for each update. The version record includes the scheduling window, task source identifier, execution queue, candidate node list, target Kunpeng computing node, alternative Kunpeng computing nodes, resource allocation request, and execution window. The pre-scheduling result of the previous version is retained in the task execution log. The pre-scheduling result of the current version is written to the scheduling strategy library call record.

[0090] Specifically, the pre-scheduling result consists of a task hierarchy record, a pre-scheduling sorting record, and a resource pre-allocation record. The task hierarchy record carries task characteristic data, execution queues, and task queue lengths. The pre-scheduling sorting record carries matching degree evaluation results, a candidate node list, data reading paths, dependent task execution status, and execution windows. The resource pre-allocation record carries the target Kunpeng computing node, candidate Kunpeng computing nodes, resource allocation requests, and execution windows. The pre-scheduling result serves as input to the S600, enabling the S600 to generate task deployment instructions and resource allocation instructions. The target Kunpeng computing node, execution queue, execution window, and data reading path are used by the S600 to generate task deployment instructions. The computing resources, storage resources, network resources, and runtime environment configuration in the resource allocation request are used by the S600 to generate resource allocation instructions. The candidate Kunpeng computing nodes are used by the S600 to generate migration instructions when the migration trigger conditions match the instruction execution status.

[0091] The technical effect of this step can be summarized as follows: the pre-scheduling result writes task classification, pre-scheduling sorting, and resource pre-allocation into the same scheduling window. The pre-scheduling result retains the target Kunpeng computing node, candidate Kunpeng computing nodes, and execution window before the task deployment instruction is generated. The pre-scheduling result provides preliminary input for the generation of S600 task deployment instructions, resource allocation instructions, and migration instructions.

[0092] S600. Generate task deployment instructions and resource allocation instructions based on the pre-scheduling results, monitor the execution status of the instructions to generate feedback data, and update the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library.

[0093] In this step, the pre-scheduling result serves as the input source. The pre-scheduling result is generated by S500 and includes a task hierarchy record, a pre-scheduling sorting record, and a resource pre-allocation record. The task hierarchy record carries task feature data, the execution queue, and the task queue length. The pre-scheduling sorting record carries the matching degree evaluation result, the candidate node list, the data reading path, the execution status of dependent tasks, and the execution window. The resource pre-allocation record carries the target Kunpeng computing node, the alternative Kunpeng computing nodes, the resource allocation request, and the execution window. After reading the pre-scheduling result, the lightweight intelligent agent module writes the target Kunpeng computing node, the execution queue, the execution window, and the data reading path into the task deployment instruction, writes the computing resources, storage resources, network resources, and runtime environment configuration into the resource allocation instruction, and sends the task deployment instruction and the resource allocation instruction to the scheduler's global control component.

[0094] Specifically, the task deployment instruction is an execution control record issued by the scheduler's global control component to the target Kunpeng computing node. The task deployment instruction includes the target Kunpeng computing node, execution queue, execution window, and data reading path. The target Kunpeng computing node is read from the resource pre-allocation record. The execution queue is read from the task hierarchy record. The execution window is read from the pre-scheduled sorting record. The data reading path is read from the matching degree evaluation result. The lightweight intelligent agent module writes the above fields into the same task deployment record, and also writes the task source identifier, task submission time, task enqueue time, and scheduling window into the task deployment record. After reading the task deployment record, the scheduler's global control component binds the task request to the target Kunpeng computing node and writes the task request into the corresponding execution queue.

[0095] Specifically, the resource allocation instruction is a resource control record received by the target Kunpeng computing node before executing the task request. The resource allocation instruction includes computing resources, storage resources, network resources, and runtime environment configuration. The computing resources are generated from the computing resources in the resource allocation request and the processor core state of the target Kunpeng computing node. The storage resources are generated from the storage resources, memory state, and storage read / write state in the resource allocation request. The network resources are generated from the network resources, network bandwidth, and latency in the resource allocation request. The runtime environment configuration is generated from the data format conversion requirements and the runtime environment. The lightweight intelligent agent module writes the resource allocation instruction into the resource allocation record and sends the resource allocation record to the target Kunpeng computing node. After receiving the resource allocation record, the target Kunpeng computing node loads the task request according to the computing resources, storage resources, network resources, and runtime environment configuration.

[0096] Furthermore, the instruction execution status is jointly collected by the node resource monitoring system, the task execution log unit, and the queue scheduling management system. The node resource monitoring system collects processor core status, memory status, storage read / write status, network bandwidth, and latency after the task deployment instruction is issued, and generates resource utilization. The task execution log unit reads the task execution log when the task starts running, ends executing, migrates, and releases resources, and generates task success rate, actual resource consumption versus estimated value deviation rate, migration execution status, and resource release information. The queue scheduling management system reads the task queue length when tasks are enqueued, dequeued, and the execution queue changes. The lightweight intelligent agent module writes resource utilization, task success rate, actual resource consumption versus estimated value deviation rate, task queue length, migration execution status, and resource release information into the feedback processing area, generating feedback data.

[0097] Specifically, when a task deployment instruction has been written to the task deployment record and a resource allocation instruction has been written to the resource allocation record, the scheduler's global control component triggers instruction execution status monitoring. The instruction execution status includes the task deployment record's issuance status, the resource allocation record's reception status, the target Kunpeng computing node's running status, and the execution queue status. The lightweight intelligent agent module reads the instruction execution status within the same scheduling window and maps its fields to the task deployment instruction and resource allocation instruction. If the task deployment record lacks a target Kunpeng computing node, execution queue, execution window, or data reading path, the lightweight intelligent agent module writes the task deployment record to the exception record area. If the resource allocation record lacks computing resources, storage resources, network resources, or runtime environment configuration, the lightweight intelligent agent module writes the resource allocation record to the exception record area. The exception record area stores the task source identifier, scheduling window, exception identifier, and original record.

[0098] Further, when the migration triggering condition matches the instruction execution state, the lightweight intelligent agent module generates a migration instruction. The migration triggering condition is provided by the scheduling strategy library and includes resource utilization deviation, task success rate change, actual resource consumption versus estimated deviation rate, task queue length change, and dependent task execution state change. The migration instruction includes the source Kunpeng computing node, target Kunpeng computing node, candidate Kunpeng computing node, migration execution state, and resource release information. The source Kunpeng computing node is read from the current task deployment record. The target Kunpeng computing node is read from the candidate Kunpeng computing nodes in the pre-scheduling results. After generating the migration instruction, the lightweight intelligent agent module writes the migration instruction to the task execution log and distributes it to the corresponding Kunpeng computing node by the scheduler's global control component. After the migration execution is completed, the task execution log unit writes the migration execution state and resource release information, which are then included in the feedback data.

[0099] In one engineering embodiment, the Kunpeng computing platform has generated a pre-scheduling result via S500. The pre-scheduling result records the target Kunpeng computing node, candidate Kunpeng computing nodes, execution queue, data read path, resource allocation request, and execution window. After reading the pre-scheduling result, the lightweight intelligent agent module generates a task deployment instruction and writes the task request to the execution queue of the target Kunpeng computing node. Subsequently, the lightweight intelligent agent module generates a resource allocation instruction based on the resource allocation request and sends the computing resources, storage resources, network resources, and runtime environment configuration to the target Kunpeng computing node. Upon receiving the task deployment instruction and resource allocation instruction, the target Kunpeng computing node begins executing the task request. The node resource monitoring system collects resource utilization during the execution process. The task execution log unit collects task success rate, actual resource consumption deviation rate, migration execution status, and resource release information. The queue scheduling management system collects task queue length. The lightweight intelligent agent module aggregates the above records and generates feedback data.

[0100] Understandably, after the feedback data is generated, the lightweight intelligent agent module performs update processing on the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library, respectively. The update processing of the task feature data includes reading the deviation rate between the actual resource consumption and the estimated value, and writing the deviation rate into the resource requirement description. The update processing of the Kunpeng computing node resource profile includes reading the resource utilization rate, task success rate, and resource release information, and writing them into the node adaptation attributes. The update processing of the scheduling strategy library includes reading the migration execution status, task queue length, and changes in the execution status of dependent tasks, and writing them into the candidate node screening rules, resource pre-allocation rules, and migration triggering conditions. After the update is completed, the current version of the task feature data, Kunpeng computing node resource profile, and scheduling strategy library are written into the task execution log. The previous version is retained in the task execution log.

[0101] Specifically, the feedback data consists of execution status, resource, queue, migration, and update fields. The execution status field carries task deployment records, resource allocation records, and instruction execution status. The resource field carries resource utilization rate, actual resource consumption deviation rate from estimated value, and resource release information. The queue field carries task queue length and execution queue changes. The migration field carries migration instructions and migration execution status. The update field carries updated task feature data, updated Kunpeng computing node resource profiles, and updated scheduling strategy libraries. The feedback data is written to the scheduling window and sent back to S200, S300, and S500 for use; wherein, the updated task feature data is used when subsequent task requests enter S200, the updated Kunpeng computing node resource profile is used when S300 generates the current version profile, and the updated scheduling strategy library is used when S500 generates pre-scheduled results.

[0102] The technical effect of this step can be summarized as follows: the feedback data writes task deployment instructions, resource allocation instructions, and instruction execution status into the same scheduling window. The feedback data updates task characteristic data, Kunpeng computing node resource profiles, and the scheduling strategy library, respectively. The feedback data is used by S200, S300, and S500 to call the current version record in subsequent scheduling windows.

[0103] Example 2: Figure 2 A structural block diagram of a multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to an embodiment of the present invention is shown. Figure 2 As shown, the structure may include:

[0104] The data acquisition module 01 is used to collect task requests through the task submission interface, collect Kunpeng computing node running status information through the node resource monitoring system, collect task running logs through the task running log unit, and collect queue scheduling information through the queue scheduling management system and the scheduler global control component. Specifically, the task submission interface receives task requests submitted by the business system and reads task resource request information, data format, data volume, priority, and data input address. The node resource monitoring system connects to the Kunpeng computing node and collects processor core status, memory status, storage read / write status, network bandwidth, latency, and task queue length. The task running log unit reads historical executed task data, actual resource consumption, task success rate, migration execution status, and resource release information. The queue scheduling management system and the scheduler global control component read execution queue, task queue length, and scheduling environment status information. When any collected field is missing, the data acquisition module retains the original record and anomaly identifier, and sends the collected task requests, Kunpeng computing node running status information, task running logs, and queue scheduling information to the time alignment module.

[0105] The time alignment module 02 is used to add timestamps to the task requests, the Kunpeng computing node running status information, the task running logs, and the queue scheduling information, and align them according to the task execution cycle and the node status cycle to generate a Kunpeng scheduling status dataset. Specifically, the time alignment module receives task requests, Kunpeng computing node running status information, task running logs, and queue scheduling information from the data acquisition module, and writes the task submission time, task enqueue time, node status sampling time, and task execution time into them, respectively. The time alignment module establishes a scheduling window based on the task execution cycle and the node status cycle, and maps the fields of the task requests, Kunpeng computing node running status information, task running logs, and queue scheduling information within the same scheduling window to generate a scheduling mapping relationship between tasks and Kunpeng computing nodes. When the timestamp is incomplete, the time alignment module writes an exception identifier and retains the original record. The Kunpeng scheduling status dataset is passed to the task feature generation module and is also available for use by the resource profile generation module.

[0106] The task feature generation module 03 is used to extract task source identifier, data format, data volume, priority, and data input address from the Kunpeng scheduling status dataset to generate task feature data. Specifically, the task feature generation module receives the Kunpeng scheduling status dataset from the time alignment module and reads the task source identifier, data format, data volume, priority, and data input address from the task request field. The task feature generation module calls the lightweight intelligent agent module to perform data field mapping on the fields written by different business systems, and combines the historical execution records of similar tasks in the task operation log to generate computational complexity features, real-time requirement features, resource dependency features, data format conversion requirements, data locality, input / output pressure, and historical resource consumption of similar tasks. When field validation fails, the task feature generation module retains the task request field and the exception identifier. The task feature data is output to the resource profile generation module and the matching degree evaluation module.

[0107] The resource profile generation module 04 is used to generate a resource profile of the Kunpeng computing node based on the task feature data and the Kunpeng scheduling status dataset. Specifically, the resource profile generation module receives task feature data from the task feature generation module and reads node status fields, log fields, queue fields, and scheduling mapping relationships from the Kunpeng scheduling status dataset. The resource profile generation module establishes node records based on the Kunpeng computing node identifier and writes node static attributes, node real-time attributes, node historical attributes, and node adaptation attributes. The node static attributes record the processor core structure, memory capacity, storage channels, network interfaces, and operating environment. The node real-time attributes record the processor core status, memory status, storage read / write status, network bandwidth, latency, and task queue length. The node historical attributes record historical resource snapshots, historical load rate sequences, and historical executed task data. The node adaptation attributes record task success rates and historical execution records of similar tasks. The Kunpeng computing node resource profile is output to the matching degree evaluation module.

[0108] The matching degree evaluation module 05 is used to perform adaptation structure difference analysis, load fluctuation analysis, data reading path processing, and dependent task status processing based on the task feature data and the Kunpeng computing node resource profile, generating a matching degree evaluation result. Specifically, the matching degree evaluation module receives task feature data from the task feature generation module and Kunpeng computing node resource profiles from the resource profile generation module. The matching degree evaluation module compares the resource requirement description in the task feature data with the node scheduling resource status in the Kunpeng computing node resource profile to generate a list of well-fitting tasks, poorly-fitting tasks, and candidate nodes. The matching degree evaluation module reads the node's historical load rate sequence, node's historical resource snapshot, node's real-time attributes, and task queue length to generate a load fluctuation record. The matching degree evaluation module generates a data reading path based on the data input address and data locality, and generates the dependent task execution status based on resource dependency characteristics and task running logs. The matching degree evaluation result is transmitted to the pre-scheduling module.

[0109] The pre-scheduling module 06 is used to classify tasks, pre-schedule and sort them, and pre-allocate resources based on the matching degree evaluation results, generating pre-scheduling results. Specifically, the pre-scheduling module receives the matching degree evaluation results from the matching degree evaluation module and reads the structure adaptation record, load fluctuation record, data reading path, execution status of dependent tasks, and candidate node list. The pre-scheduling module writes tasks with good structure adaptation into the execution queue and writes tasks with unbalanced structure adaptation into the exception record area. The pre-scheduling module classifies tasks according to priority, real-time requirements, computational complexity, resource dependency, and input / output pressure. The pre-scheduling module arranges the candidate node list according to the matching degree evaluation results, task queue length, and execution window, generating target Kunpeng computing nodes and alternative Kunpeng computing nodes. The pre-scheduling module generates resource allocation requests based on computing resources, storage resources, network resources, and runtime environment configuration. The pre-scheduling results are transmitted to the instruction generation module.

[0110] The instruction generation module 07 is used to generate task deployment instructions and resource allocation instructions based on the pre-scheduling results. Specifically, the instruction generation module receives the pre-scheduling results from the pre-scheduling module and reads the target Kunpeng computing node, execution queue, execution window, data reading path, and resource allocation request. The instruction generation module writes the target Kunpeng computing node, execution queue, execution window, and data reading path into the task deployment instructions, and writes the computing resources, storage resources, network resources, and runtime environment configuration into the resource allocation instructions. The instruction generation module sends the task deployment instructions and resource allocation instructions to the scheduler global control component, which then distributes them to the target Kunpeng computing node. When an instruction field is missing, the instruction generation module writes an exception flag and retains the original record of the pre-scheduling results. The task deployment instructions and the resource allocation instructions are transmitted to the feedback update module to register the instruction execution status.

[0111] The feedback update module 08 is used to monitor the instruction execution status, generate feedback data, and update the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library. Specifically, the feedback update module receives task deployment instructions and resource allocation instructions from the instruction generation module, and collects instruction execution status through the node resource monitoring system, the task execution log unit, and the queue scheduling management system. The feedback update module reads resource utilization rate, task success rate, actual resource consumption deviation rate, task queue length, migration execution status, and resource release information to generate feedback data. The feedback update module updates the resource requirement description in the task feature data according to the actual resource consumption deviation rate, updates the node adaptation attributes in the Kunpeng computing node resource profile according to the resource utilization rate, task success rate, and resource release information, and updates the candidate node selection rules, resource pre-allocation rules, and migration triggering conditions in the scheduling strategy library according to the migration execution status and task queue length. The feedback data is then sent back to the task feature generation module, the resource profile generation module, and the pre-scheduling module.

Claims

1. A method for optimizing multi-source data task scheduling based on Kunpeng computing power intelligent agents, characterized in that, include: S100: Collect task requests, Kunpeng computing node running status information, task running logs and queue scheduling information, add timestamps and align them with the task execution cycle and node status cycle to generate Kunpeng scheduling status dataset; S200. Based on the Kunpeng scheduling status dataset, the task source identifier, data format, data volume, priority, and data input address are extracted by the lightweight intelligent agent module to generate task feature data. S300. Generate a resource profile of the Kunpeng computing node based on the task feature data and the Kunpeng scheduling status dataset; S400. Based on the task feature data and the Kunpeng computing node resource profile, perform adaptation structure difference analysis, load fluctuation analysis, data reading path processing and dependent task status processing to generate a matching degree evaluation result. S500: Based on the matching degree evaluation results, perform task classification, pre-scheduling sorting, and resource pre-allocation to generate pre-scheduling results; S600. Generate task deployment instructions and resource allocation instructions based on the pre-scheduling results, monitor the execution status of the instructions to generate feedback data, and update the task feature data, the Kunpeng computing node resource profile, and the scheduling strategy library.

2. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 1, characterized in that, In step S100, the task request is collected by the task submission interface; the Kunpeng computing node running status information is collected by the node resource monitoring system; the task running log is collected by the task running log unit; and the queue scheduling information is collected by the queue scheduling management system and the scheduler global control component.

3. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 1, characterized in that, The time alignment includes writing the task submission time, task enqueue time, node status sampling time, and task execution time into the same scheduling window to generate a scheduling mapping relationship between the task and the Kunpeng computing node.

4. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 1, characterized in that, In step S200, the task characteristic data includes computational complexity characteristics, real-time requirement characteristics, resource dependency characteristics, data format conversion requirements, data locality, input / output pressure, and resource consumption of similar historical tasks.

5. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 1, characterized in that, In S300, the Kunpeng computing node resource profile includes node static attributes, node real-time attributes, node historical attributes, and node adaptation attributes; the node static attributes include Kunpeng computing node identifier, processor core structure, memory capacity, storage channel, network interface, and operating environment; the node real-time attributes include processor core status, memory status, storage read / write status, network bandwidth and latency, and task queue length; the node historical attributes include node historical resource snapshots, node historical load rate sequences, and historical executed task data; the node adaptation attributes include task success rate and historical execution records of similar tasks.

6. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 1, characterized in that, In step S400, the adaptation structure difference analysis includes: comparing the resource requirement description in the task feature data with the node scheduling resource status in the Kunpeng computing node resource profile to generate a list of tasks with good structure adaptation, tasks with unbalanced structure adaptation, and candidate nodes.

7. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 4, characterized in that, In step S400, the data reading path processing includes: generating a data reading path based on the data input address and the data locality; the dependent task status processing includes: generating a dependent task execution status based on the resource dependency characteristics and the task execution log; and writing the data reading path and the dependent task execution status into the matching degree evaluation result.

8. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 6, characterized in that, In step S500, task classification includes writing the task feature data into the execution queue; pre-scheduling sorting includes arranging the candidate node list according to the matching degree evaluation result, task queue length, and execution window; and resource pre-allocation includes generating target Kunpeng computing nodes, alternative Kunpeng computing nodes, resource allocation requests, and execution windows.

9. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 8, characterized in that, In S600, the task deployment instruction includes the target Kunpeng computing node, the execution queue, the execution window, and the data reading path; the resource allocation instruction includes computing resources, storage resources, network resources, and runtime environment configuration; when the migration triggering condition matches the instruction execution status, a migration instruction is generated.

10. The multi-source data task scheduling optimization method based on Kunpeng computing power intelligent agent driven according to claim 1, characterized in that, In step S600, the feedback data includes resource utilization rate, task success rate, deviation rate between actual resource consumption and estimated value, task queue length, migration execution status, and resource release information; and updates the resource requirement description in the task feature data, the node adaptation attributes in the Kunpeng computing node resource profile, and the candidate node screening rules, resource pre-allocation rules, and migration triggering conditions in the scheduling strategy library based on the feedback data.

Images