A method for implementing a run management service for large sample job simulation

Abstract

The application provides a running management service implementation method for large sample operation simulation, and belongs to the technical field of computer simulation. The application realizes the real-time and comprehensiveness of state monitoring in a large sample simulation scene by building a three-level combined query function of operation-node-task and a priority queue scheduling, avoids blocking of high-priority operations, and flexibly adjusts operation parameters without resubmission. The application realizes efficient adaptation of cluster resources and simulation tasks, improves resource utilization, and avoids repeated scheduling by realizing real-time collection of resources through interfacing with cluster management software, screening available nodes according to rules, and uniformly distributing tasks. The application realizes accurate identification and automatic processing of abnormal samples by establishing a monitoring dimension-exception type mapping relationship, a hierarchical disposal strategy and a round jump mechanism, minimizes the influence range of faults, avoids blocking the overall process in a single round of exceptions, and guarantees stable operation of thousands / millions of parallel tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer simulation technology, and in particular relates to a method for implementing operation management services for large-sample job simulation. Background Technology

[0002] With the deep application of simulation technology in various fields, large-sample job simulation (i.e., large-scale parallel computing scenarios that require running thousands or tens of thousands of simulation tasks at the same time) has become a core demand of scientific research and industry.

[0003] However, existing technologies still have shortcomings. (1) Insufficient real-time and comprehensiveness of status monitoring: Traditional simulation systems are mostly designed for small sample tasks, and the monitoring granularity only covers single nodes or small-scale clusters, which is difficult to cope with the distributed architecture of tens of thousands of nodes and millions of tasks in large sample scenarios. (2) Lack of flexibility in node engine collaborative control: Large sample simulation tasks usually have strong parallelism but weak correlation (such as Monte Carlo simulation), or weak parallelism but strong dependence (such as multi-stage fluid simulation). (3) Low degree of automation in abnormal sample handling: In large sample simulation, the probability of abnormal samples (such as crashes caused by incorrect input data, "zombie tasks" that are not completed due to computation timeout, and dirty data generated by hardware failure) increases exponentially with the sample size. Existing technologies mostly rely on manual investigation (such as checking logs and manually restarting tasks), which leads to long abnormal recovery time, high labor costs, and easy omission of potential risks (such as small-scale abnormalities spreading into global failures). Summary of the Invention

[0004] In view of the above-mentioned technical status, this invention proposes a method for implementing operation management services for large-sample job simulation.

[0005] The method for implementing the operation management service includes implementing job management service, computing resource management service, and exception handling service; wherein: The job management service includes job status monitoring, job status query, and job submission. It is used to collect, aggregate, and analyze the status data of each node and task in the cluster in real time, serving as a basis for decision-making. For job status monitoring, a distributed data acquisition agent is used, deployed on each cluster node, to synchronously collect two types of core data: node-level metrics and task-level metrics. For job status queries, it supports combined queries by job attribute, node attribute, and task attribute to meet the needs of different scenarios; For job submissions, priorities are set for jobs, and a built-in task queue pool is used to sort jobs by priority and submission time. The queue also supports pause / resume functionality. The computing resource management service includes node status monitoring, resource status monitoring, and computing resource allocation. It is used to collect and schedule computing node resources, adapt to the current cluster environment based on the node status, establish communication channels with the cluster management software, and obtain the cluster resource status. The anomaly handling service includes anomaly sample identification, anomaly tiered processing, and round-trip mechanism. This service automatically identifies anomaly sample types and triggers tiered processing to minimize the impact of faults. For anomaly sample identification, based on the anomaly scenarios of large-sample simulation operations, a correspondence between monitoring dimensions and anomaly types is established to ensure coverage of anomalies across all scenarios; For anomaly classification and handling, anomalies are classified based on the impact range and spread risk of abnormal samples, and differentiated handling strategies are adopted to achieve local anomaly handling and global anomaly risk control, so as to avoid a single anomaly from causing cluster-level failures. For the round jump mechanism, in the scenario of multi-round parallel computing for large sample simulation, the round jump mechanism is used to solve the problem of overall process blockage caused by single round anomalies, so as to achieve "abnormal round jump and sufficient sample guarantee".

[0006] The prerequisites for implementing the operation management service method include: building a cluster environment that supports large-scale parallel computing, ensuring that the number of cluster nodes meets the requirements for parallel execution of thousands / tens of thousands of simulation tasks, that each node has the hardware foundation to stably run the simulation engine, support multi-core CPU computing, meet memory / network bandwidth requirements, and avoid task abnormalities due to insufficient hardware performance.

[0007] The software deployment of the operation management service implementation method includes: Deploy distributed data collection agents: Pre-install agent programs on each node of the cluster to ensure that the agents can collect node and task data normally and establish communication with the job monitoring module; Configure cluster management software: used to implement basic management of cluster node resources and establish communication channels with the computing resource management module; Build a task queue pool: Built-in queue management function, supports sorting by priority and submission time, and has queue pause / resume capabilities.

[0008] Job status monitoring includes: Agent deployment and data collection configuration: Deploy a distributed data collection agent on each cluster node and set the data collection frequency; Configure the Agent to collect two types of data: Node-level metrics: CPU utilization, memory usage, network bandwidth, disk I / O; Task-level metrics: progress and runtime of a single simulation task.

[0009] Data aggregation and transmission: The Agent uploads the collected node / task data in real time, aggregates multi-source data, and forms a unified data view that can be used for scheduling decisions.

[0010] Job status query includes: A combined query function is built based on a three-level dimension of job, node, and task. Configure query dimensions: Job Dimension: Supports setting query conditions by job ID, job name, submission time, and job type. The returned results include the overall job progress, the number of assigned nodes, and the number of completed / abnormal / running tasks. Node dimension: Supports setting query conditions by node IP and node status, and the returned results include the ID of all tasks on the node, the job to which it belongs, the runtime, and the resource usage. Task dimension: Supports setting query conditions by task ID, task status, and timeout threshold. The returned results include the detailed log path of the task, the input data validation results, and the current execution steps.

[0011] Call the query function: Users can select query dimensions and conditions through the module's interactive interface, retrieve data in real time, and receive the results.

[0012] Assignment submission includes: Configure job parameters: When submitting a job, set the job priority, which is used for queue sorting; fill in the basic job information and specify the resource requirements for a single task; Management queue scheduling: After a job is submitted, it automatically enters the built-in task queue pool. The queue is sorted by priority + submission time, and high-priority jobs are scheduled first to avoid being blocked by low-priority tasks. If job parameters are temporarily adjusted, the task issuance of the job can be stopped by pausing the queue. The queue will be restored after the parameter adjustment is completed.

[0013] For computing resource management services: Node status monitoring includes: Read the configuration file; load the resource management configuration file and obtain key parameters, including cluster node communication addresses, resource allocation rules, and resource reservation ratios; Resource status monitoring includes: Establish a communication channel with the cluster management software to obtain the cluster resource status in real time, including basic node information, real-time node load, and allocated resources; The collected resource data is cleaned and classified, and online and non-high-load available nodes are selected to form a list of available resources; Computing resource allocation includes: Receive resource request requests; formulate a resource allocation plan based on the available resource list and job requirements: distribute tasks to multiple available nodes according to the principle of uniform distribution; mark the allocated nodes as occupied to avoid duplicate scheduling by other jobs; update the resource allocation plan synchronously, and distribute tasks to the corresponding nodes through the cluster management software to trigger task execution.

[0014] Anomaly sample identification includes: Establish a mapping relationship between monitoring dimensions, abnormal behavior characteristics, and corresponding abnormal types to cover anomalies across all scenarios; Real-time monitoring and matching; receive node / task data, match it with the mapping relationship in real time, and determine whether there are any anomalies and the type of anomaly.

[0015] Anomaly classification and handling include: Anomaly classification criteria were established, and anomalies were divided into three levels based on their impact scope and spread risk: Level 1 anomaly: A single task or single node anomaly with no risk of propagation; Level 2 anomaly: Multiple tasks or nodes are experiencing anomalies, posing a risk of local spread; Level 3 anomaly: An anomaly at the entire job or cluster level, posing a risk of global spread; Configure differentiated handling strategies: Automatic handling of Level 1 anomalies; Automatically handle level 2 anomalies and send alarm notifications to the administrator; For Level 3 anomalies, immediately suspend the scheduling of related tasks, release all occupied resources, send an emergency alert to the administrator, and wait for intervention and investigation.

[0016] Round-jumping mechanisms include: Round anomaly detection: First, clarify the triggering conditions for round anomalies, including: Abnormal task percentage: The proportion of abnormal tasks in a single round to the total number of tasks in that round is ≥10%; Round time exceeded the limit: The actual time spent in a single round exceeded the estimated time by 200% and ≥20% of the tasks were still not completed; The judgment process is as follows: During the operation of a round, a round health check is performed every 5 minutes. If any of the above conditions are met, a round abnormality warning is triggered. After 1 minute, the check is checked again. If the abnormality is confirmed, the round is judged to be abnormal. Jump trigger logic: Depending on the severity of the round anomaly, partial or full round jumps may be used to balance efficiency and data validity. Partial jump: Only abnormal tasks are reassigned resources and restarted, while normal tasks continue to execute; if an abnormal task still cannot be completed after restarting, that part of the abnormal task is skipped, the round does not jump as a whole, and the next round begins after the normal task is completed. Full Round Jump: Immediately terminates all tasks in the current round and releases all resources occupied by the round.

[0017] In summary, this invention achieves real-time and comprehensive status monitoring in large-sample simulation scenarios by establishing a three-level combined query function for jobs, nodes, and tasks, along with priority queue scheduling. This avoids blocking high-priority jobs and allows for flexible adjustment of job parameters without resubmission. By interfacing with cluster management software, this invention enables real-time resource collection, rule-based filtering of available nodes, and even task allocation, achieving efficient adaptation between cluster resources and simulation tasks, improving resource utilization, and avoiding redundant scheduling. Furthermore, by establishing a mapping relationship between monitoring dimensions and anomaly types, a tiered handling strategy, and a round-jumping mechanism, this invention achieves accurate identification and automated processing of abnormal samples, minimizing the impact of faults, preventing single-round anomalies from blocking the overall process, and ensuring the stable operation of thousands / tens of thousands of parallel tasks. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the implementation method of a runtime management service for large-sample job simulation according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the interaction between the various modules of the operation management service according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention proposes a method for implementing a runtime management service for large-sample job simulation. For example... Figure 1As shown, the method for implementing the operation management service includes implementing job management service, computing resource management service, and exception handling service.

[0022] Job monitoring includes job status monitoring, job status query, and job submission functions. It is used to collect, aggregate, and analyze the status data of each node and task in the cluster in real time, providing a basis for decision-making by the control module.

[0023] Job status monitoring: Distributed data acquisition agents are deployed on each cluster node to synchronously collect two types of core data: node-level metrics (CPU utilization, memory usage, network bandwidth, disk I / O) and task-level metrics (progress of a single simulation task, task runtime).

[0024] Job status query: Supports combined queries by "job attribute - node attribute - task attribute" to meet the needs of different scenarios. Job Dimension: Query by job ID, job name, submission time, job type (e.g., Monte Carlo simulation, fluid simulation) to return the overall progress of the job, the number of assigned nodes, and the number of completed / abnormal / running tasks; Node dimension: Query by node IP and node status (online / offline / high load) to return the ID, job, runtime, and resource consumption of all tasks on that node; Task dimension: Query by task ID, task status (initialization / running / complete / abnormal), and timeout threshold, and return the detailed log path of the task, input data validation results, and current execution steps.

[0025] Job submission: Supports setting priorities for jobs (high / medium / low levels); the system has a built-in task queue pool, which sorts tasks by "priority + submission time" to prevent high-priority jobs from being blocked by low-priority tasks; it also supports queue pause / resume function. If a job needs to temporarily adjust parameters, its queue can be paused and the scheduling can be resumed after the adjustment without resubmitting.

[0026] Computing resource management includes: collecting and scheduling computing node resources; adapting to the current cluster environment based on the node status; establishing communication channels with the cluster management software; and obtaining cluster resource status. This includes configuration file reading, resource collection, and resource allocation.

[0027] Anomaly handling includes addressing the potential for anomalous samples due to memory and bandwidth limitations, communication failures, etc. Without monitoring and handling, these anomalous samples will continuously consume CPU resources, accumulating in large-scale jobs and causing resource degradation, leading to job congestion or even crashes. Therefore, this module automatically identifies anomalous sample types and triggers tiered processing to minimize the impact of failures. This includes anomalous sample identification, tiered anomaly processing, and a round-robin switching mechanism.

[0028] Anomaly Sample Identification: Based on the anomaly scenarios of large-sample simulation jobs (such as "input data error crash, computation timeout zombie task, hardware failure dirty data" mentioned in the background technology), establish the correspondence between "monitoring dimension - anomaly type" to ensure coverage of anomalies in all scenarios.

[0029] Table 1

[0030] Anomaly classification and handling: Based on the "impact range and spread risk" of abnormal samples, anomalies are divided into three levels and differentiated handling strategies are adopted to achieve "local handling of small anomalies and global risk control of large anomalies", avoiding a single anomaly from causing cluster-level failures.

[0031] Round jump mechanism: For large-sample simulation scenarios with "multi-round parallel computing" (such as Monte Carlo simulation requiring tens of thousands of rounds of computing, and multi-stage simulation requiring round-by-round advancement), the round jump mechanism solves the problem of "single round anomalies causing overall process blockage", and achieves "flexible jump to abnormal rounds and sufficient guarantee of effective samples".

[0032] In one embodiment, such as Figure 2 As shown, a cluster environment that supports large-scale parallel computing needs to be built. The number of cluster nodes needs to meet the requirement of "parallel execution of thousands / tens of thousands of simulation tasks". Each node needs to have the hardware foundation to run the simulation engine stably (such as supporting multi-core CPU operation, meeting memory / network bandwidth requirements, and avoiding task abnormalities due to insufficient hardware performance).

[0033] In one embodiment, a distributed data collection agent is deployed: an agent program is pre-installed on each node of the cluster to ensure that the agent can collect node and task data normally and establish communication with the job monitoring module; cluster management software is configured: such as Slurm, YARN, etc., to realize basic management of cluster node resources, and a communication channel needs to be established with the computing resource management module; a task queue pool is built: a task queue management function is built-in, which supports sorting by priority and submission time, and has the ability to pause / resume the queue.

[0034] In one embodiment, the implementation of job status monitoring includes: Agent deployment and data collection configuration: Deploying distributed collection agents on each cluster node, setting the data collection frequency (ensuring real-time performance and avoiding data latency); configuring the agents to collect two types of core data: ① Node-level metrics: CPU utilization, memory usage, network bandwidth, disk I / O; ② Task-level metrics: progress of a single simulation task (e.g., completion percentage), task runtime. Data aggregation and transmission: The agents upload the collected node / task data to the job monitoring module in real time. The module aggregates the multi-source data to form a unified data view that can be used for scheduling decisions.

[0035] In one embodiment, the implementation for job status query is as follows: a combined query function is built based on the three-level dimensions of "job-node-task", and the specific implementation logic is as follows.

[0036] Query dimension configuration: Job Dimension: Supports setting query conditions by job ID, job name, submission time, and job type (such as Monte Carlo simulation, fluid simulation). The returned results include the overall job progress, the number of assigned nodes, and the number of completed / abnormal / running tasks. Node dimension: Supports setting query conditions by node IP and node status (online / offline / high load), and the returned results include the ID of all tasks on the node, the job to which it belongs, the runtime, and the resource usage. Task dimension: Supports setting query conditions by task ID, task status (initialization / running / complete / abnormal), and timeout threshold. The returned results include the detailed log path of the task, the input data validation results, and the current execution steps.

[0037] Query function call: Users select query dimensions and conditions through the module's interactive interface (such as a management terminal or visualization platform), and the module retrieves data and returns the results in real time.

[0038] In one embodiment, the process for submitting a job specifically includes the following steps.

[0039] Job parameter configuration: When submitting a job, you can set the job priority (high / medium / low levels), which is used for queue sorting; fill in the basic job information (job name, type, number of tasks, etc.) and specify the resource requirements for a single task (such as the number of CPU cores and the maximum memory usage).

[0040] Queue scheduling management: After a job is submitted, it automatically enters the built-in task queue pool. The queue is sorted by "priority + submission time," with higher-priority jobs scheduled first to avoid being blocked by lower-priority tasks. If job parameters need to be temporarily adjusted, the task issuance for that job can be stopped using the "queue pause" function. After the parameters are adjusted, "queue resume" can be executed without resubmitting the job.

[0041] In one embodiment, the computing resource management module is designed around "resource awareness and adaptive scheduling", with the core being to achieve efficient matching of cluster resources and simulation tasks.

[0042] Configuration file reading: Load the resource management configuration file and obtain key parameters: cluster node communication address (used for interfacing with cluster management software); resource allocation rules (such as node load threshold: CPU utilization ≥80% is considered high load and no new tasks are allocated); resource reservation ratio (reserving some nodes as backups for abnormal task reassignment).

[0043] Resource Collection: Establish a communication channel with the cluster management software (via API or protocol) to obtain real-time cluster resource status: Node basic information: node IP, online / offline status, hardware configuration (number of CPU cores, memory capacity); Real-time node load: current CPU utilization, memory usage, and network bandwidth usage for each node; Allocated resources: number of nodes occupied by other jobs, and CPU / memory resources. Clean and classify the collected resource data, and filter out available nodes that are "online and not under high load" to form an "available resource list".

[0044] Resource Allocation: Receives the "Resource Request" (including the number of job tasks and resource requirements per task) sent by the job monitoring module; Based on the "Available Resource List" and job requirements, formulates a resource allocation plan: Distributes tasks to multiple available nodes according to the "uniform distribution" principle (e.g., 10,000 tasks are distributed to 100 nodes, with each node handling 100 tasks); Marks the allocated nodes as "occupied" to prevent them from being repeatedly scheduled by other jobs; Synchronizes the resource allocation plan to the job monitoring module and distributes the tasks to the corresponding nodes through the cluster management software, triggering task execution.

[0045] In one embodiment, the anomaly handling module is designed around "accurate identification, hierarchical handling, and round-by-round protection" to solve the problems of resource waste and process blockage caused by abnormal samples in large-sample simulation.

[0046] Anomaly Sample Identification: Establish a mapping relationship of "monitoring dimension - anomaly manifestation characteristics - corresponding anomaly type" to cover anomalies across all scenarios (based on the three types of anomalies mentioned in the background technology document). Real-time Monitoring and Matching: The anomaly handling module receives node / task data transmitted from the job monitoring module and performs real-time matching with the above mapping rules to determine whether anomalies exist and their types.

[0047] Anomaly classification and handling: Anomalies are classified into three levels based on "scope of impact and risk of spread".

[0048] Level 1 anomalies: Single task or single node anomalies (such as a single task memory overflow or a single node going offline), with no risk of spread; Level 2 anomalies: Multi-task or multi-node anomalies (such as 10 tasks crashing or 3 nodes being overloaded in a job), with a risk of local spread; Level 3 anomalies: Job-wide or cluster-level anomalies (such as 50% of tasks in a job being abnormal or most nodes going offline), with a risk of global spread.

[0049] Differentiated handling strategies: Level 1 anomalies: Automatic handling (e.g., terminating and restarting the abnormal task, releasing resources of the abnormal node, and reallocating the task to a backup node); Level 2 anomalies: After automatic handling, send an alarm notification to the administrator (informing them of the anomaly type and scope of impact); Level 3 anomalies: Immediately suspend the task scheduling of the relevant jobs, release all occupied resources, send an emergency alarm to the administrator, and wait for manual intervention for investigation.

[0050] Round jump mechanism implementation: For large-sample simulation scenarios of "multi-round parallel computing", avoid single-round abnormal blocking of the overall process.

[0051] Round Anomaly Detection: First, clarify the triggering conditions for "round anomalies" to avoid blindly jumping to new rounds due to minor anomalies (wasting computing resources). The detection conditions are: ① Percentage of abnormal tasks: The proportion of abnormal tasks in a single round to the total number of tasks in that round is ≥10%; ② Round time exceeds the limit: The actual time of a single round exceeds 200% of the "estimated time" (estimated time is calculated based on the time of similar rounds in history), and ≥20% of tasks are still not completed (determined as overall round lag). The detection process is as follows: During round operation, a "round health check" is performed every 5 minutes. If any of the above conditions are met, a "round anomaly warning" is triggered. After 1 minute, it is checked again (to avoid instantaneous fluctuations). If an anomaly is confirmed, it is determined as a "round anomaly".

[0052] Jump trigger logic: Depending on the severity of the round's anomaly, either a "partial jump" or a "full round jump" is adopted to balance efficiency and data validity: ① Partial jump (abnormal tasks account for 10%-20%): Only abnormal tasks are "reassigned resources - restarted", while normal tasks continue to execute. If an abnormal task still cannot be completed after restarting (failed twice consecutively), then that part of the abnormal task is skipped (marked as "invalid sample"), and the round as a whole is not jumped. After the normal tasks are completed, the next round begins (the number of invalid samples can be supplemented by "recalculating rounds" in subsequent rounds); ② Full round jump (abnormal tasks account for ≥20%, round time exceeds limit, critical task anomaly): All tasks in the current round (including normal tasks) are immediately terminated, and all resources occupied by the round are released.

[0053] In one embodiment, the complete execution of a large-sample simulation job is taken as the main line to realize the closed-loop linkage of the three major modules of "job monitoring - computing resource management - exception handling". The specific process is as follows.

[0054] Job Initiation and Resource Request: After receiving the simulation job submitted by the user, the job monitoring module generates a "Resource Request" (including the number of tasks, resource requirements per task, and priority) and sends it to the computing resource management module.

[0055] Resource allocation and task assignment: The computing resource management module formulates a resource allocation plan based on the available resources in the current cluster, assigns tasks to the corresponding nodes through the cluster management software, and synchronizes the "resource allocation results" back to the job monitoring module.

[0056] Task execution and data feedback: When cluster nodes execute tasks, the distributed agent collects node load and task status data in real time and uploads it to the job monitoring module; the job monitoring module synchronizes the data to the anomaly handling module for anomaly monitoring.

[0057] Anomaly Handling and Result Feedback: The anomaly handling module analyzes data in real time. If an anomaly is detected, it performs handling operations according to the hierarchical strategy and synchronizes the "anomaly handling results" (such as task restart status and round jump status) to the job monitoring module and the computing resource management module. The job monitoring module updates the job progress and status, and the computing resource management module adjusts resource allocation (such as releasing abnormal nodes and calling backup nodes), forming a closed-loop control system.

[0058] In summary, this invention achieves real-time and comprehensive status monitoring in large-sample simulation scenarios by establishing a three-level combined query function of "job-node-task" and priority queue scheduling, avoiding blocking of high-priority jobs and allowing flexible adjustment of job parameters without resubmission. By interfacing with cluster management software, this invention enables real-time resource collection, rule-based selection of available nodes, and even task allocation, achieving efficient adaptation of cluster resources and simulation tasks, improving resource utilization, and avoiding redundant scheduling. Furthermore, by establishing a "monitoring dimension-anomaly type" mapping relationship, a hierarchical handling strategy, and a round-jumping mechanism, this invention achieves accurate identification and automated processing of abnormal samples, minimizing the impact of faults, preventing single-round anomalies from blocking the overall process, and ensuring the stable operation of thousands / tens of thousands of parallel tasks.

[0059] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A method for implementing operation management services for large-sample job simulation, characterized in that, The method for implementing the operation management service includes implementing job management service, computing resource management service, and exception handling service; wherein: The job management service includes job status monitoring, job status query, and job submission. It is used to collect, aggregate, and analyze the status data of each node and task in the cluster in real time, serving as a basis for decision-making. For job status monitoring, a distributed data acquisition agent is used, deployed on each cluster node, to synchronously collect two types of core data: node-level metrics and task-level metrics. For job status queries, it supports combined queries by job attribute, node attribute, and task attribute to meet the needs of different scenarios; For job submissions, priorities are set for jobs, and a built-in task queue pool is used to sort jobs by priority and submission time. The queue also supports pause / resume functionality. The computing resource management service includes node status monitoring, resource status monitoring, and computing resource allocation. It is used to collect and schedule computing node resources, adapt to the current cluster environment based on the node status, establish communication channels with the cluster management software, and obtain the cluster resource status. The anomaly handling service includes anomaly sample identification, anomaly tiered processing, and round-trip mechanism. This service automatically identifies anomaly sample types and triggers tiered processing to minimize the impact of faults. For anomaly sample identification, based on the anomaly scenarios of large-sample simulation operations, a correspondence between monitoring dimensions and anomaly types is established to ensure coverage of anomalies across all scenarios; For anomaly classification and handling, anomalies are classified based on the impact range and spread risk of abnormal samples, and differentiated handling strategies are adopted to achieve local anomaly handling and global anomaly risk control, so as to avoid a single anomaly from causing cluster-level failures. For the round jump mechanism, in the scenario of parallel computing of multiple rounds in large sample simulation, the round jump mechanism is used to solve the problem of the overall process being blocked due to the abnormality of a single round, so as to achieve "abnormal round jump and sufficient sample guarantee".

2. The method for implementing operation management services for large-sample job simulation according to claim 1, characterized in that, The prerequisites for implementing the operation management service method include: building a cluster environment that supports large-scale parallel computing, ensuring that the number of cluster nodes meets the requirements for parallel execution of thousands / tens of thousands of simulation tasks, that each node has the hardware foundation to stably run the simulation engine, support multi-core CPU computing, meet memory / network bandwidth requirements, and avoid task abnormalities due to insufficient hardware performance.

3. The method for implementing operation management services for large-sample job simulation according to claim 2, characterized in that, The software deployment of the operation management service implementation method includes: Deploy distributed data collection agents: Pre-install agent programs on each node of the cluster to ensure that the agents can collect node and task data normally and establish communication with the job monitoring module; Configure cluster management software: used to implement basic management of cluster node resources and establish communication channels with the computing resource management module; Build a task queue pool: Built-in queue management function, supports sorting by priority and submission time, and has queue pause / resume capabilities.

4. The method for implementing operation management services for large-sample job simulation according to claim 3, characterized in that, Operation status monitoring includes: Agent deployment and data collection configuration: Deploy a distributed data collection agent on each cluster node and set the data collection frequency; Configure the Agent to collect two types of data: Node-level metrics: CPU utilization, memory usage, network bandwidth, disk I / O; Task-level metrics: progress and runtime of a single simulation task. Data aggregation and transmission: The Agent uploads the collected node / task data in real time, aggregates multi-source data, and forms a unified data view that can be used for scheduling decisions.

5. The method for implementing operation management services for large-sample job simulation according to claim 4, characterized in that, Job status query includes: A combined query function is built based on a three-level dimension of job, node, and task. Configure query dimensions: Job Dimension: Supports setting query conditions by job ID, job name, submission time, and job type. The returned results include the overall job progress, the number of assigned nodes, and the number of completed / abnormal / running tasks. Node dimension: Supports setting query conditions by node IP and node status, and the returned results include the ID of all tasks on the node, the job to which it belongs, the runtime, and the resource usage. Task dimension: Supports setting query conditions by task ID, task status, and timeout threshold. The returned results include the detailed log path of the task, the input data validation results, and the current execution steps. Call the query function: Users can select query dimensions and conditions through the module's interactive interface, retrieve data in real time, and receive the results.

6. The method for implementing operation management services for large-sample job simulation according to claim 5, characterized in that, Assignment submission includes: Configure job parameters: When submitting a job, set the job priority, which is used for queue sorting; fill in the basic job information and specify the resource requirements for a single task; Management queue scheduling: After a job is submitted, it automatically enters the built-in task queue pool. The queue is sorted by priority + submission time, and high-priority jobs are scheduled first to avoid being blocked by low-priority tasks. If job parameters are temporarily adjusted, the task issuance of the job can be stopped by using the queue pause function. The queue will be restored after the parameter adjustment is completed.

7. The method for implementing operation management services for large-sample job simulation according to claim 6, characterized in that, For computing resource management services: Node status monitoring includes: Read the configuration file; load the resource management configuration file and obtain key parameters, including cluster node communication addresses, resource allocation rules, and resource reservation ratios; Resource status monitoring includes: Establish a communication channel with the cluster management software to obtain the cluster resource status in real time, including basic node information, real-time node load, and allocated resources; The collected resource data is cleaned and classified, and online and non-high-load available nodes are selected to form a list of available resources; Computing resource allocation includes: Receive resource request requests; formulate a resource allocation plan based on the available resource list and job requirements: distribute tasks to multiple available nodes according to the principle of uniform distribution; mark the allocated nodes as occupied to avoid duplicate scheduling by other jobs; update the resource allocation plan synchronously, and distribute tasks to the corresponding nodes through the cluster management software to trigger task execution.

8. The method for implementing operation management services for large-sample job simulation according to claim 7, characterized in that, Anomaly sample identification includes: Establish a mapping relationship between monitoring dimensions, abnormal behavior characteristics, and corresponding abnormal types to cover anomalies across all scenarios; Real-time monitoring and matching; receive node / task data, match it with the mapping relationship in real time, and determine whether there are any anomalies and the type of anomaly.

9. The method for implementing operation management services for large-sample job simulation according to claim 8, characterized in that, Anomaly classification and handling include: Anomaly classification criteria were established, and anomalies were divided into three levels based on their impact scope and spread risk: Level 1 anomaly: A single task or single node anomaly with no risk of propagation; Level 2 anomaly: Multiple tasks or nodes are experiencing anomalies, posing a risk of local spread; Level 3 anomaly: An anomaly at the entire job or cluster level, posing a risk of global spread; Configure differentiated handling strategies: Automatic handling of Level 1 anomalies; Automatically handle level 2 anomalies and send alarm notifications to the administrator; For Level 3 anomalies, immediately suspend the task scheduling of related operations, release all occupied resources, send an emergency alarm to the administrator, and wait for intervention and investigation.

10. The method for implementing operation management services for large-sample job simulation according to claim 9, characterized in that, Round-jumping mechanisms include: Round anomaly detection: First, clarify the triggering conditions for round anomalies, including: Abnormal task percentage: The proportion of abnormal tasks in a single round to the total number of tasks in that round is ≥10%; Round time exceeded the limit: The actual time spent in a single round exceeded the estimated time by 200% and ≥20% of the tasks were still not completed; The judgment process is as follows: During the operation of a round, a round health check is performed every 5 minutes. If any of the above conditions are met, a round abnormality warning is triggered. After 1 minute, the check is checked again. If the abnormality is confirmed, the round is judged to be abnormal. Jump trigger logic: Depending on the severity of the round anomaly, partial or full round jumps may be used to balance efficiency and data validity. Partial jump: Only abnormal tasks are reassigned resources and restarted, while normal tasks continue to execute; if an abnormal task still cannot be completed after restarting, that part of the abnormal task is skipped, the round does not jump as a whole, and the next round begins after the normal task is completed. Full Round Jump: Immediately terminates all tasks in the current round and releases all resources occupied by the round.

Images