Resource optimization-oriented big data scheduling method, apparatus and device, and storage medium

Abstract

The invention belongs to the technical field of big data processing and resource management, and discloses a resource optimization-oriented big data scheduling method and device, equipment and a storage medium. The method comprises the following steps: synchronizing metadata of an off-line production task from an existing data processing system through a preset interface, and carrying out sorting and resource allocation on a task queue by adopting a hybrid scheduling algorithm according to a task priority, a predicted execution time length and a task dependency relationship, so as to generate a task scheduling sequence and a resource allocation scheme, then starting and executing the task, monitoring the task execution state and the resource use condition in real time in the execution process, and dynamically adjusting the task execution sequence to realize bidirectional synchronization of production data and scheduling information without additionally designing an external data acquisition component; the method completely depends on off-line production data acquired by an existing data processing system of a semiconductor enterprise, realizes efficient and reasonable allocation of resources, and remarkably improves the scheduling efficiency and the overall performance of the system.

Description

Technical Field

[0001] This invention relates to the field of big data processing and resource management technology, and in particular to a big data scheduling method, apparatus, device and storage medium for resource optimization. Background Technology

[0002] In offline processing scenarios relying on existing data processing systems, semiconductor companies face significant differences in resource requirements for different offline tasks. Traditional scheduling methods often employ static or simple dynamic resource allocation strategies, which struggle to accurately match the real-time resource needs of offline tasks. For example, when two types of offline tasks coexist—"cleaning based on collected wafer etching parameter data" and "yield aggregation based on collected chip test data"—failure to allocate CPU, memory, and other computing resources reasonably can lead to resource-intensive yield aggregation tasks processing slowly due to insufficient resources, while resource-inefficient etching parameter cleaning tasks may waste resources due to excessive consumption, severely impacting the normal progress of semiconductor production processes. Furthermore, as the amount of production data collected by existing data processing systems in semiconductor companies increases dramatically, the number and complexity of offline tasks based on this data also increase simultaneously. Existing scheduling algorithms are prone to scheduling delays and task backlogs when handling large-scale offline task queues. For example, although the first-in-first-out scheduling algorithm is simple to implement, it cannot effectively guarantee the timely execution of critical tasks when faced with tasks of different priorities, such as "generating chip test data yield reports" and "archiving historical packaging and testing data", which ultimately leads to unreasonable resource allocation and low scheduling efficiency.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a big data scheduling method, apparatus, device, and storage medium for resource optimization, aiming to solve the technical problems of unreasonable resource allocation and low scheduling efficiency.

[0005] To achieve the above objectives, the present invention provides a resource-optimized big data scheduling method, which includes the following steps: The metadata of offline production tasks is synchronized from the existing data processing system through a preset interface. The metadata includes task type, execution cycle, resource requirements and task dependencies. Based on the metadata of the offline production tasks, a hybrid scheduling algorithm is used to sort and allocate resources in the task queue according to task priority, expected execution time and task dependencies, generating a task scheduling order and resource allocation scheme. Tasks are initiated and executed according to the task scheduling order and resource allocation scheme. During the execution process, the task execution status and resource usage are monitored in real time, and resource allocation and task execution order are dynamically adjusted based on the monitoring results. The task execution results and resource usage data obtained from monitoring are fed back to the existing data processing system to achieve two-way synchronization of production data and scheduling information.

[0006] In some embodiments, the task queue is sorted and resource-allocated using a hybrid scheduling algorithm based on the metadata of the offline production task, according to task priority, expected execution time, and task dependencies, to generate a task scheduling order and resource allocation scheme, including: Based on the metadata of the offline production tasks, obtain the business importance identifier and urgency identifier, and determine the dynamic priority value of each offline production task according to the business importance identifier and urgency identifier. Based on the dynamic priority value, the task queue is divided into three sub-queues with high, medium, and low priorities. For tasks within subqueues of the same priority, a second sorting is performed using the shortest job first algorithm based on the expected execution time in the corresponding metadata. Iterate through the twice-sorted task queue and check if each task has any preceding dependent tasks. If there are prerequisite tasks, then move the prerequisite tasks in the queue to before the corresponding tasks; After completing the checks and adjustments of the prerequisite tasks, the task scheduling order is obtained. A resource allocation scheme is generated based on the task scheduling order.

[0007] In some embodiments, generating a resource allocation scheme based on the task scheduling order includes: Based on the task scheduling order, the resource requirement metadata of each task in the sequence is obtained in turn, and the total resource requirement curve for the recent period is formed by summarizing them. The total resource demand curve is compared with the real-time and predicted system available resources obtained synchronously from the existing data processing system to detect whether there are any points in time when resources are over-allocated. If such occurrences are detected, then, while ensuring that task dependencies are met, some low-priority tasks on non-critical paths will be shifted backward on the timeline to eliminate resource spikes. The adjusted task start time is bound to the resource allocation quota to generate a resource allocation plan.

[0008] In some embodiments, the dependency relationship is obtained by constructing a dependency graph based on task dependency metadata synchronized from an existing data processing system, and the method further includes: Each offline production task is abstracted as a node in a graph. Directed edges between nodes are constructed based on the dependency metadata between tasks, forming a directed acyclic graph of task dependencies. The directed acyclic graph that the task depends on is subjected to loop detection. If a loop dependency is found, the abnormal information is reported to the existing data processing system and the scheduling of the relevant tasks is suspended. Perform topological sorting on the detected task dependency directed acyclic graph to generate one or more potential task execution sequences that satisfy all dependencies; The potential sequences generated by topological sorting are fused and optimized with the sequences generated by sorting based on priority and shortest job principles, and the dependencies are constructed using the topological sequences as constraints.

[0009] In some embodiments, the dynamic adjustment of resource allocation based on monitoring results includes: Obtain the actual resource utilization of tasks currently being executed based on monitoring results; If the actual resource utilization rate is detected to be lower than the preset allocation threshold, the amount of resources allocated to the corresponding task will be automatically reduced, and the idle resources will be released to the public resource pool. If the actual resource utilization rate is detected to have reached the upper limit of the preset allocation limit, it is determined whether there are remaining resources in the public resource pool, and if there are remaining resources, the amount of resources allocated to the corresponding task is automatically increased.

[0010] In some embodiments, dynamically adjusting the task execution order based on monitoring results includes: Based on monitoring results, tasks whose actual execution time exceeds the expected execution time and reaches the timeout threshold are identified. The task queue containing the task is paused, and a rescheduling is triggered. The new task or timed-out task and all remaining pending tasks are added to the new task queue, and a new task scheduling order and resource allocation scheme are generated based on the new task queue. Based on the new task scheduling order and resource allocation scheme, new high-priority tasks will be launched first.

[0011] In some embodiments, the task execution status includes successful execution, failed execution, and suspended execution. If the task execution status is successful, the task execution result and resource usage data are fed back to the existing data processing system. If the task execution status is failed, the task execution result is directly fed back to the existing data processing system. If the task execution status is suspended, the task execution status and corresponding logs are recorded, and the system waits for the next task execution to retry.

[0012] Furthermore, to achieve the above objectives, the present invention also proposes a resource-optimized big data scheduling device, which includes: The acquisition module is used to synchronize the metadata of offline production tasks from the existing data processing system through a preset interface. The metadata includes task type, execution cycle, resource requirements and task dependencies. The generation module is used to sort and allocate resources in the task queue based on the metadata of the offline production tasks according to task priority, expected execution time and task dependencies, and generate task scheduling order and resource allocation scheme. The execution module is used to start and execute tasks according to the task scheduling order and resource allocation scheme. At the same time, it monitors the task execution status and resource usage in real time during the execution process, and dynamically adjusts resource allocation and task execution order based on the monitoring results. The feedback module is used to feed back the task execution results and resource usage data obtained from monitoring to the existing data processing system, so as to achieve two-way synchronization of production data and scheduling information.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes a resource-optimized big data scheduling device, which includes: a memory, a processor, and a resource-optimized big data scheduling program stored in the memory and executable on the processor. The resource-optimized big data scheduling program is configured to implement the steps of the resource-optimized big data scheduling method described above.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a resource-optimized big data scheduler, wherein the resource-optimized big data scheduler, when executed by a processor, implements the steps of the resource-optimized big data scheduling method described above.

[0015] This invention is based on synchronizing the metadata of offline production tasks from the existing data processing system through a preset interface. According to the task priority, expected execution time and task dependencies, a hybrid scheduling algorithm is used to sort the task queue and allocate resources to generate a task scheduling order and resource allocation scheme. Then, the task is started and executed. During the execution process, the task execution status and resource usage are monitored in real time, and the task execution order is dynamically adjusted to achieve bidirectional synchronization of production data and scheduling information. There is no need to design additional external data acquisition components. It relies entirely on the offline production data already collected by the semiconductor company's existing data processing system to achieve efficient and reasonable allocation of resources, significantly improving scheduling efficiency and overall system performance. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the first embodiment of the resource-optimized big data scheduling method of the present invention. Figure 2 This is a system architecture diagram of the resource-optimized big data scheduling method of the present invention; Figure 3 This is a structural block diagram of the first embodiment of the resource-optimized big data scheduling device of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0019] This invention provides a big data scheduling method for resource optimization, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of a resource-optimized big data scheduling method according to the present invention.

[0020] In this embodiment, the resource-optimized big data scheduling method includes the following steps: Step S10: Synchronize the metadata of offline production tasks from the existing data processing system through a preset interface.

[0021] In this embodiment, the execution entity is a resource-optimized big data scheduling device. This resource-optimized big data scheduling device has functions such as data processing, data communication, and program execution. The resource-optimized big data scheduling device can be a computer terminal device or other network device, or other devices with similar functions. This embodiment does not limit the scope of the application.

[0022] It's important to note that in offline processing scenarios relying on existing data processing systems, semiconductor companies exhibit significant differences in resource requirements across different offline tasks. Traditional scheduling methods often employ static or simple dynamic resource allocation strategies, which struggle to accurately match the real-time resource needs of offline tasks. For instance, when two types of offline tasks coexist—"cleaning based on collected wafer etching parameter data" and "yield aggregation based on collected chip test data"—failure to allocate CPU, memory, and other computing resources reasonably can lead to resource-intensive yield aggregation tasks processing slowly due to insufficient resources, while resource-intensive etching parameter cleaning tasks may waste resources due to excessive consumption, severely impacting the normal progress of semiconductor production processes. Furthermore, as the amount of production data collected by existing data processing systems in semiconductor companies increases dramatically, the number and complexity of offline tasks based on this data also rise simultaneously. Existing scheduling algorithms are prone to scheduling delays and task backlogs when handling large-scale offline task queues. For example, although the first-in-first-out scheduling algorithm is simple to implement, it cannot effectively guarantee the timely execution of critical tasks when faced with tasks of different priorities, such as "generating chip test data yield reports" and "archiving historical packaging and testing data", which ultimately leads to unreasonable resource allocation and low scheduling efficiency.

[0023] To address the aforementioned technical issues, this embodiment synchronizes the metadata of offline production tasks from the existing data processing system via a preset interface. Based on task priority, estimated execution time, and task dependencies, a hybrid scheduling algorithm is used to sort and allocate resources in the task queue, generating a task scheduling order and resource allocation scheme. Then, the tasks are started and executed. Simultaneously, the task execution status and resource usage are monitored in real time during execution, and the task execution order is dynamically adjusted to achieve bidirectional synchronization of production data and scheduling information. No additional external data acquisition components are required. It relies entirely on the offline production data already collected by the semiconductor company's existing data processing system to achieve efficient and reasonable resource allocation, significantly improving scheduling efficiency and overall system performance. Specifically, it can be implemented as follows.

[0024] In this implementation, the metadata of offline production tasks needs to be synchronized from the existing data processing system through a preset interface. This metadata includes task type, execution cycle, resource requirements, and task dependencies. Task types include high-priority tasks and low-priority tasks, execution cycles include a 2-hour cleaning task for production line A and a 4-hour cleaning task for production line B, resource requirements are the resources needed to execute different tasks, and task dependencies include a wafer etching parameter cleaning task being a prerequisite task for wafer processing yield analysis.

[0025] It should be noted that this embodiment can be deeply integrated with the existing data processing system of semiconductor companies through native adaptation design, directly acquiring offline production data (such as raw chip testing data and packaging process inspection data) and task metadata already collected within the system. This reduces system integration costs, avoids data transmission delays and format compatibility issues, and improves the overall efficiency of semiconductor production data processing workflows. The scheduling engine interfaces with the existing data processing system of semiconductor companies through a preset interface, eliminating the need for additional data acquisition modules. The specific synchronization logic is as follows: Task information synchronization: Directly obtain the metadata of offline production tasks already created within the existing data processing system, including task type (such as yield analysis, parameter cleaning), execution cycle (such as daily early morning, monthly end of month), resource requirements (such as number of CPU cores, memory size), etc. Execution result feedback: The execution status of offline tasks (success / failure, such as successful generation of yield report, failure of parameter cleaning) and resource usage (such as CPU time consumed by the task, peak memory usage) are fed back to the existing data processing system to achieve two-way synchronization of production data and scheduling information, which makes it easier for production managers to track task progress.

[0026] The system primarily uses Java as its development language, is compatible with both Windows and Linux operating systems (semiconductor companies often use Linux for their production data servers), and achieves cross-system deployment through containerization technologies (such as Docker). It employs a B / S architecture (allowing production managers to access the scheduling console via a browser). Java boasts concise and efficient language support, rich library support (such as the Apache Commons library for handling large-scale production data), and excellent scalability, enabling the rapid implementation of complex scheduling algorithms and system functions, fully meeting the development needs of semiconductor companies' offline production data processing and scheduling engines.

[0027] The software infrastructure is built on Apache Hadoop and remains compatible with the existing data processing infrastructure of semiconductor companies (such as the Hadoop 3.0 version and HDFS production data storage clusters used in the adaptation). Apache Hadoop provides distributed storage and computing capabilities, which can directly interface with the storage modules of the existing data processing system (for storing collected offline production data, such as raw wafer testing data and packaging inspection logs), without the need to build an additional storage system, significantly reducing system integration costs and avoiding the risk of migrating production data between different storage systems.

[0028] The software adopts a simplified layered architecture of "adaptive interface layer + core scheduling layer", without an independent data acquisition layer. The functions of each layer are as follows, which are tailored to the production data processing needs of semiconductor companies: Adaptive interface layer: Responsible for interface interaction with the existing data processing system of semiconductor companies, synchronizing production task information and resource status, and feeding back execution results to achieve seamless binding between the two and avoid production data connection gaps; Task scheduling layer: Based on the production task information and dependencies synchronized with the adapter interface layer (such as parameter cleaning → yield analysis), the scheduling algorithm is used to complete task scheduling and execution monitoring (such as tracking the progress of yield analysis tasks and triggering alarms when anomalies occur). Application Interface Layer: Provides semiconductor enterprise production managers with interfaces for task status query (such as whether a batch yield task has been completed) and scheduling strategy configuration (such as adjusting the priority rules of yield tasks). It supports users to interact with the scheduling engine through the production console of the existing data processing system or a separate interface, without the need to develop an additional user operation system, thus reducing the usage cost for production managers.

[0029] Adaptive interface module: It connects with the existing data processing system of semiconductor companies through a RESTful interface, and synchronizes offline production task metadata and system resource status on a regular basis without the need to collect external data; at the same time, it feeds back the task execution results and resource usage data to the existing data processing system to ensure bidirectional synchronization of production data and scheduling information, which facilitates production management traceability.

[0030] The task scheduling module obtains offline production task information (such as task name "Q2 Wafer Yield Summary" and execution cycle "end of each month") and dependencies (such as "Q2 Wafer Test Data Cleaning → Q2 Wafer Yield Summary") from the adapter connection module. It then uses a scheduling algorithm (such as a hybrid scheduling algorithm) to determine the task execution order and resource allocation scheme. During task execution, it monitors the execution progress (such as yield summary reaching 60%) and resource usage (such as whether memory limits are exceeded) in real time. Based on actual needs, it pauses (such as when data is abnormal), resumes (such as after the abnormality is resolved), or reschedules (such as when resources are insufficient). The task status (such as "paused" or "resume execution") is synchronized to the existing data processing system, allowing production managers to monitor task dynamics in real time.

[0031] Application Interface Module: Provides a unified calling interface (such as task status query interface and scheduling strategy configuration interface) for semiconductor enterprise production management personnel. It supports users to interact with the scheduling engine through the production console of the existing data processing system (such as the production data management platform) or a separate interface (such as querying the execution log of a certain batch of yield tasks). There is no need to develop an additional user operation system, which reduces the learning and usage costs of production management personnel and improves work efficiency.

[0032] The overall system architecture diagram can be referenced. Figure 2As shown in the diagram, this architecture intuitively illustrates the binding relationship between the scheduling engine and the existing data processing system of semiconductor companies: the adapter interface layer serves as the core interaction module, responsible for all necessary information such as task results and status feedback, and resource usage data feedback; the task scheduling layer completes core scheduling logic (such as allocating resources for yield analysis tasks) based on synchronized production data; and the application interface layer supports linkage with the existing system's production console (such as production managers querying task status through the console). Through this architecture, deep collaboration between the scheduling engine and the existing data processing system of semiconductor companies is achieved, avoiding independent data acquisition design, reducing system complexity, and aligning with the data processing needs of semiconductor production.

[0033] Step S20: Based on the metadata of the offline production tasks, according to task priority, expected execution time and task dependencies, a hybrid scheduling algorithm is used to sort the task queue and allocate resources to generate a task scheduling order and resource allocation scheme.

[0034] In this implementation, a hybrid scheduling algorithm is used to sort and allocate resources in the task queue, generating a task scheduling order and resource allocation scheme. The specific generation process involves obtaining the business importance and urgency indicators based on the metadata of the offline production tasks; determining the dynamic priority value of each offline production task based on these indicators; dividing the task queue into high, medium, and low priority sub-queues based on the dynamic priority values; for tasks within the same priority sub-queue, performing a secondary sorting using the shortest job first algorithm based on the expected execution time in the corresponding metadata; traversing the secondary sorted task queue and checking if each task has any prerequisite tasks; if prerequisite tasks exist, adjusting the prerequisite tasks in the queue to precede the corresponding task; after completing the prerequisite task check and adjustment, obtaining the task scheduling order; and generating a resource allocation scheme based on the task scheduling order.

[0035] It's important to note that the scheduling engine parses the business attributes of each task from the synchronized metadata, including the importance weight of the business module to which the task belongs and the urgency flag marked when the task was created. Using predefined priority calculation rules, such as priority = business weight × 0.7 + urgency × 0.3, a dynamic priority value is generated for each task. Then, based on preset priority thresholds, such as high priority > 0.8, 0.5 < medium priority ≤ 0.8, and low priority ≤ 0.5, tasks are divided into three priority sub-queues. Within the same priority queue, tasks are sorted in ascending order according to their expected execution time, ensuring that shorter tasks are executed first. After initial sorting, the entire task queue is traversed, checking the pre-dependencies of each task. When task B depends on task A, task A's position in the queue is located; if task A is after task B, it is automatically moved before task B. Finally, based on the final determined task execution order, a detailed resource allocation plan is generated according to the resource requirements of each task, including allocating specific CPU cores, memory size, and storage space for each task. For example, Task A is wafer yield analysis (business weight 0.9, urgency 1.0), Task B is historical data archiving (business weight 0.3, urgency 0.2), and Task C is parameter cleaning (business weight 0.6, urgency 0.7). The priorities of each task are calculated as follows: Task A = 0.93, Task B = 0.27, and Task C = 0.63. Based on this, Task A is placed in the high-priority queue, Task C in the medium-priority queue, and Task B in the low-priority queue. In the medium-priority queue, assuming Task C's estimated execution time is 2 hours and another medium-priority task D's estimated execution time is 1 hour, then Task D is placed before Task C. If it is detected that Task A depends on Task C, and Task C is currently placed after Task A, the system will move Task C to a position before Task A. The final execution order is: Task C → Task A → Task D → Task B.

[0036] Furthermore, generating a resource allocation scheme based on the task scheduling order specifically includes: acquiring the resource requirement metadata of each task in the sequence according to the task scheduling order, and summarizing it to form a recent total resource requirement curve; comparing the total resource requirement curve with the real-time and predicted system available resources obtained synchronously from the existing data processing system to detect whether there are any time points of resource over-allocation; if such points are detected, shifting some low-priority tasks on non-critical paths backward on the timeline to eliminate resource peaks, while satisfying task dependencies; and binding the adjusted task start time with the resource allocation quota to generate a resource allocation scheme.

[0037] It should be noted that, following the task scheduling order, the resource requirements declared in the metadata of each task are read sequentially, including the number of CPU cores, memory size, and storage space. These requirements are then accumulated according to the planned execution time period of the tasks to generate a total system resource demand curve for a future period (e.g., 24 hours). Simultaneously, current system resource status information is obtained from the existing data processing system, including the number of available CPU cores and the amount of free memory on each computing node, as well as the predicted future resource availability trend based on historical data. The resource demand curve is compared and analyzed with the available resource curve to identify time periods of resource over-allocation. When resource conflicts are detected, the critical path in the current task queue is analyzed to identify tasks that are not on the critical path and have lower priority. These non-critical tasks are shifted backward on the timeline to avoid peak resource usage periods, while ensuring that this adjustment does not violate any task dependency constraints. Finally, a specific start time and resource quota are assigned to each task to form the final resource allocation scheme. For example, in a nighttime batch processing scenario at a semiconductor factory, a scheduling sequence containing 10 tasks revealed that during the 2:00-3:00 AM period, CPU demand reached 256 cores, while the system's maximum available CPU was only 200 cores. After detecting this conflict, analysis showed that task F (historical log compression, low priority, not on the critical path) was scheduled to execute during this time period. The system postponed the execution of task F to after 4:00 AM, when system resources were more abundant. Through this adjustment, the CPU demand during the 2:00-3:00 AM period decreased to 180 cores, eliminating the resource conflict. This ultimately led to the generation of a resource allocation scheme, clearly defining the precise start time and resource quota for each task.

[0038] In one embodiment, the dependency relationship is constructed based on task dependency metadata synchronized from the existing data processing system to build a dependency graph. The construction process involves abstracting each offline production task as a node in the graph, constructing directed edges between nodes based on the task dependency metadata, forming a task dependency directed acyclic graph; performing loop detection on the task dependency directed acyclic graph, and if a loop dependency is found, reporting the anomaly information to the existing data processing system and suspending the scheduling of the relevant tasks; performing topological sorting on the detected task dependency directed acyclic graph to generate one or more potential task execution sequences that satisfy all dependencies; fusing and optimizing the potential sequences generated by the topological sorting with the sequences generated by sorting based on priority and shortest job principles, and constructing the dependency relationship using the topological sequence as a constraint.

[0039] It should be noted that each offline production task is modeled as a node in a graph structure, and directed edges between nodes are established based on the dependency metadata between tasks. For example, if task B depends on task A, a directed edge is established from A to B, ultimately forming a directed acyclic graph of task dependencies. A depth-first search algorithm is used to detect loops in the constructed graph. If a cyclic dependency is found, such as task A→B→C→A, an anomaly alarm is immediately sent to the existing data processing system, and the scheduling process of the relevant tasks is suspended. For the acyclic graph that passes the detection, a topology sorting algorithm is used to generate possible task execution sequences. The sequences generated by the topology sorting are merged with sequences generated based on priority and shortest job principles: taking the dependency constraints of the topology sequence as a premise, and while satisfying the dependency relationships, tasks are sorted as much as possible according to priority and job length. For example, in a chip test data analysis scenario, the system identifies the following dependency relationship: the yield analysis task depends on the parameter cleaning task, and the report generation task depends on the yield analysis task, forming a dependency chain of "parameter cleaning → yield analysis → report generation". These tasks are constructed as a directed acyclic graph, with parameter cleaning as the starting node and report generation as the ending node. The task sequence generated by topology sorting must ensure that parameter cleaning is executed first and report generation is executed last. Under this dependency constraint, if other independent tasks exist simultaneously, they will be sorted according to their priority and expected execution time. For example, a high-priority device status check task, even if unrelated to the above dependency chain, will be prioritized for execution.

[0040] Step S30: Start and execute tasks according to the task scheduling order and resource allocation scheme. At the same time, monitor the task execution status and resource usage in real time during the execution process, and dynamically adjust resource allocation and task execution order based on the monitoring results.

[0041] In practice, the process of dynamically adjusting resource allocation based on monitoring results involves obtaining the actual resource utilization rate of the currently executing task based on the monitoring results; if the actual resource utilization rate is detected to be lower than the preset allocation threshold, the amount of resources allocated to the corresponding task is automatically reduced, and the idle resources are released to the public resource pool; if the actual resource utilization rate is detected to reach the upper limit of the preset allocation threshold, it is determined whether there are remaining resources in the public resource pool, and if there are remaining resources, the amount of resources allocated to the corresponding task is automatically increased.

[0042] It's important to note that the monitoring agent collects real-time resource usage metrics for running tasks, including CPU utilization, memory usage, disk I / O, and network bandwidth. These metrics are collected at a rate of one second and analyzed by the scheduling engine. A dynamic adjustment threshold for resource usage is set for each task: a resource reclamation mechanism is triggered when the actual resource usage of a task is below 60% of its allocated quota for, for example, for 5 consecutive minutes; a resource expansion mechanism is triggered when the resource usage consistently exceeds 85% of the allocated quota. During resource reclamation, the task's resource quota is gradually reduced (e.g., 10% reduction in memory allocation each time), and the reclaimed resources are released to the shared resource pool. During resource expansion, the availability of resources in the shared resource pool is checked; if resources are sufficient, the resource quota is increased for the task as needed. For example, in a wafer fabrication parameter analysis task, 32GB of memory was initially allocated. Monitoring revealed that the task's actual memory usage remained stable at 18-20GB during operation, consistently below the preset threshold. The task's memory quota was automatically adjusted from 32GB to 24GB, and the freed-up 8GB of memory was returned to the shared resource pool. Simultaneously, when another yield analysis task showed memory usage reaching 90%, an additional 4GB of memory was immediately allocated from the shared resource pool to that task, preventing execution delays due to insufficient memory.

[0043] Furthermore, the process of dynamically adjusting the task execution order based on monitoring results involves identifying tasks whose actual execution time exceeds the expected execution time and reaches the timeout threshold based on monitoring results; pausing the task queue of the task and triggering a rescheduling, adding the new task or timed-out task and all remaining pending tasks to the new task queue, and generating a new task scheduling order and resource allocation scheme based on the new task queue; and prioritizing the startup of new high-priority tasks according to the new task scheduling order and resource allocation scheme.

[0044] It's important to note that the execution progress of each task is monitored in real time, comparing the actual execution time with the estimated execution time. When the actual execution time of a task exceeds 150% of the estimated time, the task is marked as a timeout task, triggering a rescheduling process. During rescheduling, all currently pending tasks are first paused, and timed-out tasks, newly arrived tasks, and existing pending tasks are merged into a new task queue. The priorities of each task are recalculated, taking into account factors such as waiting time and the severity of timeouts. Based on the updated task information and resource status, the hybrid scheduling algorithm is rerun to generate a new task scheduling order. In the new scheduling scheme, newly arrived high-priority tasks and severely timed-out tasks are prioritized to ensure that critical tasks are processed in a timely manner. For example, in a semiconductor packaging test data processing scenario, a chip test analysis task expected to take 2 hours to complete is still not finished after 3 hours. Upon detecting this timeout, rescheduling is immediately triggered. At this time, an urgent wafer yield analysis task (high priority) is synchronized. The current queue is paused, and the timed-out test analysis task, the new yield analysis task, and other pending tasks are reordered. The new scheduling scheme prioritizes yield analysis tasks, downgrades timed-out test analysis tasks to low priority, and assigns them dedicated computing nodes to continue execution, thus avoiding impacting the normal operation of other tasks. Simultaneously, the system sends task timeout alerts to the management platform, prompting administrators to pay attention to this anomaly.

[0045] It should be understood that, for ease of understanding, the above is merely an example, and the listed parameters and threshold settings are not intended to limit this embodiment.

[0046] Step S40: Feed back the task execution results and resource usage data obtained from monitoring to the existing data processing system to achieve two-way synchronization of production data and scheduling information.

[0047] It should be noted that the task execution status includes successful execution, failed execution, and suspended execution. If the task execution status is successful, the task execution result and resource usage data will be fed back to the existing data processing system. If the task execution status is reported during execution, the task execution result will be directly fed back to the existing data processing system. If the task execution status is suspended, the task execution status and corresponding log will be recorded, and the task will be retried in the next execution.

[0048] In this embodiment, based on synchronizing the metadata of offline production tasks from the existing data processing system through a preset interface, and according to task priority, expected execution time, and task dependencies, a hybrid scheduling algorithm is used to sort the task queue and allocate resources, generating a task scheduling order and resource allocation scheme. Then, the task is started and executed. During the execution process, the task execution status and resource usage are monitored in real time, and the task execution order is dynamically adjusted to achieve bidirectional synchronization of production data and scheduling information. No additional external data acquisition components are required. It relies entirely on the offline production data already collected by the semiconductor company's existing data processing system to achieve efficient and reasonable allocation of resources, significantly improving scheduling efficiency and overall system performance.

[0049] Furthermore, this embodiment of the invention also proposes a storage medium storing a resource-optimized big data scheduler, which, when executed by a processor, implements the steps of the resource-optimized big data scheduling method described above.

[0050] Reference Figure 3 , Figure 3 This is a structural block diagram of the first embodiment of the resource-optimized big data scheduling device of the present invention.

[0051] like Figure 3 As shown, the resource-optimized big data scheduling device proposed in this embodiment of the invention includes: The acquisition module 10 is used to synchronize the metadata of offline production tasks from the existing data processing system through a preset interface. The metadata includes task type, execution cycle, resource requirements and task dependencies. The generation module 20 is used to sort and allocate resources in the task queue based on the metadata of the offline production tasks according to task priority, expected execution time and task dependency, and generate task scheduling order and resource allocation scheme. The execution module 30 is used to start and execute tasks according to the task scheduling order and resource allocation scheme, and at the same time monitor the task execution status and resource usage in real time during the execution process, and dynamically adjust resource allocation and task execution order based on the monitoring results. Feedback module 40 is used to feed back the task execution results and resource usage data obtained from monitoring to the existing data processing system to achieve two-way synchronization of production data and scheduling information.

[0052] In this embodiment, based on synchronizing the metadata of offline production tasks from the existing data processing system through a preset interface, and according to task priority, expected execution time, and task dependencies, a hybrid scheduling algorithm is used to sort the task queue and allocate resources, generating a task scheduling order and resource allocation scheme. Then, the task is started and executed. During the execution process, the task execution status and resource usage are monitored in real time, and the task execution order is dynamically adjusted to achieve bidirectional synchronization of production data and scheduling information. No additional external data acquisition components are required. It relies entirely on the offline production data already collected by the semiconductor company's existing data processing system to achieve efficient and reasonable allocation of resources, significantly improving scheduling efficiency and overall system performance.

[0053] In one embodiment, the generation module 20 is used to obtain a business importance identifier and an urgency identifier based on the metadata of the offline production task, and to determine a dynamic priority value for each offline production task based on the business importance identifier and the urgency identifier. Based on the dynamic priority value, the task queue is divided into three sub-queues with high, medium, and low priorities. For tasks within subqueues of the same priority, a second sorting is performed using the shortest job first algorithm based on the expected execution time in the corresponding metadata. Iterate through the twice-sorted task queue and check if each task has any preceding dependent tasks. If there are prerequisite tasks, then move the prerequisite tasks in the queue to before the corresponding tasks; After completing the checks and adjustments of the prerequisite tasks, the task scheduling order is obtained. A resource allocation scheme is generated based on the task scheduling order.

[0054] In one embodiment, the generation module 20 is used to sequentially obtain the resource requirement metadata of each task in the sequence according to the task scheduling order, and summarize them to form a recent total resource requirement curve. The total resource demand curve is compared with the real-time and predicted system available resources obtained synchronously from the existing data processing system to detect whether there are any points in time when resources are over-allocated. If such occurrences are detected, then, while ensuring that task dependencies are met, some low-priority tasks on non-critical paths will be shifted backward on the timeline to eliminate resource spikes. The adjusted task start time is bound to the resource allocation quota to generate a resource allocation plan.

[0055] In one embodiment, the dependency relationship is obtained by constructing a dependency relationship graph based on task dependency relationship metadata synchronized from the existing data processing system. The generation module 20 is used to abstract each offline production task as a node in the graph, construct directed edges between nodes according to the dependency relationship metadata between tasks, and form a task dependency directed acyclic graph. The directed acyclic graph that the task depends on is subjected to loop detection. If a loop dependency is found, the abnormal information is reported to the existing data processing system and the scheduling of the relevant tasks is suspended. Perform topological sorting on the detected task dependency directed acyclic graph to generate one or more potential task execution sequences that satisfy all dependencies; The potential sequences generated by topological sorting are fused and optimized with the sequences generated by sorting based on priority and shortest job principles, and the dependencies are constructed using the topological sequences as constraints.

[0056] In some embodiments, the execution module 30 is configured to obtain the actual resource utilization rate of the task being executed based on monitoring results; If the actual resource utilization rate is detected to be lower than the preset allocation threshold, the amount of resources allocated to the corresponding task will be automatically reduced, and the idle resources will be released to the public resource pool. If the actual resource utilization rate is detected to have reached the upper limit of the preset allocation limit, it is determined whether there are remaining resources in the public resource pool, and if there are remaining resources, the amount of resources allocated to the corresponding task is automatically increased.

[0057] In some embodiments, the execution module 30 is used to determine, based on monitoring results, tasks whose actual execution time exceeds the expected execution time and reaches a timeout threshold; The task queue containing the task is paused, and a rescheduling is triggered. The new task or timed-out task and all remaining pending tasks are added to the new task queue, and a new task scheduling order and resource allocation scheme are generated based on the new task queue. Based on the new task scheduling order and resource allocation scheme, new high-priority tasks will be launched first.

[0058] In some embodiments, the task execution status includes execution success, execution failure, and execution pause. The feedback module 40 is used to feed back the task execution result and resource usage data to the existing data processing system if the task execution status is execution success, to feed back the task execution result directly to the existing data processing system if the task execution status is execution failure, and to record the task execution status and corresponding logs if the task execution status is execution pause, and to wait for the next task execution to retry.

[0059] This application embodiment also provides a resource-optimized big data scheduling device, including a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other through the communication bus. The memory is used to store the resource-optimized big data scheduling program. When the processor executes the program stored in the memory, it implements the above-mentioned resource-optimized big data scheduling method.

[0060] The communication bus mentioned in the above-mentioned big data scheduling equipment for resource optimization can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0061] The communication interface is used for communication between the aforementioned resource-optimized big data scheduling equipment and other devices.

[0062] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0063] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0064] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0068] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0069] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0070] In addition, for technical details not described in detail in this embodiment, please refer to the resource-optimized big data scheduling method provided in any embodiment of the present invention, which will not be repeated here.

[0071] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0072] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0074] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

[0075] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

Claims

1. A big data scheduling method for resource optimization, characterized in that, The resource-optimized big data scheduling method includes: The metadata of offline production tasks is synchronized from the existing data processing system through a preset interface. The metadata includes task type, execution cycle, resource requirements and task dependencies. Based on the metadata of the offline production tasks, a hybrid scheduling algorithm is used to sort and allocate resources in the task queue according to task priority, expected execution time and task dependencies, generating a task scheduling order and resource allocation scheme. Tasks are initiated and executed according to the task scheduling order and resource allocation scheme. During the execution process, the task execution status and resource usage are monitored in real time, and resource allocation and task execution order are dynamically adjusted based on the monitoring results. The task execution results and resource usage data obtained from monitoring are fed back to the existing data processing system to achieve two-way synchronization of production data and scheduling information.

2. The big data scheduling method for resource optimization as described in claim 1, characterized in that, The metadata based on the offline production tasks, according to task priority, expected execution time, and task dependencies, employs a hybrid scheduling algorithm to sort the task queue and allocate resources, generating a task scheduling order and resource allocation scheme, including: Based on the metadata of the offline production tasks, obtain the business importance identifier and urgency identifier, and determine the dynamic priority value of each offline production task according to the business importance identifier and urgency identifier. Based on the dynamic priority value, the task queue is divided into three sub-queues with high, medium, and low priorities. For tasks within subqueues of the same priority, a second sorting is performed using the shortest job first algorithm based on the expected execution time in the corresponding metadata. Iterate through the twice-sorted task queue and check if each task has any preceding dependent tasks. If there are prerequisite tasks, then move the prerequisite tasks in the queue to before the corresponding tasks; After completing the checks and adjustments of the prerequisite tasks, the task scheduling order is obtained. A resource allocation scheme is generated based on the task scheduling order.

3. The big data scheduling method for resource optimization as described in claim 2, characterized in that, The process of generating a resource allocation scheme based on the task scheduling order includes: Based on the task scheduling order, the resource requirement metadata of each task in the sequence is obtained in turn, and the total resource requirement curve for the recent period is formed by summarizing them. The total resource demand curve is compared with the real-time and predicted system available resources obtained synchronously from the existing data processing system to detect whether there are any points in time when resources are over-allocated. If such occurrences are detected, then, while ensuring that task dependencies are met, some low-priority tasks on non-critical paths will be shifted backward on the timeline to eliminate resource spikes. The adjusted task start time is bound to the resource allocation quota to generate a resource allocation plan.

4. The big data scheduling method for resource optimization as described in claim 3, characterized in that, The dependency relationship is obtained by constructing a dependency graph based on task dependency metadata synchronized from the existing data processing system. The method further includes: Each offline production task is abstracted as a node in a graph. Directed edges between nodes are constructed based on the dependency metadata between tasks, forming a directed acyclic graph of task dependencies. The directed acyclic graph that the task depends on is subjected to loop detection. If a loop dependency is found, the abnormal information is reported to the existing data processing system and the scheduling of the relevant tasks is suspended. Perform topological sorting on the detected task dependency directed acyclic graph to generate one or more potential task execution sequences that satisfy all dependencies; The potential sequences generated by topological sorting are fused and optimized with the sequences generated by sorting based on priority and shortest job principles, and the dependencies are constructed using the topological sequences as constraints.

5. The big data scheduling method for resource optimization as described in claim 1, characterized in that, The dynamic adjustment of resource allocation based on monitoring results includes: Obtain the actual resource utilization of tasks currently being executed based on monitoring results; If the actual resource utilization rate is detected to be lower than the preset allocation threshold, the amount of resources allocated to the corresponding task will be automatically reduced, and the idle resources will be released to the public resource pool. If the actual resource utilization rate is detected to have reached the upper limit of the preset allocation limit, it is determined whether there are remaining resources in the public resource pool, and if there are remaining resources, the amount of resources allocated to the corresponding task is automatically increased.

6. The big data scheduling method for resource optimization as described in claim 1, characterized in that, The method of dynamically adjusting the task execution order based on monitoring results includes: Based on monitoring results, tasks whose actual execution time exceeds the expected execution time and reaches the timeout threshold are identified. The task queue containing the task is paused, and a rescheduling is triggered. The new task or timed-out task and all remaining pending tasks are added to the new task queue, and a new task scheduling order and resource allocation scheme are generated based on the new task queue. Based on the new task scheduling order and resource allocation scheme, new high-priority tasks will be launched first.

7. The resource-optimized big data scheduling method as described in any one of claims 1 to 6, characterized in that, The task execution status includes successful execution, failed execution, and paused execution. If the task execution status is successful, the task execution result and resource usage data are fed back to the existing data processing system. If the task execution status is failed, the task execution result is directly fed back to the existing data processing system. If the task execution status is paused, the task execution status and corresponding log are recorded, and the system waits for the next task execution to retry.

8. A big data scheduling device for resource optimization, characterized in that, The resource-optimized big data scheduling device includes: The acquisition module is used to synchronize the metadata of offline production tasks from the existing data processing system through a preset interface. The metadata includes task type, execution cycle, resource requirements and task dependencies. The generation module is used to sort and allocate resources in the task queue based on the metadata of the offline production tasks according to task priority, expected execution time and task dependencies, and generate task scheduling order and resource allocation scheme. The execution module is used to start and execute tasks according to the task scheduling order and resource allocation scheme. At the same time, it monitors the task execution status and resource usage in real time during the execution process, and dynamically adjusts resource allocation and task execution order based on the monitoring results. The feedback module is used to feed back the task execution results and resource usage data obtained from monitoring to the existing data processing system, so as to achieve two-way synchronization of production data and scheduling information.

9. A big data scheduling device for resource optimization, characterized in that, The resource-optimized big data scheduling device includes: a memory, a processor, and a resource-optimized big data scheduling program stored in the memory and executable on the processor, wherein the resource-optimized big data scheduling program is configured to implement the steps of the resource-optimized big data scheduling method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a resource-optimized big data scheduler, which, when executed by a processor, implements the steps of the resource-optimized big data scheduling method as described in any one of claims 1 to 7.

Images