A system for a scalable distribution service for parallel training of machine learning models and data transformation.

Abstract

A system (100) for a scalable dispatch service for simultaneous training of machine learning models and data transformation, comprising: a task capture module configured to receive a variety of job requests, including data transformation tasks and machine learning model training tasks, from one or more sources; a task classification module that is functionally coupled with the task capture module and configured to categorize the multitude of job requests based on task type, priority, and dependency relationships; a planning and distribution engine that is functionally coupled with the task classification module and configured to determine an execution order, allocate computing resources, and distribute the multitude of job requests for simultaneous execution across distributed computing nodes; a dependency management module that is functionally coupled with the scheduling and distribution engine and configured to identify and enforce dependencies between multiple job requests, so that required data transformation tasks are completed before the corresponding model training tasks; a resource management module that is functionally coupled with the planning and distribution engine and configured to dynamically allocate and scale computing resources based on workload requirements and system states (100); a monitoring and fault management module configured to track the execution of numerous job requests, detect task failures, and initiate retry or recovery operations; and an optimization and feedback module that is functionally coupled with the planning and distribution engine as well as the monitoring and error management module and is configured to refine planning guidelines and resource allocation strategies based on historical execution data and performance indicators, wherein the system (100) is configured to enable scalable, parallel execution of tasks for training machine learning models and data transformation within cloud-native infrastructures, thereby improving resource utilization, reducing execution latency and increasing the reliability of the system (100).

Description

[0001] The present invention relates generally to the field of distributed data processing and machine learning infrastructure. In particular, the invention relates to a system for a scalable dispatch service for simultaneously performing training for machine learning models and data transformations in cloud-native and distributed environments.

[0002] In recent years, the rapid growth of data-driven applications has led to an increasing reliance on machine learning systems for tasks such as prediction, classification, and data transformation. These systems typically involve complex workflows encompassing data preprocessing, feature engineering, model training, validation, and deployment. With the advent of cloud-native architectures and distributed computing platforms like Kubernetes, organizations are increasingly running these workflows across multiple compute nodes to achieve scalability and performance. However, traditional approaches to managing machine learning workloads often lack efficient mechanisms for coordinating the concurrent execution of training jobs and data transformation tasks.Existing systems typically rely on static scheduling or manual orchestration, which can lead to inefficient resource utilization, increased execution latency, and difficulties handling large-scale workloads. Furthermore, traditional job schedulers are not specifically optimized for the unique requirements of machine learning pipelines, such as dependency management between data transformation stages and model training processes. Additionally, machine learning workflows often involve interdependent tasks that must be executed in a specific sequence, while simultaneously supporting parallel execution where possible. Current systems struggle to reconcile these requirements, resulting in bottlenecks, underutilized resources, or execution failures.The lack of intelligent distribution mechanisms further limits the ability to dynamically allocate resources based on workload characteristics and system conditions. Furthermore, existing solutions offer limited support for fault tolerance, retry mechanisms, and real-time monitoring of distributed tasks. In large-scale environments, the failure of a single task can disrupt entire pipelines, leading to delays and increased operational overhead. The absence of centralized control to manage concurrent workloads also complicates task execution tracking, policy enforcement, and ensuring consistent performance. Therefore, there is a need for an improved system that provides a scalable dispatch service capable of efficiently managing concurrent tasks such as machine learning model training and data transformation.Such a system should enable dynamic planning, efficient resource utilization, robust fault tolerance, and seamless orchestration of distributed workloads in cloud-native environments.

[0003] To solve this problem, the present invention provides a system for a scalable dispatch service for the simultaneous training of machine learning models and data transformation.

[0004] The system enables a scalable and efficient distribution of concurrent tasks for training machine learning models and transforming data across distributed environments.

[0005] The system improves resource utilization by dynamically allocating computing resources based on workload requirements and system conditions.

[0006] The system reduces execution latency by enabling the parallel processing of independent tasks while adhering to dependency constraints.

[0007] The system increases throughput by supporting the simultaneous execution of multiple machine learning pipelines.

[0008] The system offers seamless integration with cloud-native platforms such as Kubernetes for managing containerized workloads.

[0009] The system improves reliability through integrated fault tolerance, retry mechanisms, and error correction strategies.

[0010] The system enables intelligent task planning and prioritization based on workload characteristics and performance metrics.

[0011] The system supports efficient dependency management between data transformation and model training tasks.

[0012] The system enables real-time monitoring and tracking of task execution across distributed nodes.

[0013] The system improves scalability to handle large-scale data processing and model training workloads.

[0014] The system reduces operational complexity by automating the orchestration and distribution of machine learning tasks.

[0015] The system enables adaptive optimization of planning guidelines based on historical execution data and system feedback.

[0016] The present invention relates to a system for a scalable distribution service for the simultaneous training of machine learning models and data transformation in distributed and cloud-native environments. The system is configured to coordinate, schedule, and distribute multiple interdependent and independent tasks across distributed computing resources, thereby enabling the efficient execution of large-scale machine learning workflows within infrastructures. In one aspect, the system comprises a task acquisition module configured to receive job requests, including data transformation and model training tasks, from one or more sources; and a task classification module configured to categorize the received tasks based on type, priority, and dependency relationships.A scheduling and distribution engine is functionally coupled with the task classification module and configured to allocate resources, determine the execution order, and distribute tasks across nodes for concurrent execution. The system also includes a dependency management module configured to identify and manage task dependencies, ensuring that necessary data transformation operations are completed before the corresponding model training tasks. A resource management module dynamically allocates compute resources based on workload requirements and system conditions. Finally, the system includes a monitoring and fault management module configured to track task execution, detect errors, and initiate retry or recovery mechanisms.An optimization and feedback module is also included to continuously refine planning guidelines and resource allocation strategies based on historical execution data and system performance metrics. Accordingly, the exposed system offers a scalable, efficient, and resilient distribution framework that enables the concurrent execution of tasks for training machine learning models and data transformation, while ensuring optimal resource utilization, reduced latency, and reliable operation in cloud-native environments.

[0017] Fig. illustrates a scalable parallel task execution system designed to handle data transformation and machine learning workloads in distributed environments.

[0018] Fig.This document presents a scalable parallel task execution system designed for processing data transformation and machine learning workloads in distributed environments. The system (100) comprises a task ingestion module configured to receive and aggregate data transformation tasks and machine learning model training tasks from one or more sources; a task classification module ( ) functionally coupled to the task ingestion module and configured to categorize received tasks based on predefined criteria, including task type, priority, and resource requirements; and a scheduling and dispatching engine functionally coupled to the task classification module and configured to determine execution order and assign tasks for processing.a dependency management module that is functionally coupled with the scheduling and distribution engine and configured to manage task dependencies and execution order constraints; a resource management module that is functionally coupled with the scheduling and distribution engine and configured to allocate computing resources, including compute units, memory, and storage; a variety of distributed compute nodes that are functionally coupled with the scheduling and distribution engine and configured to execute assigned tasks in parallel; a monitoring and fault management module that is functionally coupled with the distributed compute nodes and configured to monitor the execution status, detect faults, and ensure the reliability of the system (100);and an optimization and feedback module that is functionally coupled with the monitoring and fault management module and the planning and distribution engine, and configured to dynamically refine planning, resource allocation, and execution strategies based on performance metrics, thereby enabling scalable and parallel execution of tasks in a distributed computing environment.

[0019] The present invention relates to a scalable parallel task execution system (100) configured to process heterogeneous workloads, including data transformation tasks and machine learning model training tasks, in a distributed computing environment. In one embodiment, the system (100) comprises a task acquisition module configured to receive tasks from one or more sources, including enterprise data pipelines, application programming interfaces, and batch processing systems. The task acquisition module standardizes the task inputs and forwards them to a task classification module configured to categorize tasks based on parameters such as task type, execution priority, computational complexity, and resource requirements.A planning and distribution engine is functionally coupled with the task classification module and configured to determine an optimal execution sequence and assign tasks to the appropriate execution units based on planning guidelines, priority levels and system utilization conditions (100).

[0020] In another embodiment, the scheduling and distribution engine is functionally coupled with a dependency management module and a resource management module. The dependency management module ensures that interdependent tasks are executed according to predefined execution constraints, while the resource management module dynamically allocates the system's (100) computing resources, including processing power, memory, and storage space, to ensure efficient utilization. The system (100) further comprises a multitude of distributed compute nodes configured to execute tasks in parallel, thus enabling high scalability and high throughput. A monitoring and fault management module continuously tracks the execution status, detects errors, and initiates corrective actions such as task retry or reassignment.Additionally, an optimization and feedback module analyzes execution metrics and performance data of the system (100) to refine planning algorithms, resource allocation strategies, and task prioritization in real time. In operation, the system (100) enables efficient, reliable, and scalable parallel execution of diverse workloads by integrating intelligent scheduling, dependency handling, dynamic resource management, and adaptive optimization within a unified distributed architecture.

Claims

[1] A system (100) for a scalable dispatch service for simultaneous training of machine learning models and data transformation, comprising: a task capture module configured to receive a variety of job requests, including data transformation tasks and machine learning model training tasks, from one or more sources; a task classification module that is functionally coupled with the task capture module and configured to categorize the multitude of job requests based on task type, priority, and dependency relationships; a planning and distribution engine that is functionally coupled with the task classification module and configured to determine an execution order, allocate computing resources, and distribute the multitude of job requests for simultaneous execution across distributed computing nodes; a dependency management module that is functionally coupled with the scheduling and distribution engine and configured to identify and enforce dependencies between multiple job requests, so that required data transformation tasks are completed before the corresponding model training tasks; a resource management module that is functionally coupled with the planning and distribution engine and configured to dynamically allocate and scale computing resources based on workload requirements and system states (100); a monitoring and fault management module configured to track the execution of numerous job requests, detect task failures, and initiate retry or recovery operations; and an optimization and feedback module that is functionally coupled with the planning and distribution engine as well as the monitoring and error management module and is configured to refine planning guidelines and resource allocation strategies based on historical execution data and performance indicators, wherein the system (100) is configured to enable scalable, parallel execution of tasks for training machine learning models and data transformation within cloud-native infrastructures, thereby improving resource utilization, reducing execution latency and increasing the reliability of the system (100). [2] System (100) according to claim 1, wherein the task intake module is configured to receive job requests from multiple sources, including application interfaces, data pipelines and external systems (100). [3] System (100) according to claim 1, wherein the task classification module is further configured to assign priority levels and categorize tasks into independent and dependent task groups. [4] System (100) according to claim 1, wherein the planning and distribution engine is configured to support the parallel execution of independent tasks and the sequential execution of dependent tasks. [5] System (100) according to claim 1, wherein the planning and distribution engine is further configured to dynamically adjust the task scheduling based on real-time system utilization and resource availability. [6] System (100) according to claim 1, wherein the dependency management module is configured to manage a task dependency graph for managing the execution order. [7] System (100) according to claim 1, wherein the resource management module is configured to scale computing resources horizontally and vertically based on the requirements of the workload. [8] System (100) according to claim 1, wherein the monitoring and error management module is configured to generate warning messages and logs when task errors or performance degradation are detected. [9] System (100) according to claim 1, wherein the monitoring and fault management module is further configured to implement automatic retry policies and checkpoint-based recovery mechanisms.

Images