DESIGN PROCESS FOR A MULTI-THREAD AUTOMATED DATA PROCESSING SYSTEM (DATAOPS) ON THE DATA LAKEHOUSE PLATFORM TO OPTIMIZE COMPUTING COSTS

Abstract

This useful solution, in the field of large-scale data management and processing, proposes an automated multi-threaded data processing workflow on the Data Lakehouse unified data warehouse platform to optimize computing costs. In traditional ETL or ELT systems that do not streamline data by characteristics, the entire payload must be scanned sequentially at the reception stage; real-time and batch data share the same resource allocation mechanism; a single error can halt the entire stream; and the computing server remains idle even when there is no further processing required, forcing the infrastructure to constantly allocate resources at maximum capacity, which does not accurately reflect actual needs. To overcome this, the process is organized into five interconnected, closed-loop steps. The first step extracts metadata from the file footer with constant algorithmic complexity, regardless of file size.The second step calculates a score from three variables—schema matching level, data freshness, and payload size—to route data to a high-priority or low-priority channel without loading the payload into memory. The third step uses a branching algorithm to isolate and self-correct each data row without interrupting the main stream. The fourth step transforms the data in place using a non-cyclic directed dependency graph, only recalculating the newly generated data. The fifth step monitors queue latency, transformation completion status, and isolation zone status to reclaim resources when there is no longer any work to be done. As a result, the process reduces memory usage, processing cycles, energy consumption, disk traffic, and network bandwidth, while converting fixed infrastructure costs into operating costs proportional to the actual workload.