System for the automated processing and analysis of large data sets in scalable network systems

Abstract

Data processing system for the automated processing and analysis of large data sets in scalable network systems, comprehensive a plurality of processing units distributed from one another, a data acquisition unit for the continuous recording and pre-structuring of incoming data, a control unit for assigning data processing tasks to the processing units, as well as a storage unit for storing raw data and processing results, characterized by the fact that A hardware-implemented preprocessing unit is provided which segments and classifies incoming data structures based on defined structure and integrity parameters and only releases data for analysis that meets predefined processing conditions, whereby the processing units execute parallel analysis processes and the results are merged in a consistent state space.

Description

Technical field

[0001] The present invention relates to the field of data processing systems, in particular systems for the automated processing and analysis of large datasets in scalable network systems. The invention lies in the technical area of ​​distributed information processing, data stream processing, and system-level control and optimization of data analysis systems. Specifically, the invention relates to a device for acquiring, preprocessing, and analyzing large datasets using multiple distributed processing units, wherein structured segmentation, classification, and priority-controlled processing of the data is performed. Furthermore, the invention includes hardware-based control and preprocessing units for coordinating parallel analysis processes and ensuring consistent and efficient processing.The technical field also extends to systems for load balancing, data classification, segmentation of data structures, and deterministic validation and merging of processing results in scalable and distributed network environments. State of the art

[0002] In the field of modern data processing, large datasets are increasingly processed in distributed network systems to enable scalable and high-performance analysis. Such systems are used particularly in data-intensive applications such as enterprise analytics, telecommunications, industrial monitoring systems, financial services, and digital platforms. Large volumes of data from diverse sources are collected, stored, and analyzed by distributed processing units.

[0003] Data processing systems based on distributed architectures and implementing automated analysis methods are known from the state of the art. These systems often use software-based frameworks for data processing, in which incoming data streams are directly forwarded to analysis processes. Processing typically occurs in parallel across multiple nodes to achieve high processing speeds.

[0004] One disadvantage of existing systems is that data preprocessing and structuring are often limited. Data is frequently processed without a comprehensive check of its structure or integrity, which can lead to inefficient analysis processes and erroneous results. In particular, there is a lack of technical mechanisms that enable targeted classification and prioritization of data before the actual analysis.

[0005] Furthermore, load balancing systems are known that distribute the processing load across multiple nodes. These systems are predominantly based on software-based algorithms that consider parameters such as system utilization or network state. However, adjustments are often delayed and lack direct hardware-based control, which can limit the efficiency of the load balancing.

[0006] Furthermore, methods for analyzing large datasets are known that divide data into sub-segments and process them in parallel. However, the merging of the results is often based on heuristic or time-based mechanisms, which does not always guarantee the consistency and order of the results. This can lead to inconsistencies in the overall result, especially with dynamic and heterogeneous data structures.

[0007] Furthermore, systems are known to implement validation mechanisms for analysis results. However, this validation usually takes place after the actual processing and is often not an integral part of the data processing system. As a result, erroneous or inconsistent results can be transferred to downstream systems before a correction can be made.

[0008] Overall, the known solutions show approaches to the automated processing and analysis of large datasets, but a technically integrated system is lacking that combines structured preprocessing, priority-controlled data classification, hardware-supported load balancing, and deterministic and consistent validation and merging of analysis results in a scalable network system. Object of the invention

[0009] The present invention is based on the objective of providing a data processing system that enables efficient, reliable and scalable automated processing and analysis of large data sets in network systems, thereby overcoming the disadvantages of known systems.

[0010] In particular, the task is to create a technical solution in which incoming data is structured, segmented and checked for integrity before the actual analysis, so that only data that meets defined quality and processing conditions is processed.

[0011] Another object of the invention is to provide priority-controlled data processing in which data is classified according to its relevance and time requirements and processed in a defined sequence.

[0012] Furthermore, the invention aims to provide a hardware-based control system for dynamic load distribution, which optimizes the allocation of processing tasks depending on current system states and network parameters.

[0013] Furthermore, a system should be created that enables deterministic validation and consistent merging of analysis results, so that a reproducible and reliable overall result is provided.

[0014] Ultimately, the object of the invention is to provide a modularly designed and flexibly expandable data processing system that can adapt to increasing data volumes and ensures continuous real-time processing with low latency. Summary of the invention

[0015] The present invention relates to a data processing system for the automated processing and analysis of large datasets in scalable network systems. The system comprises a data acquisition unit for receiving incoming data, a plurality of distributed processing units for performing analysis processes, and a control unit for assigning processing tasks.

[0016] A hardware-implemented preprocessing unit is provided, which segments, classifies, and prioritizes incoming data based on defined structural and integrity parameters, ensuring that only suitable data is released for further processing. The processing units execute parallel analysis processes, the results of which are checked and combined by a result validation and merging unit using deterministic rules.

[0017] Adaptive control logic enables dynamic load balancing, taking into account current system states and network parameters. The invention ensures consistent, efficient, and scalable processing of large data volumes with improved data quality and reproducible analysis results. Detailed description of the invention

[0018] The present invention relates to a data processing system for the automated processing and analysis of large data sets in scalable network systems, in which efficient, consistent and low-latency processing of large amounts of data is enabled.

[0019] The data processing system comprises a data acquisition unit designed to continuously collect incoming data from various sources and provide it in a structured format. The collected data is fed to a hardware-implemented preprocessing unit, which serves as the first processing level. This preprocessing unit is configured to analyze, segment, and classify the data based on defined structural and integrity parameters. In particular, data formats, completeness, temporal assignment, and structural properties are taken into account.

[0020] Data that does not meet the specified conditions will be excluded from further processing or handled separately.

[0021] The preprocessing unit is also configured to prioritize data. The data is divided into different classes, each assigned a priority. Prioritization is based on defined criteria such as temporal relevance, data source, or application requirements. This ensures that particularly relevant data is processed preferentially.

[0022] After preprocessing, the data is forwarded to a multiple distributed processing units. These processing units are designed to perform parallel analysis processes. Each processing unit comprises a segmented analysis pipeline in which the data is divided into several sequential processing sections. Within these sections, specific analysis operations are performed, tailored to the respective data class.

[0023] A control unit is provided to coordinate processing and assign data processing tasks. This control unit includes hardware-based load balancing logic that continuously monitors the status of the processing units. Parameters such as current utilization, processing time, and network latency are recorded and evaluated. Based on these parameters, data is dynamically assigned to the processing units, ensuring even load distribution and efficient use of system resources.

[0024] The partial results generated by the processing units are passed to a result validation and merging unit. This unit is configured to check the partial results against predefined consistency and integrity rules. This ensures that only those results that meet the defined quality requirements are included in further processing.

[0025] The merging of partial results is carried out according to deterministic ordering rules, which ensure a consistent and reproducible combination of the results. Delayed, incomplete, or inconsistent partial results are identified and excluded from the merging or corrected accordingly.

[0026] The system is further configured to manage a global state representation in which the validated results are consolidated. This global state representation serves as the basis for downstream applications and enables a unified and consistent data foundation.

[0027] In a preferred embodiment, the data processing system has a modular design, allowing additional processing units to be added flexibly. This enables adaptation to increasing data volumes and expansion of processing capacity without disrupting ongoing operations.

[0028] In another embodiment, the preprocessing unit and the control unit are designed to operate adaptively and dynamically adjust their parameters. Historical data and current system states are taken into account to continuously optimize data processing efficiency.

[0029] The present invention thus enables a technical solution in which data is structured, classified, and prioritized prior to the actual analysis, thereby achieving improved data quality and more efficient processing. The combination of hardware-supported preprocessing, parallel data analysis, adaptive load balancing, and deterministic result validation ensures reliable and scalable processing of large datasets.

[0030] The system according to the invention is particularly suitable for applications where large amounts of data need to be processed in real time, such as in industrial systems, financial analyses, telecommunications networks, or data-intensive enterprise applications. The system can be used in both centralized and distributed network architectures.

[0031] It is understood that the described embodiments are merely exemplary and that changes and adaptations are possible without leaving the scope of protection of the invention.

Claims

[1] Data processing system for the automated processing and analysis of large data sets in scalable network systems, comprising a plurality of processing units distributed from one another, a data acquisition unit for the continuous recording and pre-structuring of incoming data, a control unit for assigning data processing tasks to the processing units, as well as a storage unit for storing raw data and processing results, characterized by , that A hardware-implemented preprocessing unit is provided which segments and classifies incoming data structures based on defined structure and integrity parameters and only releases data for analysis that meets predefined processing conditions, whereby the processing units execute parallel analysis processes and the results are merged in a consistent state space. [2] Data processing system according to claim 1, characterized by that the control unit includes hardware-based adaptive load balancing logic, which continuously adjusts the allocation of data processing tasks depending on current processing times, memory states and network delays. [3] Data processing system according to claim 1, characterized by, that the preprocessing unit is set up to divide data into multiple classes with different priorities and to control the processing of the data according to a defined priority order. [4] Data processing system according to claim 1, characterized by , that each processing unit has a segmented analysis pipeline in which data is broken down into independent processing units, the partial results of which are then reconstructed into an overall result taking structural dependencies into account. [5] Data processing system according to claim 1, characterized by , that a result validation unit is provided which checks the results generated by the processing units against predefined consistency and integrity rules and only releases validated results for further use.