Multi-engine collaborative data processing and task scheduling system and method

Abstract

The invention provides a multi-engine collaborative data processing and task scheduling system and method.The multi-engine collaborative data processing and task scheduling system comprises a task scheduler, a data processing engine, a monitoring module, a data cleaning module and a unified data management unit, the task scheduler receives an uploaded task request and allocates a scheduling task according to the task request; the tasks are scheduled into the corresponding data processing engines according to a preset scheduling rule; through parallel processing and intelligent scheduling, the system significantly improves the data processing efficiency, and can complete the processing task of large-scale data in a short time. Due to the high efficiency, enterprises can obtain data insight more quickly and make timely decisions, so that the method has advantages in the market with fierce competition. The combination of real-time monitoring and an automatic repair mechanism ensures the stability and reliability of the task processing process. When the system detects abnormity, a response can be quickly made, potential risks and losses are reduced, and the safety of the system is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of collaborative data processing, and in particular relates to a multi-engine collaborative data processing and task scheduling system and method. Background Art

[0002] Traditional single-engine system: Use a single data processing engine to perform all tasks. This solution may not fully utilize the performance of multi-core processors, resulting in slow processing speed and limited scalability. With the surge in data volume and the complexity of computing tasks, manual task management can no longer meet the needs of modern business. There are many shortcomings in current technology:

[0003] Manual task scheduling and monitoring: Rely on operations personnel to manually start, monitor, and adjust tasks. This approach is prone to errors, inefficient, and difficult to manage when there are a large number of tasks. Batch data processing: Process data regularly (such as every night or every week) instead of real-time or near-real-time processing. This may cause data processing delays and is not suitable for scenarios with high timeliness requirements. No data cleaning or simplified data cleaning process: No data cleaning is performed after data processing, or a relatively simple data cleaning process is implemented. This may result in low data quality and affect the final data analysis results. Independent data import and quality inspection tools: Use independent tools to import data and check data quality instead of operating through a unified interface. This will increase operational complexity and learning costs, and reduce work efficiency. Rule-based system: Use fixed rules for task scheduling and data processing instead of dynamically adjusting according to the real-time status of the system. Such a system is not flexible enough to adapt to changing loads and demands.

[0004] No monitoring module: No monitoring of task execution status is implemented, or only basic status viewing functions are provided. This will make the system lack the ability to automatically intervene and recover when anomalies occur. Restrictive user interface: Providing a user interface with limited functions or non-intuitiveness makes it difficult for operation and maintenance personnel to efficiently manage data and check quality. Non-automated data cleaning: Relying on manual data cleaning is not only inefficient but also prone to human errors. Independent systems without collaborative work: Each system runs independently and cannot work together, resulting in low resource utilization and inefficient data processing. Script-based system: Relying on scripts to manage task scheduling and data processing, which requires operation and maintenance personnel to have programming skills and is difficult to adapt to rapidly changing business needs. . Summary of the invention

[0005] The present invention proposes a multi-engine collaborative data processing and task scheduling system and method, which solves the problem that tasks are not started in time in the prior art, resulting in data processing delays. There is a lack of automatic monitoring during task execution, and abnormalities cannot be discovered and handled in time. After the task is completed, data cleaning needs to be done manually, which is inefficient. Operation and maintenance personnel do not have a unified data management portal, and the process of checking data quality is cumbersome. Interfaces and manually sorted data need to be manually imported into the database, affecting work efficiency.

[0006] The technical solution of the present invention is implemented as follows: a multi-engine collaborative data processing and task scheduling system includes a task scheduler, a data processing engine, a monitoring module, a data cleaning module and a unified data management unit, wherein the task scheduler receives uploaded task requests, allocates scheduling tasks according to the task requests, and schedules the tasks to the corresponding data processing engines according to preset scheduling rules;

[0007] The data processing engines are provided with a plurality of data processing engines, and the plurality of data processing engines perform parallel processing at the same time, each data processing engine processes a corresponding task independently, and each data processing engine is associated with at least three other data processing engines;

[0008] The monitoring module monitors the tasks in real time, and sets an abnormal feature recognition library in the monitoring module, monitors the status of the monitored tasks through the pre-stored data in the abnormal feature recognition library, and increases the monitoring data upload frequency and marks it when abnormal features appear; when the abnormal features exceed the set threshold, starts automatic repair and synchronously resets and updates the tasks;

[0009] The data cleaning module is used to summarize the task information and automatically clean and remove noise from the data of the completed task after the data processing engine completes the task, and import the cleaned data into the unified data management unit;

[0010] The unified data management unit is provided with an operation interface, which displays the task data after the data cleaning module completes the cleaning and the cleaning parameters of the data cleaning module, compares the monitoring data and the cleaning data of each task, generates a quality inspection report in the operation interface, and adjusts the threshold parameters in the monitoring module according to the quality inspection report.

[0011] Compared with existing technologies, this multi-engine collaborative data processing and task scheduling system shows significant advantages and innovations.

[0012] Existing data processing systems often rely on a single data processing engine or linear processing model, lacking parallel processing capabilities, resulting in slow processing speed and low efficiency when facing large-scale data. This system can achieve parallel processing of tasks by setting up multiple data processing engines. Each engine independently performs the corresponding task, fundamentally improving the efficiency of data processing. This parallel processing design can not only significantly reduce the time of data processing, but also effectively cope with the challenges in the big data environment and improve the scalability of the system.

[0013] The introduction of the task scheduler enables the system to intelligently allocate tasks based on real-time task requests. Existing technologies often rely on static scheduling strategies and cannot flexibly respond to dynamically changing task requirements. The task scheduler of this system can flexibly adjust task allocation based on uploaded task requests and preset scheduling rules, improve resource utilization, and ensure that different tasks are efficiently executed on the appropriate engine.

[0014] The design of the monitoring module is relatively rare in the existing technology. Many systems lack real-time monitoring of task status and are unable to identify and handle abnormal situations in a timely manner. This system monitors the task status in real time by setting up an abnormal feature recognition library, and dynamically adjusts the monitoring data upload frequency when abnormal features appear to ensure timely response to potential problems. When the abnormal features exceed the set threshold, the system can automatically start the repair mechanism and reset the task. This self-healing capability significantly enhances the stability and reliability of the system.

[0015] After the task is completed, the data cleaning module summarizes and removes noise from the data to ensure the quality of the output data. In the prior art, data cleaning is often an independent step that lacks close integration with the data processing process. However, this system closely integrates data cleaning with task processing to ensure that the cleaned data can be seamlessly connected to the unified data management unit, providing a high-quality data foundation for subsequent data analysis and decision-making. Finally, the design of the unified data management unit provides a visual operation interface that allows users to intuitively view the cleaned data and monitoring data and generate quality inspection reports. This design improves the operability and user experience of the system, allowing users to more conveniently perform data analysis and adjust monitoring parameters.

[0016] As a preferred implementation, the task scheduler performs preliminary classification according to task priority, type and required resources, calls the current load status of the system, allocates tasks according to the load status, and sends classification information to the data processing engine for associated engine mobilization.

[0017] As a preferred implementation, the data processing engine performs real-time data transmission through data nodes when performing association, and each data processing engine is individually provided with a processing range interval. When the processing range exceeds the set interval, the excess data is split and operated by the associated data processing engine using the data nodes, and feedback information from the associated data processing engine is received.

[0018] As a preferred implementation, the abnormal feature identification library in the monitoring module records the abnormal feature values ​​corresponding to different tasks. When the data processing engine receives the task, the monitoring module calibrates the specific data interval for the data processing engine. When a characteristic data interval appears in the data processing engine, the monitoring data upload frequency is increased and marked. When the abnormal feature exceeds the set threshold, it is repaired and reset.

[0019] As a preferred implementation, the data cleaning module obtains the data generated after completing the task from the data processing engine, and the data includes the task status, execution time and resource usage. The data cleaning module performs a preliminary evaluation on the acquired data, identifies the noise data and outliers therein, and according to the set rules, screens out the data points that meet the expectations, removes the outliers, and fills in the missing values, and at the same time corrects the erroneous data and simultaneously completes the data format unification.

[0020] A multi-engine collaborative data processing and task scheduling method, the method comprising the following steps:

[0021] S1: Receive the uploaded task request through the task scheduler, assign the scheduling task according to the task request, and schedule the task to the corresponding data processing engine according to the preset scheduling rules;

[0022] S2: several data processing engines are processed in parallel at the same time, each data processing engine processes the corresponding task independently, and each data processing engine is associated with at least three other data processing engines;

[0023] S3: Monitor the task in real time through the monitoring module, set an abnormal feature recognition library in the monitoring module, monitor the status of the monitored task through the pre-stored data in the abnormal feature recognition library, and increase the monitoring data upload frequency and mark it when abnormal features appear; when the abnormal features exceed the set threshold, start automatic repair and synchronously reset and update the task;

[0024] S4: The data cleaning module receives the task completion information of the data processing engine, summarizes the task information, automatically cleans and removes noise from the data of the completed task, and imports the cleaned data into the unified data management unit;

[0025] S5: The unified data management unit is provided with an operation interface, which displays the task data after the data cleaning module completes the cleaning and the cleaning parameters of the data cleaning module, and compares the monitoring data and the cleaning data of each task, generates a quality inspection report in the operation interface, and adjusts the threshold parameters in the monitoring module according to the quality inspection report.

[0026] After adopting the above technical solution, the beneficial effect of the present invention is that through parallel processing and intelligent scheduling, the system significantly improves the efficiency of data processing and can complete large-scale data processing tasks in a short time. This high efficiency enables enterprises to obtain data insights more quickly and make timely decisions, thereby gaining an advantage in the highly competitive market. The combination of real-time monitoring and automatic repair mechanism ensures the stability and reliability of the task processing process. When the system detects an abnormality, it can respond quickly to reduce potential risks and losses and enhance the security of the system. The effective integration of the data cleaning module ensures the high quality of the output data, reduces decision-making errors caused by data problems, and thus improves the credibility of data analysis. The visual operation interface provided by the unified data management unit enhances the user's control and understanding of the system, allowing users to more easily manage data and adjust monitoring parameters. All of this makes the system show higher flexibility and scalability in practical applications, and provides strong support for enterprises in data processing and management. In short, this innovative data processing and task scheduling system promotes the development of data management technology and improves the efficiency and security of enterprises in the process of digital transformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Example:

[0031] A multi-engine collaborative data processing and task scheduling system, comprising a task scheduler, a data processing engine, a monitoring module, a data cleaning module and a unified data management unit. The task scheduler receives uploaded task requests, allocates scheduling tasks according to the task requests, and schedules the tasks to the corresponding data processing engine according to preset scheduling rules;

[0032] The data processing engines are provided with a plurality of data processing engines, and the plurality of data processing engines perform parallel processing at the same time, each data processing engine processes a corresponding task independently, and each data processing engine is associated with at least three other data processing engines;

[0033] The monitoring module monitors the tasks in real time, and sets an abnormal feature recognition library in the monitoring module, monitors the status of the monitored tasks through the pre-stored data in the abnormal feature recognition library, and increases the monitoring data upload frequency and marks it when abnormal features appear; when the abnormal features exceed the set threshold, starts automatic repair and synchronously resets and updates the tasks;

[0034] The data cleaning module is used to summarize the task information and automatically clean and remove noise from the data of the completed task after the data processing engine completes the task, and import the cleaned data into the unified data management unit;

[0035] The unified data management unit is provided with an operation interface, which displays the task data after the data cleaning module completes the cleaning and the cleaning parameters of the data cleaning module, compares the monitoring data and the cleaning data of each task, generates a quality inspection report in the operation interface, and adjusts the threshold parameters in the monitoring module according to the quality inspection report.

[0036] The core of the system is the task scheduler, which is responsible for receiving uploaded task requests. Users or other systems submit task requests to the task scheduler, which assigns corresponding processing tasks based on the nature, priority, and resource requirements of the tasks. This process follows preset scheduling rules, such as the type of task, data volume, processing time, etc., to ensure that tasks can be efficiently assigned to appropriate data processing engines. This intelligent scheduling mechanism improves resource utilization and ensures the flexibility and efficiency of the system in processing different tasks.

[0037] The system is equipped with multiple data processing engines. These engines can work in parallel, and each engine is responsible for processing specific tasks and completing data processing quickly. By introducing multiple data processing engines, the system can achieve high concurrency processing and improve overall processing capabilities. At the same time, the interconnection between data processing engines increases the flexibility of the system. For example, the intermediate results completed by one engine can be directly used by other engines, thereby reducing the time of data transmission and improving processing efficiency.

[0038] During the data processing process, the monitoring module monitors the status of the task in real time. By setting up an abnormal feature recognition library, the monitoring module can identify abnormal situations in task execution, such as long processing time, results that do not meet expectations, etc. When abnormal features are monitored, the system will automatically increase the frequency of uploading monitoring data and mark these tasks for subsequent tracking and analysis. This real-time monitoring capability ensures that the system can detect and handle potential problems in a timely manner, reducing the risk of failure.

[0039] If the monitored abnormal characteristics exceed the set threshold, the system will start the automatic repair mechanism. This mechanism may involve rescheduling tasks, adjusting resource allocation, or reprocessing data to ensure that the system can quickly return to normal. In addition, the reset and update of tasks can ensure the accuracy and consistency of data and avoid data loss due to interruptions or errors.

[0040] The introduction of the data cleaning module is another important component of the system. Its main task is to summarize the task information after the data processing engine completes the task, and automatically clean the processed data to remove noise and redundant information. The data cleaning process ensures the quality of the final data and improves the credibility of subsequent analysis. The cleaned data will be imported into the unified data management unit to provide a reliable basis for subsequent operations and analysis.

[0041] The unified data management unit is the control center of the entire system. It is equipped with a user interface to display the task data and corresponding cleaning parameters after the data cleaning module is completed. The operation interface provides users with intuitive data monitoring and management capabilities. Users can use the interface to compare the monitoring data and cleaning data of each task and generate quality inspection reports. These reports provide users with clear feedback on task execution, data quality, and processing efficiency. Based on the quality inspection report, the system can intelligently adjust the threshold parameters in the monitoring module. This feedback mechanism enables the system to self-optimize based on historical execution and data quality, thereby continuously improving overall performance and reliability.

[0042] The setting of this structure and operation mode is mainly reflected in the following aspects: Resource optimization: through the combination of task scheduler and multi-data processing engine, efficient resource utilization is achieved, and idle time and resource waste are reduced; Flexible processing: The intelligent task scheduling enables the system to adapt to tasks of different types and priorities, and improves the processing capacity; Real-time monitoring: The introduction of the monitoring module ensures the transparency and timeliness of task processing, and can respond quickly to abnormal situations; Data quality assurance: The setting of the data cleaning module makes the final data analysis results more reliable and reduces the possibility of data errors; User-friendly: The unified data management unit provides an intuitive operation interface, allowing users to easily monitor and manage data; Self-optimization: Through the feedback of monitoring data and quality inspection reports, the system can continuously adjust and optimize its own performance, making it more efficient in actual applications.

[0043] The task scheduler performs preliminary classification according to task priority, type and required resources, calls the current load status of the system, allocates tasks according to the load status, and sends classification information to the data processing engine for associated engine mobilization.

[0044] Task scheduling often uses a static or simple priority mechanism, lacking dynamic monitoring and adjustment of the overall system load status, resulting in uneven task distribution under high load conditions, affecting the system's operating efficiency and response time. The design of this task scheduler closely combines task scheduling with the actual use of system resources by calling the system's load status in real time. It can flexibly adjust the task allocation strategy under different load conditions, thereby optimizing resource utilization and improving system performance.

[0045] The multi-dimensional classification mechanism of this task scheduler is also a significant difference from the prior art. The prior art usually only focuses on the priority or type of the task, lacks comprehensive consideration of the resources required for the task, which may lead to resource bottlenecks in the task allocation process, affecting the overall processing capacity of the system. However, this application comprehensively considers the priority, type and required resources of the task to ensure that each task can get the most appropriate processing resources when allocated, thereby improving the intelligence level and flexibility of task scheduling. This comprehensive classification method enables the task scheduler to better cope with complex and dynamically changing workloads, ensuring that the system can maintain an efficient operating state in different scenarios.

[0046] After completing the task assignment, this task scheduler sends classification information to the data processing engine to mobilize the association engine. The design of this process also reflects the difference from the existing technology. Traditional task scheduling systems often lack subsequent tracking and adjustment mechanisms after task assignment, which may cause delays or waste of resources during task execution. However, this design can mobilize the association engine in a timely manner by sending classification information in real time to ensure that the execution of tasks can adapt to the dynamic changes of resources. This mechanism not only improves the response speed of the system, but also enhances the reliability and accuracy of task execution, allowing the system to maintain efficient operation in a complex working environment.

[0047] When the data processing engine is performing association, real-time data transmission is performed through the data nodes. A processing range interval is independently set for each data processing engine. When the processing range exceeds the set interval, the excess data is split and calculated by the associated data processing engine using the data nodes, and feedback information from the associated data processing engine is received.

[0048] The data processing engine realizes the design of real-time data transmission through data nodes when performing data association, which shows obvious innovation and advantages compared with the prior art. Traditional data processing systems usually adopt a centralized data flow method, lacking dynamic monitoring of data flow and processing capacity, which causes the system to be prone to bottlenecks when facing large-scale data and reduces processing efficiency. The data processing engine of the present application can effectively monitor and manage the load of each engine by setting an independent processing range interval for each engine, ensuring that the data always remains within the optimal range during the processing process, greatly improving the flexibility and response speed of data processing.

[0049] When the processing range exceeds the set interval, the data processing engine uses the design of splitting the excess data for calculation with the help of data nodes, which reflects the effective optimization of data processing capabilities. When processing excess data, the prior art often adopts a simple discard or queuing strategy, lacks effective use of data, and causes data delay or loss. In contrast, the present application distributes excess data to the associated data processing engine for parallel processing by splitting operations, which significantly enhances the system's ability and efficiency when processing large-scale data. This design enables the data processing engine to make full use of the computing resources of each node, thereby maintaining the stability and efficiency of the system under high load conditions.

[0050] In terms of receiving feedback information, the prior art usually lacks an effective feedback mechanism, resulting in a weak correlation between the processing results and the input data, making it difficult to achieve dynamic adjustment. However, after completing the split operation, the data processing engine of the present application can promptly receive feedback information from the associated data processing engine. This design enables the entire data processing system to have adaptive capabilities. By analyzing the feedback information, the system can adjust the subsequent data processing strategy in real time to cope with the changing input data and processing requirements, thereby optimizing the overall data flow efficiency. This real-time feedback mechanism is relatively rare in the prior art, further enhancing the intelligence level of the system.

[0051] The abnormal feature recognition library in the monitoring module records the abnormal feature values ​​corresponding to different tasks. When the data processing engine receives the task, the monitoring module calibrates the specific data interval of the data processing engine. When the data processing engine shows a characteristic data interval, the monitoring data upload frequency is increased and marked. When the abnormal feature exceeds the set threshold, the monitoring data is repaired and reset.

[0052] Traditional monitoring systems often rely on static threshold settings and one-time data analysis, and are unable to identify and respond to abnormal situations in real time during task execution. This approach is prone to delayed response when faced with complex and dynamically changing environments, which in turn causes greater system losses. The monitoring module proposed in this application can record the abnormal feature values ​​corresponding to different tasks in real time, and by calibrating the specific data intervals of the data processing engine, the system can respond quickly when abnormal features appear.

[0053] When the data processing engine receives a task, the monitoring module will immediately calibrate it to a specific data interval. This process ensures that the system can accurately monitor the possible abnormal data range during operation, avoiding the monitoring blind spots caused by the lack of dynamic adjustment in the existing technology. By setting a specific data interval, the monitoring module can effectively improve the ability to identify abnormal features, thereby enhancing the security and stability of the system.

[0054] When the monitoring module detects that the characteristic data of the data processing engine enters the set abnormal range, it will automatically increase the upload frequency of the monitoring data and mark it. This mechanism is not common in the existing technology. Traditional monitoring systems usually use a fixed upload frequency and lack the ability to respond quickly to abnormal situations. By dynamically adjusting the upload frequency, it can ensure that key data is recorded and analyzed in a timely manner when an abnormality occurs, thus providing an important basis for subsequent troubleshooting and system optimization.

[0055] More importantly, when the monitoring module identifies that the abnormal characteristics exceed the set threshold, it will immediately start the repair and reset mechanism. This kind of active repair measures are relatively rare in the prior art. Many systems only issue alarms when facing anomalies and lack effective automatic repair capabilities. However, by integrating repair and reset functions, this application can take timely measures after detecting a problem, reduce system downtime, and improve system reliability and stability.

[0056] The data cleaning module obtains the data generated after completing the task from the data processing engine. The data includes the task status, execution time and resource usage. The data cleaning module performs a preliminary evaluation on the acquired data, identifies the noise data and outliers therein, and filters out the data points that meet the expectations according to the set rules, removes the outliers, and fills in the missing values. At the same time, it corrects the erroneous data and simultaneously completes the data format unification.

[0057] The design idea of ​​the data cleaning module in this application fully considers the efficiency and accuracy of data processing, and aims to improve data quality through systematic steps. In modern data processing and analysis, raw data often contains noise data, outliers, and missing values. These inaccurate or incomplete data will directly affect the results of subsequent analysis and the effectiveness of decision-making. Therefore, it is particularly important to design an efficient data cleaning module.

[0058] The data cleaning module obtains the original data generated after completing the task from the data processing engine, ensuring that the module can obtain the latest data related to the actual operation. This design idea emphasizes the real-time and accuracy of data acquisition, allowing the cleaning module to process newly generated data in a timely manner, improving the response speed and flexibility of the entire system. Compared with the prior art, many data cleaning solutions often rely on static data sets and lack the ability to update in real time. This application ensures the timeliness of data cleaning through dynamic data acquisition.

[0059] The data cleaning module performs a preliminary evaluation of the acquired data and achieves basic control of data quality by identifying noise data and outliers. This process is carried out through set rules and algorithms to ensure the scientificity and accuracy of the cleaning operation. In the prior art, many data cleaning methods often lack systematic evaluation criteria, resulting in ineffective identification and processing of abnormal data. This application uses a clear evaluation mechanism to more accurately identify data points that do not meet expectations, thereby improving the overall quality of the data.

[0060] After screening out the expected data points, the data cleaning module also fills in the missing values ​​and corrects the erroneous data. This design not only improves the integrity of the data, but also enhances the reliability of data analysis. Missing value filling and error correction are crucial links in data cleaning. The prior art usually uses a simple mean filling or deletion method for missing values, which may lead to information loss or deviation. However, this application ensures the comprehensiveness and accuracy of the data through a more complex and reasonable filling and correction method.

[0061] The data cleaning module also synchronizes the data format, which is an important step to improve data availability. Data from different sources often have inconsistent formats, which makes subsequent processing and analysis difficult. By unifying the data format, the module can ensure consistency and compatibility in the subsequent analysis process, improving the efficiency of the entire data processing system. This feature is often overlooked in the existing technology, resulting in unnecessary troubles and obstacles in the use of data.

[0062] like Figure 1 As shown, a multi-engine collaborative data processing and task scheduling method comprises the following steps:

[0063] S1: Receive the uploaded task request through the task scheduler, assign the scheduling task according to the task request, and schedule the task to the corresponding data processing engine according to the preset scheduling rules;

[0064] S2: several data processing engines are processed in parallel at the same time, each data processing engine processes the corresponding task independently, and each data processing engine is associated with at least three other data processing engines;

[0065] S3: Monitor the task in real time through the monitoring module, set an abnormal feature recognition library in the monitoring module, monitor the status of the monitored task through the pre-stored data in the abnormal feature recognition library, and increase the monitoring data upload frequency and mark it when abnormal features appear; when the abnormal features exceed the set threshold, start automatic repair and synchronously reset and update the task;

[0066] S4: The data cleaning module receives the task completion information of the data processing engine, summarizes the task information, automatically cleans and removes noise from the data of the completed task, and imports the cleaned data into the unified data management unit;

[0067] S5: The unified data management unit is provided with an operation interface, which displays the task data after the data cleaning module completes the cleaning and the cleaning parameters of the data cleaning module, and compares the monitoring data and the cleaning data of each task, generates a quality inspection report in the operation interface, and adjusts the threshold parameters in the monitoring module according to the quality inspection report.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-engine collaborative data processing and task scheduling system, characterized by: It includes a task scheduler, a data processing engine, a monitoring module, a data cleaning module and a unified data management unit. The task scheduler receives uploaded task requests, allocates scheduling tasks according to the task requests, and schedules the tasks to the corresponding data processing engine according to preset scheduling rules; The data processing engines are provided with a plurality of data processing engines, and the plurality of data processing engines perform parallel processing at the same time, each data processing engine processes a corresponding task independently, and each data processing engine is associated with at least three other data processing engines; The monitoring module monitors the tasks in real time, and sets an abnormal feature recognition library in the monitoring module, monitors the status of the monitored tasks through the pre-stored data in the abnormal feature recognition library, and increases the monitoring data upload frequency and marks it when abnormal features appear; When the abnormal characteristics exceed the set threshold, automatic repair is initiated and the task is reset and updated synchronously; The data cleaning module is used to summarize the task information and automatically clean and remove noise from the data of the completed task after the data processing engine completes the task, and import the cleaned data into the unified data management unit; The unified data management unit is provided with an operation interface, which displays the task data after the data cleaning module completes the cleaning and the cleaning parameters of the data cleaning module, compares the monitoring data and the cleaning data of each task, generates a quality inspection report in the operation interface, and adjusts the threshold parameters in the monitoring module according to the quality inspection report.

2. The multi-engine collaborative data processing and task scheduling system according to claim 1, characterized in that: The task scheduler performs preliminary classification according to task priority, type and required resources, calls the current load status of the system, allocates tasks according to the load status, and sends classification information to the data processing engine for associated engine mobilization.

3. The multi-engine collaborative data processing and task scheduling system according to claim 1, characterized in that: When the data processing engine is performing association, real-time data transmission is performed through the data nodes. A processing range interval is independently set for each data processing engine. When the processing range exceeds the set interval, the excess data is split and calculated by the associated data processing engine using the data nodes, and feedback information from the associated data processing engine is received.

4. The multi-engine collaborative data processing and task scheduling system according to claim 1, characterized in that: The abnormal feature recognition library in the monitoring module records the abnormal feature values ​​corresponding to different tasks. When the data processing engine receives the task, the monitoring module calibrates the specific data interval of the data processing engine. When the data processing engine shows a characteristic data interval, the monitoring data upload frequency is increased and marked. When the abnormal feature exceeds the set threshold, the monitoring data is repaired and reset.

5. The multi-engine collaborative data processing and task scheduling system according to claim 1, characterized in that: The data cleaning module obtains the data generated after completing the task from the data processing engine. The data includes the task status, execution time and resource usage. The data cleaning module performs a preliminary evaluation on the acquired data, identifies the noise data and outliers therein, and filters out the data points that meet the expectations according to the set rules, removes the outliers, and fills in the missing values. At the same time, it corrects the erroneous data and simultaneously completes the data format unification.

6. A multi-engine collaborative data processing and task scheduling method, characterized in that: The method comprises the following steps: S1: Receive the uploaded task request through the task scheduler, allocate the scheduling task according to the task request, and schedule the task to the corresponding data processing engine according to the preset scheduling rules; S2: several data processing engines are processed in parallel at the same time, each data processing engine processes the corresponding task independently, and each data processing engine is associated with at least three other data processing engines; S3: Monitor the task in real time through the monitoring module, set an abnormal feature recognition library in the monitoring module, monitor the status of the monitored task through the pre-stored data in the abnormal feature recognition library, and increase the monitoring data upload frequency and mark it when abnormal features appear; when the abnormal features exceed the set threshold, start automatic repair and synchronously reset and update the task; S4: The data cleaning module receives the task completion information of the data processing engine, summarizes the task information, automatically cleans and removes noise from the data of the completed task, and imports the cleaned data into the unified data management unit; S5: The unified data management unit is provided with an operation interface, which displays the task data after the data cleaning module completes the cleaning and the cleaning parameters of the data cleaning module, and compares the monitoring data and the cleaning data of each task, generates a quality inspection report in the operation interface, and adjusts the threshold parameters in the monitoring module according to the quality inspection report.

Images