A massive data processing system and coordinated control method, device and storage medium

Abstract

The present invention relates to the field of data processing technology, and discloses a massive data processing system and a coordinated control method, device, and storage medium. The system includes: a coordinated control subsystem, a data splitting subsystem, a data comparison and update subsystem, a data prediction subsystem, and a data evaluation subsystem. The present invention processes the massive data to be processed respectively through the data splitting subsystem, the data comparison and update subsystem, the data prediction subsystem, and the data evaluation subsystem, thereby achieving the three control tasks of comparison, prediction, and evaluation of massive data, making the processing of massive data more comprehensive and accurate. Furthermore, the coordinated control subsystem can control the operation of the data splitting subsystem, the data comparison and update subsystem, the data prediction subsystem, and the data evaluation subsystem during the massive data processing process, thereby achieving coordinated control of massive data processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a massive data processing system and a coordinated control method, device and storage medium. Background Art

[0002] With the widespread adoption of the internet, the Internet of Things, and mobile devices, the generation of large-scale data is experiencing explosive growth. This data encompasses a wide variety of information, such as large-scale engineering project data, equipment cluster monitoring data, and traffic data from large road networks. However, extracting useful information from this massive amount of data has become a significant challenge.

[0003] Data collection and storage are crucial for large-scale data processing. The development of sensor networks, cloud computing, and distributed storage systems has made data collection and storage more efficient and reliable. At the same time, as data volumes increase, data backup and protection become increasingly important.

[0004] Large-scale data often contains noise, missing values, and outliers, which can hinder subsequent data analysis and modeling. Therefore, data cleaning and preprocessing are essential steps in large-scale data processing. Using statistical and machine learning methods, data can be cleaned and preprocessed to improve both quality and usability.

[0005] Massive data is a development trend, and data analysis and mining are becoming increasingly important. Extracting useful information from massive data is important and urgent, which requires accurate processing, high precision, short processing time, and quick acquisition of valuable information. Therefore, research on massive data is very promising and worthy of extensive and in-depth research.

[0006] However, massive data is difficult to process due to its large volume. Therefore, it is urgent to propose a method for processing massive data. Summary of the Invention

[0007] In view of this, the present invention provides a massive data processing system and a coordinated control method, device and storage medium to solve the problem that massive data is difficult to process due to its large amount of data.

[0008] In a first aspect, the present invention provides a massive data processing system, the system comprising: a coordination and control subsystem, a data splitting subsystem, a data comparison and updating subsystem, a data prediction subsystem, and a data evaluation subsystem;

[0009] The coordination control subsystem is used to obtain the massive data to be processed and the information of multiple tasks to be processed, and when the information of multiple tasks to be processed meets the preset first condition and the massive data to be processed meets the preset second condition, the massive data to be processed is sent to the data splitting subsystem, and the multiple tasks to be processed are sent to the data splitting subsystem, the data comparison and update subsystem, the data prediction subsystem and the data evaluation subsystem; the data splitting subsystem is used to process the massive data to be processed and the multiple tasks to be processed to obtain multiple related data blocks, and send the multiple related data blocks to the data comparison and update subsystem, the data prediction subsystem and the data evaluation subsystem; the data comparison and update subsystem The subsystem is used to compare multiple related data blocks and multiple preset standard data based on multiple task information to be processed, obtain data comparison results, and send the data comparison results to the data evaluation subsystem; the data prediction subsystem is used to perform data prediction using a smoothing exponential prediction method based on preset target requirements, multiple related data blocks and multiple task information to be processed, obtain data prediction results of massive data to be processed, and send the data prediction results to the data evaluation subsystem; the data evaluation subsystem performs data evaluation based on multiple task information to be processed, multiple related data blocks, data comparison results and data prediction results, and obtains data evaluation results of massive data to be processed.

[0010] The mass data processing system provided by the present invention processes the massive data to be processed through a data splitting subsystem, a data comparison and updating subsystem, a data prediction subsystem, and a data evaluation subsystem. This system can achieve the three control tasks of comparing, predicting, and evaluating the massive data, making the processing of the massive data more comprehensive and accurate. Furthermore, the coordinated control subsystem can control the operation of the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem, and the data evaluation subsystem during the massive data processing process, thus achieving coordinated control of the massive data processing.

[0011] In an optional embodiment, the data splitting subsystem includes: a node identification module, a splitting module and an association module;

[0012] The node identification module is used to use AI technology to identify multiple pending task information, obtain multiple task key nodes and multiple data milestone information, and send the multiple task key nodes and multiple data milestone information to the splitting module, and send the multiple pending task information to the association module; the splitting module is used to split the massive data to be processed based on multiple task key nodes and multiple data milestone information, obtain multiple data blocks, and send the multiple data blocks to the association module; the association module is used to associate multiple data blocks with multiple pending task information based on data visas to obtain multiple associated data blocks.

[0013] The present invention can obtain multiple mission key nodes and multiple data milestone information through AI technology identification. Furthermore, multiple data milestone information can comprehensively and accurately reflect the overall status of massive data to be processed. Furthermore, it can improve the accuracy of subsequent data splitting and classification and the rationality of processing and analysis, and provide data support for subsequent data prediction, comparison and other processing.

[0014] In an optional embodiment, the data comparison and update subsystem includes: a data acquisition module and a comparison module;

[0015] The data acquisition module is used to acquire multiple preset standard data and send the multiple preset standard data to the comparison module; the comparison module is used to compare multiple related data blocks with the preset standard data, obtain data comparison results, and send the data comparison results to the data evaluation subsystem.

[0016] In an optional implementation, the data comparison and update subsystem further includes: an update module, configured to update a plurality of preset standard data based on a plurality of to-be-processed task information.

[0017] The present invention can update corresponding preset standard data in real time according to the received multiple task information to be processed.

[0018] In an optional embodiment, the data prediction subsystem includes: a classification editing module and a prediction module;

[0019] The classification and editing module is used to classify and edit multiple related data blocks according to preset target requirements, obtain multiple data groups, and send the multiple data groups to the prediction module; the prediction module is used to process the multiple data groups through the smoothing index prediction method based on multiple pending task information, obtain data prediction results, and send the data prediction results to the data evaluation subsystem.

[0020] The present invention classifies and edits associated data blocks through a classification editing module, which can facilitate subsequent data prediction. Furthermore, combined with a smoothing index prediction method, it can realize prediction processing of massive data.

[0021] In an optional embodiment, the data evaluation subsystem includes: a group coding module, a first model building module, a second model building module and a calculation evaluation module;

[0022] A group coding module is used to group code multiple related data blocks based on multiple task information to be processed to obtain multiple block coding sequences, and send the multiple block coding sequences to the calculation and evaluation module, where each block coding sequence corresponds to an associated data block; a first model construction module is used to establish a first data evaluation standard model based on the data comparison result, and send the first data evaluation standard model to the calculation and evaluation module; a second model construction module is used to establish a second data evaluation standard model based on the data prediction result, and send the second data evaluation standard model to the calculation and evaluation module; the calculation and evaluation module is used to obtain a data evaluation result based on multiple block coding sequences through a preset calculation method, the first data evaluation standard model and the second data evaluation standard model.

[0023] By constructing a first data evaluation standard model and a second data evaluation standard model, the present invention can comprehensively analyze the massive data to be processed from different angles, thereby improving the accuracy of the data evaluation results.

[0024] In an optional embodiment, the calculation and evaluation module includes: a first calculation submodule, a first processing submodule, a second processing submodule, and a second calculation submodule;

[0025] The first calculation submodule is used to obtain multiple first data evaluation indicators and multiple second data evaluation indicators based on multiple block coding sequences through a preset calculation method, and send the multiple first data evaluation indicators to the first processing submodule, and send the multiple second data evaluation indicators to the second processing submodule; the first processing submodule is used to input the multiple first data evaluation indicators into the first data evaluation standard model to obtain multiple first data evaluation values, and send the multiple first data evaluation values ​​to the second calculation submodule; the second processing submodule is used to input the multiple second data evaluation indicators into the second data evaluation standard model to obtain multiple second data evaluation values, and send the multiple second data evaluation values ​​to the second calculation submodule; the second calculation submodule is used to calculate the data evaluation result based on the multiple first data evaluation values ​​and the multiple second data evaluation values.

[0026] In an optional embodiment, the first calculation submodule includes: a first calculation unit, a second calculation unit, and a third calculation unit;

[0027] The first calculation unit is used to obtain multiple data block coding comparison values ​​based on multiple block coding sequences and a preset first relational expression, and send the multiple data block coding comparison values ​​to the second calculation unit; the second calculation unit is used to obtain multiple data block coding prediction values ​​based on multiple block coding sequences and multiple data block coding comparison values ​​and a preset second relational expression, and send the multiple data block coding prediction values ​​to the third calculation unit; the third calculation unit is used to obtain multiple first data evaluation indicators and multiple second data evaluation indicators based on the multiple data block coding prediction values ​​and a preset third relational expression.

[0028] In an optional embodiment, the data evaluation subsystem further includes: a detection module for receiving the data evaluation results sent by the calculation evaluation module, and detecting whether the data evaluation results meet the preset standards based on a preset evaluation threshold range to obtain a detection result.

[0029] The present invention can detect whether the data evaluation results meet the preset standards by presetting the evaluation threshold range, thereby providing support for subsequent data evaluation.

[0030] In an optional embodiment, the data comparison and update subsystem is further configured to send the data comparison result to the data evaluation subsystem via the data prediction subsystem.

[0031] In an optional embodiment, the coordination control subsystem is also used to send the massive data to be processed to the data comparison and update subsystem, the data prediction subsystem and the data evaluation subsystem for processing respectively when multiple task information to be processed meets the preset first condition and the massive data to be processed does not meet the preset second condition, and obtain the data evaluation results of the massive data to be processed.

[0032] The present invention can control the operation of the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem and the data evaluation subsystem in the process of massive data processing through the coordinated control subsystem, thereby realizing coordinated control of massive data processing.

[0033] In an optional embodiment, the coordination control subsystem includes: an acquisition module, a judgment module, a comparison module and a determination module;

[0034] The acquisition module is used to obtain the data volume and data processing speed of the massive data to be processed, and send the data volume to the judgment module and the data processing speed to the comparison module; the judgment module is used to judge whether the data volume meets the preset third condition and send the judgment result to the determination module; the comparison module is used to compare the data processing speed with the preset threshold and send the comparison result to the determination module; the determination module is used to determine whether the massive data to be processed meets the preset second condition based on the judgment result or the comparison result.

[0035] The present invention can determine whether the massive data to be processed meets the preset second condition based on the data volume or data processing speed of the massive data to be processed, thereby providing support for the coordinated control of the subsequent coordinated control subsystem.

[0036] In an optional embodiment, the determination module is used to determine that the massive data to be processed does not meet the preset second condition when the judgment result is that the data volume meets the preset third condition, or the comparison result is that the data processing speed is greater than the preset threshold; the determination module is also used to determine that the massive data to be processed meets the preset second condition when the judgment result is that the data volume does not meet the preset third condition, or the comparison result is that the data processing speed is less than the preset threshold.

[0037] In an optional embodiment, the coordination control subsystem is also used to determine the target control subsystem when multiple pending task information does not meet the preset first condition, and control all subsystems except the target control subsystem in the data splitting subsystem, data comparison and update subsystem, data prediction subsystem and data evaluation subsystem to be paused. The target control subsystem is determined based on multiple pending task information and is one of the data splitting subsystem, data comparison and update subsystem, data prediction subsystem and data evaluation subsystem.

[0038] In a second aspect, the present invention provides a coordinated control method for use in a coordinated control subsystem within a massive data processing system according to the first aspect or any corresponding embodiment thereof; the method comprising:

[0039] Obtain information on a large amount of data to be processed and a plurality of tasks to be processed; obtain a coordinated control strategy based on the information on the large amount of data to be processed and the plurality of tasks to be processed through a preset judgment method; utilize the coordinated control strategy to coordinately control the processing flow of the large amount of data to be processed within the large amount of data processing system to obtain a coordinated control result.

[0040] The coordinated control method provided by the present invention can determine the coordinated control strategy of the coordinated control subsystem through the judgment result of whether the massive data to be processed meets the preset second condition and the information of multiple tasks to be processed. Furthermore, the coordinated control strategy is used to realize the coordinated control of the processing flow of the massive data to be processed in the massive data processing system.

[0041] In a third aspect, the present invention provides a coordinated control device for executing the coordinated control method provided in the second aspect; the device comprises:

[0042] The acquisition unit is used to obtain information on massive data to be processed and multiple tasks to be processed; the judgment unit is used to obtain a coordinated control strategy based on the massive data to be processed and the multiple tasks to be processed through a preset judgment method; the control unit is used to coordinate and control the processing flow of the massive data to be processed in the massive data processing system using the coordinated control strategy to obtain a coordinated control result.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the coordinated control method provided in the second aspect above.

[0044] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, where the computer instructions are used to enable a computer to execute the coordinated control method provided in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 is a structural block diagram of a massive data processing system according to an embodiment of the present invention;

[0047] Figure 2 is a structural block diagram of a coordinated control subsystem according to an embodiment of the present invention;

[0048] Figure 3 is a structural block diagram of a data splitting subsystem according to an embodiment of the present invention;

[0049] Figure 4 is a structural block diagram of a data comparison and update subsystem according to an embodiment of the present invention;

[0050] Figure 5 is a structural block diagram of a data prediction subsystem according to an embodiment of the present invention;

[0051] Figure 6 is a structural block diagram of a data evaluation subsystem according to an embodiment of the present invention;

[0052] Figure 7 is a structural block diagram of a massive data processing and analysis system according to an embodiment of the present invention;

[0053] Figure 8 is a flow chart of a coordinated control method according to an embodiment of the present invention;

[0054] Figure 9 is a structural block diagram of a coordinated control device according to an embodiment of the present invention;

[0055] Figure 10 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0057] According to an embodiment of the present invention, a massive data processing system is provided. Figure 1 As shown, the massive data processing system 1 includes: a coordination and control subsystem 11, a data splitting subsystem 12, a data comparison and updating subsystem 13, a data prediction subsystem 14 and a data evaluation subsystem 15.

[0058] Preferably, Figure 2 As shown, the coordination control subsystem 11 includes: an acquisition module 111 , a judgment module 112 , a comparison module 113 and a determination module 114 .

[0059] Preferably, Figure 3 As shown, the data splitting subsystem 12 includes: a node identification module 121, a splitting module 122 and an association module 123.

[0060] Preferably, Figure 4 As shown, the data comparison and update subsystem 13 includes: a data acquisition module 131 , a comparison module 132 and an update module 133 .

[0061] Preferably, Figure 5 As shown, the data prediction subsystem 14 includes: a classification editing module 141 and a prediction module 142.

[0062] Preferably, Figure 6 As shown, the data evaluation subsystem 15 includes: a group coding module 151, a first model building module 152, a second model building module 153, a calculation and evaluation module 154 and a detection module 155.

[0063] The calculation and evaluation module 154 includes: a first calculation submodule 1541 , a first processing submodule 1542 , a second processing submodule 1543 and a second calculation submodule 1544 .

[0064] Furthermore, the first calculation submodule 1541 includes: a first calculation unit 15411 , a second calculation unit 15412 and a third calculation unit 15413 .

[0065] Furthermore, the functions of each device in the above system are described.

[0066] Specifically, the coordination control subsystem 11 can control the operation of all or individual subsystems.

[0067] Preferably, the coordination control subsystem 11 is used to obtain massive data to be processed and multiple task information to be processed, and when multiple task information to be processed meets the preset first condition and the massive data to be processed meets the preset second condition, the massive data to be processed is sent to the data splitting subsystem, and the multiple task information to be processed is sent to the data splitting subsystem 12, the data comparison and update subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15.

[0068] Among them, the preset first condition can be used to confirm whether it is necessary to control the operation of all other subsystems. When multiple pending task information meets the preset first condition, it means that the current data processing requires the cooperation of all subsystems to complete; the preset second condition can be used to determine whether the massive data to be processed needs to be split. When the massive data to be processed meets the preset second condition, it means that the massive data to be processed needs to be split.

[0069] Specifically, the data volume and data processing speed of the massive data to be processed are acquired in the acquisition module 111 , and the data volume is sent to the judgment module 112 , and the data processing speed is sent to the comparison module 113 .

[0070] Furthermore, the judging module 112 may judge whether the amount of the massive data to be processed satisfies a preset third condition, and send the obtained judgment result to the determining module 114 .

[0071] The third condition is preset to reflect the size of the massive data to be processed.

[0072] Specifically, if the amount of the massive data to be processed meets the preset third condition, it means that the amount of the massive data to be processed is small; if the amount of the massive data to be processed meets the preset third condition, it means that the amount of the massive data to be processed is large.

[0073] Furthermore, the data processing speed is compared with a preset threshold in the comparison module 113 , and the obtained comparison result is sent to the determination module 114 .

[0074] Furthermore, in the determination module 114 , it can be determined whether the massive data to be processed needs to be split according to the received judgment result and comparison result, that is, whether the preset second condition is met.

[0075] Specifically, if the judgment result is that the data volume meets the preset third condition, or the comparison result is that the data processing speed is greater than the preset threshold, it means that the massive data to be processed does not need to be split, that is, the massive data to be processed does not meet the preset second condition;

[0076] If the judgment result is that the data volume does not meet the preset third condition, or the comparison result is that the data processing speed is less than the preset threshold, it means that the massive data to be processed needs to be split, that is, the massive data to be processed meets the preset second condition.

[0077] Furthermore, if the massive data to be processed needs to be split, the acquired massive data to be processed will be sent to the data splitting subsystem 12, and at the same time, multiple task information to be processed will be sent to the data splitting subsystem 12, the data comparison and update subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15.

[0078] Furthermore, the data splitting subsystem is controlled to split the data to be processed and analyzed into multiple data blocks, and then the subsystems are synchronously controlled in batches according to the actual order to complete the data analysis.

[0079] Preferably, the data splitting subsystem 12 is used to process the massive data to be processed and multiple task information to be processed, obtain multiple related data blocks, and send the multiple related data blocks to the data comparison and update subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15.

[0080] First, AI technology is used in the node identification module 121 to identify multiple pending task information to obtain multiple task key nodes and multiple data milestone information, and the multiple task key nodes and multiple data milestone information are sent to the splitting module 122, and the multiple pending task information is sent to the association module 123.

[0081] Specifically, data milestone information is used to indicate the processing completion status of each key task node in the massive data to be processed. There can be multiple milestone information, which can comprehensively and accurately reflect the overall status of the data, improve the accuracy of data splitting and classification and the rationality of processing and analysis, and facilitate subsequent data prediction, comparative analysis and other processing.

[0082] Furthermore, pending task information typically provides required processing and analysis parameters, such as system processing speed, required processing time, and pending plans. Therefore, AI technology can analyze the amount of data required to complete each plan based on the system's historical processing speed and the upper and lower limits of required processing time, thereby determining the locations of key nodes and milestones for each task.

[0083] For example, if the processing time required to complete all plans is long and the current system processing speed is high, more key nodes need to be set to reduce the amount of data in each data block, so that synchronous processing and analysis can be completed more quickly. Conversely, if the processing time required to complete one of the plans is short and the current system processing speed is high, a small number of key nodes need to be set, and the amount of data in each database can be large, so that a faster completion speed can be maintained during synchronous processing and analysis.

[0084] For example, for data processing and analysis of large-scale engineering projects, milestone-based data forecasting and evaluation complements the task-based forecasting and evaluation capabilities of the project data processing and analysis system. Milestone-based forecasting and evaluation is not only applicable to project progress data, but also to equipment monitoring data, regional environmental data, and other data. Milestone-based forecasting and evaluation can be well-linked to tasks, making data forecasting and evaluation more realistic.

[0085] Next, the splitting module 122 splits the massive data to be processed according to the multiple mission-critical nodes and the multiple data milestone information to obtain multiple data blocks, and sends the multiple data blocks to the association module, wherein each data block starts or ends with a mission-critical node and a data milestone.

[0086] Finally, in the association module 123 , data visas can be used to associate multiple data blocks with multiple task information to be processed to obtain multiple associated data blocks, and the obtained multiple associated data blocks are sent to the data comparison and update subsystem 13 and the data prediction subsystem 14 .

[0087] Furthermore, data visas can be used to conduct integrated control of comparison, prediction and evaluation of data blocks.

[0088] Specifically, in the data visa structure, an association level between the data to be processed and analyzed and the data to be processed and analyzed is set. Each level corresponds to different tasks and required data, which may include: data comparison layer - setting the association between data comparison tasks, standard data and data to be processed and analyzed; data prediction layer - setting the association between data comparison tasks, historical data, prediction models and data to be processed and analyzed; data evaluation layer - setting the association between data comparison tasks, evaluation models and data to be processed and analyzed.

[0089] Furthermore, after the massive data to be processed is split into multiple data blocks, when the massive data to be processed is obtained and a comparison task needs to be processed, the required data at the same level is obtained at the same time, and then multiple data blocks at this level are obtained.

[0090] By adopting this hierarchical association form, multi-dimensional information in engineering project data analysis and processing such as data splitting, comparison, updating, prediction and evaluation is linked, ensuring the information association and continuous traceability of engineering data. At the same time, it can also adapt to the large-scale integrated control needs of massive data such as large-scale equipment cluster monitoring, large-scale urban road network traffic, etc.

[0091] Furthermore, once a task occurs at the same level, the structure cannot be changed, but new processed data blocks can be added. All analysis and processing of the same data block should be included in the same level.

[0092] Preferably, after receiving multiple related data blocks, the data comparison and update subsystem 13 compares the multiple related data blocks with multiple preset standard data based on multiple pending task information to obtain data comparison results, and sends the data comparison results to the data evaluation subsystem 15.

[0093] Specifically, a plurality of preset standard data are acquired in the data acquisition module 131 , and the plurality of preset standard data are sent to the comparison module 132 .

[0094] Among them, each associated level corresponding to each associated data block has corresponding preset standard data.

[0095] Furthermore, the comparison module 132 compares the received multiple associated data blocks with multiple preset standard data to generate corresponding data comparison results, and sends the data comparison results to the data evaluation subsystem 15 .

[0096] Furthermore, the data comparison result can also be sent to the data evaluation subsystem 15 through the data prediction subsystem 14.

[0097] Furthermore, the obtained data comparison result can be directly output and displayed.

[0098] Furthermore, the acquired multiple preset standard data may be updated in real time in the updating module 133 according to the received multiple pieces of to-be-processed task information.

[0099] In one embodiment, taking engineering project data processing and analysis as an example, the data acquisition module 131 acquires project progress data or multiple equipment monitoring parameter data, and simultaneously acquires project progress standard data for the project progress data or monitoring parameter standard data for the equipment monitoring parameter data.

[0100] Furthermore, when the comparison module 132 compares and analyzes the project progress data with the project progress standard data, it calculates the advance / delay of the project based on the accumulated actual project completion amount, generates an estimated project completion time that is different from the planned project completion time based on the advance / delay measurement of the project, and calculates the updated project progress and time change based on the estimated project completion time.

[0101] Furthermore, the comparison module 132 determines whether to take corrective measures after comparison. If it is determined that corrective measures are to be taken, the comparison module 132 identifies the project data changes of the corrective measures and generates a project progress data comparison table based on the project data changes.

[0102] Furthermore, for the monitoring parameter data of multiple engineering equipment, when the equipment monitoring parameter data is compared and analyzed with the monitoring parameter standard data using the comparison module 132, the parameter advance / lag amount of the equipment monitoring parameter data during the current equipment operation process is generated according to the weighting coefficient, and parameter adjustment data different from the monitoring parameter standard data is generated based on the parameter advance / lag measurement, and the updated equipment monitoring parameter data is calculated based on the parameter adjustment data.

[0103] Furthermore, the comparison module 132 determines whether to take corrective measures after comparison. If it is determined to take corrective measures, the comparison module 132 identifies the changes in the project data of the corrective measures and generates an equipment monitoring parameter data comparison table based on the updated equipment monitoring parameter data.

[0104] Preferably, after receiving multiple related data blocks, the data prediction subsystem 14 uses a smoothing index prediction method to perform data prediction based on preset target requirements, multiple related data blocks and multiple task information to be processed, obtains data prediction results for the massive data to be processed, and sends the data prediction results to the data evaluation subsystem 15.

[0105] First, in the classification and editing module 141 , multiple related data blocks are classified and edited according to preset target requirements to obtain multiple data groups, and the multiple data groups are sent to the prediction module 142 .

[0106] Among them, the preset target requirements can be determined based on actual conditions, such as progress data of different projects in large-scale projects, equipment monitoring data in different areas, or driving monitoring data and traffic flow data in large urban transportation networks.

[0107] Furthermore, classification editing is to classify and organize multiple related data blocks according to preset target requirements to form data groups with different classifications based on unified model standards, which is convenient for subsequent prediction calculations.

[0108] Specifically, by classifying and editing multiple related data blocks according to different preset target requirements, a data classification and editing list of the current task to be processed can be extracted, wherein the data classification and editing list is composed of multiple data groups.

[0109] Then, in the prediction module 142 , based on the information of the multiple tasks to be processed, the multiple data groups are processed by the smoothing index prediction method to obtain data prediction results, and the data prediction results are sent to the data evaluation subsystem 15 .

[0110] Specifically, the processing of the smoothing index prediction method includes:

[0111] (1) For the data corresponding to the current task to be processed, the predicted value of the classified data group i is calculated as shown in the following relationship (1):

[0112]

[0113] Where: represents the predicted value of data group i; D i-1 represents the actual value of the previous same task in the history database; α represents the balance index; y i-1 Indicates the predicted value of the previous same task in the history database.

[0114] (2) Calculate the predicted value P * , that is, the data prediction result:

[0115] Specifically, let the preset standard value of the i-th data block be P i , and its prediction error is Then the predicted value of the i-1th data group is As shown in the following equation (2):

[0116]

[0117] Where: α i It represents the degree of similarity and can be obtained through the similarity calculation formula.

[0118] Further, calculate the predicted value P * , which is the average of the sum of the predicted values ​​of all data groups.

[0119] The sum of the predicted values ​​is expressed as follows:

[0120]

[0121] Further, calculate the average value P of the sum of the predicted values * , as shown in the following relation (4):

[0122]

[0123] Preferably, the data evaluation subsystem 15 performs data evaluation based on the received multiple task information to be processed, multiple related data blocks, data comparison results and data prediction results to obtain a data evaluation result of the massive data to be processed.

[0124] First, in the group coding module 151 , multiple associated data blocks are group-coded based on multiple pieces of to-be-processed task information to obtain multiple block coding sequences, and the multiple block coding sequences are sent to the calculation and evaluation module 154 .

[0125] Each block coding sequence corresponds to an associated data block.

[0126] In one example, taking large-scale engineering project data as an example, the engineering project data is decomposed to obtain multiple unit engineering data blocks. The unit engineering data block is a 1-bit structure with a code of 1. Under the unit engineering data block, multiple sub-engineering data blocks are included. The distributed engineering data block is 2 bits, plus 1 bit of the unit engineering data block, and the code is 101. (3) Under the sub-engineering, multiple sub-item engineering data blocks are included. The sub-item engineering data block is 2 bits + plus 1 bit of the unit engineering data block + 2 bits of the sub-engineering data block, and the sub-item engineering code is: 10101.

[0127] Secondly, a first data evaluation standard model is established in the first model building module 152 based on the data comparison result, and the first data evaluation standard model is sent to the calculation evaluation module 154.

[0128] At the same time, a second data evaluation standard model is established in the second model building module 153 based on the data prediction result, and the second data evaluation standard model is sent to the calculation evaluation module 154.

[0129] Finally, in the calculation and evaluation module 154, based on the multiple block coding sequences, a data evaluation result is obtained through processing using a preset calculation method, a first data evaluation standard model, and a second data evaluation standard model.

[0130] Specifically, in the first calculation submodule 1541, based on multiple block coding sequences, a preset calculation method is used to obtain multiple first data evaluation indicators and multiple second data evaluation indicators, and the multiple first data evaluation indicators are sent to the first processing submodule 1542, and the multiple second data evaluation indicators are sent to the second processing submodule 1543.

[0131] First, in the first calculation unit 15411, based on multiple block coding sequences, a preset first relationship is calculated to obtain multiple data block coding comparison values, and the multiple data block coding comparison values ​​are sent to the second calculation unit 15412.

[0132] Specifically, the comparison result of each data block can be obtained according to the multiple block coding sequences. Further, the data block coding comparison value corresponding to each data block can be calculated using the preset first relationship, as shown in the following relationship (5):

[0133]

[0134] Where: B e Indicates the data block coding comparison value; n indicates the total number of data blocks; F i represents the encoding of the i-th data block; H represents the preset classification interval.

[0135] Secondly, in the second calculation unit 15412, based on multiple block coding sequences and multiple data block coding comparison values, a preset second relationship is calculated to obtain multiple data block coding prediction values, and the multiple data block coding prediction values ​​are sent to the third calculation unit 15413.

[0136] Specifically, the comparison result of each data block can be obtained according to the multiple block coding sequences. Further, the corresponding data block coding prediction value can be calculated using the preset second relationship, as shown in the following relationship (6):

[0137] Z id =1BID+B e (6)

[0138] Where: Z id Indicates the data block encoding prediction value; BID indicates the task ID corresponding to the data block.

[0139] Finally, in the third calculation unit 15413, based on the multiple data block coding prediction values, a preset third relationship is calculated to obtain multiple first data evaluation indicators and multiple second data evaluation indicators.

[0140] Specifically, the first data evaluation index and the second data evaluation index corresponding to each data block can be calculated using the following relationship (7):

[0141]

[0142] In the formula: A represents the first data evaluation index; B represents the second data evaluation index; Y() represents the function of taking the remainder; s1 represents the evaluation coefficient one; s2 represents the evaluation coefficient two.

[0143] Furthermore, after receiving the plurality of first data evaluation indicators, the first processing submodule 1542 inputs the plurality of first data evaluation indicators into the first data evaluation standard model to obtain the corresponding plurality of first data evaluation values ​​C j , and the plurality of first data evaluation values ​​C jSend to the second calculation submodule 1544.

[0144] At the same time, after receiving the plurality of second data evaluation indicators, the second processing submodule 1543 inputs the plurality of second data evaluation indicators into the second data evaluation standard model to obtain the corresponding plurality of second data evaluation values ​​D j , and the plurality of second data evaluation values ​​D j Send to the second calculation submodule 1544.

[0145] Furthermore, the second calculation submodule 1544 receives a plurality of first data evaluation values ​​C j and a plurality of second data evaluation values ​​D j After that, the total evaluation value V of each corresponding data block can be calculated j =C j +D j , and then the data evaluation results of the massive data to be processed can be obtained according to the total evaluation value of each data block.

[0146] Furthermore, the calculation and evaluation module 154 may also send the obtained data evaluation results to the detection module 155 .

[0147] Furthermore, after receiving the data evaluation result, the detection module 155 can detect whether the data evaluation result meets the preset standard based on the preset evaluation threshold range to obtain the detection result.

[0148] Specifically, the number of data blocks that are not within the preset evaluation threshold range can be determined. If the number is less than 5% of the number of data blocks, the data evaluation result is considered to meet the standard. Conversely, if the number is greater than or equal to 5% of the number of data blocks, the data evaluation result is considered to not meet the standard.

[0149] Preferably, the coordination control subsystem 11 is also used to send the massive data to be processed to the data comparison and update subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15 for processing respectively when multiple task information to be processed meets the preset first condition and the massive data to be processed does not meet the preset second condition, and obtain the data evaluation results of the massive data to be processed.

[0150] Specifically, if multiple pending task information satisfies the first preset condition and the bulk data to be processed does not meet the second preset condition, then the bulk data to be processed does not need to be split, and therefore, data splitting subsystem 12 does not need to be executed. In this case, the acquired bulk data to be processed is directly sent to data comparison and update subsystem 13, data prediction subsystem 14, and data evaluation subsystem 15 for processing, and the final data evaluation results are output. The specific processing process is described above with reference to the functional descriptions of data comparison and update subsystem 13, data prediction subsystem 14, and data evaluation subsystem 15, and will not be repeated here.

[0151] Preferably, the coordination control subsystem 11 is also used to determine the target control subsystem when multiple pieces of task information to be processed do not meet the preset first condition, and control all subsystems except the target control subsystem in the data splitting subsystem, data comparison and update subsystem, data prediction subsystem and data evaluation subsystem to be paused.

[0152] Among them, the target control subsystem is determined based on multiple pending task information and is one of the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem and the data evaluation subsystem.

[0153] Specifically, when multiple pieces of task information to be processed do not meet the preset first condition, it means that the current data processing requires the separate control of a subsystem, namely the target control subsystem. At this time, the operation of other subsystems except the target control subsystem is temporarily stopped.

[0154] Furthermore, the coordination control subsystem 11 can receive the operation results (computational load and speed) of each subsystem, and can perform resource scheduling according to the operation results of each subsystem to ensure the processing efficiency of the entire system.

[0155] The massive data processing system provided in this embodiment processes the massive data to be processed through the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem, and the data evaluation subsystem. This achieves the three control tasks of massive data comparison, prediction, and evaluation, making the processing of massive data more comprehensive and accurate. Furthermore, the coordinated control subsystem controls the operation of the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem, and the data evaluation subsystem during the massive data processing process, thus achieving coordinated control of massive data processing.

[0156] In one embodiment, a system for processing and analyzing massive data is provided, such as Figure 7 As shown, it includes: coordination control subsystem, data splitting subsystem, data comparison and update subsystem, data prediction subsystem and data evaluation subsystem.

[0157] The coordination and control subsystem is used to control the operation of all or individual subsystems and to schedule resources according to the computing load and speed of each subsystem.

[0158] The data splitting subsystem is used to decompose the data to be processed and analyzed into multiple unit data blocks using task information;

[0159] The data comparison and update subsystem is used to obtain the entire data or individual data blocks to be processed and analyzed, and compare the entire data or data blocks with the preset standard data, and then output or send the comparison results to the data prediction subsystem or the data evaluation subsystem;

[0160] The data prediction subsystem is used to classify the overall data or data blocks to be predicted, and use quantitative calculation methods to perform data prediction based on historical prediction data to obtain the data prediction results required for observation or evaluation;

[0161] The data evaluation subsystem is used to obtain data evaluation results based on the comparison and prediction results of the overall data or data blocks and the current task requirements.

[0162] Preferably, when the coordinated control subsystem controls the operation of all subsystems, it allocates each data block to each subsystem for synchronous processing according to the required task objectives based on the order of data processing and analysis; when controlling the operation of a single subsystem, it controls the operation of other subsystems based on the priority and resource conditions of the current task.

[0163] Preferably, the coordination control subsystem is also used to obtain the data volume in the current data processing and analysis task, and analyze it according to the processing speed of each subsystem, split the data volume that exceeds the preset value of the processing time per unit time, and process the split data blocks synchronously.

[0164] Preferably, the data splitting subsystem specifically includes a node identification module, a splitting module, and an association module;

[0165] Node identification module, used to use AI technology to identify key task nodes and data milestone information;

[0166] The splitting module is used to divide the data to be processed and analyzed into multiple data blocks according to the generated key project nodes and project milestones. Each data block starts or ends at a key project node or project milestone;

[0167] The association module provides integrated control of data block comparison, prediction and evaluation based on data visas.

[0168] Preferably, the data comparison and update subsystem specifically includes a data acquisition module, a comparison module, and an update module;

[0169] The data acquisition module is used to directly obtain the data to be processed and analyzed from the outside according to the task information, and send it to the comparison module for subsequent overall data comparison and analysis, or obtain each data block from the data splitting subsystem, send it to the comparison module for subsequent comparison and analysis of each data block, and integrate the analysis results;

[0170] A comparison module is used to automatically compare the current overall data with the preset standard data, generate data comparison results and directly output them for display or send them to the data prediction subsystem or data evaluation subsystem;

[0171] The update module is used to modify the preset standard data used for data comparison according to changes in task information or data prediction evaluation feedback information.

[0172] Preferably, the data prediction subsystem specifically includes a classification editing module and a prediction module;

[0173] The classification editing module is used to classify and edit the entire data or data blocks based on different target requirements, propose a data classification editing list for the current task, and then output the data classification editing list for display or send it to the data evaluation subsystem;

[0174] The prediction module is used to predict data based on the quantitative calculation method, and the quantitative calculation method is the smoothing exponential prediction method.

[0175] Preferably, the data evaluation subsystem includes a group coding module, a model building module, a detection module, and a calculation evaluation module;

[0176] The group coding module is used to group code the entire data or data blocks based on the current task to obtain block coding sequences representing different levels;

[0177] The model building module is used to establish a data evaluation standard model 1 based on the comparison results, and at the same time, establish a data evaluation standard model 2 based on the prediction results.

[0178] Preferably, the specific processing process of calculating and evaluating the entire data or data blocks using the constructed model is as follows:

[0179] (1) Based on the block coding sequence, obtain the comparison result of each data block and calculate the data block coding comparison value using the first calculation formula;

[0180] (2) Based on the block coding sequence, obtain the comparison result of each data block and calculate the data block coding prediction value using the second calculation formula;

[0181] (3) Calculating a data evaluation index 1 and a data evaluation index 2 of the data using a third calculation formula according to the data block coding comparison value and the data block coding prediction value;

[0182] (4) inputting a data evaluation index 1 of one of the data blocks into a data evaluation standard model 1 to obtain a plurality of first data evaluation values, and inputting a data evaluation index 2 of one of the data blocks into a data evaluation standard model 2 to obtain a plurality of second data evaluation values;

[0183] (5) Obtaining a total evaluation value of one of the data blocks, and then determining the number of data blocks that are not within the evaluation threshold range based on a preset evaluation threshold range.

[0184] Preferably, data processing and analysis include data splitting, data comparison and updating, data prediction, and data evaluation in sequence.

[0185] Preferably, a corresponding association between each data block and a processing and analysis task is established through a data visa, and the associated data is sent to one or more of the data comparison and update subsystem, the data prediction subsystem, and the data evaluation subsystem.

[0186] The massive data processing and analysis system provided in this example can improve the accuracy of data segmentation and classification and the rationality of processing and analysis, ensuring the processing efficiency of the entire system. At the same time, the use of coordinated control subsystems and data visas for integrated control achieves coordination and unification of comparison, prediction, and evaluation.

[0187] Furthermore, the massive data processing and analysis system provided by this example can be applied to multiple large-scale projects and scenarios that require massive data analysis and processing, such as data processing and analysis of large-scale engineering projects, analysis and monitoring of large-scale urban operation data, and prediction and evaluation of related data of large-scale regional cluster systems.

[0188] Furthermore, the massive data processing and analysis system provided by this example utilizes a variety of data analysis and processing technologies to enable more comprehensive and accurate analysis and processing of massive amounts of data from large-scale engineering projects, large-scale urban operation data, or data related to large-scale regional cluster systems. Furthermore, it can achieve the three control objectives of comparison, prediction, and evaluation for any engineering project data with project management characteristics.

[0189] Furthermore, the massive data processing and analysis system provided in this example is "mass data-centric, task-oriented, and data-oriented, enabling comprehensive, multi-faceted analysis and processing of large-scale engineering project data." It can support a variety of large-scale tasks and regional operations and control involving massive amounts of data. By continuously improving the data analysis and processing model as tasks evolve, the system can adapt to various operating environments and changing task requirements. Furthermore, the system supports various data processing granularities, enabling analysis depth to be varied, from deep to shallow, and from coarse to fine.

[0190] Furthermore, by applying this system, various data can be analyzed, processed, and monitored in real time, achieving higher precision and accuracy for large-scale tasks. For example, in the control of large-scale engineering projects, after using the system, each data block in the task can be calculated with a predicted value and an evaluation value. Once entered into the system, if the predicted value is exceeded, the system will issue an alarm, and if the evaluation value does not meet the preset range, it will also be output and displayed. This allows large-scale engineering projects to focus on these tasks, complete the corresponding processing or problem tracing in a timely manner, and ultimately achieve the purpose of control.

[0191] According to an embodiment of the present invention, an embodiment of a coordinated control method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0192] In this embodiment, a coordinated control method is provided, which can be used in the coordinated control subsystem 11 in the above-mentioned massive data processing system 1. Figure 8 is a flow chart of a coordinated control method according to an embodiment of the present invention. Figure 8 As shown, the process includes the following steps:

[0193] Step S801: Obtain information on a large amount of data to be processed and a plurality of tasks to be processed.

[0194] Step S802: Based on the massive amount of data to be processed and the information of the multiple tasks to be processed, a coordinated control strategy is obtained through processing using a preset judgment method.

[0195] Specifically, by judging whether the massive data to be processed meets the preset second condition, and judging whether the information of multiple tasks to be processed meets the preset first condition, the coordinated control strategy of the data splitting subsystem 12, data comparison and update subsystem 13, data prediction subsystem 14 and data evaluation subsystem 15 in the massive data processing system 1 can be determined.

[0196] The specific judgment conditions and judgment process refer to the above functional description of the coordination control subsystem 11 and will not be repeated here.

[0197] Step S803 : Coordinate and control the processing flow of the massive data to be processed in the massive data processing system using a coordinated control strategy to obtain a coordinated control result.

[0198] The specific control process refers to the above functional description of the coordination control subsystem 11, data splitting subsystem 12, data comparison and update subsystem 13, data prediction subsystem 14 and data evaluation subsystem 15, which will not be repeated here.

[0199] The coordinated control method provided in this embodiment can determine the coordinated control strategy of the coordinated control subsystem through the judgment result of whether the massive data to be processed meets the preset second condition and the information of multiple tasks to be processed. Furthermore, the coordinated control strategy is used to realize the coordinated control of the processing flow of the massive data to be processed in the massive data processing system.

[0200] This embodiment also provides a coordinated control device for implementing the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0201] This embodiment provides a coordinated control device for executing the coordinated control method provided in the above embodiment of the present invention; Figure 9 As shown, the device includes:

[0202] The acquisition unit 901 is used to acquire information of a large amount of data to be processed and a plurality of tasks to be processed.

[0203] The judgment unit 902 is used to obtain a coordinated control strategy based on the massive data to be processed and the information of multiple tasks to be processed through a preset judgment method.

[0204] The control unit 903 is configured to coordinate and control the processing flow of the massive data to be processed in the massive data processing system by using a coordinated control strategy to obtain a coordinated control result.

[0205] The further functional description of each of the above units is the same as that of the above corresponding embodiments and will not be repeated here.

[0206] The coordination control device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0207] The embodiment of the present invention also provides a computer device having the above Figure 9 The coordinated control device shown.

[0208] See also Figure 10 , Figure 10 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 10 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 10 A processor 10 is taken as an example.

[0209] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0210] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0211] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0212] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0213] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0214] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0215] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0216] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A massive data processing system, characterized in that: The system includes: a coordination and control subsystem, a data splitting subsystem, a data comparison and updating subsystem, a data prediction subsystem and a data evaluation subsystem; The coordination and control subsystem is configured to obtain information on a large amount of data to be processed and a plurality of tasks to be processed, and when the information on the plurality of tasks to be processed satisfies a preset first condition and the large amount of data to be processed satisfies a preset second condition, send the large amount of data to be processed to the data splitting subsystem, and send the information on the plurality of tasks to be processed to the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem, and the data evaluation subsystem; The data splitting subsystem is used to process the massive data to be processed and the multiple task information to be processed to obtain multiple related data blocks, and send the multiple related data blocks to the data comparison and update subsystem, the data prediction subsystem and the data evaluation subsystem; The data comparison and updating subsystem is configured to compare the plurality of associated data blocks with a plurality of preset standard data based on the plurality of to-be-processed task information, obtain data comparison results, and send the data comparison results to the data evaluation subsystem; The data prediction subsystem is configured to perform data prediction using a smoothing index prediction method based on preset target requirements, the plurality of associated data blocks, and the plurality of to-be-processed task information, obtain a data prediction result for the to-be-processed massive data, and send the data prediction result to the data evaluation subsystem; The data evaluation subsystem performs data evaluation based on the plurality of task information to be processed, the plurality of associated data blocks, the data comparison result, and the data prediction result to obtain a data evaluation result of the massive data to be processed; Wherein, the data splitting subsystem includes: a node identification module, a splitting module and an association module; The node identification module is used to use AI technology to identify the multiple pending task information to obtain multiple task key nodes and multiple data milestone information, and send the multiple task key nodes and the multiple data milestone information to the splitting module, and send the multiple pending task information to the association module; The splitting module is configured to split the to-be-processed massive data based on the multiple mission-critical nodes and the multiple data milestone information to obtain multiple data blocks, and send the multiple data blocks to the association module; The association module is configured to associate the plurality of data blocks with the plurality of to-be-processed task information based on the data visa to obtain a plurality of associated data blocks; The data evaluation subsystem includes: a group coding module, a first model building module, a second model building module and a calculation evaluation module; The group coding module is configured to perform group coding on the multiple associated data blocks based on the multiple pieces of information about the tasks to be processed to obtain multiple block coding sequences, and send the multiple block coding sequences to the calculation and evaluation module, where each block coding sequence corresponds to one associated data block; The first model building module is used to establish a first data evaluation standard model based on the data comparison result, and send the first data evaluation standard model to the calculation evaluation module; The second model building module is used to establish a second data evaluation standard model based on the data prediction result, and send the second data evaluation standard model to the calculation evaluation module; The calculation and evaluation module is used to obtain the data evaluation result based on the multiple block coding sequences through a preset calculation method, the first data evaluation standard model and the second data evaluation standard model.

2. The system according to claim 1, wherein: The data comparison and update subsystem includes: a data acquisition module and a comparison module; The data acquisition module is configured to acquire the plurality of preset standard data and send the plurality of preset standard data to the comparison module; The comparison module is used to compare the multiple associated data blocks with the preset standard data to obtain a data comparison result, and send the data comparison result to the data evaluation subsystem.

3. The system according to claim 2, characterized in that The data comparison and update subsystem further includes: An updating module is used to update the plurality of preset standard data based on the plurality of to-be-processed task information.

4. The system according to claim 1, wherein: The data prediction subsystem includes: a classification editing module and a prediction module; The classification and editing module is used to classify and edit the multiple related data blocks according to the preset target requirements to obtain multiple data groups, and send the multiple data groups to the prediction module; The prediction module is used to process the multiple data groups through the smoothing index prediction method based on the multiple task information to be processed, obtain the data prediction results, and send the data prediction results to the data evaluation subsystem.

5. The system according to claim 1, wherein: The calculation and evaluation module includes: a first calculation submodule, a first processing submodule, a second processing submodule, and a second calculation submodule; The first calculation submodule is configured to obtain, based on the multiple block coding sequences and using the preset calculation method, multiple first data evaluation indicators and multiple second data evaluation indicators, and send the multiple first data evaluation indicators to the first processing submodule, and send the multiple second data evaluation indicators to the second processing submodule; The first processing submodule is configured to input the plurality of first data evaluation indicators into the first data evaluation standard model to obtain a plurality of first data evaluation values, and send the plurality of first data evaluation values ​​to the second calculation submodule; the second processing submodule being configured to input the plurality of second data evaluation indicators into the second data evaluation standard model to obtain a plurality of second data evaluation values, and to send the plurality of second data evaluation values ​​to the second calculation submodule; The second calculation submodule is configured to calculate the data evaluation result based on the multiple first data evaluation values ​​and the multiple second data evaluation values.

6. The system according to claim 5, characterized in that The first calculation submodule includes: a first calculation unit, a second calculation unit and a third calculation unit; The first calculation unit is configured to calculate, based on the multiple block code sequences, a preset first relationship to obtain multiple data block code comparison values, and send the multiple data block code comparison values ​​to the second calculation unit; The second calculation unit is configured to calculate, based on the multiple block coding sequences and the multiple data block coding comparison values, a preset second relationship to obtain multiple data block coding prediction values, and send the multiple data block coding prediction values ​​to the third calculation unit; The third calculation unit is used to obtain the multiple first data evaluation indicators and the multiple second data evaluation indicators based on the prediction values ​​of the multiple data block codes through calculation using a preset third relationship.

7. The system according to claim 1, wherein: The data evaluation subsystem further includes: The detection module is used to receive the data evaluation result sent by the calculation and evaluation module, and detect whether the data evaluation result meets the preset standard based on a preset evaluation threshold range to obtain a detection result.

8. The system according to claim 1, wherein: The data comparison and updating subsystem is further configured to send the data comparison result to the data evaluation subsystem via the data prediction subsystem.

9. The system according to claim 1, wherein: The coordination control subsystem is also used to send the massive data to be processed to the data comparison and update subsystem, the data prediction subsystem and the data evaluation subsystem for processing respectively when the multiple task information to be processed meets the preset first condition and the massive data to be processed does not meet the preset second condition, and obtain the data evaluation results of the massive data to be processed.

10. The system according to claim 9, characterized in that The coordination control subsystem includes: an acquisition module, a judgment module, a comparison module and a determination module; an acquisition module, configured to acquire the data volume and data processing speed of the massive data to be processed, and send the data volume to the judgment module, and send the data processing speed to the comparison module; The judging module is configured to judge whether the data volume satisfies a preset third condition and send the judgment result to the determining module; The comparison module is configured to compare the data processing speed with a preset threshold and send the comparison result to the determination module; The determination module is configured to determine whether the massive data to be processed meets the preset second condition based on the judgment result or the comparison result.

11. The system according to claim 10, wherein: The determining module is configured to determine that the massive amount of data to be processed does not satisfy the preset second condition when the judgment result is that the data volume satisfies the preset third condition, or when the comparison result is that the data processing speed is greater than the preset threshold; The determination module is further configured to determine that the massive amount of data to be processed meets the preset second condition when the judgment result is that the data volume does not meet the preset third condition, or the comparison result is that the data processing speed is less than the preset threshold.

12. The system according to claim 1, wherein: The coordination control subsystem is also used to determine the target control subsystem when the multiple pending task information does not meet the preset first condition, and control the data splitting subsystem, the data comparison and update subsystem, the data prediction subsystem and the data evaluation subsystem and other subsystems except the target control subsystem to be paused. The target control subsystem is determined based on the multiple pending task information and is one of the data splitting subsystem, the data comparison and update subsystem, the data prediction subsystem and the data evaluation subsystem.

13. A coordinated control method, characterized in that: Used for coordinating a control subsystem in a massive data processing system according to any one of claims 1 to 12; the method comprising: Obtain information on massive amounts of data to be processed and multiple tasks to be processed; Based on the massive data to be processed and the information of the multiple tasks to be processed, a coordinated control strategy is obtained through processing using a preset judgment method; The coordinated control strategy is used to coordinate and control the processing flow of the massive data to be processed in the massive data processing system to obtain a coordinated control result.

14. A coordinated control device, characterized in that: Used to execute the coordinated control method according to claim 13; the device comprises: An acquisition unit, used to acquire massive data to be processed and information on multiple tasks to be processed; A judgment unit, configured to obtain a coordinated control strategy based on the massive data to be processed and the information of the plurality of tasks to be processed by a preset judgment method; The control unit is used to coordinate and control the processing flow of the massive data to be processed in the massive data processing system by using the coordinated control strategy to obtain a coordinated control result.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the coordinated control method according to claim 13.

16. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the coordinated control method according to claim 13 .

Images