Massive data processing system, coordinated control method and apparatus, and storage medium

Abstract

A massive data processing system (1), a coordinated control method and apparatus, and a storage medium. The system (1) comprises: a coordinated control subsystem (11), a data splitting subsystem (12), a data comparison and update subsystem (13), a data prediction subsystem (14), and a data evaluation subsystem (15). By using the data splitting subsystem (12), the data comparison and update subsystem (13), the data prediction subsystem (14), and the data evaluation subsystem (15) to separately process massive data to be processed, the task objective of implementing control over massive data in three aspects including comparison, prediction and evaluation can be achieved, making processing of the massive data more comprehensive and accurate.

Description

Massive data processing system and coordinated control method, device and storage medium TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a massive data processing system and a coordinated control method, device and storage medium. BACKGROUND

[0002] With the popularity of the Internet, the Internet of Things and mobile devices, the generation of large-scale data is showing an explosive growth trend. These data contain a variety of information, such as large engineering project data, device cluster monitoring data, large road network traffic data, etc. However, how to extract useful information from these massive data has become a great challenge.

[0003] In large-scale data processing, data collection and storage are the first steps. The development of sensor networks, cloud computing and distributed storage systems makes data collection and storage more efficient and reliable. At the same time, with the increase of data volume, data backup and protection become particularly important.

[0004] Large-scale data often contains noise, missing values and outliers, which will cause problems in subsequent data analysis and modeling. Therefore, data cleaning and preprocessing are indispensable steps in large-scale data processing. By using statistical and machine learning methods, data can be cleaned and preprocessed to improve data quality and usability.

[0005] Massive data is a development trend, and data analysis and mining are becoming more and more important. It is important and urgent to extract useful information from massive data, which requires accurate processing, high precision, short processing time and fast access to valuable information. Therefore, the research on massive data is promising and worthy of extensive and in-depth research.

[0006] However, massive data is difficult to process due to large data volume. Therefore, it is urgent to propose a processing method for massive data.

[0007] SUMMARY

[0008] Therefore, the present application provides a massive data processing system and a coordinated control method, device and storage medium to solve the problem of processing difficulty caused by large data volume of massive data.

[0009] In a first aspect, the present application provides a massive data processing system, which comprises a coordinated control subsystem, a data splitting subsystem, a data comparison and updating subsystem, a data prediction subsystem and a data evaluation subsystem.

[0010] The coordination control subsystem is used for obtaining mass data to be processed and a plurality of task information to be processed, and when the plurality of task information to be processed meets a preset first condition and the mass data to be processed meets a preset second condition, the mass data to be processed is sent to the data splitting subsystem, and the plurality of task information to be processed is sent to the data splitting subsystem, the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem; the data splitting subsystem is used for processing the mass data to be processed and the plurality of task information to be processed, obtaining a plurality of associated data blocks, and sending the plurality of associated data blocks to the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem; the data comparison updating subsystem is used for comparing the plurality of associated data blocks with a plurality of preset standard data based on the plurality of task information to be processed, obtaining a data comparison result, and sending the data comparison result to the data evaluation subsystem; the data prediction subsystem is used for predicting data based on a preset target demand, the plurality of associated data blocks and the plurality of task information to be processed by using a smoothing index prediction method, obtaining a data prediction result of the mass data to be processed, and sending the data prediction result to the data evaluation subsystem; and the data evaluation subsystem is used for evaluating data based on the plurality of task information to be processed, the plurality of associated data blocks, the data comparison result and the data prediction result, and obtaining a data evaluation result of the mass data to be processed.

[0011] The mass data processing system provided by the application can process the mass data to be processed by the data splitting subsystem, the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem respectively, can realize the task target of three controls of comparison, prediction and evaluation of mass data, and can make the processing of mass data more comprehensive and accurate. Further, the operation of the data splitting subsystem, the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem in the mass data processing process can be controlled by the coordination control subsystem, and the coordination control of mass data processing is realized.

[0012] In an alternative embodiment, the data splitting subsystem comprises a node identification module, a splitting module and an association module.

[0013] The node identification module is used for identifying the plurality of task information to be processed by using an AI technology, obtaining a plurality of task key nodes and a plurality of data milestone information, and sending the plurality of task key nodes and the plurality of data milestone information to the splitting module and sending the plurality of task information to be processed to the association module; the splitting module is used for splitting the mass data to be processed based on the plurality of task key nodes and the plurality of data milestone information, obtaining a plurality of data blocks, and sending the plurality of data blocks to the association module; and the association module is used for associating the plurality of data blocks with the plurality of task information to be processed based on data certification, and obtaining a plurality of associated data blocks.

[0014] The application can obtain multiple task key nodes and multiple data milestone information through AI technology, further, the overall condition of the massive data to be processed can be comprehensively and accurately reflected through the multiple data milestone information, further, the accuracy of subsequent data splitting and classification and the rationality of processing analysis can be improved, and data support is provided for subsequent data prediction, comparison and other processing.

[0015] In an optional implementation, the data comparison updating subsystem comprises a data acquisition module and a comparison module.

[0016] The data acquisition module is configured to acquire multiple preset standard data and send the multiple preset standard data to the comparison module.

[0017] In an optional implementation, the data comparison updating subsystem further comprises an updating module configured to update the multiple preset standard data based on the multiple to-be-processed task information.

[0018] The application can update the corresponding preset standard data in real time according to the received multiple to-be-processed task information.

[0019] In an optional implementation, the data prediction subsystem comprises a classification editing module and a prediction module.

[0020] The classification editing module is configured to classify and edit the multiple associated data blocks according to a preset target demand to obtain multiple data groups and send the multiple data groups to the prediction module.

[0021] The application can facilitate subsequent data prediction by classifying and editing the associated data blocks through the classification editing module, and further, the prediction processing of the massive data can be realized by combining the smooth exponential prediction method.

[0022] In an optional implementation, the data evaluation subsystem comprises a grouping encoding module, a first model construction module, a second model construction module and a calculation evaluation module.

[0023] The packet coding module is configured to group and encode a plurality of associated data blocks based on a plurality of to-be-processed task information, to obtain a plurality of block coding sequences, and to send the plurality of block coding sequences to the calculation and evaluation module, each block coding sequence corresponding to an associated data block; the first model construction module is configured to establish a first data evaluation standard model based on the data comparison result, and to send the first data evaluation standard model to the calculation and evaluation module; the second model construction module is configured to establish a second data evaluation standard model based on the data prediction result, and to send the second data evaluation standard model to the calculation and evaluation module; and the calculation and evaluation module is configured to obtain a data evaluation result by processing the plurality of block coding sequences through a pre-designed calculation method, the first data evaluation standard model and the second data evaluation standard model.

[0024] The first data evaluation standard model and the second data evaluation standard model are constructed, so that the to-be-processed massive data can be comprehensively analyzed from different angles, and the accuracy of the data evaluation result can be improved.

[0025] In an optional implementation, the calculation and evaluation module comprises a first calculation submodule, a first processing submodule, a second processing submodule and a second calculation submodule.

[0026] The first calculation submodule is configured to obtain a plurality of first data evaluation indexes and a plurality of second data evaluation indexes based on the plurality of block coding sequences through a pre-designed calculation method, to send the plurality of first data evaluation indexes to the first processing submodule, and to send the plurality of second data evaluation indexes to the second processing submodule; the first processing submodule is configured to input the plurality of first data evaluation indexes into the first data evaluation standard model to obtain a plurality of first data evaluation values, and to send the plurality of first data evaluation values to the second calculation submodule; the second processing submodule is configured to input the plurality of second data evaluation indexes into the second data evaluation standard model to obtain a plurality of second data evaluation values, and to send the plurality of second data evaluation values to the second calculation submodule; and the second calculation submodule is configured to obtain a data evaluation result based on the plurality of first data evaluation values and the plurality of second data evaluation values.

[0027] In an optional implementation, the first calculation submodule comprises a first calculation unit, a second calculation unit and a third calculation unit.

[0028] The first calculation unit is configured to calculate a plurality of data block encoding contrast values based on the plurality of block encoding sequences and a preset first relationship, and send the plurality of data block encoding contrast values to the second calculation unit; the second calculation unit is configured to calculate a plurality of data block encoding prediction values based on the plurality of block encoding sequences and the plurality of data block encoding contrast values and a preset second relationship, and send the plurality of data block encoding prediction values to the third calculation unit; and the third calculation unit is configured to calculate a plurality of first data evaluation indexes and a plurality of second data evaluation indexes based on the plurality of data block encoding prediction values and a preset third relationship.

[0029] In an optional implementation, the data evaluation subsystem further comprises a detection module configured to receive the data evaluation result sent by the calculation evaluation module, and detect whether the data evaluation result meets a preset standard based on a preset evaluation threshold range to obtain a detection result.

[0030] The preset evaluation threshold range can be used to detect whether the data evaluation result meets the preset standard, thereby providing support for subsequent data evaluation.

[0031] In an optional implementation, the data comparison updating subsystem is further configured to send the data comparison result to the data prediction subsystem and the data evaluation subsystem through the data prediction subsystem.

[0032] In an optional implementation, the coordination control subsystem is further configured to send the to-be-processed massive data to the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem for processing when the plurality of to-be-processed task information meets the preset first condition and the to-be-processed massive data does not meet the preset second condition, and obtain a data evaluation result of the to-be-processed massive data.

[0033] The coordination control subsystem can control the operation of the data splitting subsystem, the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem in the massive data processing process, and coordination control of massive data processing is achieved.

[0034] In an optional implementation, the coordination control subsystem comprises an acquisition module, a judgment module, a comparison module and a determination module.

[0035] The acquisition module is configured to acquire a data amount and a data processing speed of the to-be-processed massive data, send the data amount to the judgment module, and send the data processing speed to the comparison module; the judgment module is configured to judge whether the data amount meets a preset third condition, and send a judgment result to the determination module; the comparison module is configured to compare the data processing speed with a preset threshold, and send a comparison result to the determination module; and the determination module is configured to determine whether the to-be-processed massive data meets the preset second condition based on the judgment result or the comparison result.

[0036] The application can determine whether the to-be-processed massive data meets the preset second condition through the data volume or the data processing speed of the to-be-processed massive data, and provides support for subsequent coordination control of the coordination control subsystem.

[0037] In an optional implementation, the determining module is configured to determine that the to-be-processed massive data does not meet the preset second condition when the determination result is that the data volume meets the preset third condition or the comparison result is that the data processing speed is greater than the preset threshold; and the determining module is further configured to determine that the to-be-processed massive data meets the preset second condition when the determination result is that the data volume does not meet the preset third condition or the comparison result is that the data processing speed is less than the preset threshold.

[0038] In an optional implementation, the coordination control subsystem is further configured to determine the target control subsystem when the plurality of to-be-processed task information does not meet the preset first condition, and control all of the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem and the data evaluation subsystem except the target control subsystem to be suspended, the target control subsystem being determined based on the plurality of to-be-processed task information and being one of the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem and the data evaluation subsystem.

[0039] In a second aspect, the application provides a coordination control method, which is used in the coordination control subsystem of the massive data processing system in the first aspect or any of the optional implementations thereof; the method comprises:

[0040] obtaining to-be-processed massive data and a plurality of to-be-processed task information; processing the to-be-processed massive data and the plurality of to-be-processed task information based on a preset determination method to obtain a coordination control strategy; and using the coordination control strategy to perform coordination control on a processing flow of the to-be-processed massive data in the massive data processing system to obtain a coordination control result.

[0041] The coordination control method provided by the application can determine the coordination control strategy of the coordination control subsystem through the determination result of whether the to-be-processed massive data meets the preset second condition and the plurality of to-be-processed task information, and further realizes the coordination control of the processing flow of the to-be-processed massive data in the massive data processing system through the coordination control strategy.

[0042] In a third aspect, the application provides a coordination control device, which is used to execute the coordination control method provided in the second aspect; the device comprises:

[0043] The acquisition unit is configured to acquire the mass data to be processed and the plurality of task information to be processed; the judgment unit is configured to obtain a coordination control strategy by processing the mass data to be processed and the plurality of task information to be processed based on a preset judgment method; and the control unit is configured to utilize the coordination control strategy to perform coordination control on a processing flow of the mass data to be processed in the mass data processing system, and obtain a coordination control result.

[0044] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium has stored computer instructions for causing a computer to execute the coordination control method according to the second aspect.

[0045] In a fifth aspect, a computer program product is provided, and the computer program product includes computer instructions for causing a computer to execute the coordination control method according to the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0047] Fig. 1 is a structural block diagram of a mass data processing system according to an embodiment of the present application;

[0048] Fig. 2 is a structural block diagram of a coordination control subsystem according to an embodiment of the present application;

[0049] Fig. 3 is a structural block diagram of a data splitting subsystem according to an embodiment of the present application;

[0050] Fig. 4 is a structural block diagram of a data comparison and update subsystem according to an embodiment of the present application;

[0051] Fig. 5 is a structural block diagram of a data prediction subsystem according to an embodiment of the present application;

[0052] Fig. 6 is a structural block diagram of a data evaluation subsystem according to an embodiment of the present application;

[0053] Fig. 7 is a structural block diagram of a mass data processing and analysis system according to an embodiment of the present application;

[0054] Fig. 8 is a flowchart of a coordination control method according to an embodiment of the present application;

[0055] Fig. 9 is a structural block diagram of a coordination control device according to an embodiment of the present application;

[0056] Fig. 10 is a schematic diagram of the hardware structure of the computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0058] According to the embodiments of the present application, a mass data processing system is provided, as shown in Fig. 1, which comprises a coordination control subsystem 11, a data splitting subsystem 12, a data comparison updating subsystem 13, a data prediction subsystem 14 and a data evaluation subsystem 15.

[0059] Preferably, as shown in Fig. 2, the coordination control subsystem 11 comprises an acquisition module 111, a judgment module 112, a comparison module 113 and a determination module 114.

[0060] Preferably, as shown in Fig. 3, the data splitting subsystem 12 comprises a node identification module 121, a splitting module 122 and a correlation module 123.

[0061] Preferably, as shown in Fig. 4, the data comparison updating subsystem 13 comprises a data acquisition module 131, a comparison module 132 and an updating module 133.

[0062] Preferably, as shown in Fig. 5, the data prediction subsystem 14 comprises a classification editing module 141 and a prediction module 142.

[0063] Preferably, as shown in Fig. 6, the data evaluation subsystem 15 comprises a grouping coding module 151, a first model construction module 152, a second model construction module 153, a calculation evaluation module 154 and a detection module 155.

[0064] The calculation evaluation module 154 comprises a first calculation sub-module 1541, a first processing sub-module 1542, a second processing sub-module 1543 and a second calculation sub-module 1544.

[0065] Further, the first calculation sub-module 1541 comprises a first calculation unit 15411, a second calculation unit 15412 and a third calculation unit 15413.

[0066] Further, the functions of the devices in the above system are described.

[0067] In particular, the coordination control subsystem 11 can control all or individual subsystems to operate.

[0068] Preferably, the coordination control subsystem 11 is configured to acquire the to-be-processed massive data and the plurality of to-be-processed task information, and send the to-be-processed massive data to the data splitting subsystem and the plurality of to-be-processed task information to the data splitting subsystem 12, the data comparison updating subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15 when the plurality of to-be-processed task information meets a preset first condition and the to-be-processed massive data meets a preset second condition.

[0069] The preset first condition can be used to determine whether other subsystems need to be controlled to operate, and when the plurality of to-be-processed task information meets the preset first condition, it indicates that the current data processing needs the cooperation of all subsystems to complete; the preset second condition can be used to determine whether the to-be-processed massive data needs to be split, and when the to-be-processed massive data meets the preset second condition, it indicates that the to-be-processed massive data needs to be split.

[0070] In particular, the data volume and the data processing speed of the to-be-processed massive data are acquired in the acquisition module 111, and the data volume is sent to the judgment module 112 and the data processing speed is sent to the comparison module 113.

[0071] Further, in the judgment module 112, it can be determined whether the data volume of the to-be-processed massive data meets a preset third condition, and the obtained determination result is sent to the determination module 114.

[0072] The preset third condition is used to reflect the size of the data volume of the to-be-processed massive data.

[0073] In particular, if the data volume of the to-be-processed massive data meets the preset third condition, it indicates that the data volume of the to-be-processed massive data is small; if the data volume of the to-be-processed massive data meets the preset third condition, it indicates that the data volume of the to-be-processed massive data is large.

[0074] Further, in the comparison module 113, the data processing speed is compared with a preset threshold, and the obtained comparison result is sent to the determination module 114.

[0075] Further, in the determination module 114, according to the received determination result and comparison result, it can be determined whether the to-be-processed massive data needs to be split, that is, whether the preset second condition is met.

[0076] In particular, if the determination result is that the data volume meets the preset third condition, or the comparison result is that the data processing speed is greater than the preset threshold, it indicates that the to-be-processed massive data does not need to be split, that is, the to-be-processed massive data does not meet the preset second condition;

[0077] If the judgment result is that the data volume does not satisfy the preset third condition, or the comparison result is that the data processing speed is less than the preset threshold, it indicates that the to-be-processed massive data needs to be split, i.e., the to-be-processed massive data satisfies the preset second condition.

[0078] Further, if the to-be-processed massive data needs to be split, the obtained to-be-processed massive data is sent to the data splitting subsystem 12, and the plurality of to-be-processed task information is sent to the data splitting subsystem 12, the data comparison and updating subsystem 13, the data prediction subsystem 14, and the data evaluation subsystem 15.

[0079] Further, the control data splitting subsystem splits the to-be-processed and analyzed data into a plurality of data blocks, and then performs subsystem batch synchronous control according to the actual order to complete data analysis.

[0080] Preferably, the data splitting subsystem 12 is used to process the to-be-processed massive data and the plurality of to-be-processed task information, obtain a plurality of associated data blocks, and send the plurality of associated data blocks to the data comparison and updating subsystem 13, the data prediction subsystem 14, and the data evaluation subsystem 15.

[0081] First, the plurality of to-be-processed task information is identified by using AI technology in the node identification module 121 to obtain a plurality of task key nodes and a plurality of data milestone information, and the plurality of task key nodes and the plurality of data milestone information are sent to the splitting module 122, and the plurality of to-be-processed task information is sent to the association module 123.

[0082] Specifically, the data milestone information is used to indicate the processing completion of each task key node in the to-be-processed massive data, and there can be multiple, which can comprehensively and accurately reflect the overall data condition, improve the accuracy of data splitting and classification, and the rationality of processing and analysis, and facilitate subsequent data prediction, comparison analysis and other processing.

[0083] Further, the to-be-processed task information can usually give the required processing and analysis parameters, such as system processing speed, required processing time, to-be-completed plan, etc. Therefore, by using AI technology, the required data volume for completing each plan can be analyzed according to the upper and lower limits of the range of the historical processing speed and the required processing time of the obtained system, to determine the positions of each task key node and milestone.

[0084] For example, if the processing time required to complete all plans is longer, more key nodes need to be set to make the data amount in each data block smaller, so that the synchronization processing and analysis can be completed faster in the current system processing speed; on the contrary, if the processing time required to complete one of the plans is shorter and the current system processing speed is faster, fewer key nodes need to be set, and the data amount in each database is larger, so that the synchronization processing and analysis can also be completed faster.

[0085] For example, for data processing and analysis of large engineering projects, data prediction and evaluation based on milestones is a supplement to the prediction and evaluation function of the engineering project data processing and analysis system according to task requirements. Milestone-based prediction and evaluation not only applies to engineering project progress data, but also applies to equipment monitoring data, regional environment data, and other data prediction and evaluation, and can well establish a relationship with the task, making the data prediction and evaluation more in line with reality.

[0086] Secondly, in the splitting module 122, the massive data to be processed is split according to the plurality of task key nodes and the plurality of data milestone information, a plurality of data blocks are obtained, and the plurality of data blocks are sent to the association module. Each data block has a task key node and a data milestone as a starting point or an end point.

[0087] Finally, in the association module 123, the plurality of data blocks can be associated with the plurality of to-be-processed task information by using data certification, a plurality of associated data blocks are obtained, and the plurality of associated data blocks are sent to the data comparison updating subsystem 13 and the data prediction subsystem 14.

[0088] Further, the data blocks can be controlled integrally by comparison, prediction and evaluation through data certification.

[0089] Specifically, in the data certification structure, an association level of to-be-processed and analyzed data and to-be-processed and analyzed data is set, each level corresponds to different tasks and required data, and can include: a data comparison layer that sets data comparison tasks, standard data and to-be-processed and analyzed data association; a data prediction layer that sets data comparison tasks, historical data, prediction models and to-be-processed and analyzed data association; and a data evaluation layer that sets data comparison tasks, evaluation models and to-be-processed and analyzed data association.

[0090] Further, after the massive data to be processed is split into a plurality of data blocks, when the massive data to be processed is obtained and needs to be processed and compared, the required data at the same level is obtained at the same time, and then a plurality of data blocks at this level are obtained.

[0091] By adopting this form of hierarchical association, the multi-dimensional information linkage in the data splitting, comparison, updating, prediction and evaluation of the engineering project data analysis and processing is ensured, the continuity and traceability of the engineering data information association are ensured, and the large-scale massive data integrated control requirements such as large equipment cluster monitoring and large city road network traffic can also be adapted.

[0092] Further, once a task occurs in the same level, the structure cannot be changed, but a new data block processed can be added, and all analysis and processing of the same data block should be included in the same level.

[0093] Preferably, after receiving the plurality of associated data blocks, the plurality of associated data blocks and the plurality of preset standard data are compared based on the plurality of to-be-processed task information in the data comparison and updating subsystem 13 to obtain a data comparison result, and the data comparison result is sent to the data evaluation subsystem 15.

[0094] Specifically, the plurality of preset standard data is obtained in the data acquisition module 131 and sent to the comparison module 132.

[0095] Each associated data block corresponds to a corresponding preset standard data of the associated level.

[0096] Further, the plurality of associated data blocks received in the comparison module 132 are compared with the plurality of preset standard data to generate a corresponding data comparison result, and the data comparison result is sent to the data evaluation subsystem 15.

[0097] Further, the data comparison result can also be sent to the data evaluation subsystem 15 through the data prediction subsystem 14.

[0098] Further, the obtained data comparison result can also be directly output and displayed.

[0099] Further, the plurality of preset standard data obtained can also be updated in real time according to the plurality of to-be-processed task information received in the updating module 133.

[0100] In an example, taking engineering project data processing and analysis as an example, the project progress data or the plurality of equipment monitoring parameter data is obtained in the data acquisition module 131, and the project progress standard data for the project progress data or the monitoring parameter standard data for the equipment monitoring parameter data is obtained.

[0101] Further, when the comparison module 132 compares the project progress data with the project progress standard data, the actual engineering completion amount is calculated according to the accumulation, the engineering advance / lag amount is calculated, the engineering estimated completion time different from the planned engineering completion time is generated according to the engineering advance / lag amount, and the updated project progress and time change are calculated according to the engineering estimated completion time.

[0102] Further, after the comparison, the comparison module 132 determines whether to take corrective measures, identifies the project data changes of the corrective measures, and generates the engineering progress data comparison table according to the updated engineering progress data.

[0103] Further, for the monitoring parameter data of multiple engineering equipment, when the comparison module 132 compares the equipment monitoring parameter data with the monitoring parameter standard data, the parameter advance / lag amount of the equipment monitoring parameter data in the current equipment operation process is generated according to the weighting coefficient, the parameter adjustment data different from the monitoring parameter standard data is generated according to the parameter advance / lag amount, and the updated equipment monitoring parameter data is calculated according to the parameter adjustment data.

[0104] Further, after the comparison, the comparison module 132 determines whether to take corrective measures, identifies the project data changes of the corrective measures, and generates the engineering progress data comparison table according to the updated engineering progress data.

[0105] Preferably, after receiving the plurality of associated data blocks, the data prediction subsystem 14 performs data prediction based on the preset target demand, the plurality of associated data blocks, and the plurality of to-be-processed task information, obtains a data prediction result of the to-be-processed massive data by using a smoothing index prediction method, and sends the data prediction result to the data evaluation subsystem 15.

[0106] First, the plurality of associated data blocks are classified and edited in the classification editing module 141 according to the preset target demand, a plurality of data groups are obtained, and the plurality of data groups are sent to the prediction module 142.

[0107] The preset target demand can be determined according to actual conditions, such as different project progress data, different regional equipment monitoring data in a large-scale project, or driving monitoring data and traffic flow data in a large-scale urban traffic network.

[0108] Further, the classification and editing are to classify and arrange the plurality of associated data blocks according to the preset target demand, form different classified data groups based on a unified model standard, and facilitate subsequent prediction calculation.

[0109] Specifically, according to different preset target requirements, the plurality of associated data blocks are classified and edited, and a data classification and editing list of the current to-be-processed task can be extracted. The data classification and editing list is composed of a plurality of data groups.

[0110] Then, in the prediction module 142, according to the plurality of to-be-processed task information, the plurality of data groups are processed by a smoothing index prediction method to obtain a data prediction result, and the data prediction result is sent to the data evaluation subsystem 15.

[0111] Specifically, the processing process of the smoothing index prediction method includes:

[0112] (1) For the data corresponding to the current to-be-processed task, the predicted value of the classified data group i is calculated, as shown in the following relationship (1):

[0113] In the formula: P i represents the predicted value of the data group i; D i-1 represents the actual value of the previous same task in the historical database; a represents the balance index; y i-1 represents the predicted value of the previous same task in the historical database.

[0114] (2) Calculate the predicted value P*, that is, the data prediction result:

[0115] Specifically, the preset standard value of the i-th data block is P i , and the prediction error is Then the predicted value of the i-1-th data group is as shown in the following relationship (2):

[0116] In the formula: a i represents the degree of similarity, which can be obtained by a similarity calculation formula.

[0117] Further, the predicted value P* is calculated, that is, the average value of the sum of the predicted values of all data groups.

[0118] In which, the sum of the predicted values is shown in the following relationship (3):

[0119] Further, the average value P* of the sum of the predicted values is calculated, as shown in the following relationship (4):

[0120] Preferably, the data evaluation subsystem 15 performs data evaluation according to the received plurality of to-be-processed task information, the plurality of associated data blocks, the data comparison result and the data prediction result, to obtain a data evaluation result of the to-be-processed massive data.

[0121] Firstly, the plurality of associated data blocks are grouped and encoded based on the plurality of to-be-processed task information in the grouping and encoding module 151, a plurality of block encoding sequences are obtained, and the plurality of block encoding sequences are sent to the calculation and evaluation module 154.

[0122] Each block encoding sequence corresponds to an associated data block.

[0123] In an example, taking large-scale engineering project data as an example, engineering project data decomposition is performed to obtain a plurality of unit engineering data blocks, the unit engineering data block is 1-bit, and the code is 1; under the unit engineering data block, a plurality of sub-division engineering data blocks are included, the sub-division engineering data block is 2-bit, plus the unit engineering data block 1-bit, and the encoding bit is 101; (3) under the sub-division engineering, a plurality of sub-item engineering data blocks are included, the sub-item engineering data block is 2-bit plus the unit engineering data block 1-bit plus the sub-division engineering data block 2-bit, and the sub-item engineering code is 10101.

[0124] Secondly, the first data evaluation standard model is established based on the data comparison result in the first model construction module 152, and the first data evaluation standard model is sent to the calculation and evaluation module 154.

[0125] Meanwhile, the second data evaluation standard model is established based on the data prediction result in the second model construction module 153, and the second data evaluation standard model is sent to the calculation and evaluation module 154.

[0126] Finally, the data evaluation result is obtained by processing the plurality of block encoding sequences through the pre-designed calculation method, the first data evaluation standard model and the second data evaluation standard model in the calculation and evaluation module 154.

[0127] Specifically, the plurality of first data evaluation indexes and the plurality of second data evaluation indexes are obtained based on the plurality of block encoding sequences through the pre-designed calculation method in the first calculation sub-module 1541, the plurality of first data evaluation indexes are sent to the first processing sub-module 1542, and the plurality of second data evaluation indexes are sent to the second processing sub-module 1543.

[0128] Firstly, the plurality of data block encoding comparison values are obtained based on the plurality of block encoding sequences through the pre-set first relationship formula calculation in the first calculation unit 15411, and the plurality of data block encoding comparison values are sent to the second calculation unit 15412.

[0129] Specifically, the comparison result of each data block can be obtained according to the plurality of block encoding sequences, and further, the data block encoding comparison value corresponding to each data block can be calculated by using the pre-set first relationship formula, as shown in the following relationship formula (5):

[0130] In the formula, B edenotes the data block encoding contrast value; n denotes the total number of data blocks; F i denotes the encoding of the i-th data block; H denotes the preset classification interval.

[0131] Secondly, based on the plurality of block encoding sequences and the plurality of data block encoding contrast values, a plurality of data block encoding prediction values are obtained by preset second relationship calculation in the second calculation unit 15412, and the plurality of data block encoding prediction values are sent to the third calculation unit 15413.

[0132] Specifically, the contrast result of each data block can be obtained according to the plurality of block encoding sequences, and further, the corresponding data block encoding prediction value can be calculated by using the preset second relationship, as shown in the following relationship (6):

[0133] Z id =1BID+B e (6)

[0134] In the formula: Z id denotes the data block encoding prediction value; BID denotes the task ID corresponding to the data block.

[0135] Finally, based on the plurality of data block encoding prediction values, a plurality of first data evaluation indexes and a plurality of second data evaluation indexes are obtained by preset third relationship calculation in the third calculation unit 15413.

[0136] Specifically, the first data evaluation index and the second data evaluation index corresponding to each data block can be calculated by using the following relationship (7):

[0137] In the formula: A denotes the first data evaluation index; T denotes the second data evaluation index; Y( ) denotes the function of taking the remainder; s1 denotes the evaluation coefficient one; s2 denotes the evaluation coefficient two.

[0138] Further, after the first processing sub-module 1542 receives the plurality of first data evaluation indexes, the plurality of first data evaluation indexes are input into the first data evaluation standard model, and the corresponding plurality of first data evaluation values C j are obtained, and the plurality of first data evaluation values C j are sent to the second calculation sub-module 1544.

[0139] At the same time, after the second processing sub-module 1543 receives the plurality of second data evaluation indexes, the plurality of second data evaluation indexes are input into the second data evaluation standard model, and the corresponding plurality of second data evaluation values D j are obtained, and the plurality of second data evaluation values D j are sent to the second calculation sub-module 1544.

[0140] Further, the second calculation sub-module 1544 receives a plurality of first data evaluation values C j and a plurality of second data evaluation values D j Then, a total evaluation value V of each data block can be calculated j = C j + D j According to the total evaluation value of each data block, the data evaluation result of the to-be-processed massive data can be obtained.

[0141] Further, the calculation evaluation module 154 can send the obtained data evaluation result to the detection module 155.

[0142] Further, after receiving the data evaluation result, the detection module 155 can detect whether the data evaluation result meets the preset standard based on a preset evaluation threshold range, to obtain a detection result.

[0143] Specifically, according to the preset evaluation threshold range, the number of data blocks that are not within the preset evaluation threshold range can be determined. If the number is less than 5% of the number of data blocks, it is considered that the data evaluation result meets the standard, otherwise, if the number is greater than or equal to 5% of the number of data blocks, it is considered that the data evaluation result does not meet the standard.

[0144] Preferably, the coordination control subsystem 11 is further configured to, when the plurality of to-be-processed task information meets the preset first condition and the to-be-processed massive data does not meet the preset second condition, send the to-be-processed massive data to the data comparison and update subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15 respectively for processing, and obtain a data evaluation result of the to-be-processed massive data.

[0145] Specifically, if the plurality of to-be-processed task information meets the preset first condition and the to-be-processed massive data does not meet the preset second condition, it indicates that the to-be-processed massive data does not need to be split, i.e., the data splitting subsystem 12 does not need to be run. At this time, the obtained to-be-processed massive data is directly sent to the data comparison and update subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15 for processing, and the final data evaluation result is output. The specific processing process is described above with reference to the functions of the data comparison and update subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15, which will not be described here.

[0146] Preferably, the coordination control subsystem 11 is further configured to, when the plurality of to-be-processed task information does not meet the preset first condition, determine a target control subsystem, and control all subsystems other than the target control subsystem to be suspended.

[0147] The target control subsystem is determined based on the plurality of to-be-processed task information.

[0148] Specifically, when the plurality of to-be-processed task information does not satisfy a preset first condition, it indicates that the current data processing needs to separately control one subsystem, i.e., the target control subsystem, to run, at this time, the running of other subsystems except the target control subsystem is temporarily stopped.

[0149] Further, the coordination control subsystem 11 can receive the running results (computing load and speed) of each subsystem, and can perform resource scheduling according to the running results of each subsystem to ensure the processing efficiency of the overall system.

[0150] The mass data processing system provided in the embodiment can process the to-be-processed mass data by the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem and the data evaluation subsystem, can achieve the task target of three controls of comparison, prediction and evaluation of mass data, and makes the processing of mass data more comprehensive and accurate. Further, the running of the data splitting subsystem, the data comparison and updating subsystem, the data prediction subsystem and the data evaluation subsystem in the mass data processing process can be controlled by the coordination control subsystem, and the coordination control of mass data processing is achieved.

[0151] In an example, a mass data processing and analysis system is provided, as shown in FIG. 7, comprising a coordination control subsystem, a data splitting subsystem, a data comparison and updating subsystem, a data prediction subsystem and a data evaluation subsystem.

[0152] The coordination control subsystem is configured to control the running of each subsystem all together or separately, and perform resource scheduling according to the computing load and speed of each subsystem.

[0153] The data splitting subsystem is configured to split the to-be-processed and analyzed data into a plurality of unit data blocks by using task information.

[0154] The data comparison and updating subsystem is configured to obtain the whole or each data block of the to-be-processed and analyzed data, compare the whole or each data block with preset standard data, and then output or send the comparison result to the data prediction subsystem or the data evaluation subsystem.

[0155] The data prediction subsystem is configured to classify the whole or each data block to be predicted, and perform data prediction by using a quantitative calculation method based on historical prediction data to obtain the data prediction result required for observation or evaluation.

[0156] A data evaluation subsystem is configured to analyze the data based on the comparison result and the prediction result of the overall data or the data blocks, and based on the current task requirement, to obtain a data evaluation result.

[0157] Preferably, the coordination control subsystem is configured to, when controlling the overall subsystem to operate, allocate the data blocks to the respective subsystems for synchronous processing according to the required task target based on the data processing and analysis sequence; and when controlling the individual subsystem to operate, control the operation of the other subsystems based on the priority of the current task and the resource condition.

[0158] Preferably, the coordination control subsystem is further configured to obtain the data volume in the current data processing and analysis task, and analyze the data volume based on the processing speed of the respective subsystems, and split the data volume that exceeds the preset processing time per unit time, and synchronously process the split data blocks.

[0159] Preferably, the data splitting subsystem specifically comprises a node identification module, a splitting module, and an association module.

[0160] The node identification module is configured to call an AI technology to identify the task critical node and the data milestone information.

[0161] The splitting module is configured to divide the overall data to be processed and analyzed into a plurality of data blocks based on the generated engineering critical node and the project milestone, and each data block takes the engineering critical node and the project milestone as the starting point or the end point.

[0162] The association module is configured to integrally control the comparison, prediction, and evaluation of the data blocks based on the data certification.

[0163] Preferably, the data comparison updating subsystem specifically comprises a data acquisition module, a comparison module, and an updating module.

[0164] The data acquisition module is configured to directly acquire the data to be processed and analyzed from the outside based on the task information, and send the data to the comparison module for subsequent overall data comparison analysis, or acquire the respective data blocks from the data splitting subsystem, and send the data blocks to the comparison module for subsequent comparison analysis of the respective data blocks, and integrate the analysis results.

[0165] The comparison module is configured to automatically compare the current overall data with the preset standard data, generate a data comparison result, and directly output and display the data comparison result or send the data comparison result to the data prediction subsystem or the data evaluation subsystem.

[0166] The updating module is configured to modify the preset standard data used for data comparison based on the change of the task information or the feedback information of the data prediction and evaluation.

[0167] Preferably, the data prediction subsystem specifically comprises a classification editing module and a prediction module.

[0168] a classification editing module, configured to edit the overall data or the data blocks based on different target requirements, to propose a data classification editing list of the current task, and to subsequently output and display or send the data classification editing list to a data evaluation subsystem;

[0169] a prediction module, configured to predict the data based on a quantitative calculation method, which is a smooth exponential prediction method.

[0170] Preferably, the data evaluation subsystem comprises a grouping coding module, a model construction module, a detection module, and a calculation evaluation module.

[0171] The grouping coding module is configured to group and code the overall data or the data blocks based on the current task, to obtain a block coding sequence representing different levels.

[0172] The model construction module is configured to establish a data evaluation standard model one based on the comparison results, and to establish a data evaluation standard model two based on the prediction results.

[0173] Preferably, the specific processing procedure of the calculation evaluation of the overall data or the data blocks based on the constructed model is as follows:

[0174] (1) Based on the block coding sequence, the comparison results of each data block are obtained, and a data block coding comparison value is calculated by using a first calculation formula.

[0175] (2) Based on the block coding sequence, the comparison results of each data block are obtained, and a data block coding prediction value is calculated by using a second calculation formula.

[0176] (3) According to the data block coding comparison value and the data block coding prediction value, a data evaluation index one and a data evaluation index two of the data are calculated by using a third calculation formula.

[0177] (4) The data evaluation index one of one data block is input into the data evaluation standard model one to obtain a plurality of data first evaluation values, and the data evaluation index two of one data block is input into the data evaluation standard model two to obtain a plurality of data second evaluation values.

[0178] (5) A total evaluation value of one data block is obtained, and then, based on a preset evaluation threshold range, the number of data blocks not in the evaluation threshold range is determined.

[0179] Preferably, the data processing and analysis comprises data splitting, data comparison updating, data prediction, and data evaluation in sequence.

[0180] Preferably, each data block is associated with a processing and analysis task by data visa, and the associated data is sent to one or more of the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem.

[0181] The mass data processing and analysis system provided by the present example can improve the accuracy of data splitting and classification and the rationality of processing and analysis, ensure the processing efficiency of the overall system, and simultaneously realize the coordination and unity of comparison, prediction and evaluation by using the integrated control of the coordination control subsystem and the data visa.

[0182] Further, the mass data processing and analysis system provided by the present example can be applied to a plurality of large projects and scenarios requiring mass data analysis and processing, such as large engineering project data processing and analysis, large city operation data analysis and monitoring, and large-scale regional cluster system related data prediction and evaluation.

[0183] Further, the mass data processing and analysis system provided by the present example can be applied to a plurality of large projects and scenarios requiring mass data analysis and processing, such as large engineering project data processing and analysis, large city operation data analysis and monitoring, and large-scale regional cluster system related data prediction and evaluation.

[0184] Further, the mass data processing and analysis system provided by the present example is "centered on mass data, oriented to task demand, oriented to data analysis and processing, and performs comprehensive and multi-angle analysis and processing on large engineering project data", and can support a variety of large tasks and regional operation and control of mass data. With the development of the task, the data analysis and processing model can be continuously improved, and the system can adapt to changes in various operating environments and task requirements and operate normally; at the same time, the system supports various data processing granularities, and the analysis depth can be deep or shallow, coarse or fine.

[0185] Further, by applying the present system, real-time analysis, processing and monitoring of various data can be achieved, thereby achieving higher precision and accuracy in large task control. For example, in large engineering project control, after using the system, each data block in the task can calculate a predicted value and an evaluation value, and once it is found that the predicted value exceeds the system will alarm, or the evaluation value does not meet the preset range, the system will output and display, so that the focus of large engineering projects is concentrated on these tasks, and timely processing or problem tracing is completed, and finally the control purpose is achieved.

[0186] According to the embodiment of the present application, a coordination control method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0187] In the present embodiment, a coordination control method is provided, which can be used in the coordination control subsystem 11 of the mass data processing system 1 described above. FIG. 7 is a flowchart of the coordination control method according to the embodiment of the present application. As shown in FIG. 8, the flowchart includes the following steps:

[0188] In step S801, the to-be-processed mass data and the plurality of to-be-processed task information are obtained.

[0189] In step S802, based on the to-be-processed mass data and the plurality of to-be-processed task information, a preset judgment method is used to process to obtain a coordination control strategy.

[0190] Specifically, by judging whether the to-be-processed mass data satisfies a preset second condition and judging whether the plurality of to-be-processed task information satisfies a preset first condition, the coordination control strategy for the data splitting subsystem 12, the data comparison and updating subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15 in the mass data processing system 1 can be determined.

[0191] The specific judgment conditions and judgment processes are described above with reference to the function description of the coordination control subsystem 11, and will not be described here.

[0192] In step S803, the coordination control strategy is used to coordinate and control the processing flow of the to-be-processed mass data in the mass data processing system to obtain a coordination control result.

[0193] The specific control process is described above with reference to the function description of the coordination control subsystem 11, the data splitting subsystem 12, the data comparison and updating subsystem 13, the data prediction subsystem 14 and the data evaluation subsystem 15, and will not be described here.

[0194] The coordination control method provided in the present embodiment can determine the coordination control strategy of the coordination control subsystem through the judgment result of whether the to-be-processed mass data satisfies the preset second condition and the plurality of to-be-processed task information. Further, the coordination control method realizes the coordination control of the processing flow of the to-be-processed mass data in the mass data processing system.

[0195] In the embodiment, a coordination control device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and the description of which has been made above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0196] The embodiment provides a coordination control device, which is used to execute the coordination control method provided by the above-mentioned embodiments of the application. As shown in FIG. 9, the device comprises:

[0197] An acquisition unit 901 is configured to acquire the to-be-processed massive data and the plurality of to-be-processed task information.

[0198] A judgment unit 902 is configured to obtain a coordination control strategy by processing the to-be-processed massive data and the plurality of to-be-processed task information based on a preset judgment method.

[0199] A control unit 903 is configured to perform coordination control on a processing flow of the to-be-processed massive data in the massive data processing system by using the coordination control strategy, and obtain a coordination control result.

[0200] Further function descriptions of the above-mentioned units are the same as those of the corresponding embodiments, and will not be described here.

[0201] The coordination control device in the embodiment is presented in the form of a functional unit, and the unit herein refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0202] The embodiment of the application also provides a computer device with the above-mentioned coordination control device shown in FIG. 9.

[0203] Referring to FIG. 10, FIG. 10 is a structural diagram of a computer device according to an optional embodiment of the present application. As shown in FIG. 10, the computer device includes one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other using different buses, and can be mounted on a common main board or mounted in other manners as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory banks, if needed. Also, multiple computer devices can be connected, each providing part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). One processor 10 is taken as an example in FIG. 10.

[0204] The processor 10 can be a central processor, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0205] The memory 20 stores instructions executable by the at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiments.

[0206] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the computer device, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some optional embodiments, the memory 20 can optionally include a memory remotely arranged with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0207] The memory 20 can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state disk; and the memory 20 can further include a combination of the above kinds of memories.

[0208] The computer device also comprises a communication interface 30 for communication of the computer device with other devices or communication networks.

[0209] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded from a network and stored in a local storage medium by computer code, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that the computer, processor, microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor or hardware, the method shown in the above embodiments is implemented.

[0210] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file, etc., and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0211] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A mass data processing system, characterized by, The system comprises a coordination control subsystem, a data splitting subsystem, a data comparison updating subsystem, a data prediction subsystem and a data evaluation subsystem; The coordination control subsystem is configured to acquire massive to-be-processed data and a plurality of to-be-processed task information, and when the plurality of to-be-processed task information meets a preset first condition and the massive to-be-processed data meets a preset second condition, send the massive to-be-processed data to the data splitting subsystem, and send the plurality of to-be-processed task information to the data splitting subsystem, the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem; The data splitting subsystem is configured to process the massive to-be-processed data and the plurality of to-be-processed task information to obtain a plurality of associated data blocks, and send the plurality of associated data blocks to the data comparison updating subsystem, the data prediction subsystem and the data evaluation subsystem; The data comparison updating subsystem is configured to compare the plurality of associated data blocks with a plurality of preset standard data based on the plurality of to-be-processed task information to obtain a data comparison result, and send the data comparison result to the data evaluation subsystem; The data prediction subsystem is configured to perform data prediction on the massive to-be-processed data based on a preset target demand, the plurality of associated data blocks and the plurality of to-be-processed task information by using a smoothing index prediction method to obtain a data prediction result of the massive to-be-processed data, and send the data prediction result to the data evaluation subsystem; The data evaluation subsystem is configured to perform data evaluation on the massive to-be-processed data based on the plurality of to-be-processed task information, the plurality of associated data blocks, the data comparison result and the data prediction result to obtain a data evaluation result of the massive to-be-processed data.

2. The system of claim 1, wherein, The data splitting subsystem comprises a node identification module, a splitting module and an association module; The node identification module is configured to identify the plurality of to-be-processed task information by using an AI technology to obtain a plurality of task key nodes and a plurality of data milestone information, and send the plurality of task key nodes and the plurality of data milestone information to the splitting module, and send the plurality of to-be-processed task information to the association module; The splitting module is configured to split the massive to-be-processed data based on the plurality of task key nodes and the plurality of data milestone information to obtain a plurality of data blocks, and send the plurality of data blocks to the association module; The association module is configured to associate the plurality of data blocks with the plurality of to-be-processed task information based on data certification to obtain a plurality of associated data blocks.

3. The system of claim 1, wherein, The data comparison updating subsystem comprises a data acquisition module and a comparison module; The data acquisition module is configured to acquire the plurality of preset standard data, and send the plurality of preset standard data to the comparison module; The comparison module is configured to compare the plurality of associated data blocks with the preset standard data to obtain a data comparison result, and send the data comparison result to the data evaluation subsystem.

4. The system of claim 3, wherein, The data comparison updating subsystem further comprises: The update module is used to update the multiple preset standard data based on the multiple tasks to be processed information.

5. The system of claim 1, wherein, The data prediction subsystem includes: a classification editing module and a prediction module; The classification and editing module is used to classify and edit the multiple related data blocks according to the preset target requirements, obtain multiple data groups, and send the multiple data groups to the prediction module; The prediction module is used to process the multiple data groups based on the multiple task information to obtain the data prediction result through the smoothing exponential prediction method, and send the data prediction result to the data evaluation subsystem.

6. The system of claim 1, wherein, The data evaluation subsystem includes: a group coding module, a first model construction module, a second model construction module, and a calculation evaluation module; The group coding module is used to group and code the multiple associated data blocks based on the multiple task information to obtain multiple block coding sequences, and send the multiple block coding sequences to the calculation and evaluation module, with each block coding sequence corresponding to one associated data block; The first model building module is used to establish a first data evaluation standard model based on the data comparison results, and send the first data evaluation standard model to the calculation evaluation module; The second model building module is used to establish a second data evaluation standard model based on the data prediction results, and send the second data evaluation standard model to the calculation evaluation module; The calculation and evaluation module is used to obtain the data evaluation result based on the multiple block encoding sequences through a preset calculation method, the first data evaluation standard model, and the second data evaluation standard model.

7. The system of claim 6, wherein, The calculation and evaluation module includes: a first calculation submodule, a first processing submodule, a second processing submodule, and a second calculation submodule; The first calculation submodule is used to obtain multiple first data evaluation indicators and multiple second data evaluation indicators based on the multiple block coding sequences and through the preset calculation method, and send the multiple first data evaluation indicators to the first processing submodule and the multiple second data evaluation indicators to the second processing submodule. The first processing submodule is used to input the plurality of first data evaluation indicators into the first data evaluation standard model to obtain a plurality of first data evaluation values, and to send the plurality of first data evaluation values ​​to the second calculation submodule; The second processing submodule is used to input the plurality of second data evaluation indicators into the second data evaluation standard model to obtain a plurality of second data evaluation values, and to send the plurality of second data evaluation values ​​to the second calculation submodule; The second calculation submodule is used to calculate the data evaluation result based on the plurality of first data evaluation values ​​and the plurality of second data evaluation values.

8. The system of claim 7, wherein, The first computing submodule includes: a first computing unit, a second computing unit, and a third computing unit; The first calculation unit is used to calculate multiple data block encoding comparison values ​​based on the multiple block encoding sequences through a preset first relational formula, and send the multiple data block encoding comparison values ​​to the second calculation unit; The second calculation unit is used to calculate multiple data block encoding prediction values ​​based on the multiple block encoding sequences and the multiple data block encoding comparison values ​​through a preset second relational formula, and then send the multiple data block encoding prediction values ​​to the third calculation unit; The third calculation unit is used to obtain the multiple first data evaluation indicators and the multiple second data evaluation indicators based on the predicted values ​​encoded by the multiple data blocks and through calculation using a preset third relational formula.

9. The system of claim 6, wherein, The data evaluation subsystem also includes: The detection module is used to receive the data evaluation result sent by the calculation and evaluation module, and detect whether the data evaluation result meets the preset standard based on the preset evaluation threshold range, and obtain the detection result.

10. The system according to claim 1, characterized in that, The data comparison and update subsystem is also used to apply the data comparison results to the data prediction. The data is sent from the subsystem to the data evaluation subsystem.

11. The system according to claim 1, characterized in that, The coordination and control subsystem is further configured to, when the multiple pending task information meets the preset first condition and the pending massive data does not meet the preset second condition, send the pending massive data to the data comparison and update subsystem, the data prediction subsystem, and the data evaluation subsystem for processing, and obtain the data evaluation result of the pending massive data.

12. The system of claim 11, wherein, The coordination and control subsystem includes: an acquisition module, a judgment module, a comparison module, and a determination module; The acquisition module is used to acquire the data volume and data processing speed of the massive data to be processed, and send the data volume to the judgment module and the data processing speed to the comparison module; The judgment module is used to determine whether the data volume meets the preset third condition, and send the judgment result to the determination module; The comparison module is used to compare the data processing speed with a preset threshold and send the comparison result to the determination module; The determining module is used to determine whether the massive amount of data to be processed meets the preset second condition based on the judgment result or the comparison result.

13. The system according to claim 12, characterized in that, The determining module is used to determine that the massive amount of data to be processed does not meet the preset second condition when the judgment result is that the amount of data meets the preset third condition, or the comparison result is that the data processing speed is greater than the preset threshold. The determining module is further configured to determine that the massive amount of data to be processed meets the preset second condition when the judgment result is that the amount of data does not meet the preset third condition, or the comparison result is that the data processing speed is less than the preset threshold.

14. The system according to claim 1, characterized in that, The coordination and control subsystem is further configured to determine a target control subsystem when the multiple pending task information does not meet a preset first condition, and to control all subsystems in the data splitting subsystem, the data comparison and update subsystem, the data prediction subsystem, and the data evaluation subsystem except for the target control subsystem to be suspended. The target control subsystem is determined based on the multiple pending task information and is one of the data splitting subsystem, the data comparison and update subsystem, the data prediction subsystem, and the data evaluation subsystem.

15. A method of coordinated control, characterized by A coordinated control subsystem within a massive data processing system as described in any one of claims 1 to 14; the method includes: Acquire information on massive amounts of data to be processed and multiple tasks to be processed; Based on the massive amount of data to be processed and the information of the multiple tasks to be processed, a coordination and control strategy is obtained after processing by a preset judgment method. The coordination and control strategy is used to coordinate and control the processing flow of the massive data to be processed within the massive data processing system, and the coordination and control results are obtained.

16. A coordinated control device characterized by comprising: For performing the coordinated control method as described in claim 15; the apparatus includes: The acquisition unit is used to acquire massive amounts of data to be processed and information on multiple tasks to be processed. The judgment unit is used to obtain a coordination control strategy based on the massive amount of data to be processed and the information of the multiple tasks to be processed, through a preset judgment method. The control unit is used to coordinate and control the processing flow of the massive data to be processed within the massive data processing system using the coordination and control strategy, and to obtain the coordination and control result.

17. A computer readable storage medium characterized by: The computer-readable storage medium stores computer instructions for causing the computer to perform the coordination control method of claim 15.

18. A computer program product, characterised in that, Includes computer instructions for causing a computer to perform the coordinated control method of claim 15.

Images