Data acquisition automatic back-filling system and method based on multi-dimensional data continuity verification

Abstract

The invention discloses a data acquisition automatic compensation system based on multi-dimensional data continuity verification, and the system comprises a data acquisition unit which is used for transmitting original data acquired from a sensor to a data acquisition platform through an Internet of Things card or wifi, and transmitting the original data to a business system at the same time; the data acquisition platform is used for receiving the original data uploaded from the data acquisition unit, storing the original data in a time sequence database, performing preliminary analysis and calculation on the original data, and transmitting a calculation result, namely relative data, to the service system; and the business system is used for receiving the original data and the relative data at the same time, further analyzing and calculating the data, presenting results in forms of reports, tables and reports for final users, and storing task results needing to be supplemented into the relational database. The invention also discloses a data acquisition automatic back-filling method based on multi-dimensional data continuity verification. According to the invention, the cost is reduced, the flexibility is improved, and the compensation condition is comprehensive.

Description

Technical Field

[0001] The invention relates to a data acquisition automatic replenishment system and method based on multi-dimensional data continuity verification. Background Art

[0002] In traditional data collection of industrial equipment, the data collection equipment will upload the data to its own collection platform and push the original real-time data to the business system. However, the data transmission of industrial equipment (such as wind turbines and photovoltaic panels) is frequently interrupted, and the system operation of the collection platform is unstable, with frequent data loss and data acquisition delays. In the case of data transmission interruption or untimely data provision, if the data is not replenished in time, it will have a serious impact on the center's analysis, calculation, and statistics of the data. At present, there are many data replenishment technologies, which are mainly based on the working status of the data collection equipment, using periodic replenishment or timed replenishment methods, combined with manual operation for post-replenishment. This method has the following main problems: Poor real-time performance: Data interruption is not discovered in a timely manner. When the data interruption is discovered, the center has already started analysis and calculation, resulting in wrong data calculation results or errors.

[0003] Inaccurate: Data interruption is judged only by the working status of the data collector or other single conditions, which results in omissions, incomplete consideration of working conditions, and failure to consider many data interruptions, leading to inaccurate results. Some data interruptions are not judged and are considered complete.

[0004] Not smart enough: Due to errors in judgment, manual intervention is often required to check multiple conditions based on human experience.

[0005] High cost: Due to frequent manual access, the labor cost is high.

[0006] The log is not detailed: it is impossible to record the data at the time when the problem occurred in time, it is impossible to trace back to reproduce the problem, it is impossible to analyze the data interruption problem, it is impossible to retransmit the data, and there is no data left for future reference to form a knowledge and experience base.

[0007] Not flexible enough: Data replenishment may fail or cause other problems. You may need to check the log message to analyze the problem and try again.

[0008] Therefore, in order to solve the above problems, a data acquisition automatic backfill system and method based on multi-dimensional data continuity verification is provided. Summary of the invention

[0009] The purpose of the present invention is to overcome the existing defects and provide a data acquisition automatic backfill system and method based on multi-dimensional data continuity verification, which reduces costs, improves flexibility, and provides comprehensive backfill conditions.

[0010] The technical solution to achieve the above purpose is: The data acquisition automatic backfilling system based on multi-dimensional data continuity verification of one embodiment of the present invention comprises: Data collector, used to send the raw data collected from the sensor to the data collection platform through the IoT card or wifi, and transmit it to the business system at the same time; The data collection platform is used to receive the raw data uploaded from the data collector, store it in the time series database, perform preliminary analysis and calculation on the raw data, and transmit the calculation results, i.e., relative data, to the business system; The business system is used to receive both raw data and relative data, further analyze and calculate the data, present the results to the end user in the form of tables, tables, and reports, and save the task results that need to be supplemented in the relational database; The data replenishment module is used to create a replenishment task based on the analysis results of the relative data and the data quality of the real-time data detection, replenish the relative data and real-time data, and store the replenished data in the time series database; Notification module, used to notify users of the status of the replenishment task; The task check module is used to perform the data integrity check task of the previous day at the configured time interval.

[0011] Preferably, the data collection platform includes: A configuration unit, used to configure a plurality of reference measurement points for checking the original data, and to configure a threshold value for a data missing time period; The first reading unit is used to read all time series data of the reference measuring point one day before.

[0012] Preferably, the business system includes: The analysis unit is used to analyze the missing time period of the time series data, check the time interval between two adjacent time points of all data, and if it exceeds the threshold, it is regarded as a data interruption, and the continuous period is calculated as the data filling time period; The checking unit is used to check the continuity of the original data in real time. If two adjacent real-time data exceed the threshold of the data missing time period, it is considered that the data is discontinuous and needs to be supplemented; The monitoring unit is used to monitor the original data and the relative data transmission status. If an abnormality occurs, it will be treated as no data, and the time interval will be recorded and the need for replenishment will be recorded; The replenishment task creation unit is used to create corresponding replenishment tasks for the data that needs to be replenished in the inspection unit and the monitoring unit, and save the replenishment task information to the relational database.

[0013] Preferably, in the monitoring unit, the anomalies include: connection interruption, transmission protocol error, device failure and interface call error.

[0014] Preferably, the data replenishment module includes: The second reading unit is used to retrieve the replenishment task information from the relational database and use multi-threading to concurrently execute multiple tasks; The thread pool unit is used to use multiple threads to concurrently execute data replenishment tasks, read the original data and relative data that need to be replenished from the business system, write them into the time series database, and write the execution log into the relational database to record the task status and statistical information; The update unit is used to update the relational database replenishment task information.

[0015] Preferably, in the notification module, notification channels include but are not limited to in-site letters, emails, text messages and WeChat.

[0016] The second method of the present invention for automatically replenishing data collection based on multi-dimensional data continuity verification includes: Step S1, the data collector sends the raw data collected from the sensor to the data collection platform through the Internet of Things card or wifi, and transmits it to the business system at the same time; Step S2, the data collection platform receives the raw data uploaded from the data collector, stores it in the time series database, performs preliminary analysis and calculation on the raw data, and transmits the calculation results, i.e., relative data, to the business system; Step S3, the business system receives the original data and relative data at the same time, performs further analysis and calculation on the data, presents the results to the end user in the form of statements, tables, and reports, and saves the task results that need to be supplemented in the relational database; Step S4: The data replenishment module creates a replenishment task based on the analysis results of the relative data and the data quality of the real-time data detection, automatically replenishes the relative data and the real-time data, and stores the replenished data in the time series database; Step S5: In addition to automatic data replenishment, manual data replenishment is supported. The user manually selects the measurement point and start and end time to create a task for replenishment; Step S6: the notification module notifies the user of the task status until all tasks are completed.

[0017] Preferably, step S2 comprises: Step S21, configuring a plurality of reference measuring points for inspection, and configuring a threshold value for a data missing time period; Step S22, reading all the time series data of the previous day of the benchmark measuring point, performing preliminary analysis and calculation, and transmitting the calculation results, i.e., relative data, to the business system; In step S2, before the task is executed on the next day, the data integrity check of the task on the previous day needs to be performed at the configured time interval.

[0018] Preferably, step S3 includes: Step S31, analyzing the missing time period of the time series data in the relative data, checking the time interval between two adjacent time points of all data, if it exceeds the threshold, it is regarded as a data interruption, and the continuous period is calculated as the data filling time period; Step S32, checking the continuity of the original data in real time. If two adjacent real-time data exceed the threshold of the data missing time period, it is considered that the data is discontinuous and needs to be supplemented; Step S33, monitor the original data and relative data transmission status, if an abnormality occurs, treat it as no data, record the time interval, and need to be supplemented; Step S34, creating a corresponding replenishment task according to the data to be replenished, and saving the replenishment task information in the relational database; Among them, in step S33, the anomalies include: connection interruption, transmission protocol error, device failure and interface call error.

[0019] Preferably, step S4 comprises: Step S41, retrieving the replenishment task information from the relational database, and using multi-threading to concurrently execute multiple tasks; Step S42, using multiple threads to concurrently execute the data replenishment task, reading the original data and relative data to be replenished from the business system, writing them into the time series database, and writing the execution log into the relational database to record the task status and statistical information; Step S43, updating the relational database to fill in the task information.

[0020] The beneficial effects of the present invention are as follows: the present invention periodically scans and monitors the uploaded relative data, determines the integrity, creates data replenishment tasks according to the results, performs regular replenishment, and ensures the integrity of the data; not only replenishes the relative data, but also monitors the uploaded real-time data in real time, and obtains the real-time situation of the integrity of the uploaded data in real time; when replenishing data, it comprehensively considers the working status, interface call status, data quality, and network conditions of the data collector, and uniformly creates replenishment tasks with comprehensive replenishment conditions; after monitoring the data missing, different types of replenishment tasks are automatically created according to different situations to reduce manual access; at the same time, it supports manual replenishment operations, and manual replenishment can be performed for the replenishment data of difficult and complicated cases, and the failed tasks can be retried, and the common tasks can be copied and executed, which is flexible and easy to analyze problems; there is less manual intervention, which reduces labor costs, and all tasks are logged, and the data replenishment process records the detailed log of data replenishment BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a module diagram of the data acquisition automatic replenishment system based on multi-dimensional data continuity verification of the present invention; Figure 2 It is a specific module diagram of the data collection platform, business system and data replenishment module in the present invention; Figure 3 It is a flow chart of the data acquisition automatic backfilling method based on multi-dimensional data continuity verification of the present invention; Figure 4 It is a specific flow chart of the data collection platform in the present invention performing preliminary analysis and calculation on the original data and transmitting the calculation results, i.e., relative data, to the business system; Figure 5 It is a specific flow chart in which the business system of the present invention further analyzes and calculates the data, presents the results to the end user in the form of statements, tables, and reports, and saves the task results that need to be supplemented in the relational database; Figure 6 It is a specific flow chart of the data replenishment module in the present invention creating a replenishment task and automatically replenishing relative data and real-time data. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0023] The present invention will be further described below in conjunction with the accompanying drawings.

[0024] like Figure 1 As shown, the data collection automatic replenishment system based on multi-dimensional data continuity verification includes: a data collector 1, a data collection platform 2, a business system 3, a data replenishment module 4, a notification module 5 and a task inspection module 6.

[0025] The data collector 1 is used to send the raw data collected from the sensor to the data collection platform 2 through the Internet of Things card or wifi, and transmit it to the business system 3 at the same time.

[0026] The data collection platform 2 is used to receive the original data uploaded from the data collector, store it in the time series database, perform preliminary analysis and calculation on the original data, and transmit the calculation results, i.e. relative data, to the business system 3.

[0027] like Figure 2 As shown, the data acquisition platform 2 includes: a configuration unit 21 and a first reading unit 22 .

[0028] The configuration unit 21 is used to configure a plurality of reference measuring points for checking the original data, and to configure a threshold value for a data missing time period.

[0029] The first reading unit 22 is used to read all time series data of the reference measuring point one day before.

[0030] Business system 3 is used to simultaneously receive the original data and relative data, further analyze and calculate the data, present the results to the end user in the form of statements, tables, and reports, and save the task results that need to be supplemented in the relational database.

[0031] like Figure 3 As shown, the business system 3 includes: an analysis unit 31 , an inspection unit 32 , a monitoring unit 33 and a replenishment task creation unit 34 .

[0032] The analysis unit 31 is used to analyze the missing time period of the time series data, check the time interval between two adjacent time points of all data, and if it exceeds a threshold, it is regarded as a data interruption, and the continuous period is calculated as the data filling time period.

[0033] The checking unit 32 is used to check the continuity of the original data in real time. If two adjacent real-time data exceed the threshold of the data missing time period, it is considered that the data is discontinuous and needs to be supplemented.

[0034] The monitoring unit 33 is used to monitor the original data and the relative data transmission status. If an abnormality occurs, it is processed as no data and the time interval is recorded, and replenishment is required.

[0035] In the embodiment, the anomalies include: connection interruption, transmission protocol error, device failure and interface call error.

[0036] The replenishment task creation unit 34 is used to create corresponding replenishment tasks for the data that needs to be replenished in the inspection unit 32 and the monitoring unit 33, and save the replenishment task information to the relational database.

[0037] The data replenishment module 4 is used to create a replenishment task based on the analysis results of the relative data and the data quality of the real-time data detection, replenish the relative data and the real-time data, and store the replenished data in the time series database.

[0038] like Figure 2 As shown, the data replenishment module 4 includes: a second reading unit 41, a thread pool unit 42 and an updating unit 43.

[0039] The second reading unit 41 is used to retrieve the replenishment task information from the relational database and use multi-threading to concurrently execute multiple tasks.

[0040] The thread pool unit 42 is used to use multiple threads to concurrently execute data replenishment tasks, read the original data and relative data that need to be replenished from the business system, write them into the time series database, and write the execution log into the relational database to record the task status and statistical information.

[0041] The updating unit 43 is used to update the replenishment task information in the relational database.

[0042] Notification module 5 is used to notify the user of the status of the replenishment task.

[0043] In the embodiment, the notification channels include but are not limited to in-site letters, emails, text messages, and WeChat.

[0044] The task checking module 6 is used to perform the task of data integrity checking on the previous day at a configured time interval.

[0045] like Figure 3As shown, the data collection automatic backfill method based on multi-dimensional data continuity verification includes: Step S1, the data collector 1 sends the original data collected from the sensor to the data collection platform 2 through the Internet of Things card or wifi, and transmits it to the business system 3 at the same time.

[0046] Step S2, the data collection platform 2 receives the original data uploaded from the data collector, stores it in the time series database, and performs preliminary analysis and calculation on the original data, and transmits the calculation results, i.e., relative data, to the business system 3.

[0047] like Figure 4 As shown, step S2 includes: Step S21, configure multiple reference measuring points for inspection, generally select representative typical measuring points, for example: a measuring point in an uploaded data packet can represent all the data in this data packet, or a measuring point in a sensor can represent the entire sensor; and configure the threshold of the data missing time period, for example: it is generally set to 5 minutes. If no data is uploaded for more than 5 minutes, it is marked as data missing and the time period of data missing begins to be recorded.

[0048] Step S22 , reading all the time series data of the previous day of the benchmark measuring point, performing preliminary analysis and calculation, and transmitting the calculation result, ie, the relative data, to the business system 3 .

[0049] In the embodiment, before the execution of the task on the second day, the data integrity check of the previous day's task needs to be performed at the configured time interval. The time interval can be configured to be once a day, running at 2 a.m. every day, to check the data integrity of the previous day, and at the same time allow enough time (2 hours) to complete the data transmission of the previous day. Because data backfill usually has a large amount of data, it is run at night, so that a large amount of data will not block the system and will not interfere with the normal operation of the business system during the day.

[0050] Step S3, the business system 3 receives the original data and relative data at the same time, performs further analysis and calculation on the data, presents the results to the end user in the form of statements, tables, and reports, and saves the task results that need to be supplemented into the relational database.

[0051] like Figure 5 As shown, step S3 includes: Step S31, analyze the time period of missing time series data in the relative data, check the time interval between two adjacent time points of all data, if it exceeds the threshold, it is regarded as a data interruption, and calculate the continuous period as the data filling time period.

[0052] Step S32, checking the continuity of the original data in real time. If two adjacent real-time data exceed the threshold of the data missing time period, it is considered that the data is discontinuous and needs to be supplemented.

[0053] Step S33, monitor the original data and relative data transmission status, if an abnormality occurs, treat it as no data, record the time interval, and need to be supplemented.

[0054] In the embodiment, the anomalies include: connection interruption, transmission protocol error, device failure and interface call error.

[0055] Step S34: Create corresponding replenishment tasks according to the data to be replenished, and save the replenishment task information in the relational database.

[0056] Step S4, the data replenishment module 4 creates a replenishment task based on the analysis results of the relative data and the data quality of the real-time data detection, automatically replenishes the relative data and the real-time data, and stores the replenished data in the time series database.

[0057] like Figure 6 As shown, step S4 includes: Step S41, retrieve the replenishment task information from the relational database, and use multi-threading to concurrently execute multiple tasks.

[0058] Step S42, use multi-threaded concurrent execution of data replenishment tasks, read the original data and relative data that need to be replenished from the business system 3, write them into the time series database, and write the execution log into the relational database to record the task status and statistical information; wherein, each task calls the interface in batches to replenish data to avoid interface blocking and avoid writing a large number of time series at one time, which leads to a decrease in system performance.

[0059] Step S43, updating the relational database to fill in the task information.

[0060] Step S5, in addition to automatic data replenishment, supports manual data replenishment. The user manually selects the measurement points and start and end times to create a task for replenishment; the user can also re-execute a failed task or copy an existing task for multiple executions.

[0061] Step S6, the notification module 5 notifies the user of the task replenishment status until all tasks are completed and replenished.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some or all of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. The data collection automatic replenishment system based on multi-dimensional data continuity verification is characterized by: include: Data collector, used to send the raw data collected from the sensor to the data collection platform through the IoT card or wifi, and transmit it to the business system at the same time; The data collection platform is used to receive the raw data uploaded from the data collector, store it in the time series database, perform preliminary analysis and calculation on the raw data, and transmit the calculation results, i.e., relative data, to the business system; The business system is used to receive both raw data and relative data, further analyze and calculate the data, present the results to the end user in the form of tables, tables, and reports, and save the task results that need to be supplemented in the relational database; The data replenishment module is used to create a replenishment task based on the analysis results of the relative data and the data quality of the real-time data detection, replenish the relative data and real-time data, and store the replenished data in the time series database; Notification module, used to notify users of the status of the replenishment task; The task check module is used to perform the data integrity check task of the previous day at the configured time interval.

2. The data acquisition automatic backfilling system based on multi-dimensional data continuity verification according to claim 1 is characterized in that: The data collection platform includes: A configuration unit, used to configure a plurality of reference measurement points for checking the original data, and to configure a threshold value for a data missing time period; The first reading unit is used to read all time series data of the reference measuring point one day before.

3. The data acquisition automatic backfilling system based on multi-dimensional data continuity verification according to claim 1 is characterized in that: Business systems include: The analysis unit is used to analyze the missing time period of the time series data, check the time interval between two adjacent time points of all data, and if it exceeds the threshold, it is regarded as a data interruption, and the continuous period is calculated as the data filling time period; The checking unit is used to check the continuity of the original data in real time. If two adjacent real-time data exceed the threshold of the data missing time period, it is considered that the data is discontinuous and needs to be supplemented; The monitoring unit is used to monitor the original data and the relative data transmission status. If an abnormality occurs, it will be treated as no data, and the time interval will be recorded and the need for replenishment will be recorded; The replenishment task creation unit is used to create corresponding replenishment tasks for the data that needs to be replenished in the inspection unit and the monitoring unit, and save the replenishment task information to the relational database.

4. The data acquisition automatic backfilling system based on multi-dimensional data continuity verification according to claim 1 is characterized in that: In the monitoring unit, anomalies include: connection interruption, transmission protocol error, device failure and interface call error.

5. The data acquisition automatic backfilling system based on multi-dimensional data continuity verification according to claim 1 is characterized in that: The data backfill module includes: The second reading unit is used to retrieve the replenishment task information from the relational database and use multi-threading to concurrently execute multiple tasks; The thread pool unit is used to use multiple threads to concurrently execute data replenishment tasks, read the original data and relative data that need to be replenished from the business system, write them into the time series database, and write the execution log into the relational database to record the task status and statistical information; The update unit is used to update the relational database replenishment task information.

6. The data acquisition automatic backfilling system based on multi-dimensional data continuity verification according to claim 1 is characterized in that: In the notification module, notification channels include but are not limited to in-site messages, emails, text messages, and WeChat.

7. The data collection automatic backfill method based on multi-dimensional data continuity verification is characterized by: include: Step S1, the data collector sends the raw data collected from the sensor to the data collection platform through the Internet of Things card or wifi, and transmits it to the business system at the same time; Step S2, the data collection platform receives the raw data uploaded from the data collector, stores it in the time series database, performs preliminary analysis and calculation on the raw data, and transmits the calculation results, i.e., relative data, to the business system; Step S3, the business system receives the original data and relative data at the same time, performs further analysis and calculation on the data, presents the results to the end user in the form of statements, tables, and reports, and saves the task results that need to be supplemented in the relational database; Step S4: The data replenishment module creates a replenishment task based on the analysis results of the relative data and the data quality of the real-time data detection, automatically replenishes the relative data and the real-time data, and stores the replenished data in the time series database; Step S5: In addition to automatic data replenishment, manual data replenishment is supported. The user manually selects the measurement point and start and end time to create a task for replenishment; Step S6: the notification module notifies the user of the task status until all tasks are completed.

8. The data acquisition automatic backfilling method based on multi-dimensional data continuity verification according to claim 7 is characterized in that: Step S2 includes: Step S21, configuring a plurality of reference measuring points for inspection, and configuring a threshold value for a data missing time period; Step S22, reading all the time series data of the previous day of the benchmark measuring point, performing preliminary analysis and calculation, and transmitting the calculation results, i.e., relative data, to the business system; In step S2, before the task is executed on the next day, the data integrity check of the task on the previous day needs to be performed at the configured time interval.

9. The data acquisition automatic backfilling method based on multi-dimensional data continuity verification according to claim 7 is characterized in that: Step S3 includes: Step S31, analyzing the missing time period of the time series data in the relative data, checking the time interval between two adjacent time points of all data, if it exceeds the threshold, it is regarded as a data interruption, and the continuous period is calculated as the data filling time period; Step S32, checking the continuity of the original data in real time. If two adjacent real-time data exceed the threshold of the data missing time period, it is considered that the data is discontinuous and needs to be supplemented; Step S33, monitor the original data and relative data transmission status, if an abnormality occurs, treat it as no data, record the time interval, and need to be supplemented; Step S34, creating a corresponding replenishment task according to the data to be replenished, and saving the replenishment task information in the relational database; Among them, in step S33, the anomalies include: connection interruption, transmission protocol error, device failure and interface call error.

10. The data acquisition automatic backfill method based on multi-dimensional data continuity verification according to claim 7, characterized in that: Step S4 includes: Step S41, retrieving the replenishment task information from the relational database, and using multi-threading to concurrently execute multiple tasks; Step S42, using multiple threads to concurrently execute the data replenishment task, reading the original data and relative data to be replenished from the business system, writing them into the time series database, and writing the execution log into the relational database to record the task status and statistical information; Step S43, updating the relational database to fill in the task information.

Images