Real-time data loading method and device

Abstract

The invention discloses a real-time data loading method and device. In the real-time data loading method and device, a preprocessed file is updated into a database by externally calling an SQLLDR, thechanging and updating of data are considered, a loading file matched with a primary key of the preprocessed file is captured through a post process, and the preprocessed file to be loaded is updatedinto a system, so that the high-speed loading performance is guaranteed, the need for correctness of the data is satisfied, and the technical problems that for data loading of a measurement system inthe prior art, different data items in the same record arrive in a master station system in different times because there is inconsistency between an acquisition task and a data storage structure, thesame acquired data item has multiple repeated deliveries of acquired data because there is time difference when the data is merged and multiple reasons such as field devices and network environment,and the data collected at the same time and the data collected at different times are inconsistent are solved.

Description

technical field [0001] The invention relates to the field of databases, in particular to a real-time data loading method and device. Background technique [0002] The metering automation system is a set of real-time production and collection systems. The province collects a large amount of data. According to the current scale, the system will generate nearly 10,000 pieces of collected data per second. At peak times, the system will generate about 20,000 pieces of data. [0003] For the data loading of the metrology system in the prior art, due to the inconsistency between the collection task and the data storage structure, the time when different data items of the same record arrive at the master station system is inconsistent. When the data is merged, there is a time difference, and due to Due to various reasons such as on-site equipment and network environment, the same collected data item may be repeatedly collected and uploaded, and the data at the same time point may be...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a real-time data loading method and device, which are used to solve the problem of data loading of the metrology system in the prior art. Due to the inconsistency between the collection task and the data storage structure, different data items of the same record will arrive at the master. The time of the station system is inconsistent. When the data is merged, there is a time difference. Due to various reasons such as field equipment and network environment, the same collection data item may repeatedly collect data and send it multiple times. The same time point The data collected at different times may be inconsistent with technical issues

Images