Method and system for collecting measurement data in real time

Abstract

The invention provides a method and a system for real-time acquisition of measurement data, and aims to solve the problems of incomplete coverage of measurement data acquisition scenes, complex operation and maintenance of multi-source data and lack of metadata management in the prior art, and specifically, by formulating a Kafka message queue, an API interface and a single-valued model data specification of a relational database, matching an IEC104 protocol and Kafka, API and RDBMS exclusive acquisition components, the measurement data can be acquired in real time, and the measurement data can be acquired in real time. The method comprises the following steps: analyzing and cleaning collected data, uniformly converting the collected data into a JSON format, collecting and storing the collected data according to a path of'data source-Kafka-time sequence database ', constructing a uniform measurement data meta-model suitable for a hydropower / wind power / photovoltaic industry state, and regularly and automatically checking through a theoretical measurement point list A and an actual measurement point list B to ensure the consistency of the meta-data; meanwhile, data verification and operation and maintenance monitoring links are added; according to the method, multi-source data full-scene coverage is realized, the operation and maintenance complexity is reduced, the metadata normalization and the data acquisition reliability are improved, and the method is suitable for energy enterprise measurement data real-time acquisition and management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a method and system for real-time acquisition of measurement data. BACKGROUND

[0002] With the continuous promotion of enterprise digital transformation, the degree of production automation of energy enterprises is increasing, and the measurement data acquisition activities for automated equipment are becoming more and more frequent.

[0003] The prior art mainly faces Kafka, MQTT, IEC104 and other data sources, and different acquisition tools and technologies are used for different data source types. For example, if the data source is a Kafka message queue, a consumer program is developed according to the subscription mechanism; if the data source is an IEC104, MQTT and other industrial protocols, the relevant protocol analysis and data writing are realized through the Internet of Things components.

[0004] Although the existing technology stack meets the real-time acquisition and writing of measurement data in most scenarios, it still has the following problems and defects: 1. The data scene coverage is not complete. When high-frequency measurement data acquisition is carried out, in addition to IEC104, MQTT and other industrial protocols and Kafka message queues, the data source types also include relational databases, API interfaces and the like. The existing technology stack lacks support for relational databases, API interfaces and the like, and needs to be combined with structured data acquisition components for high-frequency scheduling combination.

[0005] 2. The operation and maintenance work is complex. Measurement data comes from different business systems or data platforms, and the data sources cover Kafka message queues, IEC104, API interfaces, relational databases and other types. The diversity of data sources leads to non-uniform acquisition technology links, and different acquisition tools and methods need to be used for different data sources. The existing technology cannot simultaneously meet the above data types, and the use of multiple acquisition tools makes the daily acquisition operation and maintenance work more complex and laborious.

[0006] 3. The measurement data metadata management is missing. The existing measurement data acquisition tool only has measurement data acquisition capability and does not have measurement data metadata management. Usually, separate offline measurement point data metadata sorting work needs to be carried out. The asynchronization of measurement data acquisition and metadata management often leads to inaccurate and timely updating of measurement data metadata information, and the manual sorting workload is large, which seriously restricts the application effect and value of measurement data. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a method and system for real-time acquisition of measurement data, to solve the problems of incomplete coverage of data scenarios, complex acquisition and operation and maintenance caused by diverse data sources, and metadata management of measurement data in the prior art, and to realize comprehensive, efficient and accurate acquisition and management of measurement data.

[0008] To solve the above technical problems, the technical solution adopted by the present application is: a method for real-time acquisition of measurement data, comprising the following steps: S1, formulating a measurement data acquisition specification: for Kafka message queues, API interfaces and relational database data sources, set the data format of a single value model, the single value model records the value of a measurement point at a certain point in time in a message body, and at least includes a measurement point code senid, a measurement value value and a timestamp timestamp field; S2, multi-source data acquisition: according to the type of data source, match the corresponding acquisition component, the data source type includes a centralized control system complying with IEC104 protocol, Kafka message queue, API interface and relational database RDBMS, and the acquisition component includes IEC104 protocol acquisition component, Kafka data acquisition component, API data acquisition component and RDBMS data acquisition component; S3, data processing: the corresponding acquisition component is used to analyze, clean and format the collected raw data, the analysis includes data frame identification, control field analysis and data field analysis, the cleaning includes removing invalid data, duplicate data and error data, and the format conversion at least includes converting the analyzed data into JSON format; S4, data collection and storage: the processed data is first sent to a specified topic of Kafka, then the topic data is subscribed by the Kafka data acquisition component, converted into data conforming to the writing specification of the time series database, and written into the time series database to form a collection path of "data source→Kafka→time series database"; S5, metadata management of measurement data: a unified measurement data meta model is constructed, the online real-time synchronization of the measurement point directory at the data source end is realized through an ETL tool or an API interface to form a theoretical measurement point list A; the actual access measurement point list B is obtained by calling the metadata capability of the time series database every day; the list A and the list B are checked periodically and automatically, new and disabled measurement point information is identified and a reminder is triggered to ensure that the metadata is consistent with the actual measurement point information.

[0009] Preferably, the data format in step S1 specifically includes: Kafka message queue data format: the top layer key is "value", and the corresponding value is a list containing at least one measurement point data, each measurement point data contains the mandatory fields of "measurement point code senid", "value", and "timestamp", and supports custom addition of attribute fields; API interface data format: contains "interface call status identification code", "interface call result description message", and "total number of returned data" general fields, and the core data is presented as a list corresponding to the "data" key, and each data in the list contains "measurement point code senid", "value", and "timestamp" fields; Relational database data format: each record in the data table corresponds to the value of a measurement point at a certain point in time, and the fields at least include the measurement point code field corresponding to "measurement point code senid", the value field corresponding to "value", and the timestamp field corresponding to "timestamp".

[0010] Preferably, in the step S2, the collection process of the IEC104 protocol collection component includes: Collection initialization: open network access strategy and configure firewall; Establish connection: establish communication connection with the source IEC104 server, and receive telemetry, telesign or remote control raw data; Data analysis: sequentially perform data frame identification (define complete data frame according to IEC104 protocol start identification and length field), control field analysis (extract data transmission direction, function code and sequence number), data field analysis (convert binary data to physical quantity or device state); Error checking and correction: use CRC cyclic redundancy check or checksum method to check the analysis result, and if there is an error, perform re-request or data error correction operation.

[0011] Preferably, each collection component in the step S2 is designed as an independent reusable module, each module has exclusive function and standard interface; according to business requirements, the components are combined and configured, when a new data source is added, only the corresponding collection component needs to be developed and integrated into the system, without adjusting the existing components; when a single component fails, only the component needs to be maintained, without affecting the operation of other components.

[0012] Preferably, in the step S4, the collection and storage process of the relational database data source further comprises: when the data supply frequency is minute level and the time series database pressure is small, directly writing the processed data into the time series database through the RDBMS data collection component; when the time series database pressure is large, first writing the processed data into the Kafka message queue, and then writing the data into the time series database according to the preset rule and frequency through the Kafka data collection component.

[0013] Preferably, in the step S5, the unified measurement data meta-model is applicable to hydropower, wind power or photovoltaic industry, and the meta-model for the hydropower industry comprises: management information: containing unit name and power station name fields; identification information: adopting a four-layer structure of system, subsystem, device and data object, and containing fields of measurement point code, measurement point name, belonging unit, production frequency, measurement point type, data format, measurement unit and data source; constraint information: containing fields of measurement point sharing type and sharing condition; extension information: reserving an interface for custom-added fields.

[0014] Preferably, it further comprises a data verification step: after the collection task is completed, connecting the time series database management page to check whether there is source data (real-time writing or time-based writing) in the library table, whether there is garbled code in the data writing process, and whether the data insertion frequency is consistent with the collection frequency, and if there is an exception, triggering an alarm and recording an exception log.

[0015] A measurement data real-time collection system implementing the method comprises: a data specification module: for storing and executing the measurement data collection specification in step S1, and verifying the format of the source input data; a multi-source data collection module: containing an IEC104 protocol collection component, a Kafka data collection component, an API data collection component and an RDBMS data collection component, for matching the components according to the data source type and completing data collection; a data processing module: for performing the data parsing, cleaning and format conversion operations in step S3; a data storage module: comprising a Kafka message queue unit and a time series database unit, the Kafka message queue unit is used for receiving source raw data or processed data and storing them into a designated topic, and the time series database unit is used for receiving data transmitted by the Kafka data collection component and persistently storing them; Metadata management module: including meta-model construction unit, synchronization unit and monitoring engine unit, the meta-model construction unit is used for constructing the unified measurement data meta-model in step S5, the synchronization unit is used for realizing the synchronization of the measurement point directory and generating list A and list B, and the monitoring engine unit is used for checking list A and list B periodically and triggering a reminder.

[0016] Preferably, the multi-source data acquisition module further comprises a data supplement acquisition component and a task monitoring component; the data supplement acquisition component is used for detecting the integrity of the acquired data, and triggering a supplement acquisition process if there is data loss; and the task monitoring component is used for monitoring the running state of each acquisition task in real time, and recording task logs (including acquisition start time, end time, data volume and abnormal information).

[0017] Preferably, the synchronization unit of the metadata management module specifically comprises: receiving the access measurement point metadata provided by the data source end management department according to the unified meta-model, realizing real-time synchronization with the measurement point directory of each data source end through an ETL tool or an API interface to generate list A; calling a metadata query interface of the time series database daily to obtain actual measurement point information currently accessed to generate list B; and the checking period of the monitoring engine unit is every week, and after checking, a pop-up window or a message notification mode is used to remind the data management personnel to handle the newly added and disabled measurement points.

[0018] The application provides a kind of measurement data real-time acquisition method and system, with the following beneficial effects: 1, acquisition scene comprehensive coverage: support IEC104, Kafka, API, relational database and other multi-type data sources, without combining multiple tools, can meet the full scene demand of high-frequency measurement data acquisition of energy enterprise.For example, a certain wind power enterprise can simultaneously acquire IEC104 protocol data of wind turbine sensor, API interface data of cloud platform and historical data of local MySQL database by the present scheme, and the acquisition coverage is improved from 65% of prior art to 100%.

[0019] 2, operation and maintenance complexity is significantly reduced: component-based management and unified operation and maintenance interface are used, operation and maintenance personnel do not need to switch multiple systems, fault positioning time is shortened from an average of 2 hours to 15 minutes; component independent deployment feature makes it only need to develop corresponding component integration when adding new data source (such as MQTT protocol equipment), without adjusting the existing system, and the development cycle is shortened by 50%.

[0020] 3, metadata management standardization: unified meta-model and automatic checking mechanism ensure the consistency of metadata and actual measurement point data, avoid manual error, metadata update delay is reduced from 24 hours to real time, data interpretability is improved by 80%, and reliable metadata support is provided for subsequent data analysis (such as power generation prediction, equipment fault diagnosis).

[0021] 4. Data collection reliability is improved: through Kafka buffering, retry mechanism, data verification and other designs, the data loss rate is reduced from 0.5% in the prior art to 0.01%; the application of WebSocket protocol shortens the time series database write delay to within 100ms, meeting the real-time monitoring requirements. BRIEF DESCRIPTION OF DRAWINGS

[0022] The application will be further described below in conjunction with the accompanying drawings and examples: Figure 1 The method flowchart of the application; Figure 2 The structural schematic diagram of the application. DETAILED DESCRIPTION

[0023] As shown in Figure 1 , 1. Formulate measurement data collection specifications For the three types of core data sources of Kafka message queue, API interface and relational database, set the single value model data format (i.e. one message body records the value of one measurement point at a certain time point), and ensure the uniformity of data structure. Among them: (1) Kafka message queue: publish data through Kafka message queue topic, the data format should be single value model, i.e. one message body records the value of one measurement point at a certain time point, and "measurement point code (senid), measurement value (value), timestamp (timestamp)" in the message data are mandatory parameters, which accurately identify the key information of measurement data, and ensure the traceability and timeliness of data. At the same time, in order to support the actual needs of different business scenarios, other attributes can be customized and added to meet the diversified data needs, for example: { "values":[ { "senid":"10000001", "value":20.01, "timestamp":"2024-04-1512:12:21:32", …… }, { "senid":"10000002", "value":19.01, "timestamp":"2024-04-1512:12:22:21", …… }, …… ]} (2) API Interface: The API interface returns data in a single-value model, where each message body records the value of a measurement point at a specific time. Besides general business-related information such as "code" (to identify the API call status), "message" (to describe the API call result), and "total" (the total number of returned data), the core data is presented in a list format. Each data entry contains basic fields such as "senid," "value," and "timestamp" to ensure a clear and consistent data structure, facilitating subsequent data parsing and processing. A specific example of the returned results is shown below: { "code":"200", "message":"API call successful", "total":1000, "data":[ { "senid":"10000001", "value":20.01, "timestamp":"2024-04-1512:12:21:32", ... }, { "senid":"10000002", "value":19.01, "timestamp":"2024-04-1512:12:22:21", ... }, ... ]} (3) Relational database: The structured measurement data provided by the relational database should be in the form of a single-value model, that is, a message body records the value of a measurement point at a certain time point. The data specifications are shown in the table below.

[0024]

[0025] 2. Multi-source data acquisition Enterprises have a rich variety of internal data sources, including Kafka message queues, IEC104, MQTT, APIs, and RDBMS, characterized by high timeliness, high frequency, and large data volume. The measurement data management system invented in this patent accurately matches the corresponding acquisition method according to different data scenarios, realizing the timed or real-time acquisition of various measurement data. The data is ultimately collected and stored according to the path of "data source → Kafka → time-series database".

[0026] The measurement management system adopts component-based management, which means that each functional module of the system is split into independent and reusable components, each with specific functions and interfaces. These components can be combined and configured according to actual needs to achieve the overall function of the system. In the measurement data management system, component-based management enables the system to easily handle different types of data sources, data formats, and collection methods.

[0027] Component-based management has the following advantages: Flexibility: Components can be flexibly selected and combined according to different business needs and data source characteristics. When new data sources or collection methods are needed, only the corresponding components need to be developed and integrated into the system, without the need for large-scale adjustments to the entire system.

[0028] Ease of maintenance: Since components are independent, when a component has a problem, only that component needs to be maintained and repaired, without affecting the normal operation of other components. This greatly reduces the difficulty and cost of system maintenance.

[0029] Reusability: Components can be reused in different projects or scenarios, improving development efficiency and reducing development workload.

[0030] The measurement data management system of the present invention commonly uses the following data collection components: IEC104 protocol collection component: used to collect measurement data that complies with the IEC104 protocol specification. This component implements IEC104 protocol parsing and communication functions, enabling data exchange with devices that support the IEC104 protocol, and sending collected data to the system.

[0031] Kafka data collection component: responsible for obtaining data from the Kafka message queue. This component can listen to messages in real time according to the configured Kafka topic and partition, and pass the messages to subsequent data processing components.

[0032] API data collection component: used to collect data sources that provide data using API interfaces. This component can send requests and parse returned data according to API interface specifications, supporting different API types such as RESTful API, SOAP API, etc.

[0033] RDBMS data collection component: The RDBMS data collection component is suitable for collecting, processing, and transmitting data from relational databases. It can adapt to multiple relational database types such as MySQL, Oracle, etc., and accurately obtain data by configuring collection rules. It has data cleaning and conversion capabilities, can unify data formats, handle invalid data, and supports writing processed data to time series databases and Kafka.

[0034] 3. Data processing Each collection component performs the "parsing -> cleaning -> conversion" process on raw data: Parsing: For IEC104 protocol data, sequentially perform data frame recognition (define complete frame according to start character "68" and length field), control field parsing (extract data transmission direction, function code), and data field parsing (convert binary data to physical quantity, such as telemetry data "00000041100000" parsed to 220.5kV voltage value); For Kafka / API data, extract core fields according to the preset JSON format; Cleaning: Remove invalid data (such as "value" is empty, "timestamp" format is incorrect), duplicate data (de-duplicate by "senid+timestamp"), and abnormal data outside the range (such as voltage value exceeding 380kV); Conversion: Convert the parsed data to JSON format to ensure that the field name matches the time series database table structure.

[0035] During processing, CRC cyclic redundancy check or checksum is used to verify data integrity. If the check fails, a retry mechanism is triggered (default retry 3 times, interval 5 seconds). If the retry fails, an error log is recorded.

[0036] 4. Data collection and storage Use the unified collection path of "data source -> Kafka -> time series database": Processed data is first sent to the specified Kafka topic to achieve data buffering (especially suitable for time series database pressure peak scenarios); The Kafka data collection component subscribes to the topic and generates an Insert statement for the time series database (such as TDengine, InfluxDB) to write to the time series database through the WebSocket protocol (supports full-duplex communication, reduces connection overhead); For relational database data sources, if the data supply frequency is minute-level and the time series database load is low, the RDBMS component can directly write to the time series database; if the time series database pressure is high (such as CPU utilization exceeding 80%), first write to Kafka, and then synchronize to the time series database when the load decreases.

[0037] The specific use scenarios are as follows: Scenario 1: Data source is IEC104 industrial protocol IEC104 protocol is a remote communication protocol based on TCP / IP, which is extended based on IEC60870-5-101 protocol, and is used for data communication between substations and control centers in power system automation field.

[0038] (1) Before the start of the collection task, the network access strategy should be opened, the firewall settings and other initialization work should be completed to prepare for data collection.

[0039] (2) Use the IEC104 protocol acquisition component to receive source data, establish a connection with the source IEC104 server, and receive sample data as follows: Telemetry raw data example: 6814000000000100040100010000010000000041100000 Telemetry raw data analysis: 68 is the starting character; 01 is the type identifier, indicating telemetry data; 00000041100000 is the corresponding data value, which represents the measurement value of a certain electrical quantity, such as voltage, current, etc.

[0040] Telesign raw data example: 6810000000000300020100020000010000000001 Telesign raw data analysis: 68 is the starting character; 03 is the type identifier, indicating telemetry data; the last 0001 is the telesign data value, representing the state of a switch, 0 indicating open and 1 indicating closed.

[0041] Remote control raw data example: 68120000000006000101000300000100000000000100 Remote control raw data analysis: 68 is the starting character; 06 type identifier is remote control command; 00000100 is the remote control command data value, where 01 indicates the remote control operation type, such as closing and opening.

[0042] (3) Use the IEC104 protocol acquisition component to analyze the received data, and the analysis steps are as follows: Data frame identification: The acquisition component first identifies the data frame of the received and spliced data. IEC104 protocol specifies the specific format of the data frame, which usually includes start identifier, length field, control field, data field and check field, etc. The component determines the start position of the data frame by identifying the start identifier, and obtains the overall length of the data frame according to the length field, so as to accurately define the range of a complete data frame, laying a foundation for subsequent analysis.

[0043] Control field analysis: Extract and analyze the control field in the data frame. The control field contains rich control information, such as data transmission direction, function code, sequence number, etc. By analyzing these information, the acquisition component can determine the purpose and transmission state of the data, whether the data is telemetry data, remote control command or other types of data, and whether there is loss or repetition in the transmission process.

[0044] Data Field Parsing: According to the IEC104 protocol specification and pre-configured parsing rules, the data field is parsed. This process requires the use of appropriate parsing methods according to the different data types, such as for telemetry data, binary data is converted into actual physical quantities such as voltage and current values according to the specified encoding method and range; for remote signaling data, the switch state of the device is determined according to the state of the data bit.

[0045] Data Conversion: The parsed data is converted into JSON format suitable for Kafka storage and subsequent processing. JSON format has good readability and universality, which is convenient for Kafka consumers to parse and process.

[0046] Data Cleaning: The parsed data is cleaned to remove invalid, duplicate or erroneous data.

[0047] Verification and Error Correction: During the parsing process, the acquisition component verifies the parsing results using the verification field. Common verification methods include CRC (Cyclic Redundancy Check), checksum, etc. If the verification finds that the data has errors, the acquisition component processes according to the pre-set error correction strategy, such as requesting the sender to resend the data, or correcting the error according to the existing data, to ensure the accuracy of the parsed data.

[0048] Telemetry data parsing example: {“measuring point code”:“1001”,“measuring value”:220.5,“timestamp”:“2024-10-1012:00:00”} Telemetry data parsing explanation: “Measuring point code” is used to uniquely identify the specific measuring point corresponding to the telemetry data, making it easy for the system to distinguish and manage data from different measuring points. “Measuring value” is the actual physical quantity value obtained after parsing and conversion, such as 220.5 here representing voltage value, and the unit can be determined according to the actual system configuration, such as KV. “Timestamp” records the accurate time of data collection, accurate to the second, which is crucial for analyzing the time series changes of the power system running state.

[0049] Remote signaling data parsing example: {“measuring point code”:“2001”,“measuring value”:“1”,“timestamp”:“2024-10-1012:05:00”} Remote signaling data parsing example: “Measuring point code” is also used to identify the remote signaling measuring point, which means “switch state”. If the “measuring value” is 1, it represents “closed” indicating that the switch is in the on state and power can pass normally, if it is 0, it represents “open” indicating that the switch is in the off state. “Timestamp” records the time when the switch state changes, which is of great significance for timely discovering the state change of the device in the power system, fault diagnosis, etc.

[0050] Remote control data analysis example: { "measuring point code": "3001", "measuring value": "1", "timestamp": "2024-10-10 12:10:00"} Remote control data analysis example: "measuring point code" is the unique identifier of this remote control command, which facilitates the system to track and manage the issued control instructions. The measuring value of 1 represents "closing" and requires the device to perform the on operation, while 0 represents "opening" and is the cut-off operation. The "timestamp" records the time when the remote control command is issued.

[0051] (4) Send the parsed data to the specified topic of Kafka, subscribe to the corresponding topic using the Kafka data collection component, and convert it to regular measuring point data in standard format, finally write it to the time series database, providing reliable data support for subsequent data analysis and application.

[0052] (5) Connect to the time series database management page and perform comprehensive data verification. Check if there is real-time data writing in the related library table, and check if there are abnormal situations such as garbled codes during data writing, to ensure the integrity and accuracy of the data.

[0053] Scenario Two: Data source is Kafka message queue (1) Before starting the collection task, the network access strategy should be opened, the firewall should be set, and other initialization work should be completed to prepare for data collection.

[0054] (2) Subscribe to the source Kafka topic, and the measurement data management system uses the Kafka collection component to obtain data. The Kafka collection component will continuously listen to the data of this topic, and continuously consume and obtain data if there is data change.

[0055] (3) After obtaining the data, the Kafka collection component will generate specific Insert statements according to the Insert statement specification of the target time series database.

[0056] (4) Use WebSocket to connect to the time series database. WebSocket is a protocol for full-duplex communication over a single TCP connection. Unlike traditional HTTP protocols, WebSocket allows real-time, bidirectional data transmission between servers and clients without frequent connection establishment and disconnection. This makes it perform well in scenarios that require real-time data interaction.

[0057] (5) The timing library executes the Insert statement to write data into the timing library. It should be noted that the data type inserted must strictly match the column data type in the table definition. The timestamp column should use a legal timestamp format, the numerical column should use a suitable numerical type, and the string column should ensure that the length and encoding meet the requirements. If the data types do not match, it may cause insertion failure or abnormal data storage, affecting subsequent data query and analysis.

[0058] (6) Connect to the timing library management page to comprehensively verify the data. Check whether there is real-time data writing in the related library table, and check whether there are abnormal situations such as garbled codes during data writing, to ensure the integrity and accuracy of the data.

[0059] Scenario Three: Data Source is API Interface (1) Before starting the collection task, the network access strategy should be opened, the firewall should be set, and other initialization work should be completed to prepare for data collection.

[0060] (2) Use API data collection components to collect, process, and write data. API data collection components have the following capabilities: Interface connection: According to the API protocol (such as HTTP, HTTPS) and related specifications, establish a connection with the target API, handle identity verification, authorization, etc.

[0061] Data request and acquisition: According to the settings, send requests to the API and get response data, support different request methods (GET, POST, etc.), and can handle paging data.

[0062] Data preprocessing: Process the acquired data, data cleaning (remove invalid or incorrect data), format conversion (such as JSON to regular measurement point data).

[0063] Exception handling: When encountering network failures, API return errors, etc., it can perform retry, log recording, etc.

[0064] Data storage: Store the collected and processed data to the specified location, such as database, Kafka.

[0065] Collection frequency configuration: The collection frequency can be configured according to the data provider's support for concurrent capability and the data consumer's receiving time.

[0066] (3) Use the API data collection component data storage capability to write the parsed data into the timing library.

[0067] (4) Connect to the timing library management page to verify whether the source data has been written into the related library table, whether the data insertion frequency is consistent with the collection frequency, and whether there are garbled codes in the data writing.

[0068] Scenario Four: Data Source is a Relational Database (1) Before the start of the collection task, the network access policy should be opened, the firewall settings and other initialization work should be completed to prepare for data collection.

[0069] (2) According to the type of the source relational database (such as MySQL, Oracle, etc.), the corresponding RDBMS data collection component is selected to ensure the pertinence and efficiency of the collection work.

[0070] (3) Through the RDBMS data collection component, the data management system completes the collection, processing and writing operations. The RDBMS data collection component has the following capabilities: Connect to the database: According to the connection parameters of the database, establish a stable connection with the source relational database to ensure smooth data transmission.

[0071] Data collection: According to the preset collection rules, extract the required data from the database, and support flexible collection of different tables and fields.

[0072] Data processing: Perform preprocessing operations such as cleaning and conversion on the collected data, such as removing duplicate data and unifying data formats, to meet the requirements of subsequent storage and analysis.

[0073] Data writing: Support writing processed data to multiple storage media, with the ability to write data to time series databases and Kafka. When writing to time series databases, the data will be accurately stored according to the data structure and writing specifications of the time series database, providing reliable data support for subsequent data analysis and application. When writing to Kafka, the data will be sent to the corresponding topic according to the Kafka topic configuration, facilitating further data flow and processing.

[0074] Exception handling: In the collection process, if network failures, database connection exceptions and other problems occur, automatic retries will be performed, and detailed error logs will be recorded for subsequent problem troubleshooting.

[0075] (4) In the scenario where the source data source is a relational database and the data supply frequency is minute-level, the data writing capability of the relational database collection component can be used to write the processed data to the time series database, realizing effective storage and management of data. However, when the time series database pressure is large, in order to ensure the stability and efficiency of data processing, the data will be written to Kafka first.

[0076] Kafka has high throughput and low latency characteristics, and can store data as a buffer when the time series database is under heavy pressure. Then, through the Kafka data collection component, the data is written from Kafka to the time series database according to certain rules and frequency. During this process, the Kafka collection component continuously listens to the data in the Kafka topic, and once new data is written, it will be timely acquired and formatted and written according to the requirements of the time series database, so that when the time series database is under heavy pressure, the stable storage and management of data can also be ensured.

[0077] (5) Connect the time series database management page to comprehensively verify the data. Check if the source data is written in time in the related database table, confirm if the data insertion frequency matches the collection frequency, and check if there are garbled codes or other abnormal situations during data writing to ensure the integrity and accuracy of the data.

[0078] 5. Metadata management A unified measurement data meta-model suitable for hydropower, wind power and photovoltaic industry is constructed. Taking hydropower industry as an example, the meta-model contains four types of information: Management information: unit name, power station name; Identification information: adopt four-layer structure of "system→subsystem→device→data object" (refer to GB / T50549-2020 standard), including measurement point code, measurement point name, belonging unit, production frequency (such as 50Hz), measurement point type (telemetry / remote signaling), unit of measurement; Constraint information: measurement point sharing type (public / internal), sharing condition; Extended information: reserved custom field interface (such as "data quality level").

[0079] Metadata synchronization and monitoring process: 1) The data source end provides measurement point metadata according to the meta-model, and synchronizes it to the collection system in real time through ETL tools or API interfaces to generate a theoretical measurement point list A; 2) Call the time series database metadata interface every day (such as 2 a.m.) to get the current actual access measurement point information, and generate an actual measurement point list B; 3) The metadata monitoring engine automatically checks A and B every week to identify new measurement points (A has, B has not) and disabled measurement points (B has, A has not), and reminds data management personnel to update metadata records through system pop-up windows or emails.

[0080] 6. Data verification and operation and maintenance monitoring Data verification: After the collection task is completed, check the data writing situation through the time series database management page: confirm that there is real-time / timed data writing in the database table, there is no garbled code, and the data insertion frequency is consistent with the collection frequency (such as API interface configuration 5 minutes once). Operation and maintenance monitoring: Provide a unified operation and maintenance interface to display the running status (running / pausing / failure) of each collection task, the amount of collected data, and the success rate. Support filtering tasks by data source type, automatically locate to specific collection components when there is a failure (such as "IEC104 collection component connection timeout"), and push alarm information.

[0081] The application provides a kind of measurement data real-time acquisition method and system, to solve the problems of existing technology measurement data acquisition scene coverage not complete, multi-source data operation and maintenance complex and metadata management missing, specifically by formulating Kafka message queue, API interface, single value model data specification of relational database, matching IEC104 protocol, Kafka, API, RDBMS special collection component, parsing, cleaning and uniformly converting to JSON format to collection data, according to the path of "data source→Kafka→time series database" path, gather and store, build unified measurement data meta model suitable for hydropower / wind power / photovoltaic industry and through theoretical measuring point list A and actual measuring point list B Regular automatic check to ensure metadata consistency, while adding data verification and operation and maintenance monitoring link;The application realizes multi-source data full scene coverage, reduces operation and maintenance complexity, improves metadata specification and data acquisition reliability, and is suitable for energy enterprise measurement data real-time acquisition and management.

[0082] The above embodiments are only preferred technical solutions of the application, and should not be regarded as a limitation of the application. The protection scope of the application should be based on the technical solutions recited in the claims, including equivalent replacement solutions of the technical features recited in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the application.

Claims

1. A method for real-time acquisition of metrology data, characterized in that, The method comprises the following steps: S1, formulating a measurement data collection specification: for a Kafka message queue, an API interface, and a relational database data source, setting a data format of a single-value model, the single-value model recording a value of a message body of a measurement point at a certain time point, and at least containing a measurement point code senid, a measurement value value, and a timestamp timestamp field; S2, multi-source data collection: according to a data source type, matching a corresponding collection component, the data source type including a centralized control system complying with an IEC104 protocol, a Kafka message queue, an API interface, and a relational database RDBMS, and the collection component including an IEC104 protocol collection component, a Kafka data collection component, an API data collection component, and an RDBMS data collection component; S3, data processing: through a corresponding collection component, analyzing, cleaning, and format-converting collected raw data, the analysis including data frame identification, control field analysis, and data field analysis, the cleaning including removing invalid data, repeated data, and error data, and the format conversion at least including converting the analyzed data into a JSON format; S4, data collection and storage: sending the processed data to a Kafka designated topic first, then subscribing to the topic data through a Kafka data collection component, converting the data into data conforming to a writing specification of a time series database, and writing the data into the time series database to form a collection path of "data source -> Kafka -> time series database"; S5, measurement data metadata management: constructing a unified measurement data metadata model, realizing online real-time synchronization of a measurement point directory at a data source end through an ETL tool or an API interface, forming a theoretical measurement point list A; calling a time series database metadata capability every day to obtain an actual access measurement point list B; periodically and automatically checking the list A and the list B, identifying new and disabled measurement point information, and triggering a reminder to ensure that the metadata is consistent with the actual measurement point information.

2. The method of real-time acquisition of metrology data according to claim 1, wherein, The data format in the step S1 specifically includes: A Kafka message queue data format: taking "measurement values values" as a top-level key, the corresponding value of which is a list containing at least one measurement point data, each measurement point data containing "measurement point code senid", "measurement values values", and "timestamp timestamp" required fields, and supporting custom-added attribute fields; An API interface data format: containing "interface call status identifier code", "interface call result description message", and "total number of returned data total" general fields, and core data being presented in a list corresponding to a "data" key, each data in the list containing "measurement point code senid", "measurement values values", and "timestamp timestamp" fields; A relational database data format: each record in a data table corresponding to a value of a measurement point at a certain time point, the fields at least including a measurement point code field corresponding to "measurement point code senid", a measurement value field corresponding to "measurement values values", and a timestamp field corresponding to "timestamp timestamp".

3. The method of claim 1, wherein the metrology data is collected in real time. The acquisition process of the IEC104 protocol acquisition component in the step S2 includes: Acquisition initialization: opening network access strategy and configuring firewall; Connection establishment: establishing communication connection with the source end IEC104 server, receiving telemetry, telesignaling or remote control raw data; Data analysis: sequentially performing data frame identification, control field analysis and data field analysis; Error checking and correction: checking and correcting the analysis result by using CRC cyclic redundancy check or checksum method, and performing re-request or data correction operation if there is error.

4. The method of real-time acquisition of metrology data according to claim 1, wherein, Each acquisition component in the step S2 is designed as an independent reusable module, each module has exclusive function and standard interface; according to business requirements, the components are combined and configured, when a new data source is added, only the corresponding acquisition component needs to be developed and integrated into the system, without adjusting the existing components; when a single component fails, only the component needs to be maintained, without affecting the operation of other components.

5. The method of real-time acquisition of metrology data according to claim 1, wherein, In the step S4, the acquisition and storage process of the relational database data source further includes: when the data supply frequency is minute level and the time series database pressure is small, the processed data is directly written into the time series database through the RDBMS data acquisition component; when the time series database pressure is large, the processed data is first written into the Kafka message queue, and then the data is written into the time series database according to the preset rule and frequency through the Kafka data acquisition component.

6. The method of real-time acquisition of metrology data according to claim 1, wherein, In the step S5, the unified measurement data meta model is applicable to hydropower, wind power or photovoltaic industry, and the meta model for hydropower industry includes: Management information: including unit name and power station name fields; Identification information: adopting four-layer structure, system, subsystem, device and data object, and including measurement point code, measurement point name, belonging unit, production frequency, measurement point type, data format, measurement unit and data source fields; Constraint information: including measurement point sharing type and sharing condition fields; Extension information: reserving the interface of custom-added fields.

7. The method of real-time acquisition of metrology data according to claim 1, wherein, It also includes a data checking step: after the acquisition task is completed, the time series database management page is connected, it is checked whether there is source data in the database table, whether there is garbled code in the data writing process, and whether the data insertion frequency is consistent with the acquisition frequency, if there is an exception, an alarm is triggered and an exception log is recorded.

8. A metrology data real-time acquisition system implementing the method of any one of claims 1 to 7, characterized in that, It includes: Data specification module: used for storing and executing the measurement data acquisition specification in the step S1, and checking the format of the source input data; Multi-source data acquisition module: including IEC104 protocol acquisition component, Kafka data acquisition component, API data acquisition component and RDBMS data acquisition component, used for matching components according to data source type and completing data acquisition; Data processing module: used for performing data analysis, cleaning and format conversion operation in the step S3; Data storage module: including Kafka message queue unit and time series database unit, the Kafka message queue unit is used for receiving source raw data or processed data and storing to the specified topic, and the time series database unit is used for receiving data transmitted by the Kafka data acquisition component and persistently storing; The metadata management module comprises a meta-model construction unit, a synchronization unit and a monitoring engine unit, the meta-model construction unit is used for constructing the unified measurement data meta-model in step S5, the synchronization unit is used for realizing the synchronization of the measurement point directory and generating the list A and the list B, and the monitoring engine unit is used for periodically checking the list A and the list B and triggering a reminder.

9. The real-time metrology data acquisition system of claim 8, wherein, The multi-source data acquisition module further comprises a data supplement acquisition component and a task monitoring component; the data supplement acquisition component is used for detecting the completeness of the acquired data, and triggering a supplement acquisition process if there is data loss; and the task monitoring component is used for monitoring the running state of each acquisition task in real time and recording a task log.

10. The metrology data acquisition system of claim 8, wherein, The synchronization unit of the metadata management module specifically comprises: receiving the access measurement point metadata provided by the data source end management department according to the unified meta-model, realizing the real-time synchronization with the measurement point directory of each data source end through an ETL tool or an API interface to generate the list A; calling a metadata query interface of the time series database at a fixed time every day, acquiring the actual measurement point information currently accessed to generate the list B; and the checking period of the monitoring engine unit is every week, and after checking, a pop-up window or a message notification mode is used to remind the data management personnel to handle the newly added and disabled measurement points.

Images