A method, device, system and storage medium for collecting device data based on multi-source heterogeneity

Abstract

The present application discloses a device data collection method, apparatus, system, and storage medium based on multi-source heterogeneity. The method includes: collecting device data from multiple devices; configuring corresponding data parsing code according to the data format and characteristics of each device, and parsing the collected device data based on the data parsing code to obtain device parsed data; extracting the attributes and point addresses of each device from the device parsed data, identifying the functional type or physical quantity information contained in the point address, and mapping it to the corresponding device data item; extracting the data value corresponding to the point address; filtering illegal data according to preset rules for the extracted data, and performing type conversion on legal data; organizing the target data after verification and type conversion into a standardized and unified JSON data structure; parsing the generated JSON data structure, and writing the parsed data into a target database for data analysis and device operation status monitoring.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, system and storage medium for collecting device data based on multi-source heterogeneity. Background Art

[0002] With the rapid development of the Industrial Internet of Things (IIoT) and intelligent manufacturing, operational status data from various scenarios and devices has become a crucial resource for companies to optimize production and improve efficiency. However, data collection methods and formats vary significantly across devices, resulting in so-called "multi-source heterogeneous data." The lack of uniformity in data formats and attributes presents significant challenges for data analysts, including:

[0003] The collected data from different devices may be in various formats such as tables, text, binary streams, etc., and the number and structure of data fields vary significantly, making it difficult to directly process them in a unified manner.

[0004] The data generated by some devices may contain irregular, missing, or illegal records, which need to be cleaned and repaired before analysis.

[0005] Due to the lack of a unified processing framework, data analysis technicians need to develop parsing code separately for each device, which increases the complexity of system development and maintenance.

[0006] While some general-purpose data analysis platforms exist on the market, these platforms typically only support simple analysis of structured data and lack the ability to uniformly process heterogeneous data from multiple sources. Furthermore, in complex application scenarios such as underground coal mines and those involving high-temperature, low-pressure environments, the diverse data collected by different devices poses a significant challenge to unified analysis and storage. Therefore, a method is urgently needed to efficiently parse, process, and store heterogeneous data from multiple sources to support subsequent equipment operating status analysis and production indicator monitoring. Summary of the Invention

[0007] In order to solve the above technical problems, the present application provides a method for collecting device data based on multi-source heterogeneity, the method comprising:

[0008] Collect device data from multiple devices;

[0009] According to the data format and characteristics of each device, a corresponding data parsing code is configured, and the collected device data is parsed based on the data parsing code to obtain device parsed data;

[0010] Extract the attributes and point addresses of each device from the device parsed data, parse the coding structure of the point addresses, identify the function type or physical quantity information contained in the point addresses according to the preset parsing rules, and map them to the corresponding device data items;

[0011] Extracting the data value corresponding to the point address;

[0012] For the extracted data, illegal data is filtered according to preset rules, and the type of legal data is converted;

[0013] Organize the target data after verification and type conversion into a standardized and unified JSON data structure;

[0014] Parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

[0015] Optionally, filtering out illegal data according to preset rules and performing type conversion on legal data from the extracted data includes:

[0016] Verify whether each extracted data item conforms to a predetermined format, including whether numerical data conforms to a numerical range and whether string data conforms to a regular expression matching rule;

[0017] Convert legal string types to floating point or integer data;

[0018] Convert state data into Boolean values ​​according to preset conversion rules;

[0019] Converts a specific data item to a predefined enumeration type or a constant value.

[0020] Optionally, configuring corresponding data parsing code according to the data format and characteristics of each device, and parsing the collected device data based on the data parsing code to obtain device parsed data includes:

[0021] Identify the data type and data structure of each device, determine the characteristics of the device data, and determine the mapping relationship between device attributes and device data;

[0022] Configuring a parsing function according to characteristics of the device data;

[0023] The parsing function is called to parse the device data of each device, and the nested structure is processed using a recursive method to obtain device parsed data.

[0024] Optionally, identifying the data type and data structure of each device, determining the characteristics of the device data, and determining the mapping relationship between the device attributes and the device data includes:

[0025] For each device, query the relevant point data from the pre-configured plc_dianbiao table according to the device name;

[0026] Use regular expressions to extract the corresponding bit number from the device name, and obtain the device's point data based on the bit number;

[0027] Use the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value;

[0028] Store all devices and their corresponding point data in the result dictionary.

[0029] Optionally, configuring a parsing function according to characteristics of the device data includes:

[0030] Configure the data parsing code through the getWh() function and dwToObject() function, and encapsulate the device's point data into a preset format.

[0031] Optionally, organizing the target data after verification and type conversion into a standardized and unified JSON data structure includes:

[0032] Determine the hierarchical structure of the data;

[0033] Define the data structure of the corresponding layer according to the business requirements of each layer;

[0034] Configure corresponding data processing strategies for each layer.

[0035] Optionally, extracting the attributes and point address of each device from the device parsed data, parsing the coding structure of the point address, identifying the function type or physical quantity information contained in the point address according to a preset parsing rule, and mapping it to the corresponding device data item includes:

[0036] Traverse the device parsing data and obtain the data points of each device one by one;

[0037] Extracting device attributes and point addresses from each data point, wherein the device attributes include device ID and device name; and the point address includes sensor location and function identifier;

[0038] and parsing the coding structure of the point address, and identifying the function type or physical quantity information contained in the point address according to a preset parsing rule;

[0039] According to the function information in the point address, the corresponding device data item is searched through the pre-configured mapping table, and the physical value and unit information are filled in.

[0040] A second aspect of the present application provides a device data acquisition apparatus based on multi-source heterogeneity, comprising:

[0041] A data acquisition unit, used for collecting device data of multiple devices;

[0042] a configuration unit, configured to configure corresponding data parsing codes according to the data format and characteristics of each device, and parse the collected device data based on the data parsing codes to obtain device parsed data;

[0043] A data mapping unit is used to extract the attributes and point addresses of each device from the device parsed data, parse the coding structure of the point addresses, identify the function type or physical quantity information contained in the point addresses according to preset parsing rules, and map them to corresponding device data items;

[0044] A data extraction unit, configured to extract a data value corresponding to the point address;

[0045] A data conversion unit is used to filter out illegal data according to preset rules and perform type conversion on legal data.

[0046] A structure generation unit is used to organize the target data after verification and type conversion into a standardized and unified JSON data structure;

[0047] The data writing unit is used to parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

[0048] A third aspect of the present application provides a device data acquisition system based on multi-source heterogeneity, the device comprising:

[0049] processor, memory, input and output units, and buses;

[0050] The processor is connected to the memory, the input and output unit, and the bus;

[0051] The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0053] It can be seen from the above technical solutions that this application has the following advantages:

[0054] 1. Using custom parsing functions (such as getWh()) can flexibly parse the function or physical quantity information in the point address to meet the needs of different scenarios.

[0055] 2. Extract attributes and point addresses from device parsed data, accurately map device data items, and reduce errors in data processing. After data parsing, filter out illegal data according to preset rules to ensure that the data entering the system is legal and valid, thereby improving data quality.

[0056] 3. Through type conversion and data normalization, data is organized into a unified JSON data structure, facilitating subsequent data analysis, transmission, and storage. JSON data structures are easy to integrate with other systems and facilitate visualization and debugging.

[0057] 4. Writing the parsed data into the target database can realize real-time monitoring of the equipment operation status and subsequent data analysis.

[0058] 5. The method design supports the rapid integration of new devices, which only requires the configuration of new parsing codes without major modifications to the system structure.

[0059] After the data is organized into a unified format (JSON), it can be easily connected to other data analysis platforms or systems, improving the system's scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 This is a flow chart of an embodiment of the method for collecting device data based on multi-source heterogeneity provided in this application;

[0062] Figure 2 This is a flowchart of an embodiment of step S102 in the multi-source heterogeneous device data collection method provided in this application;

[0063] Figure 3 This is a flowchart of an embodiment of step S106 in the multi-source heterogeneous device data collection method provided in this application;

[0064] Figure 4 This is a schematic structural diagram of an embodiment of the multi-source heterogeneous device data acquisition device provided in this application;

[0065] Figure 5 This is a structural diagram of an embodiment of a multi-source heterogeneous device data acquisition system provided in this application. DETAILED DESCRIPTION

[0066] It should be noted that the method provided in this application can be applied to a terminal, a system, or a server. For example, the terminal can be a smartphone, a computer, a tablet computer, a smart TV, a smartwatch, a portable computer terminal, or a fixed terminal such as a desktop computer. For ease of explanation, this application uses a terminal as the execution subject for example.

[0067] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the method provided in this application, which includes:

[0068] S101, collecting device data of multiple devices;

[0069] The system collects data generated by multiple devices during operation. These devices can be industrial equipment, sensor modules, or other data-generating devices. Communication with these devices can be accomplished through various communication protocols, such as UART, SPI, I2C, Modbus, and TCP / IP. Device data collection frequencies and trigger conditions (such as timed or event-triggered) can be defined. The collected raw data is then streamed or batched to the data processing module.

[0070] Specifically, initially select appropriate hardware based on the device's communication interface, such as a serial-to-USB module, Ethernet port, or I2C / SPI interface chip. If data is transmitted via a wireless protocol (such as Wi-Fi, Zigbee, or LoRa), a gateway device must be configured.

[0071] You can define the collection frequency and trigger conditions, for example, using a scheduler (such as Python's schedule library or cron service) to periodically trigger the collection task. Here is an example:

[0072] import schedule

[0073] import time

[0074] def collect_data():

[0075] print("Collecting data from devices...")

[0076] schedule.every(10).seconds.do(collect_data)

[0077] while True:

[0078] schedule.run_pending()

[0079] time.sleep(1)

[0080] You can also trigger acquisition based on device event signals (such as interrupt signals and state changes). For example, a GPIO interrupt can trigger the reading of sensor data.

[0081] Data collection can use synchronous collection or asynchronous collection strategies.

[0082] Synchronous acquisition:

[0083] Single thread polls data from each device one by one.

[0084] Example (serial communication):

[0085] import serial

[0086] ser = serial.Serial('COM3', 9600, timeout=1)while True:

[0087] data = ser.readline().decode('utf-8').strip()

[0088] print("Data received:", data)

[0089] Asynchronous collection:

[0090] Use asynchronous IO or multi-threading to concurrently collect data from multiple devices.

[0091] Example (asynchronous TCP / IP communication):

[0092] import asyncio

[0093] async def collect_device_data(reader, writer):

[0094] data = await reader.read(100)

[0095] print(f"Data received: {data.decode()}")

[0096] writer.close()

[0097] async def main():

[0098] server = await asyncio.start_server(collect_device_data, '127.0.0.1',8888)

[0099] await server.serve_forever()

[0100] asyncio.run(main())

[0101] The collected data can be continuously sent to the data processing module for real-time monitoring. For example, the data stream can be sent to the server via protocols such as WebSocket and MQTT. Before transmission, the data can be formatted in a basic way, such as using the following methods:

[0102] {

[0103] "deviceId": "12345",

[0104] "timestamp": "2025-01-01T10:00:00Z",

[0105] "data": {

[0106] "temperature": 25.3,

[0107] "humidity": 60.5

[0108] }

[0109] }

[0110] This method allows for efficient data collection from multiple devices, meeting real-time or batch processing requirements. It also flexibly supports multiple protocols and data transmission modes, making it suitable for industrial, IoT, and other scenarios.

[0111] S102. Configure corresponding data parsing code according to the data format and characteristics of each device, and parse the collected device data based on the data parsing code to obtain device parsed data;

[0112] In this step, create a description file (such as JSON or XML format) for each device, defining the format and parsing rules of the device data. An example is as follows:

[0113] {

[0114] "deviceId": "12345",

[0115] "protocol": "custom",

[0116] "dataFormat": "binary",

[0117] "fields": [

[0118] {"name": "temperature", "offset": 0, "length": 2, "type": "int16", "unit": "C"},

[0119] {"name": "humidity", "offset": 2, "length": 2, "type": "int16", "unit": "%"} ]

[0121] }

[0122] Then, for each protocol (such as Modbus, UART, TCP / IP), implement an adapter module to communicate with the device and obtain the raw data. You can use the adapter pattern to encapsulate the communication logic, as shown below:

[0123] class ProtocolAdapter:

[0124] def __init__(self, protocol_type):

[0125] self.protocol_type = protocol_type

[0126] def fetch_data(self, raw_data):

[0127] if self.protocol_type == "modbus":

[0128] return self._parse_modbus(raw_data)

[0129] elif self.protocol_type == "custom":

[0130] return self._parse_custom(raw_data)

[0131] # Add more protocol adaptation logic

[0132] def _parse_modbus(self, raw_data):

[0133] # Modbus data parsing logic

[0134] pass

[0135] def _parse_custom(self, raw_data):

[0136] # Customize protocol parsing logic

[0137] pass

[0138] Parse data based on the device description file. The following are some possible data types and corresponding parsing logic:

[0139] String format data: Use the delimiter to split the fields. The following is an example:

[0140] raw_data = "25.3,60.5"

[0141] fields = raw_data.split(",")

[0142] parsed_data = {"temperature": float(fields[0]), "humidity": float(fields[1])}

[0143] Binary format data: Parse according to field offset and length. The following is an example:

[0144] import struct

[0145] raw_data = b'\x19\x04\x2C\x01' # Sample binary data

[0146] temperature = struct.unpack_from("h", raw_data, 0)[0] / 10.0# Parse int16 type temperature

[0147] humidity = struct.unpack_from("h", raw_data, 2)[0] / 10.0# Parsing int16 type humidity

[0148] parsed_data = {"temperature": temperature, "humidity": humidity}

[0149] For custom protocol data, the frame header, frame body, and checksum field can be parsed according to the protocol document.

[0150] Finally, fields are dynamically extracted based on the device description file, supporting general parsing logic.

[0151] In this step, the device description file is a structured file that defines the data format and parsing rules for each device. It includes the following:

[0152] Device identification information: uniquely identifies the device (such as device ID or type).

[0153] Field definition: describes the field of device data, including field name, data offset, data length, data type and unit.

[0154] Protocol Type: Specifies the communication protocol used by the device (such as Modbus, custom protocol, etc.).

[0155] In this step, you need to extract the corresponding field value from the device's raw data based on the offset and length in the field definition. The offset determines the starting position of the field in the data stream. The length determines the size of the field data.

[0156] The data extracted from the device is usually raw binary or text data and needs to be converted according to the data type in the description file.

[0157] Integer type (int16): Parses binary data into integers.

[0158] Float: Parses binary data into floating-point numbers.

[0159] String type (string): directly intercept and convert to text.

[0160] The system also needs to perform the following dynamic mapping operations on each field:

[0161] Traverse the field definitions in the description file.

[0162] Read field name (name): Generate a unified field name to facilitate subsequent data processing.

[0163] Extract field value: Extract the corresponding value from the data stream based on the offset and length.

[0164] Type and unit conversion: Ensures that values ​​conform to defined types and units.

[0165] As attached Figure 2 As shown, in an optional implementation, the specific implementation of this step may include:

[0166] S1021. Identify the data type and data structure of each device, determine the characteristics of the device data, and determine the mapping relationship between device attributes and device data;

[0167] In this step, the device data type and data structure need to be identified. For the device data type, the data can have various forms depending on the device type, for example, it can include:

[0168] Numerical data: For example, sensor readings such as temperature, humidity, and pressure.

[0169] String data: such as device status, alarm information, etc.

[0170] Binary data: The device may send raw binary data, which needs to be decoded to obtain valid information.

[0171] JSON format data: Some devices may directly transmit data in JSON format.

[0172] Custom protocol data: Some devices use custom data protocols, which may contain specific fields such as header, body, and checksum.

[0173] Analyze the original data structures of different devices and identify the main fields contained in the data.

[0174] For example, for a sensor device, the data structure may contain fields such as timestamp, sensorId, value, and unit.

[0175] For devices based on the Modbus protocol, data may be transmitted in the form of function codes, data registers, CRC checks, etc., which need to be unpacked.

[0176] Determine the mapping between device attributes and device data. First, identify the device's primary attributes (such as device ID, type, model, and operating status). For example, a temperature and humidity sensor might have attributes such as sensorId, value, and unit. Define mapping rules for each device to its data fields. For example, if a sensor's raw data includes sensorId and reading, map sensorId to deviceId and reading to value.

[0177] S1022. Configuring an analysis function according to characteristics of the device data;

[0178] In this step, you first define a parsing function template and configure the parsing function based on the different data formats of the device. Each device type can have a dedicated parsing function, or a parsing function can be dynamically selected based on the data type. If the device returns numeric data (such as temperature), you can use a simple type conversion function to parse the data, for example:

[0179] def parse_numeric_data(data):

[0180] try:

[0181] return float(data)

[0182] except ValueError:

[0183] return None

[0184] For string data, parsing functions can handle string cutting, cleaning, or regular expression matching, for example:

[0185] def parse_string_data(data):

[0186] if isinstance(data, str):

[0187] return data.strip()# remove spaces, etc.

[0188] return None

[0189] For binary data, you need to write a parsing function to disassemble the binary data packet and extract the valid fields, for example:

[0190] def parse_binary_data(data):

[0191] # Assuming the data structure is known, decompose the binary data into fields

[0192] field1 = data[0:4]# Assume field 1 occupies 4 bytes

[0193] field2 = data[4:8]# Assume field 2 occupies 4 bytes

[0194] return field1, field2

[0195] If the device returns data in JSON format, you can directly use the JSON library to parse it.

[0196] Finally, configure a dedicated parsing function for the different data formats of each device, or use the factory mode to dynamically select the parsing function, or dynamically configure the parsing function according to the device type.

[0197] S1023: Call the parsing function to parse the device data of each device, and use a recursive method to process the nested structure to obtain device parsed data.

[0198] In this step, based on the device type, the corresponding parsing function is obtained from step S1022 and used to parse the raw data. For simple data, the corresponding parsing function, such as parse_numeric_data, can be directly called. For nested data structures (such as JSON containing sub-objects or arrays), the nested fields are parsed recursively. Nested data (such as arrays or dictionaries in JSON objects) requires recursive parsing. Nested fields can be parsed using a recursive function. The following is an example of nested data processing:

[0199] Assume that the device data structure is as follows:

[0200] "sensor": {

[0201] "id": "sensor_1",

[0202] "data": {

[0203] "temperature": "23.5",

[0204] "humidity": "55%"

[0205] For nested sensor data structures, a recursive method can be used to parse:

[0206] parsed_data = recursive_parse(raw_data).

[0207] Ultimately, the recursively parsed data will be returned and organized into a unified device parsing data format.

[0208] In the above implementation, the data types and formats of different devices are analyzed to determine the mapping between device attributes and data items. The corresponding parsing function is configured based on the device's data format, and the appropriate parsing function can be dynamically selected through a factory pattern. For data containing nested structures, a recursive method is used to ensure complete data parsing. This approach offers excellent flexibility and scalability, addressing the needs of a wide range of device types and data formats.

[0209] Furthermore, identifying the data type and data structure of each device, determining the characteristics of the device data, and determining the mapping relationship between the device attributes and the device data includes: for each device, querying the relevant point data from the pre-configured plc_dianbiao table according to the device name; using regular expressions to extract the corresponding bit number through the device name, and obtaining the point data of the device according to the bit number; using the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value; storing all devices and their corresponding point data in the result dictionary.

[0210] In this further embodiment, the goal of the step is to query the device point data from the preconfigured database table plc_dianbiao by the device name, extract the corresponding bit number by regular expression, and finally encapsulate the queried data into a dictionary object for storage.

[0211] First, query the point data from the plc_dianbiao table. You can use the device name as the query condition to find the relevant point data of the device from the plc_dianbiao database table. Assume that each device has multiple point records in the database, and each record includes information such as the device name, point name, and point address. An example SQL is as follows:

[0212] SELECT point_name, point_address, description

[0213] FROM plc_dianbiao

[0214] WHERE device_name = ?

[0215] Where device_name is the input device name, and the query returns all point data related to the device name.

[0216] Execute the above query and get the results. You can use a database connection library (such as pyodbc, pymysql, psycopg2, etc.) to extract data from the database.

[0217] Next, use a regular expression to extract the device's tag number. Regular expressions are used to extract the device's tag number (i.e., point address) from the point name or description. Assume that the point data includes the device name and a specific tag number structure. Tag numbers may appear in a specific format (such as "point_1" or "point001"), which requires regular expression extraction. For each point data entry found, use a regular expression to extract the tag number from the point address. For example, if the point name is "point_001" or "point_42," the regular expression will extract "001" or "42" as the tag number.

[0218] Use the dwToObject() function to encapsulate point data. The purpose of the dwToObject() function is to encapsulate the queried point data (such as point name, point address, description, etc.) into a dictionary object and convert it into a unified data format. Assume that this function creates a dictionary object with the description field as the key and the point_address field as the value.

[0219] Finally, the device and point data are stored in the result dictionary, with the device name as the key and the point data (in dictionary format) as the value. Each device's point data is encapsulated as a dictionary object and stored in the result by device name. This dictionary can be used for subsequent data processing, transmission, or storage (for example, storing data in a database, file, or cache). During data parsing, the result dictionary can be used to quickly find devices and their corresponding point data.

[0220] S103, extracting the attributes and point address of each device from the device parsed data, parsing the coding structure of the point address, identifying the function type or physical quantity information contained in the point address according to a preset parsing rule, and mapping it to the corresponding device data item;

[0221] In this step, device attributes are basic identification information of the device, such as device ID, device type, etc. This information is extracted to accurately classify and manage the device in subsequent operations.

[0222] Extract attribute fields from device parsed data. Example fields include: deviceId, deviceType, and location. Store the extracted attributes in a unified structure to identify the data source. Here's a code example:

[0223] def extract_device_properties(parsed_data):

[0224] device_properties = {

[0225] "deviceId": parsed_data.get("deviceId"),

[0226] "deviceType": parsed_data.get("deviceType"),

[0227] "location": parsed_data.get("location"),

[0228] }

[0229] return device_properties

[0230] In this step, the point address is a field in the data that indicates the monitoring point of the device. The point address can contain hierarchical information such as device module, physical quantity type, or specific function.

[0231] Specifically, when parsing a point address field, it is necessary to split the point address field into multiple parts, for example, by using separators (such as . or _), and separate the information related to the function or physical quantity.

[0232] Point address example:

[0233] Point address format: Mod1.Sensor1.Temp

[0234] In this example, Mod1 indicates the module number, Sensor1 indicates the sensor number, and Temp indicates the physical quantity type (such as temperature).

[0235] Then use the getWh() function to extract physical quantity information. getWh() is a custom function used to extract function or physical quantity information (such as temperature, pressure, current, etc.) from the field of the point address.

[0236] Specifically, the parsed field is passed to the getWh() function, and the physical quantity type is identified using predefined mapping rules (such as a dictionary or regular expression).

[0237] Rule example:

[0238] "Temp" maps to temperature in degrees Celsius.

[0239] "Pres" maps to pressure (unit: Pa).

[0240] "Curr" is mapped to current (unit: A).

[0241] Finally, the custom function outputs the physical quantity type and unit.

[0242] Here is a code example of a mapping:

[0243] def getWh(parameter):

[0244] # Predefined mapping rules

[0245] mapping = {

[0246] "Temp": {"type": "Temperature", "unit": "Celsius"},

[0247] "Pres": {"type": "Pressure", "unit": "Pa"},

[0248] "Curr": {"type": "Current", "unit": "A"}

[0249] }

[0250] return mapping.get(parameter, {"type": "Unknown", "unit": ""})

[0251] Establish clear associations between extracted physical quantity information and device data items to facilitate subsequent data analysis and monitoring. Device attributes, point addresses, and physical quantity information can be combined to organize parsed data items into a unified structure.

[0252] S104, extracting the data value corresponding to the point address;

[0253] In this step, the point address is used to locate the specific monitoring point (e.g., sensor number, functional module, etc.) of the device from the parsed data. The data value is the actual monitoring value or data set corresponding to the point address, which is used to reflect the operating status or output of the device.

[0254] Specifically, first record the device parsed data one by one to ensure that all monitoring points are covered. Determine the location field of the data value through the parsed point address. The point address can be matched in the following ways:

[0255] Exact match: The point address is exactly the same as the field in the data record.

[0256] Fuzzy match: A portion of the point address matches a field in the data record (such as module number or function type).

[0257] Based on the matching results, the corresponding data value is extracted from the record. The data value can be:

[0258] Single value: single point reading such as temperature, pressure, current, etc.

[0259] Data array: such as spectrum analysis data, time series data, etc.

[0260] Storing the extracted data values ​​in a structured format (such as a dictionary or JSON) ensures ease of subsequent processing.

[0261] An example of device parsing data is as follows: [

[0263] {"pointAddress": "Mod1.Sensor1.Temp", "value": 25.5},

[0264] {"pointAddress": "Mod1.Sensor2.Pres", "value": 101325},

[0265] {"pointAddress": "Mod2.Sensor3.Freq", "value": [100, 200, 300]} ]

[0267] The following is an example of code implementation for extracting data values:

[0268] def extract_data_values(parsed_data, point_addresses):

[0269] """

[0270] Extract the monitoring value corresponding to the point address.

[0271] :param parsed_data: Device parsed data (list form).

[0272] :param point_addresses: The list of point addresses to be extracted.

[0273] :return: Structured data containing the point address and the corresponding monitoring value.

[0274] """

[0275] extracted_values ​​= []

[0276] for point_address in point_addresses:

[0277] for record in parsed_data:

[0278] # Matching point address

[0279] if record.get("pointAddress") == point_address:

[0280] extracted_values.append({

[0281] "pointAddress": point_address,

[0282] "value": record.get("value")

[0283] })

[0284] break# Exit the inner loop directly after extracting the value

[0285] return extracted_values

[0286] In this application, fuzzy matching can also be used to match point addresses. An example of fuzzy matching is as follows:

[0287] def extract_data_with_partial_match(parsed_data, point_prefix):

[0288] """

[0289] Extraction functions that support fuzzy matching.

[0290] :param parsed_data: Device parsed data (list form).

[0291] :param point_prefix: Point address prefix (for fuzzy matching).

[0292] :return: Structured data containing the point address and the corresponding monitoring value.

[0293] """

[0294] extracted_values ​​= []

[0295] for record in parsed_data:

[0296] if record.get("pointAddress", "").startswith(point_prefix):

[0297] extracted_values.append({

[0298] "pointAddress": record.get("pointAddress"),

[0299] "value": record.get("value")

[0300] })

[0301] return extracted_values.

[0302] S105: For the extracted data, filter out illegal data according to preset rules and perform type conversion on legal data;

[0303] In this embodiment, the extracted data may contain outliers, format errors, or missing values. These values ​​must be filtered out or corrected before they can be processed further. Pre-set rules are used to verify and filter valid data and perform necessary type conversions (e.g., data format and unit conversion) to ensure data normalization.

[0304] Based on business needs, you can set rules to filter illegal data, for example:

[0305] The values ​​must be within a reasonable range (e.g. temperature: -50°C to 150°C).

[0306] Make sure the value is of the correct type (e.g. it must be a floating point number).

[0307] Field cannot be empty or missing.

[0308] Filter data based on rules, filtering out illegal data or marking it as anomaly. Perform conversions on data that meets the rules, such as converting strings to floating-point numbers or converting temperatures from Fahrenheit to Celsius.

[0309] The following is further explained through examples:

[0310] Assume the extracted data is as follows: [

[0312] {"pointAddress": "Sensor1.Temp", "value": "25.5"},

[0313] {"pointAddress": "Sensor2.Pres", "value": 101325},

[0314] {"pointAddress": "Sensor3.Humidity", "value": null},

[0315] {"pointAddress": "Sensor4.Temp", "value": "-200"},

[0316] {"pointAddress": "Sensor5.Temp", "value": "InvalidData"} ]

[0318] The following is a code example of filtering and data conversion based on the extracted data:

[0319] def validate_and_convert_data(data, rules):

[0320] """

[0321] Filter illegal data and perform type conversion.

[0322] :param data: The extracted data list, each record contains the point address and value.

[0323] :param rules: Validation rule dictionary, defining the range and type of each data.

[0324] :return: Two lists: valid data and invalid data.

[0325] """

[0326] valid_data = []

[0327] invalid_data = []

[0328] for record in data:

[0329] point_address = record.get("pointAddress")

[0330] value = record.get("value")

[0331] # Find the corresponding rules

[0332] rule = rules.get(point_address)

[0333] if not rule:

[0334] # If no rules are defined, mark it as illegal data

[0335] invalid_data.append({**record, "reason": "No rule defined"})

[0336] continue

[0337] # Data integrity check

[0338] if value is None or value == "":

[0339] invalid_data.append({**record, "reason": "Value is missing"})

[0340] continue

[0341] # Data type check and attempt type conversion

[0342] try:

[0343] converted_value = float(value)

[0344] except ValueError:

[0345] invalid_data.append({**record, "reason": "Invalid format"})

[0346] continue

[0347] # Data range check

[0348] if not (rule["min"]<= converted_value<= rule["max"]):

[0349] invalid_data.append({**record, "reason": "Out of range"})

[0350] continue

[0351] # Pass verification and add legal data

[0352] valid_data.append({

[0353] "pointAddress": point_address,

[0354] "value": converted_value

[0355] })

[0356] return valid_data, invalid_data.

[0357] In a possible implementation, this step includes:

[0358] S1051. Verify whether each extracted data item conforms to a predetermined format, wherein the predetermined format includes whether numerical data conforms to a numerical range and whether string data conforms to a regular expression matching rule;

[0359] Define the value range according to the specific requirements of each data item. For example, the temperature data is required to be between -50 degrees Celsius and 150 degrees Celsius, and the pressure data is required to be between 0 and 1000 bar.<min_value or value> Max_value) to verify that the extracted value falls within the permitted range. Define regular expressions to match data in a specific format. Use regular expression tools (such as Python's re.match) to verify that the string conforms to the expected format. If the data does not conform to the predetermined format, mark it as invalid and record the reason for the invalidity. Use the logging system to record detailed information for subsequent analysis.

[0360] S1052. Convert the legal string type into floating point number or integer data;

[0361] Determine whether the data item is of string type (such as isinstance(value, str)).

[0362] Ensure that the string conforms to a valid numeric format, for example, by checking whether it is a pure number or a floating-point number with a decimal point. Perform type conversion. If the string is in a valid integer format, use int() to convert it to an integer. If the string is in a valid floating-point format, use float() to convert it to a floating-point number. If the conversion fails (for example, if the data contains illegal characters), catch the exception and log it. Mark the data that cannot be converted as invalid and return it to the processing module.

[0363] S1053, converting the status data into a Boolean value according to a preset conversion rule;

[0364] Define the possible values ​​of state data, such as "ON" / "OFF", "TRUE" / "FALSE", or "1" / "0". Use a dictionary or mapping relationship to define the conversion rules. For example:

[0365] status_mapping = {"ON": True, "OFF": False, "1": True, "0": False}

[0366] Perform the conversion, mapping the status value to a Boolean value according to the preset rules. If the status value does not match the rules, it is marked as invalid data and recorded. If the status data is missing or formatted abnormally, a default value (such as False) can be set to ensure that subsequent processing is not affected.

[0367] S1054. Convert the specific data item into a predefined enumeration type or constant value.

[0368] Define enumeration rules. The enumeration type lists all possible values ​​of the data and the corresponding constant values. For example:

[0369] device_type_mapping = {

[0370] "TemperatureSensor": 1,

[0371] "PressureSensor": 2,

[0372] "FlowSensor": 3

[0373] }

[0374] The mapping rule searches for a corresponding enumeration value based on the data content. If a match is found, the data is replaced with the corresponding constant value. If the data item does not match an enumeration value, it is marked as invalid or set to the default value (such as UNKNOWN = -1).

[0375] Through the above steps, this implementation systematically verifies data, converts formats, and unifies types, ensuring that input data meets standardization requirements and laying a solid foundation for subsequent processing and storage. The rule configuration and logging mechanisms in each step facilitate dynamic adjustment and maintenance, while also enhancing the robustness of the system.

[0376] S106, organizing the target data after verification and type conversion into a standardized and unified JSON data structure;

[0377] In this embodiment, in order to ensure that data from different devices can be uniformly processed and transmitted, a standardized JSON data structure must be defined. The following is an example of a possible JSON data structure:

[0378] {

[0379] "deviceId": "string", / / device unique identifier

[0380] "timestamp": "string", / / Timestamp of data collection time

[0381] "dataItems": [ / / Data item list

[0382] {

[0383] "address": "string", / / Point address (such as sensor number or function module address)

[0384] "value": "number|string", / / data value (may be a numerical value or a status value)

[0385] "unit": "string", / / data unit (such as C, kPa, etc.)

[0386] "status": "string" / / Data status (such as legal, illegal, missing, etc., optional)

[0387] } ]

[0389] }

[0390] In the above examples, the device identifier (deviceId) is provided by the device description file or acquisition module. It is usually the unique ID or name of the device.

[0391] For example: "deviceId":"sensor_12345"

[0392] The timestamp is the time when the data was collected, formatted in the ISO8601 standard format (such as "2025-01-01T10:00:00Z"). The timestamp can be obtained through the time library of the programming language, for example:

[0393] from datetime import datetime

[0394] timestamp=datetime.utcnow().isoformat()+"Z"

[0395] Each data item in dataItems can include the following fields:

[0396] address: Point address, provided by device parsing data.

[0397] value: data value, the legal value after verification and conversion.

[0398] unit: The unit information of the data, obtained from the description file or point address parsing.

[0399] status: Optional field that records the data status (such as "valid", "invalid").

[0400] The specific way to organize and convert to JSON objects can be to use a dictionary or object in a programming language (such as Python) as a template, for example:

[0401] json_data = {

[0402] "deviceId": device_id,

[0403] "timestamp": timestamp,

[0404] "dataItems": []

[0405] }

[0406] Traverse the verified and converted data and construct a dataItems list. For example:

[0407] for data in validated_data:# validated_data is the validated data list

[0408] data_item = {

[0409] "address": data["address"],

[0410] "value": data["value"],

[0411] "unit": data.get("unit", ""), # The default unit is empty

[0412] "status": data.get("status", "valid")# The default status is valid

[0413] }

[0414] json_data["dataItems"].append(data_item)

[0415] Use the JSON library to serialize the dictionary object into a JSON format string. For example:

[0416] import json

[0417] json_string=json.dumps(json_data,indent=4)

[0418] Finally, the generated JSON data can be transmitted through a network interface (such as HTTP / HTTPS), and the JSON data can be stored in a file system or database. Specifically, the JSON data can be stored in a relational database (such as MySQL's JSON field) or a non-relational database (such as MongoDB).

[0419] The following is an example of organized JSON data:

[0420] {

[0421] "deviceId": "sensor_12345",

[0422] "timestamp": "2024-11-01T10:00:00Z",

[0423] "dataItems": [

[0424] {

[0425] "address": "point1",

[0426] "value": 23.5,

[0427] "unit": "C",

[0428] "status": "valid"

[0429] },

[0430] {

[0431] "address": "point2",

[0432] "value": 101.3,

[0433] "unit": "kPa",

[0434] "status": "valid"

[0435] },

[0436] {

[0437] "address": "point3",

[0438] "value": null,

[0439] "unit": "",

[0440] "status": "invalid"

[0441] } ]

[0443] }

[0444] As attached Figure 3 As shown, in a possible embodiment, an implementation of this step includes:

[0445] S1061. Determine the hierarchical structure of data;

[0446] This step aims to determine the hierarchical structure of the entire data model, ensuring that the data organization can be clearly mapped to actual business needs. The data hierarchy can be divided into multiple levels, such as the device level, the point level, and the data value level.

[0447] An example of a hierarchy is as follows:

[0448] Device layer: basic information about the device itself, such as device ID, device name, device type, etc.

[0449] Point layer: information of each monitoring point in the device, including point address, description, unit, etc.

[0450] Data value layer: actual measured values ​​or calculated results, such as temperature, pressure, humidity, etc.

[0451] First, determine the role and data content of each layer, and determine the hierarchical relationship between the data. For example:

[0452] Device layer: Contains device identification information, which may include device hardware information, connection status, etc.

[0453] Point layer: contains information about each monitoring point in the device and defines the specific attributes of each monitoring point (such as point address, function, unit, etc.).

[0454] Data layer: corresponds to the data collected in real time at the monitoring point, which may include sensor readings, calculated parameters, status values, etc.

[0455] More levels can be divided according to different business needs, such as "sub-device layer" or "functional module layer".

[0456] S1062. Define the data structure of each layer according to the business requirements of the layer;

[0457] In this step, you need to define the data structures required for each layer based on the data requirements of different layers. These data structures must be able to meet business needs and support efficient storage and access of data.

[0458] Device layer data structure definition:

[0459] Device ID (device_id): The ID that uniquely identifies the device.

[0460] Device name (device_name): The name or description of the device.

[0461] Device type (device_type): The category or type of device, such as "temperature sensor" or "pressure sensor".

[0462] Point layer data structure definition:

[0463] Point address (point_address): The address that uniquely identifies the point.

[0464] Description: The function or description of the point, such as "temperature sensor".

[0465] Unit: Data unit, such as "Celsius", "Pa", etc.

[0466] Data layer data structure definition:

[0467] Data value: The data value read from the device, such as temperature, pressure, etc.

[0468] Timestamp: The time when the data was collected.

[0469] Unit: The unit of data, such as "Celsius", "Pascal", etc.

[0470] S1063. Configure corresponding data processing strategies for each layer.

[0471] Define appropriate processing strategies for data at each layer, including data verification, format conversion, cleaning and aggregation, to ensure the standardization and consistency of the data processing process.

[0472] Device layer data processing strategy:

[0473] Check the validity and uniqueness of the device ID and device name to ensure the accuracy of device information.

[0474] Monitor the connection status and working status of the device to ensure that the device is in normal working condition.

[0475] Point layer data processing strategy:

[0476] Check whether the point address conforms to the predetermined format to avoid data errors.

[0477] For some point data, it may be necessary to aggregate the data according to specific rules, such as aggregating multiple temperature sensor data of a certain device into the average value of a region.

[0478] Data value layer data processing strategy:

[0479] Verify each collected data value, for example, check whether the value is within a reasonable range, check whether the units are consistent, etc.

[0480] Ensure data type consistency and convert data values ​​to a standard format, such as converting string values ​​to floating-point numbers or integers.

[0481] Filter out outliers based on set rules (such as temperature range limits).

[0482] In this implementation, the data roles and hierarchical relationships at each layer are clearly defined to ensure clear data processing logic. A data structure that meets business requirements is designed for each layer to ensure flexible and efficient data storage and access. Appropriate data processing strategies are configured for each layer to ensure data quality, reliability, and consistency.

[0483] S107: Parse the generated JSON data structure and write the parsed data into the target database for data analysis and device operation status monitoring.

[0484] In this step, the content is extracted from the JSON data structure generated in the previous step and converted into a format suitable for storage.

[0485] If the JSON data is stored in a file or received from an interface, load the data first. An example of loading data is as follows:

[0486] import json

[0487] # Load from string

[0488] json_string = '{"deviceId": "sensor_12345", "timestamp": "2025-01-01T10:00:00Z", "dataItems": [...]}'

[0489] data = json.loads(json_string)

[0490] # Load from file

[0491] with open("data.json", "r") as file:

[0492] data = json.load(file)

[0493] Extract device ID, timestamp, and data items, for example:

[0494] device_id = data["deviceId"]

[0495] timestamp = data["timestamp"]

[0496] data_items = data["dataItems"]

[0497] Before writing to the database, you need to ensure that the target database has been created and the corresponding table structure has been configured. An example of a database table is as follows:

[0498] CREATE TABLE DeviceData (

[0499] id INT AUTO_INCREMENT PRIMARY KEY,

[0500] deviceId VARCHAR(255),

[0501] timestamp DATETIME,

[0502] address VARCHAR(255),

[0503] value FLOAT,

[0504] unit VARCHAR(50),

[0505] status VARCHAR(50) );

[0507] You can use an appropriate database driver to connect to a database, such as MySQL, PostgreSQL, or SQLite.

[0508] The specific implementation of the data write operation is as follows:

[0509] First, prepare the SQL statement and construct the insert SQL statement to write the extracted data into the database, for example:

[0510] INSERT INTO DeviceData (deviceId, timestamp, address, value, unit, status)

[0511] VALUES (%s, %s, %s, %s, %s, %s);

[0512] Iterate through the dataItems and write them to the database one by one. Once the data is written, commit the transaction to save the changes. This step can also catch potential exceptions in database operations to prevent program crashes. For example, before writing, verify the integrity and validity of the data, ensuring that all required fields (such as deviceId and address) are not null, and that the value field contains a valid value. Once the data is written to the database, query operations can be used to monitor device operating status and analyze the data.

[0513] The above embodiments describe the device data collection method based on multi-source heterogeneity provided by this application. The following describes embodiments of the device, system, and storage medium provided in this application:

[0514] See Figure 4 The present application provides an embodiment of a device data acquisition apparatus based on multi-source heterogeneity, the embodiment comprising:

[0515] The data collection unit 401 is used to collect device data of multiple devices;

[0516] The configuration unit 402 is configured to configure corresponding data parsing codes according to the data format and characteristics of each device, and parse the collected device data based on the data parsing codes to obtain device parsed data;

[0517] The data mapping unit 403 is used to extract the attributes and point addresses of each device from the device parsed data, and extract the function or physical quantity information in the point address through the custom parsing function getWh(), and map it to the corresponding device data item;

[0518] A data extraction unit 404 is used to extract the data value corresponding to the point address;

[0519] The data conversion unit 405 is used to filter out illegal data according to preset rules and perform type conversion on legal data.

[0520] The structure generation unit 406 is used to organize the target data after verification and type conversion into a standardized and unified JSON data structure;

[0521] The data writing unit 407 is used to parse the generated JSON data structure and write the parsed data into the target database for data analysis and device operation status monitoring.

[0522] Optionally, the data conversion unit 405 is specifically configured to:

[0523] Verify whether each extracted data item conforms to a predetermined format, including whether numerical data conforms to a numerical range and whether string data conforms to a regular expression matching rule;

[0524] Convert legal string types to floating point or integer data;

[0525] Convert state data into Boolean values ​​according to preset conversion rules;

[0526] Converts a specific data item to a predefined enumeration type or a constant value.

[0527] Optionally, the configuration unit 402 is specifically configured to:

[0528] Identify the data type and data structure of each device, determine the characteristics of the device data, and determine the mapping relationship between device attributes and device data;

[0529] Configuring a parsing function according to characteristics of the device data;

[0530] The parsing function is called to parse the device data of each device, and the nested structure is processed using a recursive method to obtain device parsed data.

[0531] Optionally, the configuration unit 402 is specifically configured to:

[0532] For each device, query the relevant point data from the pre-configured plc_dianbiao table according to the device name;

[0533] Use regular expressions to extract the corresponding bit number from the device name, and obtain the device's point data based on the bit number;

[0534] Use the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value;

[0535] Store all devices and their corresponding point data in the result dictionary.

[0536] Optionally, the configuration unit 402 is specifically configured to:

[0537] Configure the data parsing code through the getWh() function and dwToObject() function, and encapsulate the device's point data into a preset format.

[0538] Optionally, the structure generating unit 406 is specifically configured to:

[0539] Determine the hierarchical structure of the data;

[0540] Define the data structure of the corresponding layer according to the business requirements of each layer;

[0541] Configure corresponding data processing strategies for each layer.

[0542] The data mapping unit 403 is specifically used for:

[0543] Traverse the device parsing data and obtain the data points of each device one by one;

[0544] Extracting device attributes and point addresses from each data point, wherein the device attributes include device ID and device name; and the point address includes sensor location and function identifier;

[0545] Parse the function information and physical quantity information in the point address through the custom parsing function getWh();

[0546] According to the function information in the point address, the corresponding device data item is searched through the pre-configured mapping table, and the physical value and unit information are filled in.

[0547] See Figure 5 , the present application also provides a device data acquisition device based on multi-source heterogeneity, comprising:

[0548] Processor 501, memory 502, input and output unit 503, bus 504;

[0549] The processor 501 is connected to the memory 502, the input and output unit 503 and the bus 504;

[0550] The memory 502 stores a program, and the processor 501 calls the program to execute any of the above-mentioned device data collection methods based on multi-source heterogeneity.

[0551] The present application also relates to a computer-readable storage medium having a program stored thereon, wherein when the program is run on a computer, the computer is caused to execute any of the above-mentioned methods for collecting device data based on multi-source heterogeneity.

[0552] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0553] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0554] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0555] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0556] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, and other media that can store program code.

Claims

1. A method for collecting device data based on multi-source heterogeneity, characterized in that: The method comprises: Collect device data from multiple devices; According to the data format and characteristics of each device, a corresponding data parsing code is configured, and the collected device data is parsed based on the data parsing code to obtain device parsed data; Extract the attributes and point addresses of each device from the device parsed data, parse the coding structure of the point addresses, identify the function type or physical quantity information contained in the point addresses according to the preset parsing rules, and map them to the corresponding device data items; Extracting data corresponding to the point address; For the extracted data, illegal data is filtered according to preset rules, and the type of legal data is converted; Organize the target data after verification and type conversion into a standardized and unified JSON data structure; Parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring; The process of extracting the attributes and point addresses of each device from the device parsed data, parsing the coding structure of the point addresses, identifying the function type or physical quantity information contained in the point addresses according to preset parsing rules, and mapping them to corresponding device data items includes: Traverse the device parsing data and obtain the data points of each device one by one; Extracting device attributes and point addresses from each data point, wherein the device attributes include device ID and device name; the point address includes sensor location and function identifier; Parsing the coding structure of the point address and identifying the function type or physical quantity information contained in the point address according to a preset parsing rule; According to the function type or physical quantity information in the point address, the corresponding device data item is searched through the pre-configured mapping table, and the physical quantity value and unit information are filled in; The step of configuring corresponding data parsing codes according to the data format and characteristics of each device and parsing the collected device data based on the data parsing codes to obtain device parsed data includes: Identify the data type and data structure of each device, determine the characteristics of the device data, and determine the mapping relationship between device attributes and device data; Configuring a parsing function according to characteristics of the device data; Calling the parsing function to parse the device data of each device, and using a recursive method to process the nested structure to obtain device parsed data; Configuring the parsing function according to the characteristics of the device data includes: The data parsing code is configured through the getWh() function and the dwToObject() function, and the device's point data is encapsulated into a preset format. The getWh() function is a custom function used to extract the function type or physical quantity information from the field of the point address. The function type or physical quantity information includes temperature, pressure, and current; the dwToObject() function is used to encapsulate the queried point data into a dictionary object and convert it into a unified data format.

2. The method for collecting device data based on multi-source heterogeneity according to claim 1, characterized in that: The extracted data is filtered according to preset rules to filter illegal data and perform type conversion on legal data, including: Verify whether each extracted data item conforms to a predetermined format, including whether numerical data conforms to a numerical range and whether string data conforms to a regular expression matching rule; Convert legal string types to floating point or integer data; Convert state data into Boolean values ​​according to preset conversion rules; Converts a specific data item to a predefined enumeration type or a constant value.

3. The method for collecting device data based on multi-source heterogeneity according to claim 1, characterized in that: The identifying of the data type and data structure of each device, determining the characteristics of the device data, and determining the mapping relationship between the device attributes and the device data includes: For each device, query the relevant point data from the pre-configured plc_dianbiao table according to the device name; Use regular expressions to extract the corresponding bit number from the device name, and obtain the device's point data based on the bit number; Use the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value; Store all devices and their corresponding point data in the result dictionary.

4. The method for collecting device data based on multi-source heterogeneity according to claim 1, characterized in that: The step of organizing the target data after verification and type conversion into a standardized and unified JSON data structure includes: Determine the hierarchical structure of the data; Define the data structure of the corresponding layer according to the business requirements of each layer; Configure corresponding data processing strategies for each layer.

5. A device data acquisition device based on multi-source heterogeneity, characterized in that: For performing the method according to any one of claims 1 to 4, the apparatus comprises: A data acquisition unit, used for collecting device data of multiple devices; a configuration unit, configured to configure corresponding data parsing codes according to the data format and characteristics of each device, and parse the collected device data based on the data parsing codes to obtain device parsed data; A data mapping unit is used to extract the attributes and point addresses of each device from the device parsed data, parse the coding structure of the point addresses, identify the function type or physical quantity information contained in the point addresses according to preset parsing rules, and map them to corresponding device data items; A data extraction unit, configured to extract data corresponding to the point address; A data conversion unit is used to filter out illegal data according to preset rules and perform type conversion on legal data. A structure generation unit is used to organize the target data after verification and type conversion into a standardized and unified JSON data structure; The data writing unit is used to parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

6. A multi-source heterogeneous equipment data acquisition system, characterized in that: include: processor, memory, input and output units, and buses; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 4. 7 . A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to claim 1 .

Images