Equipment data collection method, device and system based on multi-source heterogeneity and storage medium

Abstract

The invention discloses an equipment data collection method, device and system based on multi-source heterogeneity and a storage medium. The method comprises the following steps: collecting device data of a plurality of devices; according to the data format and characteristics of each device, configuring a corresponding data analysis code, and analyzing the collected device data based on the data analysis code to obtain device analysis data; the attribute and the point address of each device are extracted from the device analysis data, and function types or physical quantity information contained in the point addresses are identified and mapped to corresponding device data items; extracting a data value corresponding to the point address; filtering illegal data from the extracted data according to a preset rule, and performing type conversion on legal data; organizing the target data subjected to verification and type conversion into a standard and unified JSON data structure; and analyzing the generated JSON data structure, and writing the analyzed data into a target database for data analysis and equipment operation state monitoring.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, system and storage medium for collecting device data based on multi-source heterogeneity. Background Art

[0002] With the rapid development of industrial Internet of Things and intelligent manufacturing, the operating status data of different scenarios and equipment has gradually become an important resource for enterprises to optimize production and improve efficiency. However, there are significant differences in the data collection methods and data formats of different devices, forming the so-called "multi-source heterogeneous data". The format and attributes of these data are not uniform, which brings huge challenges to data analysis technicians, specifically: The collected data from different devices may be in various formats such as tables, texts, binary streams, etc., and there are significant differences in the number and structure of data fields, making it difficult to directly process them in a unified manner.

[0003] The data generated by some devices may contain irregular, missing or illegal records, which need to be cleaned and repaired before analysis.

[0004] Due to the lack of a unified processing framework, data analysis technicians need to develop parsing codes separately for each device, which increases the complexity of system development and maintenance.

[0005] At present, although there are some general data analysis platforms on the market, these platforms usually only support simple analysis of structured data and have weak unified processing capabilities for multi-source heterogeneous data. At the same time, in complex application scenarios such as underground coal mines, high temperature and low pressure, the data collected by different devices are of various types, making unified analysis and storage more difficult. Therefore, there is an urgent need for a method that can efficiently parse, process and store multi-source heterogeneous data to provide support for subsequent equipment operation status analysis and production indicator monitoring. Summary of the invention

[0006] In order to solve the above technical problems, the present application provides a method for collecting device data based on multi-source heterogeneity, the method comprising: Collect device data from multiple devices; According to the data format and characteristics of each device, a corresponding data parsing code is configured, and the collected device data is parsed based on the data parsing code to obtain device parsing data; Extract the attributes and point addresses of each device from the device parsing data, parse the coding structure of the point addresses, identify the function type or physical quantity information contained in the point addresses according to the preset parsing rules, and map them to the corresponding device data items; Extracting the data value corresponding to the point address; For the extracted data, illegal data is filtered according to preset rules, and the legal data is converted into different types; Organize the target data after verification and type conversion into a standardized and unified JSON data structure; Parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

[0007] Optionally, filtering the extracted data according to preset rules to identify illegal data and performing type conversion on legal data includes: Verify whether each extracted data item conforms to a predetermined format, wherein the predetermined format includes whether the numerical data conforms to a numerical range and whether the string data conforms to a regular expression matching rule; Convert legal string types to floating point or integer data; Convert state data into Boolean values ​​according to preset conversion rules; Converts a specific data item to a predefined enumeration type or a constant value.

[0008] Optionally, configuring a corresponding data parsing code according to the data format and characteristics of each device, and parsing the collected device data based on the data parsing code to obtain device parsed data includes: Identify the data type and data structure of each device, determine the characteristics of the device data, and determine the mapping relationship between device attributes and device data; Configuring a parsing function according to characteristics of the device data; The parsing function is called to parse the device data of each device, and the nested structure is processed using a recursive method to obtain device parsing data.

[0009] Optionally, the identifying the data type and data structure of each device, determining the characteristics of the device data, and determining the mapping relationship between the device attributes and the device data includes: For each device, query the relevant point data from the pre-configured plc_dianbiao table according to the device name; Use regular expressions to extract the corresponding bit number through the device name, and obtain the point data of the device according to the bit number; Use the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value; Store all devices and their corresponding point data in the result dictionary.

[0010] Optionally, configuring the parsing function according to the characteristics of the device data includes: The data parsing code is configured through the getWh() function and dwToObject() function, and the device's point data is encapsulated into a preset format.

[0011] Optionally, organizing the target data after verification and type conversion into a standardized and unified JSON data structure includes: Determine the hierarchical structure of the data; Define the data structure of the corresponding layer according to the business requirements of each layer; Configure corresponding data processing strategies for each layer.

[0012] Optionally, extracting the attributes and point address of each device from the device parsed data, parsing the coding structure of the point address, identifying the function type or physical quantity information contained in the point address according to a preset parsing rule, and mapping to the corresponding device data item includes: Traverse the device parsing data and obtain the data points of each device one by one; Extracting device attributes and point addresses from each data point, wherein the device attributes include device ID and device name; the point address includes sensor location and function identifier; and parsing the coding structure of the point address, and identifying the function type or physical quantity information contained in the point address according to a preset parsing rule; According to the function information in the point address, the corresponding device data item is searched through the preconfigured mapping table, and the physical value and unit information are filled in.

[0013] The second aspect of the present application provides a device data acquisition apparatus based on multi-source heterogeneity, comprising: A data collection unit, used for collecting device data of multiple devices; A configuration unit, configured to configure a corresponding data parsing code according to the data format and characteristics of each device, and parse the collected device data based on the data parsing code to obtain device parsing data; A data mapping unit, used to extract the attributes and point addresses of each device from the device parsing data, parse the coding structure of the point address, identify the function type or physical quantity information contained in the point address according to a preset parsing rule, and map it to the corresponding device data item; A data extraction unit, used to extract the data value corresponding to the point address; A data conversion unit, used to filter out illegal data according to preset rules and perform type conversion on legal data for the extracted data; A structure generation unit, used to organize the target data after verification and type conversion into a standardized and unified JSON data structure; The data writing unit is used to parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

[0014] The third aspect of the present application provides a device data acquisition system based on multi-source heterogeneity, the device comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0015] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0016] It can be seen from the above technical solutions that this application has the following advantages: 1. Use custom parsing functions (such as getWh()) to flexibly parse the function or physical quantity information in the point address to meet the needs of different scenarios.

[0017] 2. Extract attributes and point addresses from device parsed data, accurately map device data items, and reduce errors in data processing. After data parsing, filter out illegal data according to preset rules to ensure that the data entering the system is legal and valid, thereby improving data quality.

[0018] 3. Through type conversion and data normalization, the data is organized into a unified JSON data structure to facilitate subsequent data analysis, transmission and storage. The JSON data structure is easy to integrate with other systems and is also convenient for visualization and debugging.

[0019] 4. Writing the parsed data into the target database can realize real-time monitoring of the equipment operation status and subsequent data analysis.

[0020] 5. The method design supports the rapid access to new devices, which only requires the configuration of new parsing codes without major modifications to the system structure.

[0021] After the data is organized into a unified format (JSON), it can be easily connected to other data analysis platforms or systems to improve the system's scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for collecting device data based on multi-source heterogeneity provided in this application; Figure 2 This is a flow chart of an embodiment of step S102 in the multi-source heterogeneous device data collection method provided in this application; Figure 3 This is a flowchart of an embodiment of step S106 in the multi-source heterogeneous device data collection method provided in this application; Figure 4 This is a schematic diagram of the structure of an embodiment of a device data acquisition apparatus based on multi-source heterogeneity provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of a multi-source heterogeneous device data acquisition system provided in this application. DETAILED DESCRIPTION

[0024] It should be noted that the method provided in this application can be applied to a terminal, a system, or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as an example for explanation.

[0025] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the method provided in this application, and the embodiment includes: S101, collecting device data of multiple devices; The system collects data generated by multiple devices during operation. These devices can be industrial devices, sensor modules, or other data generating devices. You can communicate with the devices through a variety of communication protocols (such as UART, SPI, I2C, Modbus, TCP / IP, etc.). Define the frequency or trigger conditions for device data collection (such as timed collection or event triggering). The collected raw data is transferred to the data processing module in the form of stream or batch processing.

[0026] Specifically, at the beginning, select appropriate hardware according to the communication interface of the device, such as a serial port to USB module, an Ethernet port, an I2C / SPI interface chip, etc. If data is transmitted via a wireless protocol (such as Wi-Fi, Zigbee, or LoRa), a gateway device needs to be configured.

[0027] You can define the collection frequency and trigger conditions, for example, use a scheduler (such as Python's schedule library or cron service) to periodically trigger the collection task. Here is an example: import schedule import time def collect_data(): print("Collecting data from devices...") schedule.every(10).seconds.do(collect_data) while True: schedule.run_pending() time.sleep(1) It is also possible to trigger acquisition based on device event signals (such as interrupt signals, state changes), such as GPIO interrupt triggering the reading of sensor data.

[0028] Data collection can use synchronous collection or asynchronous collection strategy.

[0029] Synchronous acquisition: Single thread polls data from device to device.

[0030] Example (serial communication): import serial ser = serial.Serial('COM3', 9600, timeout=1)while True: data = ser.readline().decode('utf-8').strip() print("Data received:", data) Asynchronous collection: Use asynchronous IO or multi-threading to concurrently collect data from multiple devices.

[0031] Example (asynchronous TCP / IP communication): import asyncio async def collect_device_data(reader, writer): data = await reader.read(100) print(f"Data received: {data.decode()}") writer.close() async def main(): server = await asyncio.start_server(collect_device_data, '127.0.0.1',8888) await server.serve_forever() asyncio.run(main()) The collected data can be continuously sent to the data processing module for real-time monitoring. For example, the data stream can be sent to the server through protocols such as WebSocket and MQTT. Before transmission, the data can be formatted in a basic way, for example, using the following method: { "deviceId": "12345", "timestamp": "2025-01-01T10:00:00Z", "data": { "temperature": 25.3, "humidity": 60.5 } } Through the above method, data from multiple devices can be efficiently collected to meet real-time or batch processing requirements. It flexibly supports multiple protocols and data transmission modes to adapt to industrial, IoT and other scenarios.

[0032] S102, configuring a corresponding data parsing code according to the data format and characteristics of each device, and parsing the collected device data based on the data parsing code to obtain device parsing data; In this step, create a description file (such as JSON or XML format) for each device to define the format and parsing rules of the device data. An example is as follows: { "deviceId": "12345", "protocol": "custom", "dataFormat": "binary", "fields": [ {"name": "temperature", "offset": 0, "length": 2, "type": "int16", "unit": "C"}, {"name": "humidity", "offset": 2, "length": 2, "type": "int16", "unit": "%"} ] } Then implement an adapter module for each protocol (such as Modbus, UART, TCP / IP), which is responsible for communicating with the device and obtaining raw data. The adapter pattern can be used to encapsulate the communication logic, as shown below: class ProtocolAdapter: def __init__(self, protocol_type): self.protocol_type = protocol_type def fetch_data(self, raw_data): if self.protocol_type == "modbus": return self._parse_modbus(raw_data) elif self.protocol_type == "custom": return self._parse_custom(raw_data) # Add more protocol adaptation logic def _parse_modbus(self, raw_data): # Modbus data parsing logic pass def _parse_custom(self, raw_data): # Customize protocol parsing logic pass Parse data based on the device description file. The following are some possible data types and corresponding parsing logic: String format data: Use the delimiter to split the fields. The following is an example: raw_data = "25.3,60.5" fields = raw_data.split(",") parsed_data = {"temperature": float(fields[0]), "humidity": float(fields[1])} Binary format data: Parse according to field offset and length. The following is an example: import struct raw_data = b'\x19\x04\x2C\x01' # Sample binary data temperature = struct.unpack_from("h", raw_data, 0)[0] / 10.0# Parse int16 type temperature humidity = struct.unpack_from("h", raw_data, 2)[0] / 10.0# Parse int16 type humidity parsed_data = {"temperature": temperature, "humidity": humidity} For custom protocol data, the frame header, frame body, and checksum fields can be parsed according to the protocol document.

[0033] Finally, fields are dynamically extracted based on the device description file, supporting general parsing logic.

[0034] In this step, the device description file is a structured file that defines the data format and parsing rules for each device. It includes the following: Device identification information: uniquely identifies the device (such as device ID or type).

[0035] Field definition: describes the field of device data, including field name, data offset, data length, data type and unit.

[0036] Protocol Type: Indicates the communication protocol used by the device (such as Modbus, custom protocol, etc.).

[0037] In this step, you need to extract the corresponding field value from the device's raw data based on the offset and length in the field definition. The offset is used to determine the starting position of the field in the data stream. The length is used to determine the size of the field data.

[0038] The data extracted from the device is usually raw binary or text data and needs to be converted according to the data type in the description file.

[0039] Integer type (int16): Parses binary data into integers.

[0040] Float: Parses binary data into floating point numbers.

[0041] String type: directly intercept and convert to text.

[0042] The system also needs to perform the following dynamic mapping operations on each field: Iterate over the field definitions in the description file.

[0043] Read field name (name): Generate a unified field name to facilitate subsequent data processing.

[0044] Extract field value: Extract the corresponding value from the data stream based on the offset and length.

[0045] Type and unit conversion: Ensure that values ​​conform to defined types and units.

[0046] As attached Figure 2 As shown, in an optional implementation, the specific implementation of this step may include: S1021, identifying the data type and data structure of each device, determining the characteristics of the device data, and determining the mapping relationship between the device attributes and the device data; In this step, the device data type and data structure need to be identified. For the device data type, the data may have various forms depending on the device type, for example, it may include: Numeric data: For example, sensor readings such as temperature, humidity, and pressure.

[0047] String data: such as device status, alarm information, etc.

[0048] Binary data: The device may send raw binary data, which needs to be decoded to obtain valid information.

[0049] JSON format data: Some devices may directly transmit data in JSON format.

[0050] Custom protocol data: Some devices use custom data protocols, which may contain specific header, body, checksum, and other fields.

[0051] Analyze the original data structures of different devices and identify the main fields contained in the data.

[0052] For example, for a sensor device, the data structure may contain fields such as timestamp, sensorId, value, and unit.

[0053] For devices based on the Modbus protocol, data may be transmitted in the form of function codes, data registers, CRC checks, etc., which need to be unpacked.

[0054] Determine the mapping relationship between device attributes and device data. First, identify the main attributes of the device (such as device ID, type, model, working status, etc.). For example, a temperature and humidity sensor may have attributes such as sensorId, value, and unit. Define the mapping rules between each device and the data field. For example: If the raw data of the sensor includes sensorId and reading, you can map sensorId to deviceId and read to value.

[0055] S1022, configuring a parsing function according to the characteristics of the device data; In this step, first define the parsing function template and configure the parsing function according to the different data formats of the device. Each device type can have a dedicated parsing function, or dynamically select the parsing function based on the data type. If the device returns numeric data (such as temperature), a simple type conversion function can be used to parse the data, for example: def parse_numeric_data(data): try: return float(data) except ValueError: return None For string data, parsing functions can handle string cutting, cleaning, or regular expression matching, for example: def parse_string_data(data): if isinstance(data, str): return data.strip()# Remove spaces, etc. return None For binary data, you need to write a parsing function to disassemble the binary data packet and extract the valid fields, for example: def parse_binary_data(data): # Assuming the data structure is known, decompose the binary data into fields field1 = data[0:4]# Assume field 1 occupies 4 bytes field2 = data[4:8]# Assume field 2 occupies 4 bytes return field1, field2 If the device returns data in JSON format, you can directly use the JSON library to parse it.

[0056] Finally, configure a dedicated parsing function for the different data formats of each device, or use the factory mode to dynamically select the parsing function, or dynamically configure the parsing function according to the device type.

[0057] S1023: Call the parsing function to parse the device data of each device, and use a recursive method to process the nested structure to obtain device parsed data.

[0058] In this step, according to the type of device, the corresponding parsing function is obtained from step S1022, and the original data is parsed using the function. For simple data, the corresponding parsing function can be directly called, such as parse_numeric_data. For nested data structures (such as sub-objects or arrays in JSON), the nested fields are parsed recursively. Nested data (such as arrays or dictionaries in JSON objects) requires recursive parsing. The nested fields can be parsed through a recursive function. The following is an example of nested data processing: Assume that the device data structure is as follows: "sensor": { "id": "sensor_1", "data": { "temperature": "23.5", "humidity": "55%" For nested sensor data structures, a recursive method can be used for parsing: parsed_data = recursive_parse(raw_data). Eventually, the recursively parsed data will be returned and organized into a unified device parsed data format.

[0059] In the above implementation, the data types and formats of different devices are analyzed to determine the mapping relationship between device attributes and data items. The corresponding parsing function is configured according to the data format of the device, and the appropriate parsing function can be dynamically selected through the factory mode. For data containing nested structures, a recursive method is used to ensure complete parsing of the data. This method has good flexibility and scalability, and can meet the needs of various device types and data formats.

[0060] Furthermore, identifying the data type and data structure of each device, determining the characteristics of the device data, and determining the mapping relationship between the device attributes and the device data includes: for each device, querying the relevant point data from the preconfigured plc_dianbiao table according to the device name; using regular expressions to extract the corresponding bit number through the device name, and obtaining the device's point data according to the bit number; using the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value; storing all devices and their corresponding point data in the result dictionary.

[0061] In this further embodiment, the goal of the step is to query the point data of the device from the preconfigured database table plc_dianbiao through the device name, extract the corresponding bit number through a regular expression, and finally encapsulate the queried data into a dictionary object for storage.

[0062] First, query the point data from the plc_dianbiao table. You can use the device name as the query condition to find the relevant point data of the device from the database table plc_dianbiao. Assume that each device has multiple point records in the database, and each record includes information such as the device name, point name, and point address. An example of SQL is as follows: SELECT point_name, point_address, description FROM plc_dianbiao WHERE device_name = ? Where device_name is the input device name, and the query returns all point data related to the device name.

[0063] Execute the above query and get the results. You can use a database connection library (such as pyodbc, pymysql, psycopg2, etc.) to extract data from the database.

[0064] Then use regular expressions to extract the bit number of the device. Regular expressions are used to extract the bit number (i.e., point address) of the device from the point name or description, assuming that the point data includes the device name and a specific bit number structure. The bit number may appear in a specific format (such as "point_1", "point001") and needs to be extracted by regular expressions. For each point data queried, use regular expressions to extract the bit number in the point address. For example, if the point name is "point_001" or "point_42", the regular expression will extract "001" or "42" as the bit number.

[0065] Use the dwToObject() function to encapsulate point data. The purpose of the dwToObject() function is to encapsulate the queried point data (such as point name, point address, description, etc.) into a dictionary object and convert it into a unified data format. Assume that this function will create a dictionary object with the description field as the key and the point_address field as the value.

[0066] Finally, the device and point data are stored in the result dictionary. The device name is the key of the result dictionary, and the point data (in dictionary format) is the value. The point data of each device is encapsulated as a dictionary object and stored in the result by device name. The dictionary can be further used for subsequent data processing, transmission or storage (for example, storing data in a database, file, cache, etc.). During the data parsing process, the device and its corresponding point data can be quickly found from the result dictionary.

[0067] S103, extracting the attributes and point address of each device from the device parsed data, parsing the coding structure of the point address, identifying the function type or physical quantity information contained in the point address according to a preset parsing rule, and mapping it to the corresponding device data item; In this step, the device attributes are the basic identification information of the device, such as device ID, device type, etc. This information is extracted to accurately classify and manage the devices in subsequent operations.

[0068] Extract attribute fields from device parsed data. Example fields: deviceId, deviceType, location. Store the extracted attributes into a unified structure to identify the source of the data. Here is a code example: def extract_device_properties(parsed_data): device_properties = { "deviceId": parsed_data.get("deviceId"), "deviceType": parsed_data.get("deviceType"), "location": parsed_data.get("location"), } return device_properties In this step, the point address is a field in the data, which is used to indicate the monitoring point of the device. The point address can contain hierarchical information, such as device module, physical quantity type or specific function.

[0069] Specifically, when parsing the point address field, the point address field needs to be split into multiple parts, for example, by separators (such as . or _), and the information related to the function or physical quantity is separated.

[0070] Point address example: Point address format: Mod1.Sensor1.Temp In this example, Mod1 indicates the module number, Sensor1 indicates the sensor number, and Temp indicates the physical quantity type (such as temperature).

[0071] Then use the getWh() function to extract the physical quantity information. getWh() is a custom function used to extract function or physical quantity information (such as temperature, pressure, current, etc.) from the field of the point address.

[0072] Specifically, the parsed fields are passed to the getWh() function, and the physical quantity type is identified using predefined mapping rules (such as a dictionary or regular expression).

[0073] Example of rule: "Temp" maps to temperature in degrees Celsius.

[0074] "Pres" maps to pressure (in Pa).

[0075] "Curr" maps to current (in A).

[0076] Finally, the custom function outputs the physical quantity type and unit.

[0077] Here is a code example of a mapping: def getWh(parameter): # Predefined mapping rules mapping = { "Temp": {"type": "Temperature", "unit": "Celsius"}, "Pres": {"type": "Pressure", "unit": "Pa"}, "Curr": {"type": "Current", "unit": "A"} } return mapping.get(parameter, {"type": "Unknown", "unit": ""}) Establish clear associations between the extracted physical quantity information and the equipment data items to facilitate subsequent data analysis and monitoring. The parsed data items can be organized into a unified structure by combining equipment attributes, point addresses, and physical quantity information.

[0078] S104, extracting the data value corresponding to the point address; In this step, the point address is used to locate the specific monitoring point of the device (such as sensor number, functional module, etc.) from the device parsed data. The data value is the actual monitoring value or data set corresponding to the point address, which is used to reflect the operating status or output result of the device.

[0079] Specifically, first record the device parsing data one by one to ensure that all monitoring points are covered. Determine the location field of the data value through the parsed point address. The point address can be matched in the following ways: Exact match: The point address is exactly the same as the field in the data record.

[0080] Fuzzy match: A portion of the point address matches a field in the data record (such as module number or function type).

[0081] Based on the matching results, the corresponding data value is extracted from the record. The data value can be: Single value: a single point reading such as temperature, pressure, current, etc.

[0082] Data array: such as spectrum analysis data, time series data, etc.

[0083] Store the extracted data values ​​in a structured format (such as a dictionary or JSON) to ensure easy subsequent processing.

[0084] An example of device parsing data is as follows: [ {"pointAddress": "Mod1.Sensor1.Temp", "value": 25.5}, {"pointAddress": "Mod1.Sensor2.Pres", "value": 101325}, {"pointAddress": "Mod2.Sensor3.Freq", "value": [100, 200, 300]} ] The following is an example of code implementation for extracting data values: def extract_data_values(parsed_data, point_addresses): """ Extract the monitoring value corresponding to the point address.

[0085] :param parsed_data: device parsed data (list form).

[0086] :param point_addresses: The list of point addresses to be extracted.

[0087] :return: Structured data containing the point address and the corresponding monitoring value.

[0088] """ extracted_values ​​= [] for point_address in point_addresses: for record in parsed_data: # Matching point address if record.get("pointAddress") == point_address: extracted_values.append({ "pointAddress": point_address, "value": record.get("value") }) break# Exit the inner loop directly after extracting the value return extracted_values In this application, fuzzy matching can also be used to match point addresses. An example of fuzzy matching is as follows: def extract_data_with_partial_match(parsed_data, point_prefix): """ Extraction functions that support fuzzy matching.

[0089] :param parsed_data: device parsed data (list form).

[0090] :param point_prefix: Point address prefix (for fuzzy matching).

[0091] :return: Structured data containing the point address and the corresponding monitoring value.

[0092] """ extracted_values ​​= [] for record in parsed_data: if record.get("pointAddress", "").startswith(point_prefix): extracted_values.append({ "pointAddress": record.get("pointAddress"), "value": record.get("value") }) return extracted_values.

[0093] S105, filtering out illegal data according to preset rules and performing type conversion on legal data for the extracted data; In this embodiment, the extracted data may contain abnormal values, format errors or missing values, which need to be filtered out or corrected before entering the subsequent processing stage. Verification is performed through preset rules, legal data is screened and necessary type conversions (such as data format and unit system conversion) are performed on it to ensure data standardization.

[0094] According to business needs, you can set rules to filter illegal data, such as: The values ​​must be within a reasonable range (e.g. temperature: -50 degrees Celsius to 150 degrees Celsius).

[0095] Make sure the value is of the correct type (e.g. it must be a floating point number).

[0096] Field cannot be empty or missing.

[0097] Filter data according to rules, filter out illegal data or mark it as abnormal. Perform conversion on data that meets the rules, such as converting string types to floating point numbers. For example, converting temperature from Fahrenheit to Celsius.

[0098] The following is further explained by examples: Assume the extracted data is as follows: [ {"pointAddress": "Sensor1.Temp", "value": "25.5"}, {"pointAddress": "Sensor2.Pres", "value": 101325}, {"pointAddress": "Sensor3.Humidity", "value": null}, {"pointAddress": "Sensor4.Temp", "value": "-200"}, {"pointAddress": "Sensor5.Temp", "value": "InvalidData"} ] The following is a code example of filtering and data conversion based on the extracted data: def validate_and_convert_data(data, rules): """ Filter illegal data and perform type conversion.

[0099] :param data: The extracted data list, each record contains the point address and value.

[0100] :param rules: Validation rule dictionary, defining the range and type of each data.

[0101] :return: Two lists: legal data and illegal data.

[0102] """ valid_data = [] invalid_data = [] for record in data: point_address = record.get("pointAddress") value = record.get("value") # Find the corresponding rules rule = rules.get(point_address) if not rule: # If no rules are defined, mark it as illegal data invalid_data.append({**record, "reason": "No rule defined"}) continue # Data integrity check if value is None or value == "": invalid_data.append({**record, "reason": "Value is missing"}) continue # Data type check and attempt type conversion try: converted_value = float(value) except ValueError: invalid_data.append({**record, "reason": "Invalid format"}) continue # Data range check if not (rule["min"]<= converted_value<= rule["max"]): invalid_data.append({**record, "reason": "Out of range"}) continue # Pass verification and add legal data valid_data.append({ "pointAddress": point_address, "value": converted_value }) return valid_data, invalid_data.

[0103] In a possible implementation, an implementation of this step includes: S1051, verifying whether each extracted data item conforms to a predetermined format, wherein the predetermined format includes whether the numerical data conforms to a numerical range, and whether the string data conforms to a regular expression matching rule; Define the value range according to the specific requirements of each data item. For example, temperature data is required to be between -50 degrees Celsius and 150 degrees Celsius, and pressure data is required to be between 0 and 1000 bar.<min_value or value> max_value) to check whether the extracted value falls within the allowed range. Define regular expressions to match data in a specific format. Use regular expression tools (such as Python's re.match) to verify whether the string conforms to the expected format. If the data does not conform to the predetermined format, mark it as illegal data and record the illegal reason. Record detailed information through the logging system for subsequent analysis.

[0104] S1052, converting a legal string type into floating point number or integer data; Determine whether the data item is of string type (such as isinstance(value, str)).

[0105] Ensure that the string conforms to the legal numerical format, such as checking whether it is a pure number or a floating point number with a decimal point. Perform type conversion. If the string is in a legal integer format, use int() to convert it to an integer. If the string is in a legal floating point format, use float() to convert it to a floating point number. If the conversion fails (for example, there are illegal characters in the data), catch the exception and log it. Mark the data that cannot be converted as illegal data and return it to the processing module.

[0106] S1053, converting the state data into a Boolean value according to a preset conversion rule; Define the possible values ​​of state data, such as "ON" / "OFF", "TRUE" / "FALSE", or "1" / "0". Use a dictionary or mapping relationship to define the conversion rules. For example: status_mapping = {"ON": True, "OFF": False, "1": True, "0": False} Perform the conversion and map the status value to a Boolean value according to the preset rules. If the status value does not match the rules, it is marked as illegal data and recorded. If the status data is missing or has an abnormal format, a default value (such as False) can be set to ensure that subsequent processing is not affected.

[0107] S1054. Convert a specific data item into a predefined enumeration type or a constant value.

[0108] Define enumeration rules. The enumeration type lists all possible values ​​of the data and the corresponding constant values. For example: device_type_mapping = { "TemperatureSensor": 1, "PressureSensor": 2, "FlowSensor": 3 } Look for the corresponding enumeration value in the mapping rule based on the data content. If a match is found, the data is replaced with the corresponding constant value. If the data item does not match the enumeration value, it is marked as illegal data or set to the default value (such as UNKNOWN=-1).

[0109] Through the above steps, this implementation can systematically verify data, convert formats, and unify types to ensure that input data meets standardization requirements, laying a good foundation for subsequent processing and storage. The rule configuration and logging mechanism in each step facilitate dynamic adjustment and maintenance, while enhancing the robustness of the system.

[0110] S106, organizing the target data after verification and type conversion into a standardized and unified JSON data structure; In this embodiment, in order to ensure that data from different devices can be uniformly processed and transmitted, a standardized JSON data structure must be defined. The following is an example of a possible JSON data structure: { "deviceId": "string", / / Device unique identifier "timestamp": "string", / / Timestamp of data collection time "dataItems": [ / / Data item list { "address": "string", / / Point address (such as sensor number or function module address) "value": "number|string", / / data value (may be a numerical value or a status value) "unit": "string", / / data unit (such as C, kPa, etc.) "status": "string" / / data status (such as legal, illegal, missing, etc., optional) } ]} In the above examples, the device identifier (deviceId) is provided by the device description file or acquisition module, which is usually the unique ID or name of the device.

[0111] For example: "deviceId":"sensor_12345" The timestamp is the time when the data is collected, formatted in the ISO8601 standard format (such as "2025-01-01T10:00:00Z"). The timestamp can be obtained through the time library of the programming language, for example: from datetime import datetime timestamp=datetime.utcnow().isoformat()+"Z" Each data item in dataItems can include the following fields: address: point address, provided by device parsing data.

[0112] value: data value, a legal value after verification and conversion.

[0113] unit: The unit information of the data, obtained from the description file or point address parsing.

[0114] status: Optional field that records the data status (such as "valid", "invalid").

[0115] The specific way to organize and convert to JSON objects can be to use a dictionary or object in a programming language (such as Python) as a template, for example: json_data = { "deviceId": device_id, "timestamp": timestamp, "dataItems": [] } Traverse the verified and converted data and construct a dataItems list. For example: for data in validated_data:# validated_data is the validated data list data_item = { "address": data["address"], "value": data["value"], "unit": data.get("unit", ""), # The default unit is empty "status": data.get("status", "valid")# The default status is valid } json_data["dataItems"].append(data_item) Use the JSON library to serialize the dictionary object into a JSON format string. For example: import json json_string=json.dumps(json_data,indent=4) Finally, the generated JSON data can be transmitted through a network interface (such as HTTP / HTTPS), and the JSON data can be stored in a file system or database. Specifically, the JSON data can be stored in a relational database (such as MySQL's JSON field) or a non-relational database (such as MongoDB). The following is an example of organized JSON data: { "deviceId": "sensor_12345", "timestamp": "2024-11-01T10:00:00Z", "dataItems": [ { "address": "point1", "value": 23.5, "unit": "C", "status": "valid" }, { "address": "point2", "value": 101.3, "unit": "kPa", "status": "valid" }, { "address": "point3", "value": null, "unit": "", "status": "invalid" } ] } As attached Figure 3 As shown, in a possible embodiment, an implementation of this step includes: S1061. Determine the hierarchical structure of data; This step aims to determine the hierarchical structure of the entire data model to ensure that the organization of the data can be clearly mapped to the actual business needs. The data hierarchy can be divided into multiple levels, such as the device level, point level, data value level, etc.

[0116] An example of a hierarchy is as follows: Device layer: basic information of the device itself, such as device ID, device name, device type, etc.

[0117] Point layer: information of each monitoring point in the device, including point address, description, unit, etc.

[0118] Data value layer: actual measured values ​​or calculated results, such as temperature, pressure, humidity, etc.

[0119] First, determine the role and data content of each layer, and determine the hierarchical relationship between the data. For example: Device layer: Contains device identification information, which may include device hardware information, connection status, etc.

[0120] Point layer: contains the information of each monitoring point in the device and defines the specific attributes of each monitoring point (such as point address, function, unit, etc.).

[0121] Data layer: corresponds to the data collected in real time at the monitoring point, which may include sensor readings, calculated parameters, status values, etc.

[0122] More levels can be divided according to different business needs, such as "sub-device layer" or "functional module layer".

[0123] S1062. Define the data structure of each layer according to the business requirements of each layer; In this step, the data structures required for each layer need to be defined according to the data requirements of different levels. These data structures need to be able to meet business requirements and support efficient storage and access of data.

[0124] Device layer data structure definition: Device ID (device_id): An ID that uniquely identifies a device.

[0125] Device name (device_name): The name or description of the device.

[0126] Device Type (device_type): The class or type of device, such as "temperature sensor" or "pressure sensor".

[0127] Point layer data structure definition: Point address (point_address): The address that uniquely identifies the point.

[0128] Description: The function or description of the point, such as "temperature sensor".

[0129] Unit: Data unit, such as "Celsius", "Pa", etc.

[0130] Data layer data structure definition: Data value: The data value read from the device, such as temperature, pressure, etc.

[0131] Timestamp: The time when the data was collected.

[0132] Unit: The unit of the data, such as "Celsius", "Pascal", etc.

[0133] S1063. Configure a corresponding data processing strategy for each layer.

[0134] Define appropriate processing strategies for each layer of data, including data verification, format conversion, cleaning and aggregation, to ensure the standardization and consistency of the data processing process.

[0135] Device layer data processing strategy: Check the validity and uniqueness of the device ID and device name to ensure the accuracy of the device information.

[0136] Monitor the connection status and working status of the device to ensure that the device is in normal working condition.

[0137] Point layer data processing strategy: Check whether the point address conforms to the predetermined format to avoid data errors.

[0138] For some point data, it may be necessary to aggregate the data according to specific rules, such as aggregating multiple temperature sensor data of a certain device into the average value of an area.

[0139] Data value layer data processing strategy: Verify each collected data value, for example, check whether the value is within a reasonable range, check whether the units are consistent, etc.

[0140] Ensure the consistency of data types and convert data values ​​to standard formats, such as converting string values ​​to floating point numbers or integers.

[0141] Filter outliers based on set rules (such as temperature range limits).

[0142] In this embodiment, the data roles and hierarchical relationships of each layer are clarified to ensure clear data processing logic. A data structure that meets business requirements is designed for each layer to ensure that data storage and access are flexible and efficient. A suitable data processing strategy is configured for each layer to ensure data quality, reliability, and consistency.

[0143] S107, parsing the generated JSON data structure, and writing the parsed data into a target database for data analysis and equipment operation status monitoring.

[0144] In this step, the content is extracted from the JSON data structure generated in the previous step and converted into a format suitable for storage.

[0145] If the JSON data is stored in a file or received from an interface, load the data first. An example of loading data is as follows: import json # Load from string json_string = '{"deviceId": "sensor_12345", "timestamp": "2025-01-01T10:00:00Z", "dataItems": [...]}' data = json.loads(json_string) # Load from file with open("data.json", "r") as file: data = json.load(file) Extract device ID, timestamp, and data items, for example: device_id = data["deviceId"] timestamp = data["timestamp"] data_items = data["dataItems"] Before writing to the database, you need to ensure that the target database has been created and the corresponding table structure has been configured. An example of a database table is as follows: CREATE TABLE DeviceData ( id INT AUTO_INCREMENT PRIMARY KEY, deviceId VARCHAR(255), timestamp DATETIME, address VARCHAR(255), value FLOAT, unit VARCHAR(50), status VARCHAR(50) ); You can use an appropriate database driver to connect to a database, such as MySQL, PostgreSQL, or SQLite.

[0146] The specific implementation of the data writing operation is as follows: First, prepare the SQL statement and construct the insert SQL statement to write the extracted data into the database, for example: INSERT INTO DeviceData (deviceId, timestamp, address, value, unit, status) VALUES (%s, %s, %s, %s, %s, %s); Traverse the dataItems and write them to the database one by one. After the data is written, commit the transaction to save the changes. In this step, you can also catch possible exceptions in database operations to prevent program crashes, such as verifying the integrity and validity of the data before writing, ensuring that all required fields (such as deviceId and address) are not empty, and ensuring that the value field is a legal value. Once the data is written to the database, you can monitor the device operation status and analyze the data through query operations.

[0147] The above embodiment describes the device data collection method based on multi-source heterogeneity provided by the present application. The following describes the embodiments of the device, system and storage medium provided in the present application: See also Figure 4 The present application provides an embodiment of a device data acquisition apparatus based on multi-source heterogeneity, the embodiment comprising: The data collection unit 401 is used to collect device data of multiple devices; The configuration unit 402 is used to configure a corresponding data parsing code according to the data format and characteristics of each device, and parse the collected device data based on the data parsing code to obtain device parsing data; The data mapping unit 403 is used to extract the attributes and point addresses of each device from the device parsing data, and extract the function or physical quantity information in the point address through the custom parsing function getWh(), and map it to the corresponding device data item; A data extraction unit 404 is used to extract the data value corresponding to the point address; The data conversion unit 405 is used to filter out illegal data according to preset rules and perform type conversion on legal data for the extracted data; The structure generation unit 406 is used to organize the target data after verification and type conversion into a standardized and unified JSON data structure; The data writing unit 407 is used to parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

[0148] Optionally, the data conversion unit 405 is specifically used for: Verify whether each extracted data item conforms to a predetermined format, wherein the predetermined format includes whether the numerical data conforms to a numerical range and whether the string data conforms to a regular expression matching rule; Convert legal string types to floating point or integer data; Convert state data into Boolean values ​​according to preset conversion rules; Converts a specific data item to a predefined enumeration type or a constant value.

[0149] Optionally, the configuration unit 402 is specifically configured to: Identify the data type and data structure of each device, determine the characteristics of the device data, and determine the mapping relationship between device attributes and device data; Configuring a parsing function according to characteristics of the device data; The parsing function is called to parse the device data of each device, and the nested structure is processed using a recursive method to obtain device parsing data.

[0150] Optionally, the configuration unit 402 is specifically configured to: For each device, query the relevant point data from the pre-configured plc_dianbiao table according to the device name; Use regular expressions to extract the corresponding bit number through the device name, and obtain the point data of the device according to the bit number; Use the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value; Store all devices and their corresponding point data in the result dictionary.

[0151] Optionally, the configuration unit 402 is specifically configured to: The data parsing code is configured through the getWh() function and dwToObject() function, and the device's point data is encapsulated into a preset format.

[0152] Optionally, the structure generating unit 406 is specifically used for: Determine the hierarchical structure of the data; Define the data structure of the corresponding layer according to the business requirements of each layer; Configure corresponding data processing strategies for each layer.

[0153] The data mapping unit 403 is specifically used for: Traverse the device parsing data and obtain the data points of each device one by one; Extracting device attributes and point addresses from each data point, wherein the device attributes include device ID and device name; the point address includes sensor location and function identifier; Parse the function information and physical quantity information in the point address through a custom parsing function getWh(); According to the function information in the point address, the corresponding device data item is searched through the preconfigured mapping table, and the physical value and unit information are filled in.

[0154] See also Figure 5 The present application also provides a device data acquisition device based on multi-source heterogeneity, comprising: Processor 501, memory 502, input and output unit 503, bus 504; The processor 501 is connected to the memory 502, the input and output unit 503 and the bus 504; The memory 502 stores a program, and the processor 501 calls the program to execute any of the above-mentioned device data collection methods based on multi-source heterogeneity.

[0155] The present application also relates to a computer-readable storage medium on which a program is stored, wherein when the program is run on a computer, the computer is enabled to execute any of the above-mentioned device data collection methods based on multi-source heterogeneity.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for collecting device data based on multi-source heterogeneity, characterized in that: The method comprises: Collect device data from multiple devices; According to the data format and characteristics of each device, a corresponding data parsing code is configured, and the collected device data is parsed based on the data parsing code to obtain device parsing data; Extract the attributes and point addresses of each device from the device parsing data, parse the coding structure of the point addresses, identify the function type or physical quantity information contained in the point addresses according to the preset parsing rules, and map them to the corresponding device data items; Extracting the data value corresponding to the point address; For the extracted data, illegal data is filtered according to preset rules, and the legal data is converted into different types; Organize the target data after verification and type conversion into a standardized and unified JSON data structure; Parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

2. The method for collecting data based on multi-source heterogeneity according to claim 1, characterized in that: The step of filtering out illegal data according to preset rules and converting the type of legal data includes: Verify whether each extracted data item conforms to a predetermined format, wherein the predetermined format includes whether the numerical data conforms to a numerical range and whether the string data conforms to a regular expression matching rule; Convert legal string types to floating point or integer data; Convert state data into Boolean values ​​according to preset conversion rules; Converts a specific data item to a predefined enumeration type or a constant value.

3. The method for collecting data based on multi-source heterogeneity according to claim 1, characterized in that: The step of configuring a corresponding data parsing code according to the data format and characteristics of each device, and parsing the collected device data based on the data parsing code to obtain device parsing data includes: Identify the data type and data structure of each device, determine the characteristics of the device data, and determine the mapping relationship between device attributes and device data; Configuring a parsing function according to characteristics of the device data; The parsing function is called to parse the device data of each device, and the nested structure is processed using a recursive method to obtain device parsing data.

4. The method for collecting data based on multi-source heterogeneity equipment according to claim 3, characterized in that: The identifying the data type and data structure of each device, determining the characteristics of the device data, and determining the mapping relationship between the device attributes and the device data includes: For each device, query the relevant point data from the pre-configured plc_dianbiao table according to the device name; Use regular expressions to extract the corresponding bit number through the device name, and obtain the point data of the device according to the bit number; Use the dwToObject() function to encapsulate the queried point data into a dictionary object, in which description is used as the key and the point address is used as the value; Store all devices and their corresponding point data in the result dictionary.

5. The method for collecting data based on multi-source heterogeneity according to claim 3, characterized in that: The configuring of the parsing function according to the characteristics of the device data comprises: The data parsing code is configured through the getWh() function and dwToObject() function, and the device's point data is encapsulated into a preset format.

6. The method for collecting data from multi-source heterogeneous devices according to claim 1, characterized in that: The step of organizing the target data after verification and type conversion into a standardized and unified JSON data structure includes: Determine the hierarchical structure of the data; Define the data structure of the corresponding layer according to the business requirements of each layer; Configure corresponding data processing strategies for each layer.

7. The method for collecting device data based on multi-source heterogeneity according to claim 1, characterized in that: The method of extracting the attributes and point address of each device from the device parsing data, parsing the coding structure of the point address, identifying the function type or physical quantity information contained in the point address according to the preset parsing rules, and mapping to the corresponding device data items includes: Traverse the device parsing data and obtain the data points of each device one by one; Extracting device attributes and point addresses from each data point, wherein the device attributes include device ID and device name; the point address includes sensor location and function identifier; Parsing the coding structure of the point address, and identifying the function type or physical quantity information contained in the point address according to a preset parsing rule; According to the function information in the point address, the corresponding device data item is searched through the preconfigured mapping table, and the physical value and unit information are filled in.

8. A device data acquisition device based on multi-source heterogeneity, characterized in that: include: A data collection unit, used for collecting device data of multiple devices; A configuration unit, configured to configure a corresponding data parsing code according to the data format and characteristics of each device, and parse the collected device data based on the data parsing code to obtain device parsing data; A data mapping unit, used to extract the attributes and point addresses of each device from the device parsing data, parse the coding structure of the point address, identify the function type or physical quantity information contained in the point address according to a preset parsing rule, and map it to the corresponding device data item; A data extraction unit, used to extract the data value corresponding to the point address; A data conversion unit, used to filter out illegal data according to preset rules and perform type conversion on legal data for the extracted data; A structure generation unit, used to organize the target data after verification and type conversion into a standardized and unified JSON data structure; The data writing unit is used to parse the generated JSON data structure and write the parsed data into the target database for data analysis and equipment operation status monitoring.

9. A multi-source heterogeneous equipment data acquisition system, characterized in that: include: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 7.

Images