Multi-protocol extensible data acquisition system architecture and method with unified interface standard

Abstract

The invention relates to the technical field of data acquisition, and discloses a multi-protocol extensible data acquisition system architecture and method with a unified interface standard, and the architecture comprises a data source management module which is used for storing data source configuration information and distributing protocol identifiers; the protocol adaptation module comprises an analysis unit and a matching unit and is used for realizing data format conversion and protocol compatibility verification; the data conversion module is used for generating target data through format conversion and semantic mapping; the data verification module and the anomaly detection module are used for comparing and monitoring data integrity based on an integrity algorithm and a threshold value; the data quality monitoring module and the early warning module are used for comparing and evaluating data quality through a consistency model and a threshold value; and the result output module transmits the early warning information to a target system. The method comprises the steps of data source configuration, protocol adaptation, data conversion, integrity and quality monitoring, early warning and the like. The system expansibility and the data reliability are improved, and the method is suitable for multi-scene data acquisition.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and in particular to a multi-protocol extensible data acquisition system architecture and method with unified interface standards. Background Art

[0002] With the rapid development of information technology, data, as a key production factor, is becoming increasingly important in terms of its collection, processing, and utilization. However, the current data collection field faces many challenges that severely restrict the efficient use of data and the scalability of systems.

[0003] From the perspective of data source diversity, the widespread adoption of technologies such as the Internet of Things, cloud computing, and big data has led to an explosive growth in data sources, with their complexity and diversity. Different data sources may utilize completely different communication protocols, such as Modbus and OPC UA, common in the industrial sector, HTTP and RESTful in the internet, and ZigBee and MQTT in sensor networks. This makes it difficult for traditional data acquisition systems to be compatible with multiple protocols, often requiring the development of separate adapter modules for each protocol. This results in high system development costs, long development cycles, and significant maintenance difficulties. Furthermore, data source configuration information management is chaotic, lacking a unified storage and management mechanism. This makes it difficult for the system to quickly identify and access configuration information from different data sources, severely impacting the efficiency and accuracy of data acquisition.

[0004] When it comes to data format conversion, different data sources generate data in a wide variety of formats, including structured data like JSON, XML, and CSV, as well as unstructured data like text, images, audio, and video. Traditional data acquisition systems lack unified standards and specifications for data format conversion, relying on manual code implementation, which is inefficient and error-prone. Furthermore, understanding and extracting data semantics is a challenge. Different data sources may have different definitions and interpretations of the same data, leading to semantic ambiguity during data sharing and interaction, making it impossible to accurately convey the data's true meaning.

[0005] Data quality assurance is another key issue in the data collection process. During data collection, transmission, and processing, problems such as missing data, data errors, and data inconsistencies may arise, severely impacting data availability and reliability. Traditional data collection systems lack comprehensive data quality monitoring and early warning mechanisms, making it difficult to promptly detect and address data quality issues. This results in low-quality data entering subsequent data processing and analysis, which in turn impacts decision-making accuracy.

[0006] Insufficient system scalability and compatibility are also major challenges facing current data collection systems. As business grows and demands change, data sources and data types may continue to increase and update. Traditional data collection systems, due to architectural limitations, struggle to adapt quickly to these changes. Integrating new data sources or supporting new protocols often requires large-scale modifications and upgrades to the entire system, which is not only costly but can also cause system downtime and impact business continuity.

[0007] Traditional data acquisition systems have obvious shortcomings in terms of multi-protocol compatibility, data format conversion, data quality assurance and system scalability. There is an urgent need for a new data acquisition system architecture and method that can unify interface standards, support multi-protocol expansion and ensure data quality to meet the diverse needs of the modern data acquisition field. Summary of the Invention

[0008] The purpose of the present invention is to provide a multi-protocol scalable data acquisition system architecture and method with a unified interface standard to solve the problems raised in the above background technology.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a multi-protocol scalable data acquisition system architecture with a unified interface standard, the architecture comprising: The data source management module stores different data source configuration information into the data source knowledge base according to the data source type and assigns a unique protocol identifier to each data source; The protocol adaptation module includes a protocol parsing unit and a protocol matching unit. After receiving external data, the protocol parsing unit calls the corresponding protocol parsing library according to the protocol identifier to convert the original data into a standard intermediate format. The protocol matching unit verifies the protocol compatibility and confirms data access after verification. A data conversion module includes a format conversion unit and a semantic mapping unit. The format conversion unit converts the intermediate format data into the target data structure using preset rules, and the semantic mapping unit extracts data semantic tags according to the metadata model. The data verification module calculates the data integrity coefficient through the data integrity algorithm based on the data semantic labels of the data conversion module; The anomaly detection module determines the data integrity status based on the data integrity coefficient of the data verification module and triggers warning rules for abnormal results; The data quality monitoring module calculates the data consistency coefficient based on the data semantic labels of the data conversion module through the data consistency model; The data quality warning module judges the data quality status based on the data consistency coefficient of the data quality monitoring module and triggers warning rules for abnormal results.

[0010] Preferably, in the data source management module, different data source configuration information is stored in a data source knowledge base according to the data source type, and the configuration information includes the communication protocol type and the data collection frequency.

[0011] Preferably, in the protocol adaptation module, the protocol parsing unit calls the corresponding protocol parsing library according to the protocol identifier, converts the original data into a standard intermediate format, the system verifies the protocol compatibility of the data source, and outputs the corresponding protocol configuration in the data source knowledge base, and the protocol matching unit verifies whether the data format complies with the target specification and whether it supports dynamic expansion, and confirms data access after verification.

[0012] Preferably, in the data conversion module, the format conversion unit uses preset rules to convert the intermediate format data into the target data structure including field mapping rules, data type conversion rules and timestamp alignment rules, and the semantic mapping unit extracts data semantic labels including entity identifiers, attribute relationship graphs and data dependency paths according to the metadata model.

[0013] Preferably, in the data verification module, based on the data semantic tag transmitted by the data conversion module, the specific content of the data integrity coefficient calculated by the data integrity algorithm is as follows: Step S01: Use the hash algorithm to generate a checksum value for the field set of the target data structure, and calculate the number of missing fields based on the checksum value. ; Step S02: Calculate the actual data field missing rate, the calculation formula is:

[0014] in represents the field missing rate, Indicates the number of missing fields, Indicates the total number of fields; Step S03: Calculate the data integrity coefficient, the calculation formula is:

[0015] in Indicates the data integrity coefficient.

[0016] Preferably, the anomaly detection module receives the data integrity coefficient transmitted by the data verification module , and the data integrity coefficient With the preset integrity threshold Compare and judge the data integrity status. If the data integrity coefficient Less than the preset integrity threshold , then trigger the data missing warning, if the data integrity coefficient Greater than or equal to the preset integrity threshold , the data is determined to be complete.

[0017] Preferably, in the data quality monitoring module, based on the semantic tags transmitted by the data conversion module, the specific content of the data consistency coefficient calculated by the data consistency model is as follows: Step S01: Calculate the entity attribute conflict rate, the calculation formula is:

[0018] in represents the attribute conflict rate, Indicates the number of conflicting attributes, Indicates the total number of associated attributes; Step S02: Calculate the data consistency coefficient, the calculation formula is:

[0019] in Represents the data consistency coefficient.

[0020] Preferably, the data quality warning module receives the data consistency coefficient transmitted by the data quality monitoring module , and the data consistency coefficient Consistency threshold with the preset Compare and judge the data quality status. If the data consistency coefficient Less than the preset consistency threshold , then trigger the data conflict warning, if the data consistency coefficient Greater than or equal to the preset consistency threshold , then the data are determined to be consistent.

[0021] Preferably, the architecture further includes: The result output module is used to receive the warning information transmitted by the anomaly detection module and the data quality warning module, and transmit the warning information to the target application system through the API interface.

[0022] Preferably, the present invention further includes a multi-protocol scalable data acquisition method with a unified interface standard, which is applied to the above-mentioned multi-protocol scalable data acquisition system architecture with a unified interface standard, and the method includes the following steps: Step S1: storing different data source configuration information into a data source knowledge base according to the data source type, and assigning a unique protocol identifier to each data source; Step S2: Call the corresponding protocol parsing library according to the protocol identifier to verify the protocol compatibility. After verification, confirm data access; Step S3: Convert the intermediate format data into the target data structure using preset rules, and extract data semantic labels based on the metadata model; Step S4: Based on the data semantic label transmitted by the data conversion module, a data integrity coefficient is calculated using a data integrity algorithm; Step S5: Determine the data integrity status based on the data integrity coefficient and trigger warning rules for abnormal results; Step S6: Based on the data semantic labels transmitted by the data conversion module, a data consistency coefficient is calculated using a data consistency model; Step S7: Determine the data quality status based on the data consistency coefficient and trigger warning rules for abnormal results.

[0023] Compared with the prior art, the present invention has the following beneficial effects: The data source management module stores different data source configuration information in a data source knowledge base based on data source type and assigns a unique protocol identifier to each data source, enabling centralized management and unified identification of multiple data source configuration information. This enables the system to quickly identify and access configuration information from different data sources, significantly improving the efficiency and accuracy of data source management. When a new data source needs to be connected, simply add its configuration information to the data source knowledge base and assign a protocol identifier to quickly access the data source, significantly reducing the cost and difficulty of system expansion.

[0024] Through the design of the protocol parsing unit and protocol matching unit, the protocol adapter module can call the corresponding protocol parsing library based on the protocol identifier, convert the raw data into a standard intermediate format, and verify protocol compatibility and whether the data format meets the target specification and supports dynamic expansion. This design effectively solves the multi-protocol compatibility problem of traditional data acquisition systems, enabling the system to seamlessly access data sources with multiple different protocols without the need to develop separate adapter modules for each protocol, significantly reducing development costs and cycle time. At the same time, the protocol matching unit's verification of the data format ensures the standardization and consistency of the accessed data, laying a good foundation for subsequent data processing.

[0025] The data conversion module's format conversion unit uses preset field mapping rules, data type conversion rules, and timestamp alignment rules to convert intermediate format data into the target data structure, achieving efficient conversion between different data formats. The semantic mapping unit extracts data semantic labels based on the metadata model, including entity identifiers, attribute relationship maps, and data dependency paths. This resolves data semantic ambiguity, ensures that data accurately conveys its true meaning during sharing and interaction, and improves data readability and usability.

[0026] The data verification module calculates the data integrity coefficient using a data integrity algorithm, accurately detecting missing data. The anomaly detection module determines the data integrity status based on the data integrity coefficient and triggers warning rules for abnormal results. This enables real-time monitoring and early warning of data integrity, allowing timely detection of missing data issues and the implementation of appropriate measures, ensuring data integrity and reliability.

[0027] The data quality monitoring module calculates the data consistency coefficient using a data consistency model. The data quality warning module uses this data consistency coefficient to determine data quality and trigger warning rules, enabling effective monitoring and management of data consistency. This enables timely detection of data conflicts and prevents low-quality data from entering subsequent processing, improving data quality and availability and providing a reliable basis for data analysis and decision-making.

[0028] The result output module transmits the warning information transmitted by the anomaly detection module and the data quality warning module to the target application system through the API interface, realizing the timely delivery and processing of warning information, enabling relevant personnel to respond quickly to data quality issues, and improving the intelligence and automation level of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a working principle diagram of the multi-protocol scalable data acquisition system architecture with a unified interface standard according to the present invention; Figure 2 This is the design diagram of the data verification module; Figure 3 This is the design diagram of the anomaly detection module. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] See also Figure 1-Figure 3 The present invention relates to a multi-protocol scalable data acquisition system architecture with a unified interface standard. The system includes a data source management module, a protocol adaptation module, a data conversion module, a data verification module, an anomaly detection module, a data quality monitoring module, and a data quality warning module. The specific implementation method is as follows: The data source management module stores different data source configuration information in a data source knowledge base based on the data source type and assigns a unique protocol identifier to each data source. The protocol adaptation module includes a protocol parsing unit and a protocol matching unit. After receiving external data, the protocol parsing unit calls the corresponding protocol parsing library based on the protocol identifier to convert the raw data into a standard intermediate format. The protocol matching unit verifies protocol compatibility and, if verified, confirms data access.

[0032] The data conversion module includes a format conversion unit and a semantic mapping unit. The format conversion unit uses preset rules to convert intermediate format data into the target data structure, and the semantic mapping unit extracts data semantic tags based on the metadata model. The data verification module calculates the data integrity coefficient based on the data semantic tags of the data conversion module using a data integrity algorithm. The anomaly detection module determines the data integrity status based on the data integrity coefficient of the data verification module and triggers warning rules for abnormal results. The data quality monitoring module calculates the data consistency coefficient based on the data semantic tags of the data conversion module using a data consistency model. The data quality warning module determines the data quality status based on the data consistency coefficient of the data quality monitoring module and triggers warning rules for abnormal results.

[0033] The present invention is further described in detail below with reference to specific embodiments: Example 1: In this embodiment, the core function of the data source management module is to implement standardized configuration management and unique identifier allocation for different types of data sources, ensure that the system can orderly identify and access various heterogeneous data sources, and provide basic support for subsequent data processing processes.

[0034] The data source management module needs to classify and manage data sources. According to the physical form, data generation method and communication characteristics of the data source, it can be divided into different types such as sensor data source, database data source, file data source, network interface data source, etc. For example, sensor data sources can be further subdivided into industrial sensors, environmental monitoring sensors, etc., whose data are usually transmitted in the form of real-time streams through specific industrial protocols; database data sources include relational databases (such as MySQL, Oracle) and non-relational databases (such as MongoDB), and data is exchanged through database connection protocols; file data sources cover files in formats such as CSV, Excel, and JSON, and data is obtained through file system interfaces or network file transfer protocols; network interface data sources include API interfaces that provide data services through protocols such as HTTP and WebSocket. Through clear type classification, the module can formulate differentiated configuration strategies based on the characteristics of different types of data sources to improve management efficiency.

[0035] The module stores the configuration information for different data sources in a data source knowledge base. This configuration information primarily includes two core parameters: the communication protocol type and the data collection frequency. It can also be expanded to include auxiliary information such as the data source name, physical address, system affiliation, update cycle, and authentication information. The communication protocol type is crucial for data exchange between the data source and the system. For example, a sensor data source may use industrial protocols such as ModbusRTU, ModbusTCP, or OPCUA. A database data source requires connection protocols such as JDBC or ODBC. A file data source specifies transfer protocols such as FTP, SFTP, or HTTP. A network interface data source requires interface protocols such as RESTful or GraphQL. The data collection frequency should be determined based on the characteristics of the data source and business requirements. For industrial sensors with extremely high real-time requirements (such as temperature and pressure sensors on a production line), a collection frequency of milliseconds or seconds can be set to ensure timely acquisition of device operating status data. For business databases with batch updates (such as transaction system databases that archive data at dawn), a daily collection frequency can be set. For unstructured log file data sources, scheduled collection tasks can be set based on the file generation cycle (e.g., hourly or daily).

[0036] When storing configuration information, the data source knowledge base uses a structured storage method. For example, a data table is constructed based on a relational database, with data source type as the classification index. Each data source corresponds to a record, and the record contains fields for various configuration parameters. Through this structured storage, the module can quickly retrieve and update data source configuration information, while also supporting configuration management for system administrators through a visual interface, such as adding new data sources, modifying existing data source configurations, and deleting invalid data sources. In addition, the knowledge base must have a data backup and recovery mechanism to ensure the security and reliability of configuration information and avoid data loss due to system failures or human errors.

[0037] Assigning a unique protocol identifier to each data source is a key function of the data source management module. This identifier is generated using specific encoding rules to ensure system-wide uniqueness and reflect the data source type and protocol characteristics. For example, an identifier can consist of a data source type code, a protocol type code, and a serial number. "S" designates a sensor data source, "DB" designates a database data source, "F" designates a file data source, and "API" designates a network interface data source. The protocol type code can be an abbreviation of the protocol name, such as "MOD" for Modbus, "OPC" for OPCUA, and "JDBC" for JDBC. The serial number is an incremental number used to distinguish different data sources of the same type and protocol. For example, the identifier for a temperature sensor data source using the ModbusTCP protocol might be "S-MOD-TCP-001," while the identifier for a MySQL database data source using the JDBC protocol might be "DB-JDBC-MySQL-002."

[0038] The unique protocol identifier plays a crucial role throughout the entire data collection process. During the protocol adaptation phase, the protocol parser uses this identifier to quickly locate the corresponding protocol parser library, automatically completing protocol matching without manual intervention. During data routing and processing, the system uses the identifier to identify the data source type and protocol characteristics, invoking the appropriate processing logic and resources. During system maintenance and expansion, the identifier provides a convenient index for data source management and tracking. For example, searching for a specific protocol identifier allows for rapid location of the relevant data source's configuration information, data flow, and processing status.

[0039] Furthermore, the data source management module must also include data source status monitoring capabilities, allowing real-time tracking of data source connection status, collection task execution, and other aspects. For sensor data sources, the module can periodically send heartbeat packets to detect device online status. If no response is received for multiple consecutive times, the module is marked as offline and triggers an alert. For database data sources, it can monitor metrics such as connection pool usage and query response time to promptly identify connection anomalies or performance bottlenecks. For file data sources, it can monitor whether files are generated on time and their size is normal, preventing data collection failures due to missing or damaged files. Through status monitoring, the module can promptly identify issues at the data source level, ensuring system stability and reliability.

[0040] The module supports dynamic adjustment of data collection frequency, either automatically or manually, based on business needs or system load. For example, during peak industrial production periods, the frequency of sensor data collection can be temporarily increased from once per minute to once per second to more intensively monitor equipment operating status. If the system detects excessive connection load on a database data source, it can automatically reduce its frequency to alleviate pressure on the database server. This dynamic adjustment mechanism, implemented through collaboration with the system's task scheduling and resource monitoring modules, ensures a balance between data collection efficiency and resource consumption.

[0041] To meet the need for multi-protocol compatibility, the data source management module supports configuring multiple backup protocols for the same data source to address situations such as primary protocol failures or system upgrades. For example, a sensor data source can be configured with both ModbusTCP and MQTT. If the primary protocol (ModbusTCP) fails due to a network port failure, the module automatically switches to the backup protocol (MQTT) for data collection, ensuring continuous data transmission. The configuration and switching logic for the backup protocols must be pre-defined in the data source knowledge base and seamlessly integrated with the protocol parsing library of the protocol adapter module.

[0042] Example 2: In this embodiment, the protocol adapter module serves as the key hub of the data acquisition system and is responsible for protocol parsing and compatibility verification during external data access. Its core function is to convert the original data of different protocols into a standard intermediate format that can be processed by the system and ensure the standardization and reliability of data access.

[0043] The work of the protocol adapter module begins with the protocol parsing unit receiving external data. When the data source sends data to the system through a physical interface (such as a serial port or an Ethernet port) or a network protocol (such as TCP / IP or HTTP), the protocol parsing unit first obtains the original format of the data and the corresponding protocol identifier. This identifier is pre-assigned by the data source management module and uniquely identifies the communication protocol used by the data source (such as Modbus, OPC UA, MQTT, etc.). Based on this identifier, the protocol parsing unit calls the corresponding parsing rules from the system's built-in protocol parsing library. The protocol parsing library is a predefined set of rules that standardizes the syntax structure, data frame format, encoding method, etc. of different protocols. For example, for the Modbus protocol, the parsing rules need to identify the slave address, function code, data field, and checksum in the data frame; for API data in JSON format, the parsing rules need to extract key-value pair information according to the JSON schema definition.

[0044] The core task of the protocol parsing unit is to convert the raw data into a standard intermediate format. The standard intermediate format is a system-defined unified data structure that masks the underlying differences between different protocols. For example, it uses key-value pairs or JSON-like structures to store data fields, which contain common fields such as timestamps, device identifiers, data types, and values. During the conversion process, the protocol parsing unit needs to decode, extract fields, and reorganize the raw data according to the protocol parsing rules. Taking the Modbus protocol as an example, the raw data frame may contain a hexadecimal byte stream. The parsing unit needs to determine the data type (such as coil status, register value) based on the function code, extract the corresponding starting address and data value, and map them to the "device address" and "measurement value" fields in the intermediate format; for XML data returned by the HTTP interface, the parsing unit needs to extract the node content through the XML parser and convert it into structured fields in the intermediate format.

[0045] After completing the data format conversion, the system enters the protocol compatibility verification phase. This verification is performed by the protocol matching unit, which aims to ensure that the data source's protocol type, version, and configuration parameters are consistent with the system's supported range. During the verification process, the system first retrieves the corresponding protocol configuration information from the data source knowledge base, including the protocol type, version number, transmission parameters (such as baud rate, data bits, stop bits), and security authentication information (such as username, password, and certificate path). The protocol matching unit then verifies the following: Protocol compatibility: Checks whether the protocol used by the data source is included in the system's supported protocol list. The system's supported protocol list is preconfigured by the developer, and includes protocols such as industrial protocols (Modbus, OPC UA, CANopen), IoT protocols (MQTT, CoAP), and database protocols (JDBC, ODBC). Protocols not included in the list will be denied access.

[0046] Protocol version compatibility: For protocols that support multiple versions (such as Modbus TCP v1.1 and v1.2), verify whether the protocol version used by the data source is compatible with the system parsing library to avoid parsing errors caused by protocol version differences.

[0047] Transmission parameter matching: For serial communication protocols (such as Modbus RTU), verify whether parameters such as baud rate, data bits, stop bits, and parity are consistent with the system configuration; for network protocols, verify whether parameters such as IP address, port number, and timeout period are correct.

[0048] Security authentication validity: If the protocol requires identity authentication (such as SSL certificate verification for HTTPS and user permission verification for databases), the protocol matching unit needs to call the corresponding authentication interface to verify whether the identity credentials of the data source are valid to prevent unauthorized devices from accessing the system.

[0049] After the protocol compatibility verification is passed, the protocol matching unit further performs data format verification to ensure that the converted intermediate format data meets the target specifications preset by the system. The target specifications include the integrity, format correctness and logical consistency of the data fields: Field integrity: Checks whether the intermediate format data contains the mandatory fields required by the system, such as timestamp, data source identifier, at least one valid data field, etc. Data missing mandatory fields will be considered invalid.

[0050] Format correctness: Verify that the format of the data field complies with preset rules. For example, timestamps must be strings in ISO8601 format, numeric fields must be integers or floating-point numbers, and enumeration fields must match the preset enumeration value list.

[0051] Logical consistency: Check whether the logical relationship between data fields is reasonable. For example, the value of the "Start Time" field cannot be later than the value of the "End Time" field. The device status field (such as "Running" and "Fault") must match the value range of other related parameters (such as current and voltage).

[0052] In addition to static verification, the protocol adapter module also supports dynamic extension mechanisms to accommodate new protocol types or data formats that may be added in the future. The dynamic extension mechanism is implemented in the following ways: Pluggable protocol parsing library: The system's modular architecture allows developers to expand protocol support by adding new protocol parsing plug-ins (such as DLL files and Python scripts). Plugins must adhere to unified interface specifications and implement core functions such as data parsing and field mapping. Integration can be completed without modifying the underlying system code.

[0053] Dynamic loading of data format templates: For custom protocols or non-standard data formats, the system supports defining data parsing templates through a visual interface or configuration files. Templates contain field extraction rules, data type conversion rules, protocol parameter configuration, and more. For example, for a proprietary protocol, you can define a rule such as "extracting four bytes starting from the third bit of the byte stream as the device ID, and bits 7 through 10 as the temperature value (hexadecimal conversion required)." Once the template is saved, it takes effect in real time.

[0054] Compatibility support for protocol version upgrades: When the existing protocol parsing library needs to be upgraded to support a new version of the protocol, the system allows you to retain multiple versions of parsing rules at the same time and specify the protocol version to use through data source configuration information to avoid the inability to access old data sources due to version upgrades.

[0055] Throughout the data access process, the protocol adapter module must record detailed log information, including data reception time, data source identifier, protocol type, key steps in the parsing process, and verification results. Log information is used for system debugging, troubleshooting, and performance analysis. For example, if a data source frequently experiences protocol parsing errors, logs can be used to determine whether the cause is an incorrect protocol parsing rule, an abnormal data format sent by the data source, or incomplete data due to packet loss during network transmission.

[0056] In addition, the protocol adapter module must also have an error handling mechanism to take appropriate measures for different types of abnormal situations: Protocol parsing failure: If the original data does not conform to the protocol syntax rules (such as Modbus frame checksum errors or JSON format syntax errors), the system will discard the data frame and record the error log. It can also be configured as a retry mechanism to re-request the data source to send data after a set time interval.

[0057] Compatibility verification failed: If the data source protocol version is incompatible or the transmission parameters are incorrect, the system will refuse to access the data source and send an early warning message to the administrator through the result output module, prompting the administrator to check the data source configuration or upgrade the protocol parsing library.

[0058] Data format verification failed: For data with missing fields or incorrect formats, the system can choose to discard the data, fill in default values, or return error information to the data source based on the configuration. For example, for data with missing timestamps, the current system time can be automatically filled in and marked as "filled in" for subsequent data verification modules to identify.

[0059] Example 3: In this embodiment, the data conversion module serves as a bridge connecting the protocol adaptation module and the subsequent data processing links. Its core function is to further convert the standard intermediate format data output by the protocol adaptation module into a data structure that meets the requirements of the target system, and extract the semantic labels of the data to provide structured and semantic input for data verification, quality monitoring and other links.

[0060] The core task of the format conversion unit is to convert the intermediate format data into the target data structure using preset rules. The preset rules are conversion logic pre-defined according to the data model and business requirements of the target system, mainly including field mapping rules, data type conversion rules and timestamp alignment rules. The field mapping rules are used to establish the correspondence between the intermediate format data fields and the target data structure fields, and to solve the problems of differences in field naming, meaning and hierarchical structure between different systems. For example, the "device_id" field in the intermediate format may need to be mapped to the "device unique identifier" field in the target data structure; if the target system uses a hierarchical data structure (such as a nested JSON object), the field mapping rules must also define the nesting level of the field, such as mapping the "sensor.temperature" field in the intermediate format to the path of "device status.sensor data.temperature value" in the target structure.

[0061] Data type conversion rules are used to address data type differences between different systems, ensuring that data can be correctly parsed and used in the target system. Common data type conversion scenarios include: converting a string timestamp in an intermediate format (such as "2023-10-01 12:00:00") to the timestamp value required by the target system (such as a Unix timestamp); converting a Boolean "running status" field (true / false) to the enumeration value required by the target system ("0" for stopped, "1" for running); and rounding or truncating floating-point temperature data (such as 23.5°C) according to the target system's accuracy requirements. Data type conversion rules must strictly adhere to the target system's data type definitions to avoid data storage errors or calculation anomalies caused by type mismatches.

[0062] Timestamp alignment rules are designed for multi-source data fusion scenarios to address inconsistent time bases across different data sources. For example, some sensor data sources may generate timestamps in the device's local time, while database data sources may use UTC time. The format conversion unit must convert timestamps from all data sources to the system's global time (e.g., UTC+8 standard time). The timestamp alignment process includes identifying the format of the original timestamp (e.g., ISO8601, Unix timestamp, custom format), parsing the time value, and adjusting the time zone offset. This ensures that all data points in the same dataset are comparable in time, providing a foundation for subsequent time series analysis, data aggregation, and other operations.

[0063] The semantic mapping unit extracts data semantic tags based on the metadata model. These tags describe the data's business meaning, entity relationships, and dependency paths, transforming the data from simple numeric values ​​or strings into business-readable information. A metadata model abstractly models the domain knowledge of the data, typically including entity definitions, attribute relationships, business rules, and more. For example, in an Industrial Internet of Things (IIoT) scenario, a metadata model might define a "sensor" entity as having attributes such as "device number," "type," and "installation location," and a "contains" relationship between the "production line" entity and the "sensor" entity.

[0064] The semantic labels extracted by the semantic mapping unit mainly include entity identifiers, attribute relationship graphs, and data dependency paths. Entity identifiers are used to uniquely identify the physical or logical entity corresponding to the data. For example, the entity identifier in sensor data can be the physical number of the sensor (such as "SN-2023001"), and the entity identifier in business data can be an order number, user ID, etc. The accuracy of the entity identifier directly affects the traceability of data in cross-system interactions. For example, when a sensor data anomaly occurs, the entity identifier can be used to quickly locate the corresponding physical device.

[0065] An attribute relationship graph is a graphical description of the relationships between entities. It uses nodes and edges to represent dependencies, inclusions, associations, and other relationships between entities and their attributes. For example, in a smart manufacturing scenario, an attribute relationship graph can represent that the "equipment" entity is connected to the "production line" entity through the "installed on" relationship, and the "sensor" entity is connected to the "equipment" entity through the "belongs to" relationship. The attributes of each entity (such as device model and sensor accuracy) are stored as the attribute values ​​of the node. Attribute relationship graphs not only help understand the business context of data, but can also be used for data verification and quality monitoring. For example, when the "device number" attribute value of a certain sensor data cannot find a corresponding record in the device entity, it can be determined to be invalid data.

[0066] The data dependency path records the source and conversion history of data during its generation, transmission, and processing. For example, a piece of temperature data may come from the original measurement value of "Sensor A" and is generated after processing steps such as protocol parsing, format conversion, and unit conversion. The extraction of the data dependency path requires combining the protocol identifier of the data source management module, the parsing log of the protocol adapter module, and the rule application record of the format conversion unit to form a complete data traceability chain. The role of the data dependency path is that when problems arise with data quality, the problem link can be located through traceability analysis. For example, if the unit conversion rule of a batch of data is found to be incorrect, it can be traced back to the corresponding rule configuration error in the format conversion unit.

[0067] In practical applications, the format conversion unit and the semantic mapping unit usually need to work together. For example, when converting intermediate format data into the target structure, the format conversion unit needs to determine the hierarchical structure of the fields based on the entity relationship in the semantic label - if a field belongs to the "location" attribute of the "device" entity, it will be placed under the path of "device information.location" in the target structure; when the semantic mapping unit extracts the attribute relationship graph, it needs to refer to the field mapping results after the format conversion to ensure that the label is consistent with the target data structure. The collaborative mechanism between the two is achieved through a shared metadata model. The metadata model not only defines the semantic rules of the data, but also stipulates the organization of the data structure, so that the format conversion and semantic extraction processes follow a unified business logic.

[0068] The data conversion module must also have a rule management function, allowing system administrators to configure and modify preset rules through a visual interface. The rule management interface usually includes modules such as the field mapping table, data type conversion dictionary, and timestamp format library. The field mapping table displays the correspondence between intermediate fields and target fields in a tabular form, supporting batch import / export and online editing; the data type conversion dictionary lists all supported type conversion rules, such as "string to integer" and "enumeration value mapping", etc. Administrators can select the appropriate conversion rule for the field through the drop-down menu; the timestamp format library predefines common time format templates (such as RFC3339, Unix timestamp), and supports custom format expressions (such as matching specific time strings through regular expressions).

[0069] To ensure the accuracy of data conversion, the module requires a built-in verification mechanism. After rule configuration is complete, the system automatically generates test cases, using sample data to verify that the conversion results meet expectations. For example, by inputting intermediate format data containing "device_id:SN-123, timestamp:2023-10-01T12:00:00Z, value:25.5," the module verifies that the converted target data correctly maps fields, converts timestamp formats, and preserves numerical precision. If rule errors are discovered during verification, the system prompts the administrator to correct them, preventing incorrect rules from being applied to actual data processing.

[0070] Furthermore, the data conversion module must support dynamic rule loading. When the target system's data model changes (such as adding new fields or adjusting field types), administrators can update pre-set rules without restarting the system, and the new rules take effect immediately. This dynamic rule loading mechanism is implemented through hot deployment techniques, such as storing rules in a JSON-formatted configuration file. The module periodically scans for configuration file changes and reloads the rules, or receives rule update requests through an API interface, which take effect immediately.

[0071] When processing large amounts of data, the data conversion module employs a pipeline architecture, splitting operations such as field mapping, type conversion, timestamp alignment, and semantic extraction into multiple processing nodes. Each node is responsible for a specific conversion task, and data flows through each node in a sequential manner. This pipeline architecture improves processing efficiency and supports parallel computing and load balancing. For example, data from different data sources can be assigned to different processing threads, preventing processing delays from a single data source from impacting the overall process.

[0072] Example 4: In this embodiment, the data verification module and the anomaly detection module together constitute a data integrity management system. Its core function is to quantitatively evaluate data integrity through algorithms based on the semantic labels output by the data conversion module and trigger early warnings for abnormal conditions.

[0073] The core of the data verification module is the execution of a data integrity algorithm, which assesses the integrity of the target data structure through field-level verification. The algorithm relies on the semantic tags provided by the data conversion module, specifically entity identifiers and attribute relationship graphs, to locate the relationships between data fields and the scope of verification. The algorithm specifically includes the following steps: A hash algorithm is used to generate a checksum reference value for the target data structure's field set. A hash algorithm is a one-way cryptographic function that converts input data of any length into a fixed-length digest value (e.g., MD5 generates a 128-bit digest, SHA-256 generates a 256-bit digest). During this step, the algorithm combines all fields in the target data structure (such as device number, measurement value, and timestamp) into a single input string and generates a unique checksum reference value using a hash function. This checksum reference value represents the complete state of the field set. Subsequent additions, deletions, or modifications to data fields will significantly change the digest value. For example, for data containing three fields: "Device Number," "Temperature Value," and "Collection Time," the combined string might be "Device Number_12345Temperature Value_23.5Collection Time_2023-10-01 12:00:00." The digest value generated by the SHA-256 algorithm serves as a reference for the complete data.

[0074] The number of missing fields is calculated based on the verification benchmark value. The system identifies the fields that are not included in the actual data by comparing the actual data fields with the field list defined by the target data structure. The field list of the target data structure is pre-defined by the metadata model, which clarifies the required fields and optional fields that each entity should contain. For example, the target structure of sensor data may stipulate that "device number", "measurement value" and "acquisition time" are required fields, and "status description" is an optional field. During the verification process, if the actual data lacks any required field (such as "acquisition time" is not transmitted), it is counted as a missing record; the missing of optional fields is usually not included in the calculation, but it can be configured to participate in the verification according to business needs. The number of missing fields is recorded as , which directly reflects the integrity defects of the data field.

[0075] Calculate the actual data field missing rate. The field missing rate is determined by the ratio of the number of missing fields to the total number of fields. The calculation formula is:

[0076] in, represents the field missing rate, Indicates the total number of fields defined in the target data structure (including required fields and optional fields that participate in validation). For example, if there are 10 total fields and 2 required fields are missing, the missing rate is 20%. The missing rate is a value between 0 and 1, with higher values ​​indicating more severe data integrity issues.

[0077] Finally, the data completeness coefficient is calculated by the field missing rate. The calculation formula is:

[0078] in, Indicates the data integrity coefficient, and its value range is 0 to 1. When , it indicates that the data field is completely complete; when For example, if the field missing rate is 20%, the completeness coefficient is 0.8, indicating that the data integrity is at a high level.

[0079] Data integrity coefficient generated by the data verification module The data is transmitted to the anomaly detection module in real time, which then performs integrity status judgment. The anomaly detection module has a built-in preset integrity threshold. The threshold is set based on business requirements and data standards, and is usually between 0.8 and 0.95 (e.g., 0.9). The judgment logic is as follows: like , the data integrity is determined to be substandard and a data missing warning is triggered. The warning information includes the data source identifier, integrity coefficient, and a list of missing fields, such as "Data source SN-001 integrity coefficient 0.75, missing fields: collection time, status description."

[0080] like , the data is judged to be complete and no warning is triggered.

[0081] In practical applications, the integrity threshold Dynamic configuration is supported. System administrators can use the backend interface to set differentiated thresholds for different data sources. For example, a higher threshold (0.95) can be set for real-time monitoring data (such as industrial sensor data) and a lower threshold (0.8) for non-real-time business data (such as log file data). Threshold configuration should be based on the importance of the data, the frequency of collection, and the historical integrity of the data. Avoid excessively strict thresholds that lead to frequent warnings, or excessively loose thresholds that lead to invalid data entering the system.

[0082] The warning trigger mechanism of the anomaly detection module is integrated with the system's message notification system, supporting multiple warning methods: Logging: Write warning information into the system log, recording the time, data source, anomaly type and coefficient value in detail for subsequent auditing and fault tracing.

[0083] Interface alert: Abnormal data sources are marked with eye-catching colors (such as red) on the system monitoring interface, and a prompt box pops up to display the warning details.

[0084] Message push: Send early warning notifications to designated administrators via email, SMS, and instant messaging tools (such as WeChat for Business and DingTalk) to ensure timely response.

[0085] API callback: Push warning information to external systems (such as operation and maintenance management platforms) through preset API interfaces, triggering automated processing processes (such as re-collecting data and starting troubleshooting scripts).

[0086] Detailed processing logs must be kept for the entire process of data verification and anomaly detection. The log contents include: Data receiving time: records the specific time when the data enters the verification module.

[0087] Data source identifier: a unique identifier assigned by the data source management module, such as "API-HTTP-003".

[0088] Verification baseline value: The field set summary value generated by the hash algorithm is used to compare whether the data has changed.

[0089] Missing field details: includes information such as the field name, the entity to which it belongs, and whether it is a required field.

[0090] Integrity coefficient calculation process: record 、 、 、 The specific value of .

[0091] Threshold comparison results: Display and Numerical comparison and conclusion.

[0092] To meet the real-time verification needs of large-scale data, the data verification module utilizes a distributed computing architecture. Multiple verification nodes process data from different data sources in parallel, distributing tasks through a load balancer to prevent overloading of any single node. Each verification node independently performs hash calculations, field comparisons, and coefficient calculations, with the results aggregated to a central controller for unified threshold determination and alert distribution. This distributed architecture significantly improves processing efficiency, supporting real-time verification of tens of thousands of data items per second and meeting the performance requirements of scenarios such as the Industrial Internet of Things and real-time data analysis.

[0093] In the dynamic expansion of data fields, the data validation module supports automatic adjustment of validation rules according to the update of metadata model. For example, when the target data structure adds a required field "geographic location", the system automatically updates The calculation scope of the new field is expanded and the new field is included in the missing check. This mechanism is implemented through real-time synchronization of the metadata model and the verification logic, ensuring that the verification rules are consistent with the data structure definition, avoiding verification failures caused by field changes.

[0094] Furthermore, for data containing nested structures or complex relationships (such as hierarchical data in JSON format), the data validation module uses recursive parsing to traverse all subfields. For example, in a three-layer nested structure of "device-sensor-measurement value," the system verifies the required fields at each level to ensure the integrity of the nested fields. This recursive parsing process, combined with the attribute relationship map within the semantic tags, clarifies the hierarchical attribution and dependencies of the fields, preventing the omission of deeply nested fields.

[0095] The design of the data integrity algorithm fully considers the balance between computational efficiency and accuracy. The hash algorithm is selected based on data volume and security requirements: for lightweight data (such as a single sensor record), a computationally efficient algorithm like MD5 is used; for large-scale data sets (such as batch file data), a more secure algorithm like SHA-256 is used. The algorithm also supports incremental verification mode. When data fields have not changed, the historical verification baseline value can be directly reused, skipping repeated calculation steps and improving processing efficiency.

[0096] Example 5: In this embodiment, the data quality monitoring module and the data quality warning module jointly construct a data consistency management system. Its core function is to quantitatively evaluate data consistency through a model based on the semantic labels output by the data conversion module and trigger warnings for conflicting states.

[0097] The core of the data quality monitoring module is the operation of the data consistency model, which assesses the logical consistency of data through entity attribute conflict analysis. The model relies on the semantic labels provided by the data conversion module, specifically the attribute relationship graph and data dependency paths, which are used to identify attribute associations and potential conflicts between entities. The model specifically includes the following steps: Calculate the entity attribute conflict rate. Entity attribute conflict refers to inconsistent attribute values ​​for the same entity in different data sources or different processing links. For example, the "Manufacturer" attribute of the same device may be "Manufacturer A" in sensor data and "Manufacturer B" in business system data. The conflict rate is determined by the ratio of the number of conflicting attributes to the total number of associated attributes. The calculation formula is:

[0098] in, represents the attribute conflict rate, Indicates the number of conflicting attributes detected, Indicates the total number of associated attributes participating in the consistency check. Associated attributes refer to attributes with business associations defined in the metadata model, such as "Device Number" and "Device Model," "Order Number" and "Order Time," etc. During the verification process, the system traverses all associated attribute pairs through the attribute relationship graph and determines whether the attribute values ​​are consistent based on business rules. For example, if the "Device Number" of a device is "SN-123," its "Device Model" attribute should be consistent in both the sensor data and the asset management system data. Any discrepancy is counted as a conflict.

[0099] The data consistency coefficient is calculated by the attribute conflict rate. The calculation formula is:

[0100] in, Indicates the data consistency coefficient, ranging from 0 to 1. When , it indicates that all associated attribute values ​​are completely consistent; when When , it indicates that all associated attribute values ​​are conflicting. For example, if the total number of associated attributes is 50 and 5 pairs of conflicting attributes are detected, the conflict rate is 10% and the consistency coefficient is 0.9, indicating that the data consistency is at a high level.

[0101] Data consistency coefficient generated by the data quality monitoring module The data is transmitted to the data quality warning module in real time, and the module performs consistency status judgment. The data quality warning module has a built-in preset consistency threshold. The threshold is set according to business rules and data standards, usually between 0.8 and 0.95 (such as 0.9). The judgment logic is as follows: like , the data is determined to have a consistency conflict and a data conflict warning is triggered. The warning information includes the data source identifier, consistency coefficient, conflict attribute list, and specific conflict values. For example, "Data source DB-002 consistency coefficient 0.85, conflict attribute: device model (sensor data is 'ModelX', business data is 'ModelY')."

[0102] like , the data is determined to be consistent and no warning is triggered.

[0103] Consistency threshold Differentiated configurations are supported, allowing system administrators to set different thresholds based on the trustworthiness of data sources. For example, a higher threshold (0.95) can be set for homogeneous data (e.g., different tables in the same database), while a lower threshold (0.85) can be set for heterogeneous data (e.g., sensor data and data from third-party systems). Threshold configuration should be tailored to the fault tolerance of the business scenario. For example, financial transaction data requires extremely high consistency (threshold 0.99), while log analysis data can tolerate a certain degree of conflict (threshold 0.8).

[0104] The data quality warning module's warning trigger mechanism is integrated with the system's event management process to support multi-level response strategies: Primary warning: Close to the threshold (such as ), a yellow warning is triggered, and the administrator is prompted to pay attention to data fluctuations through notifications within the system.

[0105] Intermediate warning: However, if the conflict level is not reached (e.g., the conflict rate is less than 30%), an orange alert will be triggered, relevant personnel will be notified via email and text message, and a conflict data list will be automatically generated.

[0106] Advanced Warning: When When the value falls below the severe conflict threshold (such as 0.5), a red alert is triggered, data access is suspended, and the automatic repair process is started (such as retrying data collection and calling the data cleaning interface), and the technical team is notified to intervene.

[0107] The entire process of data quality monitoring and early warning requires detailed audit logs, which include: Data processing time: records the specific time when data enters the monitoring module.

[0108] Data source ID: a unique identifier associated with the data source management module, such as "F-SFTP-001".

[0109] Association attribute verification details: includes attribute pair name, data source A value, data source B value, and conflict determination rules (such as "string exact match" and "fuzzy match threshold 80%").

[0110] Calculation process of consistency coefficient: record 、 、 、 Specific value of .

[0111] Threshold comparison results: Display and Numerical comparison and warning level.

[0112] To improve the efficiency of consistency verification for large-scale data, the data quality monitoring module utilizes distributed in-memory computing technologies (such as Spark Streaming). By loading the attribute relationship graph into distributed memory, each compute node can quickly query entity relationships and perform attribute value comparisons in parallel. For example, for data on millions of devices, the system can bucket the device IDs by hashing them into different nodes. Each node independently verifies the consistency of the device attributes in the corresponding bucket, and the final results are aggregated by the aggregation node, achieving a response time of seconds.

[0113] When processing cross-entity linked data (such as the association between order data and inventory data), the data quality monitoring module supports delayed consistency checks based on time windows. For example, the "inventory quantity" attribute of a particular order data must be consistent with the inventory system's data within the same time window (e.g., 10 minutes). If a temporary conflict occurs due to data synchronization delay, the system can set a buffer period (e.g., 30 minutes) and automatically retry the check within the buffer period to avoid misidentification of data conflicts due to network delays.

[0114] The conflict resolution rules of the data consistency model support custom configuration to adapt to diverse business logic. For example: Exact match rule: requires that attribute values ​​are exactly the same (such as ID number, device unique identifier), applicable to primary key attributes.

[0115] Fuzzy matching rule: allows certain differences in attribute values ​​(such as simplified and traditional Chinese characters for names, different expressions of addresses), and calculates the matching degree using similarity algorithms (such as edit distance and cosine similarity). When the matching degree falls below a preset threshold (such as 80%), it is determined to be a conflict.

[0116] Business rule matching: Consistency is determined based on industry standards or enterprise-defined rules (e.g., the date format must be YYYY-MM-DD, and the value must be within a specified range). For example, the "age" attribute value must be greater than 0 and less than 150, otherwise it is considered a conflict.

[0117] In addition, the data quality monitoring module supports comparative analysis of historical data. By tracking the time series changes in the attribute values ​​of the same entity, it can identify gradual conflicts (such as historical data inconsistencies caused by slow changes in device models). The system traces the source of attribute values ​​through data dependency paths, compares the differences in values ​​at different time points, and generates conflict evolution trend reports, providing a basis for decision-making in data governance.

[0118] The data quality warning module works in conjunction with the result output module, pushing warning information to the target application system via an API. For example, if a conflict occurs between order data and logistics data on an e-commerce platform, the warning information can trigger the order system to automatically mark the order as abnormal, the logistics system to generate an abnormal work ticket, and the customer service system to simultaneously send a notification to the user, forming a cross-system collaborative processing process.

[0119] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0120] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-protocol scalable data acquisition system architecture with a unified interface standard, characterized by: include: The data source management module stores different data source configuration information into the data source knowledge base according to the data source type and assigns a unique protocol identifier to each data source; The protocol adaptation module includes a protocol parsing unit and a protocol matching unit. After receiving external data, the protocol parsing unit calls the corresponding protocol parsing library according to the protocol identifier to convert the original data into a standard intermediate format. The protocol matching unit verifies the protocol compatibility and confirms data access after verification. A data conversion module includes a format conversion unit and a semantic mapping unit. The format conversion unit converts the intermediate format data into the target data structure using preset rules, and the semantic mapping unit extracts data semantic tags according to the metadata model. The data verification module calculates the data integrity coefficient through the data integrity algorithm based on the data semantic labels of the data conversion module; The anomaly detection module determines the data integrity status based on the data integrity coefficient of the data verification module and triggers warning rules for abnormal results; The data quality monitoring module calculates the data consistency coefficient based on the data semantic labels of the data conversion module through the data consistency model; The data quality warning module judges the data quality status based on the data consistency coefficient of the data quality monitoring module and triggers warning rules for abnormal results.

2. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: In the data source management module, different data source configuration information is stored in a data source knowledge base according to the data source type, and the configuration information includes the communication protocol type and the data collection frequency.

3. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: In the protocol adaptation module, the protocol parsing unit calls the corresponding protocol parsing library according to the protocol identifier, converts the original data into a standard intermediate format, and the system verifies the protocol compatibility of the data source and outputs the corresponding protocol configuration in the data source knowledge base. The protocol matching unit verifies whether the data format complies with the target specification and whether it supports dynamic expansion. After verification, data access is confirmed.

4. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: In the data conversion module, the format conversion unit uses preset rules to convert the intermediate format data into the target data structure, including field mapping rules, data type conversion rules, and timestamp alignment rules. The semantic mapping unit extracts data semantic labels including entity identifiers, attribute relationship graphs, and data dependency paths based on the metadata model.

5. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: In the data verification module, based on the data semantic tags transmitted by the data conversion module, the data integrity coefficient is calculated by the data integrity algorithm as follows: Step S01: Use the hash algorithm to generate a checksum value for the field set of the target data structure, and calculate the number of missing fields based on the checksum value. ; Step S02: Calculate the actual data field missing rate, the calculation formula is: in represents the field missing rate, Indicates the number of missing fields, Indicates the total number of fields; Step S03: Calculate the data integrity coefficient, the calculation formula is: in Indicates the data integrity coefficient.

6. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: The anomaly detection module receives the data integrity coefficient transmitted by the data verification module , and the data integrity coefficient With the preset integrity threshold Compare and judge the data integrity status. If the data integrity coefficient Less than the preset integrity threshold , then trigger the data missing warning, if the data integrity coefficient Greater than or equal to the preset integrity threshold , the data is determined to be complete.

7. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: In the data quality monitoring module, based on the semantic tags transmitted by the data conversion module, the data consistency coefficient is calculated by the data consistency model as follows: Step S01: Calculate the entity attribute conflict rate, the calculation formula is: in represents the attribute conflict rate, Indicates the number of conflicting attributes, Indicates the total number of associated attributes; Step S02: Calculate the data consistency coefficient, the calculation formula is: in Represents the data consistency coefficient.

8. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: The data quality warning module receives the data consistency coefficient transmitted by the data quality monitoring module , and the data consistency coefficient Consistency threshold with the preset Compare and judge the data quality status. If the data consistency coefficient Less than the preset consistency threshold , then trigger the data conflict warning, if the data consistency coefficient Greater than or equal to the preset consistency threshold , then the data are determined to be consistent.

9. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: The architecture also includes: The result output module is used to receive the warning information transmitted by the anomaly detection module and the data quality warning module, and transmit the warning information to the target application system through the API interface.

10. A multi-protocol scalable data acquisition method with a unified interface standard, applied to a multi-protocol scalable data acquisition system architecture with a unified interface standard as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: Step S1: storing different data source configuration information into a data source knowledge base according to the data source type, and assigning a unique protocol identifier to each data source; Step S2: Call the corresponding protocol parsing library according to the protocol identifier to verify the protocol compatibility. After verification, confirm data access; Step S3: Convert the intermediate format data into the target data structure using preset rules, and extract data semantic labels based on the metadata model; Step S4: Based on the data semantic label transmitted by the data conversion module, a data integrity coefficient is calculated using a data integrity algorithm; Step S5: Determine the data integrity status based on the data integrity coefficient and trigger warning rules for abnormal results; Step S6: Based on the data semantic labels transmitted by the data conversion module, a data consistency coefficient is calculated using a data consistency model; Step S7: Determine the data quality status based on the data consistency coefficient and trigger warning rules for abnormal results.

Images