Unified interface standard multiprotocol extensible data acquisition system architecture and method

Abstract

The application relates to the technical field of data collection and discloses a multi-protocol extensible data collection system architecture and method of a unified interface standard, which comprises a data source management module, a protocol adaptation module, a data conversion module, a data verification module and an exception detection module, a data quality monitoring module and a warning module and a result output module. The data source management module is used for storing data source configuration information and distributing protocol identifiers. The protocol adaptation module comprises an analysis unit and a matching unit and is used for realizing data format conversion and protocol compatibility verification. The data conversion module is used for generating target data through format conversion and semantic mapping. The data verification module and the exception detection module are used for monitoring data integrity based on an integrity algorithm and threshold comparison. The data quality monitoring module and the warning module are used for evaluating data quality through a consistency model and threshold comparison. The result output module is used for transmitting warning information to a target system. The method comprises the steps of data source configuration, protocol adaptation, data conversion, integrity and quality monitoring and warning and the like. The application improves system expansibility and data reliability and is suitable for multi-scene data collection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection, in particular to a multi-protocol extensible data collection system architecture and method with unified interface standards. BACKGROUND

[0002] In today's rapidly developing information technology, data as a key production factor, its collection, processing and utilization of the importance of increasingly prominent. However, the current data collection field is facing many challenges, which seriously restricts the efficient use of data and the expansion ability of the system.

[0003] From the perspective of data source diversity, with the wide application of Internet of Things, cloud computing, big data and other technologies, data sources show explosive growth and complex types. Different data sources may use completely different communication protocols, such as Modbus, OPCUA protocols commonly used in the industry, HTTP, RESTful protocols in the Internet field, and ZigBee, MQTT protocols in sensor networks. This makes it difficult for traditional data collection systems to support multiple protocols, often requiring separate development of adaptation modules for each protocol, resulting in high development costs, long development cycles, and great difficulty in maintenance. At the same time, the configuration information management of data sources is chaotic, lacking a unified storage and management mechanism, making it difficult for the system to quickly identify and call the configuration information of different data sources, seriously affecting the efficiency and accuracy of data collection.

[0004] In terms of data format conversion, the data formats produced by different data sources are different, including structured data such as JSON, XML, CSV, and unstructured data such as text, images, audio and video. Traditional data collection systems lack unified standards and specifications when converting data formats, and the conversion process relies on manual code writing, which is not only inefficient but also prone to errors. In addition, the understanding and extraction of data semantics is also a problem. Different data sources may have different definitions and interpretations of the same data, leading to semantic ambiguity in data sharing and interaction, and failing to accurately convey the true meaning of the data.

[0005] Data quality assurance is another key issue in the data collection process. During data collection, transmission and processing, problems such as data missing, data errors, and data inconsistency may occur, which will seriously affect the usability and reliability of the data. Traditional data collection systems lack a perfect data quality monitoring and early warning mechanism, and cannot timely discover and handle data quality problems, resulting in low-quality data entering the subsequent data processing and analysis link, and further affecting the accuracy of decision-making.

[0006] The lack of scalability and compatibility of the system is also an important challenge faced by current data collection systems. With the development of business and changes in demand, data sources and data types may continue to increase and update. Due to the limitations of the architecture design, the traditional data collection system is difficult to quickly adapt to these changes. When a new data source needs to be accessed or a new protocol needs to be supported, a large-scale modification and upgrade of the entire system is often required, which not only has high cost, but also may cause system downtime and affect business continuity.

[0007] The traditional data collection system has obvious deficiencies in the aspects of multi-protocol compatibility, data format conversion, data quality guarantee and system scalability, and there is an urgent need for a new data collection system architecture and method that can unify interface standards, support multi-protocol expansion and guarantee data quality to meet the diversified needs of modern data collection. SUMMARY

[0008] The purpose of the present application is to provide a multi-protocol scalable data collection system architecture with unified interface standards to solve the problems raised in the background art.

[0009] To achieve the above purpose, the present application provides the following technical solution: a multi-protocol scalable data collection system architecture with unified interface standards, comprising:

[0010] A data source management module stores different data source configuration information in a data source knowledge base according to the data source type, and assigns a unique protocol identifier to each data source;

[0011] A protocol adaptation module includes a protocol parsing unit and a protocol matching unit. The protocol parsing unit receives external data and calls the corresponding protocol parsing library according to the protocol identifier to convert the raw data into a standard intermediate format. The protocol matching unit verifies the protocol compatibility and confirms data access after verification;

[0012] A data conversion module includes a format conversion unit and a semantic mapping unit. The format conversion unit converts the intermediate format data into a target data structure using a preset rule. The semantic mapping unit extracts data semantic labels according to a metadata model;

[0013] A data verification module calculates a data completeness coefficient based on the data semantic labels of the data conversion module through a data integrity algorithm;

[0014] An anomaly detection module judges the data integrity state based on the data completeness coefficient of the data verification module and triggers an early warning rule for abnormal results;

[0015] A data quality monitoring module calculates a data consistency coefficient based on the data semantic labels of the data conversion module through a data consistency model.

[0016] The data quality early warning module determines the data quality state based on the data consistency coefficient of the data quality monitoring module, and triggers an early warning rule for an abnormal result.

[0017] Preferably, in the data source management module, different data source configuration information is stored in the data source knowledge base according to data source types, and the configuration information includes a communication protocol type and a data acquisition frequency.

[0018] Preferably, in the protocol adaptation module, a protocol parsing unit converts the original data into a standard intermediate format according to a protocol identifier, calls a corresponding protocol parsing library, and verifies the protocol compatibility of the data source. The protocol matching unit verifies whether the data format conforms to the target specification and whether it supports dynamic expansion, and confirms data access after verification.

[0019] Preferably, in the data conversion module, the format conversion unit converts the intermediate format data into a target data structure using preset rules, including field mapping rules, data type conversion rules, and timestamp alignment rules. The semantic mapping unit extracts data semantic labels according to a metadata model, including entity identifiers, attribute relationship graphs, and data dependency paths.

[0020] Preferably, in the data verification module, based on the data semantic labels transmitted by the data conversion module, the data completeness coefficient is calculated by a data integrity algorithm, and the specific content is as follows:

[0021] Step S01: Generate a verification reference value for the field set of the target data structure using a hash digest algorithm, and calculate the number of missing fields based on the verification reference value ;

[0022] Step S02: Calculate the actual data field missing rate, and the calculation formula is:

[0023]

[0024] Wherein represents the field missing rate, represents the number of missing fields, represents the total number of fields.

[0025] Step S03: Calculate the data completeness coefficient, and the calculation formula is:

[0026]

[0027] Wherein represents the data completeness coefficient.

[0028] Preferably, the abnormality detection module receives the data integrity coefficient transmitted by the data verification module , and compares the data integrity coefficient with a preset integrity threshold to determine the data integrity state. If the data integrity coefficient is less than the preset integrity threshold , a data loss warning is triggered. If the data integrity coefficient is greater than or equal to the preset integrity threshold , the data is determined to be complete.

[0029] Preferably, in the data quality monitoring module, the data consistency coefficient is calculated based on the semantic label transmitted by the data conversion module through a data consistency model, and the specific content is as follows:

[0030] Step S01: Calculate the entity attribute conflict rate, and the calculation formula is:

[0031]

[0032] wherein represents the attribute conflict rate, represents the number of conflict attributes, and represents the total number of associated attributes.

[0033] Step S02: Calculate the data consistency coefficient, and the calculation formula is:

[0034]

[0035] wherein represents the data consistency coefficient.

[0036] Preferably, the data quality warning module receives the data consistency coefficient transmitted by the data quality monitoring module , and compares the data consistency coefficient with a preset consistency threshold to determine the data quality state. If the data consistency coefficient is less than the preset consistency threshold , a data conflict warning is triggered. If the data consistency coefficient is greater than or equal to the preset consistency threshold , the data is determined to be consistent.

[0037] Preferably, the architecture further comprises:

[0038] A result output module is configured to receive the warning information transmitted by the abnormality detection module and the data quality warning module, and transmit the warning information to a target application system through an API interface.

[0039] Preferably, the application further comprises a unified interface standard multi-protocol extensible data acquisition method applied to the unified interface standard multi-protocol extensible data acquisition system architecture, and the method comprises the following steps:

[0040] Step S1: storing different data source configuration information into a data source knowledge base according to data source types, and assigning a unique protocol identifier to each data source;

[0041] Step S2: calling a corresponding protocol analysis library according to the protocol identifier, verifying protocol compatibility, and confirming data access after verification;

[0042] Step S3: converting intermediate format data into a target data structure using a preset rule, and extracting data semantic labels according to a metadata model;

[0043] Step S4: calculating a data integrity coefficient through a data integrity algorithm based on data semantic labels transmitted by a data conversion module;

[0044] Step S5: judging data integrity status according to the data integrity coefficient, and triggering an early warning rule for abnormal results;

[0045] Step S6: calculating a data consistency coefficient through a data consistency model based on data semantic labels transmitted by the data conversion module;

[0046] Step S7: judging data quality status according to the data consistency coefficient, and triggering an early warning rule for abnormal results.

[0047] Compared with the prior art, the application has the following beneficial effects:

[0048] The data source management module stores different data source configuration information into a data source knowledge base according to data source types, and assigns a unique protocol identifier to each data source, thereby realizing centralized management and unified identification of a variety of data source configuration information. This enables the system to quickly identify and call configuration information of different data sources, greatly improving the efficiency and accuracy of data source management. When a new data source needs to be accessed, its configuration information can be added to the data source knowledge base and assigned a protocol identifier, so that the data source can be quickly accessed, significantly reducing the expansion cost and difficulty of the system.

[0049] The protocol adaptation module can call the corresponding protocol analysis library according to the protocol identifier, convert the original data into a standard intermediate format, and verify whether the protocol compatibility and the data format conform to the target specification and whether dynamic expansion is supported, by the design of the protocol analysis unit and the protocol matching unit. This design effectively solves the problem of multi-protocol compatibility of traditional data acquisition systems, enables the system to seamlessly access data sources of various protocols, eliminates the need to develop an adaptation module for each protocol, and greatly reduces development costs and cycles. At the same time, the protocol matching unit verifies the data format to ensure the standardization and consistency of the accessed data, laying a good foundation for subsequent data processing.

[0050] The format conversion unit of the data conversion module converts the intermediate format data into a target data structure by using preset field mapping rules, data type conversion rules and timestamp alignment rules, realizing efficient conversion between different data formats. The semantic mapping unit extracts data semantic labels including entity identifiers, attribute relationship graphs and data dependency paths according to the metadata model, solves the problem of data semantic ambiguity, enables data to accurately convey its true meaning in the process of sharing and interaction, and improves the readability and usability of data.

[0051] The data verification module can accurately detect data missing conditions by calculating the data integrity coefficient based on the data integrity algorithm. The anomaly detection module judges the data integrity state based on the data integrity coefficient, and triggers the early warning rule for abnormal results, realizing real-time monitoring and early warning of data integrity, discovering data missing problems in time and taking corresponding measures, and guaranteeing the integrity and reliability of data.

[0052] The data quality monitoring module calculates the data consistency coefficient based on the data consistency model, and the data quality warning module judges the data quality state based on the data consistency coefficient and triggers the early warning rule, realizing effective monitoring and management of data consistency. Timely discovery of data conflict problems avoids low-quality data entering subsequent processing links, improves the quality and usability of data, and provides a reliable basis for data analysis and decision-making.

[0053] The result output module transmits the early warning information transmitted by the anomaly detection module and the data quality warning module to the target application system through the API interface, realizes the timely transmission and processing of the early warning information, and enables relevant personnel to respond to data quality problems quickly, improving the intelligent and automated level of the system. BRIEF DESCRIPTION OF DRAWINGS

[0054] Fig. 1 The working principle diagram of the unified interface standard multi-protocol extensible data acquisition system architecture described in the application;

[0055] Fig. 2 The design diagram of the data verification module;

[0056] Fig. 3 The design diagram of the anomaly detection module. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0058] Please refer to Figs. 1-3 The present application relates to a unified interface standard multi-protocol extensible data acquisition system architecture, which comprises a data source management module, a protocol adaptation module, a data conversion module, a data verification module, an anomaly detection module, a data quality monitoring module and a data quality early warning module. The specific implementation is as follows:

[0059] The data source management module stores different data source configuration information into a data source knowledge base according to data source types, and allocates a unique protocol identifier to each data source. The protocol adaptation module comprises a protocol parsing unit and a protocol matching unit. The protocol parsing unit receives external data, calls a corresponding protocol parsing library according to the protocol identifier, and converts the original data into a standard intermediate format. The protocol matching unit verifies protocol compatibility, and confirms data access after verification.

[0060] The data conversion module comprises a format conversion unit and a semantic mapping unit. The format conversion unit converts the intermediate format data into a target data structure by using a preset rule. The semantic mapping unit extracts data semantic labels according to a metadata model. The data verification module calculates a data integrity coefficient by using a data integrity algorithm based on the data semantic labels of the data conversion module. The anomaly detection module judges data integrity states based on the data integrity coefficient of the data verification module, and triggers an early warning rule for an abnormal result. The data quality monitoring module calculates a data consistency coefficient by using a data consistency model based on the data semantic labels of the data conversion module. The data quality early warning module judges data quality states based on the data consistency coefficient of the data quality monitoring module, and triggers an early warning rule for an abnormal result.

[0061] The present application will be further described below in combination with specific embodiments:

[0062] Embodiment 1:

[0063] In this embodiment, the core function of the data source management module is to implement standardized configuration management and unique identification allocation for different types of data sources, ensuring that the system can orderly identify and access various heterogeneous data sources, and providing a foundation for subsequent data processing procedures.

[0064] The data source management module needs to classify and manage data sources. According to the physical form, data generation method and communication characteristics of the data source, it is divided into sensor data source, database data source, file data source, network interface data source and other types. For example, sensor data sources can be further divided into industrial sensors, environmental monitoring sensors, etc., and their data is usually transmitted in real-time stream form through specific industrial protocols; database data sources include relational databases (such as MySQL, Oracle) and non-relational databases (such as MongoDB), and data is interacted through database connection protocols; file data sources include CSV, Excel, JSON and other formats of files, and data is obtained through file system interface or network file transfer protocol; network interface data source includes API interface that provides data service through HTTP, WebSocket and other protocols. Through clear type division, the module can develop differentiated configuration strategies for the characteristics of different types of data sources, improving management efficiency.

[0065] The module needs to store the configuration information of different data sources in the data source knowledge base. Configuration information mainly includes two core parameters: communication protocol type and data acquisition frequency, and can also include auxiliary information such as data source name, physical address, system affiliation, update cycle, permission authentication information. Communication protocol type is the key basis for data source and system to interact with data, for example, sensor data source may use ModbusRTU, ModbusTCP, OPCUA industrial protocol, database data source needs to configure JDBC, ODBC connection protocol, file data source needs to specify FTP, SFTP, HTTP transmission protocol, network interface data source needs to specify RESTful, GraphQL interface protocol. The setting of data acquisition frequency needs to be combined with the characteristics of data source and business demand. For industrial sensors with very high real-time requirements (such as temperature, pressure sensors on the production line), millisecond or second level acquisition frequency can be set to ensure timely acquisition of equipment running state data; for batch update business database (such as transaction system database that archives data at dawn every day), the acquisition frequency can be set to once a day; for unstructured log file data source, you can set a timed collection task according to the file generation period (such as hourly, daily).

[0066] When storing configuration information, the data source knowledge base adopts a structured storage method, such as building a data table based on a relational database, taking the data source type as a classification index, each data source corresponding to a record, and the record containing the fields of various configuration parameters. Through this structured storage, the module can quickly retrieve and update data source configuration information, while supporting configuration management by the system administrator through a visual interface, such as adding a new data source, modifying existing data source configurations, deleting invalid data sources, and other operations. In addition, the knowledge base needs to have a data backup and recovery mechanism to ensure the security and reliability of the configuration information and to avoid data loss due to system failure or human error.

[0067] Assigning a unique protocol identifier to each data source is one of the key functions of the data source management module. The identifier is generated using a specific coding rule to ensure its uniqueness within the entire system and to reflect the type and protocol characteristics of the data source. For example, the identifier can be composed of data source type code, protocol type code, and serial number: "S" represents a sensor data source, "DB" represents a database data source, "F" represents a file data source, and "API" represents a network interface data source; the protocol type code can use the abbreviation of the protocol name, such as "MOD" for Modbus protocol, "OPC" for OPCUA protocol, and "JDBC" for JDBC protocol; the serial number is used to distinguish different data sources under the same type and protocol, and is coded with an incremental number. For example, the identifier of a temperature sensor data source using ModbusTCP protocol can be defined as "S-MOD-TCP-001", and the identifier of a MySQL database data source based on JDBC protocol can be defined as "DB-JDBC-MySQL-002".

[0068] The unique protocol identifier plays a role throughout the entire data acquisition process. In the protocol adaptation phase, the protocol parsing unit quickly locates the corresponding protocol parsing library through the identifier, and automatically completes protocol matching without human intervention; during data routing and processing, the system can identify the type and protocol characteristics of the data source according to the identifier, and call the corresponding processing logic and resources; when the system is maintained and expanded, the identifier provides a convenient index for the management and tracking of data sources, such as quickly locating the configuration information, data flow, and processing status of related data sources by searching for a specific protocol identifier.

[0069] In addition, the data source management module also needs to have a data source state monitoring function, which tracks the connection state of the data source, the execution of the collection task, etc. For sensor data sources, the module can periodically send heartbeat packets to detect whether the device is online. If no response is received for a continuous number of times, it is marked as offline and a warning is triggered. For database data sources, the use of the connection pool, query response time, and other indicators can be monitored to promptly discover connection abnormalities or performance bottlenecks. For file data sources, the module can monitor whether the file is generated on time and whether the file size is normal to avoid data collection failures due to missing or damaged files. Through state monitoring, the module can promptly discover problems at the data source level, providing a guarantee for the stability and reliability of the system.

[0070] In terms of dynamic adjustment of data collection frequency, the module supports automatic or manual adjustment of collection frequency based on business needs or system load. For example, during peak periods of industrial production, the collection frequency of sensor data sources can be temporarily increased from once every minute to once every second to more closely monitor device operating status. When the system detects that the connection load of a certain database data source is too high, it can automatically reduce the collection frequency to reduce the pressure on the database server. The dynamic adjustment mechanism is realized through linkage with the system's task scheduling module and resource monitoring module, ensuring a balance between efficiency and resource consumption in data collection.

[0071] For the requirement of multi-protocol compatibility, the data source management module supports configuring multiple backup protocols for the same data source to deal with situations such as main protocol failure or system upgrade. For example, a sensor data source can be configured with both ModbusTCP and MQTT protocols. When the main protocol (ModbusTCP) cannot communicate due to network port failure, the module can automatically switch to the backup protocol (MQTT) for data collection, ensuring the continuity of data transmission. The configuration and switching logic of the backup protocol need to be defined in advance in the data source knowledge base and seamlessly integrated with the protocol parsing library of the protocol adaptation module.

[0072] Embodiment 2:

[0073] In this embodiment, the protocol adaptation module serves as the key hub of the data collection system, taking on the tasks of protocol parsing and compatibility verification when external data is accessed. Its core function is to convert raw data from different protocols into a standard intermediate format that the system can process, and to ensure the standardization and reliability of data access.

[0074] The protocol adaptation module's operation begins with the protocol parsing unit receiving external data. When a data source sends data to the system through a physical interface (such as a serial port or Ethernet port) or a network protocol (such as TCP / IP or HTTP), the protocol parsing unit first obtains the raw data format and the corresponding protocol identifier—this identifier is pre-assigned by the data source management module and uniquely identifies the communication protocol used by the data source (such as Modbus, OPCUA, MQTT, etc.). Based on this identifier, the protocol parsing unit calls the corresponding parsing rules from the system's built-in protocol parsing library. The protocol parsing library is a predefined set of rules that standardizes the syntax structure, data frame format, encoding method, etc., of different protocols. For example, for the Modbus protocol, the parsing rules need to identify the slave address, function code, data fields, and checksum in the data frame; for JSON-formatted API data, the parsing rules need to extract key-value pair information according to the JSON schema definition.

[0075] The core task of the protocol parsing unit is to convert raw data into a standard intermediate format. This standard intermediate format is a system-defined, unified data structure that masks the underlying differences between protocols. For example, it uses key-value pairs or a JSON-like structure to store data fields, including common fields such as timestamps, device identifiers, data types, and values. During the conversion process, the protocol parsing unit needs to decode, extract, and reassemble the raw data according to the protocol parsing rules. Taking the Modbus protocol as an example, the raw data frame may contain a hexadecimal byte stream. The parsing unit needs to determine the data type (such as coil status or register value) based on the function code, extract the corresponding starting address and data value, and map them to the "device address" and "measurement value" fields in the intermediate format. For XML data returned by the HTTP interface, the parsing unit needs to extract the node content using an XML parser and convert it into structured fields in the intermediate format.

[0076] After completing the data format conversion, the system enters the protocol compatibility verification phase. Protocol compatibility verification is performed by the protocol matching unit to ensure that the protocol type, version, and configuration parameters of the data source are consistent with the system's supported range. During the verification process, the system first retrieves the corresponding protocol configuration information for the data source from the data source knowledge base, including protocol type, version number, transmission parameters (such as baud rate, data bits, and stop bits), and security authentication information (such as username, password, and certificate path). Then, the protocol matching unit verifies the following:

[0077] Protocol type compatibility: Check whether the protocol used by the data source is on the system's list of supported protocols. The list of supported protocols is pre-configured by the developers, including industrial protocols (Modbus, OPCUA, CANopen), IoT protocols (MQTT, CoAP), database protocols (JDBC, ODBC), etc. Protocols not on the list will be rejected from access.

[0078] Protocol version compatibility: For protocols that support multiple versions (such as ModbusTCPv1.1, v1.2), verify whether the protocol version used by the data source is compatible with the system parsing library to avoid parsing errors caused by protocol version differences.

[0079] Transmission parameter matching: For serial communication protocols (such as Modbus RTU), verify that parameters such as baud rate, data bits, stop bits, and parity are consistent with the system configuration; for network protocols, verify that parameters such as IP address, port number, and timeout are correct.

[0080] Security authentication validity: If the protocol requires identity authentication (such as SSL certificate verification for HTTPS, user permission verification for the database), the protocol matching unit needs to call the corresponding authentication interface to verify whether the identity credentials of the data source are valid, so as to prevent unauthorized devices from accessing the system.

[0081] After the protocol compatibility verification is passed, the protocol matching unit further performs data format verification to ensure that the converted intermediate format data conforms to the system's preset target specifications. The target specifications include the completeness of data fields, correct format, and logical consistency.

[0082] Field Integrity: Check whether the intermediate format data contains the required fields required by the system, such as timestamp, data source identifier, and at least one valid data field. Data with missing required fields will be considered invalid.

[0083] Format correctness: Verify whether the format of the data field conforms to the preset rules. For example, timestamps must be strings in ISO8601 format, numeric fields must be integers or floating-point numbers, and enumeration fields must match the preset enumeration value list.

[0084] Logical consistency: Check whether the logical relationship between data fields is reasonable. For example, the value of the "start time" field cannot be later than the value of the "end time" field, and the device status field (such as "running" or "fault") must match the value range of other related parameters (such as current and voltage).

[0085] In addition to static verification, the protocol adaptation module also supports a dynamic extension mechanism to accommodate future additions of protocol types or data formats. This dynamic extension mechanism is implemented in the following ways:

[0086] The protocol parsing library features a pluggable design: the system adopts a modular architecture, allowing developers to extend protocol support by adding new protocol parsing plugins (such as DLL files or Python scripts). Plugins must adhere to a unified interface specification and implement core functions such as data parsing and field mapping, achieving integration without modifying the underlying system code.

[0087] Dynamic loading of data format templates: For custom protocols or non-standard data formats, the system supports defining data parsing templates through a visual interface or configuration file. Templates include field extraction rules, data type conversion rules, protocol parameter configurations, etc. For example, for a specific private protocol, rules such as "extract 4 bytes from the 3rd bit of the byte stream as the device ID, and the 7th to 10th bits as the temperature value (hexadecimal to decimal conversion is required)" can be defined. The template takes effect immediately after being saved.

[0088] Compatibility support for protocol version upgrades: When the existing protocol parsing library needs to be upgraded to support the new version of the protocol, the system allows multiple versions of parsing rules to be retained at the same time, and the protocol version to be used can be specified through the data source configuration information to avoid the old data source from being unable to be accessed due to version upgrades.

[0089] Throughout the data access process, the protocol adaptation module needs to record detailed log information, including data reception time, data source identifier, protocol type, key steps in the parsing process, and verification results. This log information is used for system debugging, troubleshooting, and performance analysis. For example, when a data source frequently experiences protocol parsing errors, the logs can be used to pinpoint whether the error is due to incorrect protocol parsing rules, abnormal data format sent by the data source, or packet loss during network transmission leading to incomplete data.

[0090] In addition, the protocol adaptation module must also have an error handling mechanism to take corresponding measures for different types of abnormal situations:

[0091] Protocol parsing failure: If the original data does not conform to the protocol syntax rules (such as Modbus frame check error, JSON format syntax error), the system will discard the data frame and record the error log. It can also be configured to retry the data source after a set time interval.

[0092] Compatibility verification failed: If the data source protocol version is incompatible or the transmission parameters are incorrect, the system will refuse to connect to the data source and send a warning message to the administrator through the result output module, prompting them to check the data source configuration or upgrade the protocol parsing library.

[0093] Data format validation failed: For data with missing fields or incorrect format, the system can choose to discard the data, fill in the default value, or return the error information to the data source according to the configuration. For example, for data with missing timestamps, the current system time can be automatically filled in and marked as "filled in" so that the subsequent data validation module can identify it.

[0094] Example 3:

[0095] In this embodiment, the data conversion module acts as a bridge connecting the protocol adaptation module and the subsequent data processing stage. Its core function is to further convert the standard intermediate format data output by the protocol adaptation module into a data structure that meets the requirements of the target system, and extract the semantic tags of the data to provide structured and semantic input for data verification, quality monitoring and other stages.

[0096] The core task of the format conversion unit is to convert intermediate format data into the target data structure using preset rules. These preset rules are conversion logic predefined based on the target system's data model and business requirements, primarily including field mapping rules, data type conversion rules, and timestamp alignment rules. Field mapping rules are used to establish the correspondence between intermediate format data fields and target data structure fields, resolving differences in field naming, meaning, and hierarchical structure between different systems. For example, the "device_id" field in the intermediate format might need to be mapped to the "device unique identifier" field in the target data structure. If the target system uses a hierarchical data structure (such as nested JSON objects), the field mapping rules also need to define the nesting level of the fields, for example, mapping the "sensor.temperature" field in the intermediate format to the path "device status.sensor data.temperature value" in the target structure.

[0097] Data type conversion rules are used to handle differences in data types between different systems, ensuring that data can be correctly parsed and used in the target system. Common data type conversion scenarios include: converting string-type timestamps in intermediate formats (such as "2023-10-01 12:00:00") to the timestamp values ​​required by the target system (such as Unix timestamps); converting boolean "running status" fields (true / false) to enumerated values ​​required by the target system ("0" indicates stopped, "1" indicates running); and rounding or truncating floating-point temperature data (such as 23.5℃) according to the precision requirements of the target system. Data type conversion rules must strictly adhere to the data type definitions of the target system to avoid data storage errors or calculation anomalies due to type mismatches.

[0098] Timestamp alignment rules are designed for multi-source data fusion scenarios to address the issue of inconsistent time bases across different data sources. For example, some sensor data sources may generate timestamps using the device's local time, while database data sources may use UTC time. The format conversion unit needs to uniformly convert the timestamps from all data sources to the system's global time (such as UTC+8 standard time). The timestamp alignment process includes steps such as identifying the format of the original timestamp (e.g., ISO8601, Unix timestamps, custom formats), parsing the time value, and adjusting the time zone offset. This ensures that all data points in the same dataset are comparable in the time dimension, providing a foundation for subsequent time series analysis, data aggregation, and other operations.

[0099] The semantic mapping unit extracts semantic tags from the metadata model. These tags describe the business meaning, entity relationships, and dependency paths of the data, transforming it from simple numerical values ​​or strings into business-readable information. The metadata model is an abstract model of domain knowledge, typically including entity definitions, attribute relationships, and business rules. For example, in an Industrial Internet of Things (IIoT) scenario, the metadata model might define the "sensor" entity as having attributes such as "device number," "type," and "installation location," and the "production line" entity as having an "inclusion" relationship with the "sensor" entity.

[0100] The semantic tags extracted by the semantic mapping unit mainly include entity identifiers, attribute relationship graphs, and data dependency paths. Entity identifiers are used to uniquely identify the physical or logical entity corresponding to the data. For example, the entity identifier in sensor data could be the sensor's physical number (such as "SN-2023001"), while the entity identifier in business data could be an order number, user ID, etc. The accuracy of entity identifiers directly affects the traceability of data in cross-system interactions. For instance, when sensor data shows an anomaly, the corresponding physical device can be quickly located through the entity identifier.

[0101] An attribute relationship graph is a graphical description of the relationships between entities. It represents the dependencies, inclusions, associations, and other relationships between entities and their attributes through nodes and edges. For example, in a smart manufacturing scenario, an attribute relationship graph can represent the "equipment" entity connected to the "production line" entity through the "installed in" relationship, and the "sensor" entity connected to the "equipment" entity through the "belongs to" relationship. The attributes of each entity (such as equipment model and sensor accuracy) are stored as attribute values ​​for nodes. Attribute relationship graphs not only help in understanding the business context of data but can also be used for data validation and quality monitoring. For example, if the "equipment number" attribute value of a sensor data cannot be found in a corresponding record within the equipment entity, it can be determined as invalid data.

[0102] Data dependency paths record the source and transformation history of data during its generation, transmission, and processing. For example, a temperature data point might originate from the raw measurement value of "sensor A," and be generated after processing steps such as protocol parsing, format conversion, and unit conversion. Extracting data dependency paths requires combining the protocol identifier from the data source management module, the parsing logs from the protocol adaptation module, and the rule application records from the format conversion unit to form a complete data traceability chain. The purpose of data dependency paths is to pinpoint the problem link through traceability analysis when data quality issues arise. For example, if an error is found in the unit conversion rule of a batch of data, it can be traced back to the corresponding rule configuration error in the format conversion unit.

[0103] In practical applications, the format conversion unit and the semantic mapping unit typically need to work together. For example, when converting intermediate format data into the target structure, the format conversion unit needs to determine the hierarchical structure of fields based on the entity relationships in the semantic tags—if a field belongs to the "location" attribute of the "device" entity, it is placed under the path "device information.location" in the target structure; when extracting the attribute relationship graph, the semantic mapping unit needs to refer to the field mapping results after format conversion to ensure that the tags are consistent with the target data structure. The collaboration mechanism between the two is achieved through a shared metadata model. The metadata model not only defines the semantic rules of the data but also specifies the organization of the data structure, ensuring that the format conversion and semantic extraction processes follow a unified business logic.

[0104] The data conversion module also needs to have rule management functionality, allowing system administrators to configure and modify preset rules through a visual interface. The rule management interface typically includes modules such as a field mapping table, a data type conversion dictionary, and a timestamp format library. The field mapping table displays the correspondence between intermediate and target fields in tabular form, supporting batch import / export and online editing; the data type conversion dictionary lists all supported type conversion rules, such as "string to integer" and "enumeration value mapping," allowing administrators to select appropriate conversion rules for fields via drop-down menus; the timestamp format library predefines common time format templates (such as RFC3339 and Unix timestamps) and supports custom format expressions (such as matching specific time strings using regular expressions).

[0105] To ensure the accuracy of data transformation, the module needs a built-in verification mechanism. After the rules are configured, the system can automatically generate test cases to verify whether the transformation results meet expectations using sample data. For example, inputting intermediate format data containing "device_id:SN-123,timestamp:2023-10-01T12:00:00Z,value:25.5", the system verifies whether the transformed target data correctly maps the fields, converts the timestamp format, and retains numerical precision. If rule errors are found during the verification process, the system will prompt the administrator to correct them, preventing erroneous rules from being applied to the actual data processing workflow.

[0106] In addition, the data transformation module needs to support dynamic rule loading. When the data model of the target system changes (such as adding fields or adjusting field types), the administrator can update the preset rules without restarting the system, and the new rules will take effect immediately. The dynamic rule loading mechanism is implemented through hot deployment technology, such as storing the rules in a JSON format configuration file, and the module periodically scanning the configuration file for changes and reloading the rules, or receiving rule update requests through an API interface and making them effective immediately.

[0107] When processing large-scale data, the data transformation module adopts a pipelined architecture design, which breaks down operations such as field mapping, type conversion, timestamp alignment, and semantic extraction into multiple processing nodes. Each node is responsible for a specific transformation task, and data flows through each node sequentially in a streaming manner. The pipelined architecture can improve processing efficiency and support parallel computing and load balancing. For example, data from different data sources can be distributed to different processing threads to avoid the processing delay of a single data source affecting the overall process.

[0108] Example 4:

[0109] In this embodiment, the data verification module and the anomaly detection module together constitute a data integrity management system. Its core function is to quantitatively evaluate the data integrity based on the semantic tags output by the data conversion module and trigger early warnings for abnormal states.

[0110] The core of the data validation module is the execution of the data integrity algorithm, which assesses the integrity of the target data structure through field-level validation. The algorithm's execution relies on semantic tags provided by the data transformation module, especially entity identifiers and attribute relationship graphs, used to locate the relationships between data fields and the scope of validation. The algorithm specifically includes the following steps:

[0111] A hash digest algorithm is used to generate a verification benchmark value for the field set of the target data structure. A hash digest algorithm is a one-way encryption function that can convert input data of arbitrary length into a fixed-length digest value (e.g., MD5 generates a 128-bit digest, SHA-256 generates a 256-bit digest). In this step, the algorithm combines all fields in the target data structure (such as device number, measurement value, timestamp, etc.) into an input string and generates a unique verification benchmark value through a hash function. This verification benchmark value represents the complete state of the field set; if subsequent data fields are added, deleted, or modified, the digest value will change significantly. For example, for data containing three fields: "device number," "temperature value," and "collection time," the combined string might be "device number_12345 temperature value_23.5 collection time_2023-10-01 12:00:00," and the digest value generated by the SHA-256 algorithm can serve as the benchmark for the complete data.

[0112] The system calculates the number of missing fields based on the verification baseline. It identifies fields not included in the actual data by comparing them with the field list defined in the target data structure. The field list of the target data structure is predefined by the metadata model, specifying the required and optional fields for each entity. For example, the target structure for sensor data might specify "Device Number," "Measurement Value," and "Acquisition Time" as required fields, while "Status Description" is optional. During verification, if any required field is missing (e.g., "Acquisition Time" is not transmitted), it is counted as a missing record. Missing optional fields are generally not included in the calculation, but their inclusion in the verification can be configured according to business needs. The number of missing fields is denoted as... This value directly reflects the integrity defects of the data field.

[0113] Calculate the field missing rate of the actual data. The field missing rate is determined by the ratio of the number of missing fields to the total number of fields, and the formula is as follows:

[0114]

[0115] in, Indicates the field missing rate. This indicates the total number of fields defined in the target data structure (including required fields and optional fields used for validation). For example, if there are 10 fields in total and 2 required fields are missing, the missing field rate is 20%. The field missing field rate is a value between 0 and 1; the higher the value, the more serious the data integrity problem.

[0116] Finally, the data integrity coefficient is calculated using the field missing rate. The calculation formula is as follows:

[0117]

[0118] in, This represents the data integrity coefficient, and its value ranges from 0 to 1. When... When, it indicates that the data field is completely complete; when A missing value indicates that all data fields are missing. For example, if the field missing rate is 20%, the integrity coefficient is 0.8, indicating that the data integrity is at a high level.

[0119] Data integrity coefficient generated by the data verification module The data is transmitted in real time to the anomaly detection module, which then performs an integrity status assessment. The anomaly detection module has a built-in preset integrity threshold. This threshold is set based on business needs and data standards, and is typically a value between 0.8 and 0.95 (e.g., 0.9). The judgment logic is as follows:

[0120] like If the data is not complete, a data missing warning will be triggered. The warning information includes the data source identifier, integrity coefficient, and a list of missing fields, such as "Data source SN-001 integrity coefficient 0.75, missing fields: collection time, status description".

[0121] like If the data is found to be complete, no warning will be triggered.

[0122] In practical applications, integrity threshold Dynamic configuration is supported. System administrators can set differentiated thresholds for different types of data sources through the backend interface. For example, a higher threshold (0.95) can be set for real-time monitoring data (such as industrial sensor data), while a lower threshold (0.8) can be set for non-real-time business data (such as log file data). Threshold configuration should be based on the importance of the data, the frequency of collection, and historical integrity performance to avoid frequent warnings due to overly strict threshold settings or invalid data flowing into the system due to overly lenient threshold settings.

[0123] The anomaly detection module's early warning triggering mechanism is integrated with the system's message notification system, supporting multiple early warning methods:

[0124] Log recording: Write the warning information to the system log, and record the time, data source, anomaly type and coefficient value in detail for subsequent auditing and fault tracing.

[0125] Interface Alarm: Abnormal data sources are marked with a conspicuous color (such as red) on the system monitoring interface, and a pop-up prompt box displays the warning details.

[0126] Push notifications: Send alerts to designated administrators via email, SMS, or instant messaging tools (such as WeChat Work and DingTalk) to ensure timely response.

[0127] API callback: Push the warning information to the external system (such as the operation and maintenance management platform) through the preset API interface to trigger the automated processing process (such as re-collecting data, starting the fault diagnosis script).

[0128] Detailed processing logs must be maintained throughout the entire data validation and anomaly detection process. These logs should include:

[0129] Data reception time: Records the specific time when data enters the verification module.

[0130] Data source identifier: A unique identifier assigned by the data source management module, such as "API-HTTP-003".

[0131] Verification baseline value: A summary value of the field set generated by the hash algorithm, used to compare whether the data has changed.

[0132] Missing field details: Includes information such as field name, the entity to which it belongs, and whether it is a required field.

[0133] Completeness coefficient calculation process: Record , , , The specific value.

[0134] Threshold comparison results: Display and Numerical comparison and judgment conclusions.

[0135] To address the real-time verification requirements of large-scale data, the data verification module adopts a distributed computing architecture. Multiple verification nodes process data from different data sources in parallel, with tasks distributed through a load balancer to avoid overloading any single node. Each verification node independently performs hash calculations, field comparisons, and coefficient calculations, and the results are aggregated to a central controller for unified threshold judgment and alert distribution. The distributed architecture significantly improves processing efficiency, supporting real-time verification of tens of thousands of data points per second, meeting the performance requirements of scenarios such as Industrial IoT and real-time data analysis.

[0136] In scenarios where data fields are dynamically expanded, the data validation module supports automatically adjusting validation rules based on updates to the metadata model. For example, when a required "geographical location" field is added to the target data structure, the system automatically updates... The calculation scope is defined, and new fields are included in the missing field validation. This mechanism is achieved through real-time synchronization between the metadata model and the validation logic, ensuring that the validation rules remain consistent with the data structure definition and preventing validation failures due to field changes.

[0137] Furthermore, for data containing nested structures or complex relationships (such as hierarchical data in JSON format), the data validation module uses a recursive parsing approach to traverse all sub-fields. For example, for a three-level nested structure of "device-sensor-measurement value," the system validates the required fields at each level to ensure the integrity of the nested fields. The recursive parsing process, combined with the attribute relationship graph in the semantic tags, clarifies the hierarchical affiliation and dependencies of fields, avoiding the omission of deeply nested fields.

[0138] The data integrity algorithm is designed with a strong balance between computational efficiency and accuracy. The choice of hash digest algorithm is based on data volume and security requirements: for lightweight data (such as a single sensor record), computationally efficient algorithms like MD5 are used; for large-scale datasets (such as batch file data), more secure algorithms like SHA-256 are used. Furthermore, the algorithm supports incremental verification mode. When data fields have not changed, historical verification baseline values ​​can be directly reused, skipping repetitive calculation steps and improving processing efficiency.

[0139] Example 5:

[0140] In this embodiment, the data quality monitoring module and the data quality early warning module jointly construct a data consistency management system. Its core function is to evaluate data consistency through model quantification based on the semantic tags output by the data transformation module, and trigger early warnings for conflict states.

[0141] The core of the data quality monitoring module is the operation of the data consistency model, which assesses the logical consistency of data through entity attribute conflict analysis. The model's operation relies on semantic tags provided by the data transformation module, especially the attribute relationship graph and data dependency paths, used to identify attribute associations and potential conflicts between entities. The model specifically includes the following steps:

[0142] Calculate the entity attribute conflict rate. Entity attribute conflict refers to inconsistent attribute values ​​for the same entity across different data sources or processing stages. For example, the "manufacturer" attribute for the same device might be "Manufacturer A" in sensor data but "Manufacturer B" in the business system data. The conflict rate is determined by the ratio of the number of conflicting attributes to the total number of associated attributes. The formula is as follows:

[0143]

[0144] in, Indicates the attribute conflict rate. This indicates the number of conflicting attributes detected. This indicates the total number of associated attributes participating in the consistency check. Associated attributes refer to attributes defined in the metadata model that have business relationships, such as "device number" and "device model," or "order number" and "order time." During the check, the system traverses all associated attribute pairs through the attribute relationship graph and determines whether the attribute values ​​are consistent based on business rules. For example, if a device has an "device number" of "SN-123," its "device model" attribute should be consistent in both the sensor data and the asset management system data; any discrepancy is counted as a conflict.

[0145] The data consistency coefficient is calculated using the attribute conflict rate. The formula is as follows:

[0146]

[0147] in, This represents the data consistency coefficient, with a value ranging from 0 to 1. When... When, it indicates that all associated attribute values ​​are completely consistent; when A 0.5% consistency coefficient indicates that all associated attribute values ​​conflict. For example, if there are 50 pairs of associated attributes and 5 pairs of conflicting attributes are detected, the conflict rate is 10% and the consistency coefficient is 0.9, indicating that the data consistency is at a high level.

[0148] Data consistency coefficient generated by the data quality monitoring module Data is transmitted in real time to the data quality early warning module, which then performs a consistency status assessment. The data quality early warning module has a built-in preset consistency threshold. This threshold is set according to business rules and data standards, and is typically a value between 0.8 and 0.95 (e.g., 0.9). The judgment logic is as follows:

[0149] like If a data inconsistency conflict is detected, a data conflict warning will be triggered. The warning information includes the data source identifier, consistency coefficient, list of conflict attributes, and specific conflict values, such as "Data source DB-002 consistency coefficient 0.85, conflict attribute: device model (sensor data is 'ModelX', business data is 'ModelY')".

[0150] like If the data is consistent, no warning will be triggered.

[0151] Consistency threshold Differentiated configuration is supported, allowing system administrators to set different thresholds based on the trust level of the data source. For example, a higher threshold (0.95) can be set for data from the same source (different tables in the same database), while a lower threshold (0.85) can be set for data from different sources (such as sensor data and data from third-party systems). Threshold configuration needs to be combined with the fault tolerance of the business scenario. For example, financial transaction data requires extremely high consistency (threshold 0.99), while log analysis data can tolerate a certain degree of conflict (threshold 0.8).

[0152] The data quality early warning module's early warning triggering mechanism is integrated with the system's event management process, supporting multi-level response strategies.

[0153] Primary warning: When Approaching threshold (e.g.) When a yellow alert is triggered, the administrator will be notified via system notification to pay attention to data fluctuations.

[0154] Intermediate warning: When However, if the conflict does not reach the level of serious conflict (e.g., the conflict rate is below 30%), an orange alert will be triggered, relevant personnel will be notified via email and SMS, and a conflict data list will be automatically generated.

[0155] Advanced warning: When When the value falls below the severe conflict threshold (e.g., 0.5), a red alert is triggered, data access is suspended and an automatic repair process is initiated (e.g., retrying data acquisition or calling the data cleaning interface), and the technical team is notified to intervene.

[0156] The entire process of data quality monitoring and early warning must be recorded in detailed audit logs, which include:

[0157] Data processing time: Records the specific time when data enters the monitoring module.

[0158] Data source identifier: A unique identifier associated with the data source management module, such as "F-SFTP-001".

[0159] Related attribute validation details: Includes attribute pair name, data source A value, data source B value, and conflict determination rules (such as "string exact match" or "fuzzy match threshold 80%).

[0160] Consistency coefficient calculation process: Record , , , The specific value.

[0161] Threshold comparison results: Display and Numerical comparison and warning levels.

[0162] To improve the efficiency of consistency verification for large-scale data, the data quality monitoring module employs distributed in-memory computing technology (such as Spark Streaming). By loading the attribute relationship graph into distributed memory, each computing node can quickly query entity relationships and perform attribute value comparisons in parallel. For example, for millions of device data points, the system can hash and bucket the device IDs onto different nodes. Each node independently verifies the consistency of device attributes in its corresponding bucket, and the final results are aggregated by an aggregation node, achieving a response time within seconds.

[0163] When handling cross-entity related data (such as the association between order data and inventory data), the data quality monitoring module supports time-window-based delayed consistency verification. For example, the "inventory quantity" attribute of an order data needs to be consistent with the data in the inventory system within the same time window (such as 10 minutes). If a temporary conflict occurs due to data synchronization delay, the system can set a buffer time (such as 30 minutes) and automatically retry the verification within the buffer period to avoid misjudging data conflicts due to network latency.

[0164] The conflict resolution rules of the data consistency model support custom configuration to adapt to diverse business logic. For example:

[0165] Exact match rule: Requires attribute values ​​to be exactly the same (such as ID card number, device unique identifier), applicable to primary key attributes.

[0166] Fuzzy matching rules: Allow for certain differences in attribute values ​​(such as simplified and traditional Chinese characters for names, or different expressions of addresses). The matching degree is calculated using similarity algorithms (such as edit distance and cosine similarity). When the matching degree is lower than a preset threshold (such as 80%), it is judged as a conflict.

[0167] Business rule matching: Consistency is determined based on industry standards or enterprise-defined rules (such as date format must be YYYY-MM-DD, and values ​​must be within a specified range). For example, the "age" attribute value must be greater than 0 and less than 150, otherwise it is considered a conflict.

[0168] In addition, the data quality monitoring module supports historical data comparison and analysis. By tracking the time-series changes of the same entity's attribute values, it identifies gradual conflicts (such as inconsistencies in historical data caused by slow changes in equipment models). The system traces the source of attribute values ​​through data dependency paths, compares the differences in values ​​at different points in time, and generates a conflict evolution trend report, providing a basis for data governance decisions.

[0169] The data quality early warning module works in conjunction with the result output module to push early warning information to the target application system via API. For example, when there is a consistency conflict between order data and logistics data on an e-commerce platform, the early warning information can trigger the order system to automatically mark the abnormal order, the logistics system to generate an abnormal work order, and the customer service system to simultaneously send a notification to the user, forming a cross-system collaborative processing flow.

[0170] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0171] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-protocol scalable data acquisition system architecture with a unified interface standard, characterized in that, include: The data source management module stores different data source configuration information in the data source knowledge base according to the data source type, and assigns a unique protocol identifier to each data source; The protocol adaptation module includes a protocol parsing unit and a protocol matching unit. After receiving external data, the protocol parsing unit calls the corresponding protocol parsing library according to the protocol identifier to convert the raw data into a standard intermediate format. The protocol matching unit verifies the protocol compatibility and confirms the data access after verification. The data conversion module includes a format conversion unit and a semantic mapping unit. The format conversion unit converts intermediate format data into a target data structure using preset rules, and the semantic mapping unit extracts data semantic tags based on the metadata model. The data verification module, based on the data semantic tags from the data transformation module, calculates the data integrity coefficient using a data integrity algorithm. The anomaly detection module determines the data integrity status based on the data integrity coefficient of the data verification module and triggers early warning rules for abnormal results. The data quality monitoring module, based on the data semantic tags of the data transformation module, calculates the data consistency coefficient through the data consistency model. The data quality early warning module determines the data quality status based on the data consistency coefficient of the data quality monitoring module and triggers early warning rules for abnormal results. In the protocol adaptation module, the protocol parsing unit calls the corresponding protocol parsing library according to the protocol identifier to convert the raw data into a standard intermediate format. The system performs protocol compatibility verification on the data source and outputs the corresponding protocol configuration in the data source knowledge base. The protocol matching unit verifies whether the data format conforms to the target specification and whether it supports dynamic expansion. After verification, the data access is confirmed. In the data verification module, based on the data semantic tags transmitted by the data conversion module, the data integrity coefficient is calculated using a data integrity algorithm, and the specific content of the data integrity coefficient is as follows: Step S01: Use a hash digest algorithm to generate a validation baseline value for the field set of the target data structure, and calculate the number of missing fields based on the validation baseline value. ; Step S02: Calculate the actual data field missing rate. The calculation formula is as follows: in Indicates the field missing rate. Indicates the number of missing fields. Indicates the total number of fields; Step S03: Calculate the data integrity coefficient. The calculation formula is as follows: in This represents the data integrity coefficient; In the data quality monitoring module, based on the semantic tags transmitted by the data conversion module, the data consistency coefficient is calculated using the data consistency model, and the specific details are as follows: Step S01: Calculate the entity attribute conflict rate. The calculation formula is as follows: in Indicates the attribute conflict rate. Indicates the number of conflicting attributes. Indicates the total number of associated attributes; Step S02: Calculate the data consistency coefficient. The calculation formula is as follows: in This represents the data consistency coefficient.

2. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: In the data source management module, different data source configuration information is stored in the data source knowledge base according to the data source type. The configuration information includes the communication protocol type and data acquisition frequency.

3. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: In the data conversion module, the format conversion unit uses preset rules to convert intermediate format data into target data structure, including field mapping rules, data type conversion rules, and timestamp alignment rules. The semantic mapping unit extracts data semantic tags based on the metadata model, including entity identifiers, attribute relationship graphs, and data dependency paths.

4. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: The anomaly detection module receives the data integrity coefficient transmitted by the data verification module. and the data integrity coefficient Compared with the preset integrity threshold Compare the data to determine its integrity status. If the data integrity coefficient is... Less than the preset integrity threshold If the data integrity coefficient is low, a data missing warning will be triggered. Greater than or equal to the preset integrity threshold If the data is complete, then the data is deemed intact.

5. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that: The data quality early warning module receives the data consistency coefficient transmitted by the data quality monitoring module. and the data consistency coefficient Consistency threshold with preset Compare the data to determine its quality; if the data consistency coefficient is... Less than the preset consistency threshold If the data consistency coefficient is low, a data conflict warning will be triggered. Greater than or equal to the preset consistency threshold If the data matches, then the data is considered consistent.

6. The multi-protocol scalable data acquisition system architecture with a unified interface standard according to claim 1, characterized in that, The architecture also includes: The result output module is used to receive warning information transmitted by the anomaly detection module and the data quality warning module, and transmit the warning information to the target application system through the API interface.

7. A multi-protocol scalable data acquisition method with a unified interface standard, applied to the multi-protocol scalable data acquisition system architecture with a unified interface standard as described in any one of claims 1-6, characterized in that: Includes the following steps: Step S1: Store the configuration information of different data sources in the data source knowledge base according to the data source type, and assign a unique protocol identifier to each data source; Step S2: Call the corresponding protocol parsing library according to the protocol identifier to verify protocol compatibility. After verification, confirm data access. Step S3: Convert the intermediate format data into the target data structure using preset rules, and extract data semantic tags based on the metadata model; Step S4: Based on the semantic tags of the data transmitted by the data conversion module, calculate the data integrity coefficient using a data integrity algorithm; Step S5: Determine the data integrity status based on the data integrity coefficient and trigger early warning rules for abnormal results; Step S6: Based on the semantic tags of the data transmitted by the data conversion module, calculate the data consistency coefficient using the data consistency model; Step S7: Determine the data quality status based on the data consistency coefficient and trigger early warning rules for abnormal results.

Images