Distributed data real-time acquisition and management system

Abstract

The invention relates to the technical field of big data processing, and discloses a distributed data real-time acquisition and management system which comprises a data processing center and a plurality of distributed edge acquisition nodes, the edge acquisition nodes are used for receiving heterogeneous source end original data streams, and protocol adaptation modules, semantic value weight modules and differential flow control modules are integrated in the edge acquisition nodes. The protocol adaptation module constructs a protocol fingerprint by using the Shannon information entropy and byte frequency distribution of the preamble segment to realize adaptive matching and decoding of a heterogeneous protocol; the semantic value weight module distributes a dynamic value weight label for the data record based on a business rule; and the differential flow control module calculates a congestion control coefficient by using a PID algorithm, and differentially adjusts the token generation rate of each level of queue. According to the invention, intelligent access of heterogeneous data sources is realized, and real-time transmission of high-priority data can be guaranteed preferentially when the network is congested.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a distributed real-time data acquisition and management system. Background Technology

[0002] With the rapid development of the Industrial Internet, the Internet of Things, and Fintech, data acquisition systems face the challenge of accessing massive heterogeneous data sources. Existing data acquisition technologies mainly suffer from two problems: First, in terms of protocol parsing, traditional solutions often rely on manually pre-configured parsing rules or hard-coded adapters. When faced with complex and non-standard proprietary protocols from industrial field devices (such as various PLC and sensor data), they lack adaptability. Once the protocol format changes or unknown devices are connected, system downtime for development is required, resulting in high maintenance costs and low efficiency. Second, in terms of data transmission and flow control, existing acquisition systems typically use global thresholds or simple FIFO (First-In, First-Out) queues for congestion control. This coarse-grained flow control mechanism lacks the ability to perceive the "business value" of the data content. When the system faces millions of concurrent connections or network bandwidth fluctuations, high-value alarm and transaction data are often blocked or even discarded by massive amounts of low-value log data, failing to meet the millisecond-level response and zero-loss reliability requirements of critical business operations.

[0003] Therefore, there is an urgent need for a data acquisition and management system that can automatically identify heterogeneous protocols and perform refined dynamic flow control based on data value. Summary of the Invention

[0004] This invention provides a distributed real-time data acquisition and management system. It utilizes protocol fingerprinting technology based on information entropy to achieve adaptive and efficient access to heterogeneous data sources. Furthermore, it ensures low latency and high reliability transmission of critical business data in high-concurrency congestion scenarios through a differential flow control mechanism that integrates PID algorithm and semantic value weight.

[0005] This invention provides a distributed real-time data acquisition and management system, including a data processing center and several distributed edge acquisition nodes that are communicatively connected to the data processing center via a dynamic transmission network;

[0006] The data processing center is used to receive and store the data collected by the distributed edge acquisition nodes. The distributed edge acquisition nodes are used to receive the raw data streams generated by heterogeneous source terminals and to standardize the raw data streams. The heterogeneous source terminals include industrial field equipment, IoT sensors, or financial transaction terminals.

[0007] The distributed edge acquisition node is internally equipped with:

[0008] The protocol adaptation module is used to receive the heterogeneous source raw data stream, extract the preamble fragment of the data packets constituting the heterogeneous source raw data stream, calculate the Shannon information entropy and byte frequency distribution of the preamble fragment to construct a protocol fingerprint feature vector, match the protocol fingerprint feature vector with a preset fingerprint feature library, dynamically load the corresponding decoding strategy according to the matching result, and convert the data packets in the heterogeneous source raw data stream into standardized data containing several data records.

[0009] The semantic value weighting module, connected to the protocol adaptation module, is used to receive the standardized data, identify the content attributes of each data record based on preset business importance rules, and assign a corresponding dynamic value weighting label to each data record.

[0010] The differential flow control module, connected to the semantic value weight module, establishes a transmission channel with the data processing center through the dynamic transmission network. It maintains multi-level transmission queues corresponding to different dynamic value weight tags. The differential flow control module monitors the transmit buffer occupancy rate and network processing latency of the distributed edge acquisition nodes in real time to generate congestion feedback signals. It calculates congestion control coefficients using a PID control algorithm and, based on the pre-stored mapping relationship between the congestion control coefficients and the dynamic value weight tags, differentially adjusts the token generation rate of each level of the multi-level transmission queues to ensure that data records with high-priority dynamic value weight tags are transmitted first.

[0011] Furthermore, in the protocol adaptation module, a preamble segment of the data packets constituting the original data stream from the heterogeneous source end is extracted, and the Shannon information entropy and byte frequency distribution of the preamble segment are calculated to construct a protocol fingerprint feature vector. Specific steps include:

[0012] (1) Define a preset sampling window length L, monitor the original data stream from the heterogeneous source end arriving at the protocol adaptation module in real time, and when a data packet is detected to enter the receiving buffer, read the continuous binary byte sequence starting from the header address of the data packet; if the length of the continuous binary byte sequence is greater than or equal to the sampling window length L, then truncate the first L bytes as the preamble segment.

[0013] (2) Establish a byte frequency statistics array of length 256, traverse the extracted preamble, count the number of times each byte value from 0 to 255 appears in the preamble, and generate a byte frequency distribution histogram.

[0014] (3) Divide the occurrence count of each byte value in the byte frequency distribution histogram by the sampling window length L to calculate the probability of occurrence of each byte value i. Using the formula Calculate the Shannon information entropy value H of the leader fragment;

[0015] (4) Based on the byte frequency distribution histogram, calculate the cumulative percentage of visible ASCII character bytes in the preamble segment. and the cumulative percentage of zero-value bytes And calculate the statistical variance of all byte values. To represent the degree of dispersion in the data distribution;

[0016] (5) The calculated Shannon information entropy value H and the statistical variance The cumulative percentage of visible ASCII character bytes and the cumulative percentage of the zero-value bytes. The protocol fingerprint feature vector is generated by normalizing and combining the features in a preset order. .

[0017] Furthermore, in the protocol adaptation module, the protocol fingerprint feature vector is matched with a preset fingerprint feature library, and the corresponding decoding strategy is dynamically loaded according to the matching result to convert the data packets in the heterogeneous source end's original data stream into standardized data containing several data records. Specific steps include:

[0018] (1) Using the protocol fingerprint feature vector V as the query vector, traverse the reference vector set in the preset fingerprint feature database. The similarity score between the query vector and each reference vector is calculated using the cosine similarity formula.

[0019] (2) Select the highest similarity score and its corresponding target reference vector, and Matching threshold with preset Compare;

[0020] (3) If If the match is successful, the corresponding decoding dynamic link library or script plugin is called from the local plugin library according to the index ID of the target reference vector. The decoding dynamic link library or script plugin parses the payload content of the data packet according to the predefined field mapping rules and reassembles it into the standardized data in key-value pair format.

[0021] (4) If If the match fails, a self-learning update mechanism is triggered, marking the preamble fragment and its associated data packet as an unknown protocol sample and uploading it to the data processing center. The data processing center uses a clustering algorithm to perform structural analysis on the unknown protocol sample to generate a new decoding strategy and a new reference vector, and then distributes the new decoding strategy and the new reference vector to the protocol adaptation module through the dynamic transmission network to update the local plugin library and the preset fingerprint feature library.

[0022] Furthermore, the specific implementation steps of the semantic value weighting module include:

[0023] (1) Parse the key-value pair structure of each data record and extract the values ​​corresponding to the preset key fields; wherein, the preset key fields include log text field, numerical measurement field and source address identifier field;

[0024] (2) Match the values ​​corresponding to the preset key fields with the multiple rule items contained in the preset business importance rules one by one;

[0025] (3) When a rule item is successfully matched, the weight level corresponding to the rule item is obtained and assigned to the data record as the dynamic value weight label; if no match is found after traversing all rule items, a default low priority weight label is assigned to the data record.

[0026] Furthermore, the preset business importance rule includes the following rule items:

[0027] Keyword matching rule item: Determine whether the log text field contains the keywords Error, Alarm, or Critical. If it does, it has higher priority.

[0028] Numerical threshold determination rule item: Determine whether the value of the numerical measurement field exceeds the preset safety alarm threshold. If it exceeds the threshold, it corresponds to high priority.

[0029] Source address whitelist rule item: Determine whether the source address identifier field belongs to the preset core device IP list. If it does, it has the highest priority.

[0030] Furthermore, the specific implementation steps of the differential flow control module include:

[0031] (1) Set a target idle rate threshold for the sending buffer, collect the current idle rate of the sending buffer in real time, calculate the difference between the target idle rate threshold and the current idle rate, and use the difference as a real-time congestion feedback signal. ;

[0032] (2) The real-time congestion feedback signal Input the PID control model to calculate the congestion control coefficients reflecting the current congestion level using the PID control algorithm. ;

[0033] (3) Based on the mapping relationship between the pre-stored congestion control coefficient and the dynamic value weight label, obtain the weight decay factor corresponding to each level of the transmission queue in the multi-level transmission queue; wherein, the weight decay factor is negatively correlated with the dynamic value weight label, that is, the higher the priority of the dynamic value weight label, the smaller the corresponding weight decay factor.

[0034] (4) Using the congestion control coefficient The weight decay factors corresponding to each level of the transmission queue are dynamically calculated and updated to determine the token generation rate of each level of the transmission queue at the current moment; wherein, when the congestion level increases, leading to... When the priority level increases, the token generation rate of the low-priority queue decreases more significantly than that of the high-priority queue.

[0035] Furthermore, the congestion control coefficient, reflecting the current level of congestion, is calculated using a PID control algorithm. The calculation formula is:

[0036]

[0037] in, This is the proportional gain coefficient, used in response to the real-time congestion feedback signal. The current amplitude; the corresponding integral term This represents the cumulative error value of the real-time congestion feedback signal from the initial time 0 to the current time t, where τ is the time variable during the integration process. The value of the real-time congestion feedback signal corresponding to time τ; The differential gain coefficient is the corresponding differential term. The congestion control coefficient is calculated by representing the rate of change of the real-time congestion feedback signal over time. It is normalized and restricted to the numerical range of [0, 1].

[0038] Furthermore, the token generation rate of each level of the transmission queue is dynamically calculated and updated at the current moment, using the following formula:

[0039]

[0040] in, Let be the token generation rate of the k-th level transmission queue at time t; The preset maximum transmission rate for this queue; The weight attenuation factor corresponding to the k-th level transmission queue; the weight attenuation factor The value range is [0, 1], and for the transmission queue with the highest priority dynamic value weight label, its corresponding Set to a minimum value of 0 or close to 0 to ensure that even in As the number of tokens increases, the token generation rate of the queue remains at the baseline level.

[0041] Furthermore, the data processing center adopts a tiered storage architecture for hot and cold data, including an in-memory database cluster and a distributed file system. For data records with high-priority dynamic value weight tags, they are preferentially written to the in-memory database cluster for real-time analysis. For data records with low-priority dynamic value weight tags, they are batch-written to the distributed file system for persistent archiving.

[0042] Furthermore, the distributed edge acquisition node adopts a dual-buffer mechanism, including a circular receiving buffer for temporarily storing the raw data streams from heterogeneous source ends and a sending buffer for temporarily storing standardized data.

[0043] The beneficial effects of this invention are as follows:

[0044] This invention significantly improves the intelligence and reliability of data acquisition by embedding a protocol adaptation module and a differential flow control module in distributed edge acquisition nodes. It utilizes Shannon information entropy and byte frequency distribution to construct protocol fingerprints, combined with cosine similarity matching and a self-learning mechanism, to achieve zero-configuration adaptive parsing of unknown heterogeneous protocols, greatly reducing the access threshold and operation and maintenance costs of multi-source heterogeneous devices. It introduces business semantic value into the flow control mechanism, using a PID algorithm to calculate congestion control coefficients in real time, and dynamically adjusting the token generation rate using a decay factor negatively correlated with value weight, thus constructing a closed-loop differential flow control system. This breaks away from the drawbacks of traditional one-size-fits-all flow control, ensuring that high-priority critical business data (such as alarms and transaction information) still receive sufficient bandwidth resources under extreme conditions of network congestion or high load, achieving optimal system throughput and timeliness of critical services. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the distributed real-time data acquisition and management system of the present invention.

[0046] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0047] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0048] like Figure 1As shown, the present invention provides a distributed real-time data acquisition and management system, including a data processing center and several distributed edge acquisition nodes. Each distributed edge acquisition node establishes a bidirectional communication connection with the data processing center through a dynamic transmission network (such as a message middleware network based on MQTT, AMQP protocol or a 5G slicing network).

[0049] The distributed edge acquisition node is used to receive the raw data stream generated by the heterogeneous source device; the heterogeneous source device serves as the system's data input source and is physically connected to the distributed edge acquisition node. The heterogeneous source device refers to various terminal devices that generate the raw data, specifically covering the following three typical scenarios:

[0050] ① Industrial field equipment: such as PLC controllers, CNC machine tools, industrial robots, etc.;

[0051] ② IoT sensors: such as temperature and humidity sensors, vibration sensors, pressure transmitters, etc. in environmental monitoring;

[0052] ③ Financial transaction terminals: such as POS machines, ATM transaction log servers, high-frequency trading gateways, etc.

[0053] The distributed edge acquisition nodes are responsible for receiving the raw data streams generated by the aforementioned heterogeneous source terminals, performing standardization processing and transmission control at the edge, and finally sending the processed data to the data processing center. The data processing center receives and stores the data collected by the distributed edge acquisition nodes. To balance performance and cost, the data processing center adopts a tiered cold and hot data storage architecture, specifically including:

[0054] In-memory database cluster (hot data layer): Deploy such as Redis Cluster or Apache Ignite to prioritize writing and storing data records with high-priority dynamic value weight labels for millisecond-level queries by the real-time analytics engine;

[0055] Distributed file system (cold data layer): Deployed such as HDFS or S3 object storage for batch writing and storing data records with low-priority dynamic value weight tags for persistent archiving.

[0056] In one embodiment, the distributed edge acquisition node is the core execution unit of the system. In terms of hardware resource allocation, this node employs a dual-buffer mechanism, including:

[0057] Ring Buffer: Used to temporarily store raw data streams written at high speed from heterogeneous sources, reducing CPU interrupts by using zero-copy technology;

[0058] Send buffer: Used to temporarily store standardized data that has been parsed and encapsulated, waiting to be sent to the dynamic transmission network.

[0059] The distributed edge acquisition node integrates and is equipped with a protocol adaptation module, a semantic value weight module, and a differential flow control module. The detailed implementation methods of these three modules are explained below.

[0060] 1. Protocol Adaptation Module

[0061] This module receives the raw data stream from the heterogeneous source, extracts the preamble fragments of the data packets constituting the raw data stream, calculates the Shannon information entropy and byte frequency distribution of the preamble fragments to construct a protocol fingerprint feature vector, matches the protocol fingerprint feature vector with a preset fingerprint feature library, dynamically loads the corresponding decoding strategy based on the matching result, and converts the data packets in the raw data stream from the heterogeneous source into standardized data containing several data records. This solves the problem of inconsistent protocols at heterogeneous source ends. More specifically, when the raw data stream from the heterogeneous source enters the circular receive buffer, the protocol adaptation module performs the following processing steps:

[0062] Step (1): Define a preset sampling window length L (in this embodiment, $L$ is set to 256 bytes), monitor the buffer in real time, and when the data packet header is detected to arrive, read the continuous binary byte sequence starting from the header start address; if the sequence length is ≥ L, then truncate the first L bytes as a preamble.

[0063] Step (2): Establish a byte frequency statistics array of length 256, traverse the extracted preamble, count the number of times each byte value from 0 to 255 appears in the preamble, and generate a byte frequency distribution histogram;

[0064] Step (3): Divide the occurrence count of each byte value in the byte frequency distribution histogram by the sampling window length L to calculate the probability of occurrence of each byte value i. Using the formula Calculate the Shannon information entropy value H of the leading fragment; the entropy value reflects the randomness of the data (e.g., compressed data has high entropy, and text data has low entropy).

[0065] Step (4): Based on the byte frequency distribution histogram, calculate the cumulative percentage of visible ASCII character (0x20-0x7E) bytes in the preamble segment. and the cumulative percentage of zero-value bytes (0x00) And calculate the statistical variance of all byte values. To represent the degree of dispersion in the data distribution;

[0066] Step (5): Calculate the Shannon information entropy value H and the statistical variance. The cumulative percentage of visible ASCII character bytes and the cumulative percentage of the zero-value bytes. The protocol fingerprint feature vector is generated by normalizing and combining the features in a preset order. .

[0067] Step (6): Using the protocol fingerprint feature vector V as the query vector, traverse the set of reference vectors in the preset fingerprint feature database. The similarity score between the query vector and each reference vector is calculated using the cosine similarity formula.

[0068] Step (7): Select the highest similarity score and its corresponding target reference vector, and Matching threshold with preset (e.g., 0.95) for comparison;

[0069] ①If If the match is successful, the corresponding decoding dynamic link library (DLL) or script plugin is called from the local plugin library according to the index ID of the target reference vector. The decoding dynamic link library or script plugin parses the payload content of the data packet according to the predefined field mapping rules and reassembles it into the standardized data in key-value pair format.

[0070] ②If If the matching fails, a self-learning update mechanism is triggered, marking the preamble fragment and its associated data packet as an unknown protocol sample and uploading it to the data processing center. The data processing center uses a clustering algorithm to perform structural analysis on the unknown protocol sample to generate a new decoding strategy and a new reference vector, and then distributes the new decoding strategy and the new reference vector to the protocol adaptation module through the dynamic transmission network to update the local plugin library and the preset fingerprint feature library, thereby achieving system evolution.

[0071] 2. Semantic Value Weighting Module

[0072] The protocol adaptation module is connected to receive the standardized data, identify the content attributes of each data record based on preset business importance rules, and assign a corresponding dynamic value weight label to each data record. Specifically, the processing steps include the following:

[0073] Step (1): Parse the key-value pair structure of each data record and extract the values ​​corresponding to the preset key fields; wherein, the preset key fields include log text field, numerical measurement field and source address identifier field;

[0074] Step (2): Match the values ​​corresponding to the preset key fields one by one with the multiple rule items contained in the preset business importance rules;

[0075] Step (3): When a rule item is successfully matched, the weight level corresponding to the rule item is obtained and assigned to the data record as the dynamic value weight label; if no match is found after traversing all rule items, a default low-priority weight label is assigned to the data record. The preset business importance rules include the following rule items:

[0076] ① Keyword matching rule item: Determine whether the log text field contains the keywords Error, Alarm or Critical. If it does, it has a higher priority (e.g., Level 1).

[0077] ② Numerical threshold determination rule item: Determine whether the value of the numerical measurement field exceeds the preset safety alarm threshold. If it exceeds the threshold, it corresponds to a higher priority (such as Level 2).

[0078] ③ Source address whitelist rule item: Determine whether the source address identifier field belongs to the preset core device IP list. If it does, it corresponds to the highest priority (Level 0).

[0079] 3. Differential Flow Control Module

[0080] The semantic value weight module is connected, and a transmission channel is established with the data processing center through the dynamic transmission network to maintain multi-level transmission queues corresponding to different dynamic value weight tags. The differential flow control module monitors the sending buffer occupancy rate and network processing latency of the distributed edge acquisition nodes in real time to generate congestion feedback signals. It uses a PID control algorithm to calculate the congestion control coefficient and, based on the pre-stored mapping relationship between the congestion control coefficient and the dynamic value weight tags, differentially adjusts the token generation rate of each level of the multi-level transmission queue to ensure that data records with high-priority dynamic value weight tags are transmitted first.

[0081] To protect critical data during network congestion, the differential flow control module performs the following differential flow control steps:

[0082] Step (1): Set the target idle rate threshold for the send buffer (e.g., The current idle rate of the sending buffer is collected in real time. Calculate the difference between the target idle rate threshold and the current idle rate, and use the difference as a real-time congestion feedback signal. The calculation formula is:

[0083]

[0084] When the buffer occupancy rate increases, the free rate decreases. A positive value that increases indicates worsening congestion.

[0085] Step (2): The real-time congestion feedback signal Input the PID control model to calculate the congestion control coefficients reflecting the current congestion level using the PID control algorithm. The calculation formula is:

[0086]

[0087] in, This is the proportional gain coefficient, used in response to the real-time congestion feedback signal. The current amplitude; the corresponding integral term This represents the cumulative error value of the real-time congestion feedback signal from the initial time 0 to the current time t, where τ is the time variable during the integration process. The value of the real-time congestion feedback signal corresponding to time τ; The differential gain coefficient is the corresponding differential term. The rate of change of the real-time congestion feedback signal over time is represented; the calculated congestion control coefficient is... It is normalized and restricted to the numerical range of [0, 1].

[0088] Step (3): Based on the pre-stored mapping relationship between congestion control coefficients and dynamic value weight labels, obtain the weight attenuation factor corresponding to each level of transmission queue in the multi-level transmission queue; wherein, the differential flow control module maintains multi-level transmission queues (such as Queue_L0, Queue_L1, Queue_Default), and each level of queue corresponds to a weight attenuation factor. This factor is negatively correlated with priority; that is, the higher the priority of the dynamic value weight label, the smaller the corresponding weight decay factor. For example:

[0089] Highest priority (Level 0) queue: Set to a minimum value of 0 or close to 0;

[0090] Low-priority (Default) queue: Set to 1.0.

[0091] Step (4): Utilize the congestion control coefficient The weight decay factor corresponding to each level of the transmission queue is dynamically calculated and updated to determine the token generation rate of each level of the transmission queue at the current moment. The calculation formula is as follows:

[0092]

[0093] in, Let be the token generation rate of the k-th level transmission queue at time t; The preset maximum transmission rate for this queue; The weight attenuation factor corresponding to the k-th level transmission queue; the weight attenuation factor The value range is [0, 1], and for the transmission queue with the highest priority dynamic value weight label, its corresponding Set to a minimum value of 0 or close to 0 to ensure that even in As the number of tokens increases, the token generation rate of the queue remains at the baseline level.

[0094] The achieved flow control effect is: when system congestion occurs... When it increases,

[0095] For low-priority queues ( Its rate It will drop significantly, even to 0, thereby actively discarding or delaying the transmission of non-critical data and freeing up bandwidth.

[0096] For high priority queues ( Its rate This means that no matter how congested the system is, the transmission rate of high-value data remains at the baseline level, thus achieving the goal of "ensuring that data records with high-priority dynamic value weight labels are transmitted first".

[0097] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0098] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A distributed real-time data acquisition and management system, characterized in that, It includes a data processing center and several distributed edge acquisition nodes that are connected to the data processing center via a dynamic transmission network. The data processing center is used to receive and store the data collected by the distributed edge acquisition nodes. The distributed edge acquisition nodes are used to receive the raw data streams generated by heterogeneous source terminals and to standardize the raw data streams. The heterogeneous source terminals include industrial field equipment, IoT sensors, or financial transaction terminals. The distributed edge acquisition node is internally equipped with: The protocol adaptation module is used to receive the heterogeneous source raw data stream, extract the preamble fragment of the data packets constituting the heterogeneous source raw data stream, calculate the Shannon information entropy and byte frequency distribution of the preamble fragment to construct a protocol fingerprint feature vector, match the protocol fingerprint feature vector with a preset fingerprint feature library, dynamically load the corresponding decoding strategy according to the matching result, and convert the data packets in the heterogeneous source raw data stream into standardized data containing several data records. The semantic value weighting module, connected to the protocol adaptation module, is used to receive the standardized data, identify the content attributes of each data record based on preset business importance rules, and assign a corresponding dynamic value weighting label to each data record. The differential flow control module, connected to the semantic value weight module, establishes a transmission channel with the data processing center through the dynamic transmission network. It maintains multi-level transmission queues corresponding to different dynamic value weight tags. The differential flow control module monitors the transmit buffer occupancy rate and network processing latency of the distributed edge acquisition nodes in real time to generate congestion feedback signals. It calculates congestion control coefficients using a PID control algorithm and, based on the pre-stored mapping relationship between the congestion control coefficients and the dynamic value weight tags, differentially adjusts the token generation rate of each level of the multi-level transmission queues to ensure that data records with high-priority dynamic value weight tags are transmitted first.

2. The distributed real-time data acquisition and management system according to claim 1, characterized in that, In the protocol adaptation module, a preamble segment of the data packets constituting the original data stream from the heterogeneous source end is extracted, and the Shannon information entropy and byte frequency distribution of the preamble segment are calculated to construct a protocol fingerprint feature vector. Specific steps include: (1) Define a preset sampling window length L, monitor the original data stream from the heterogeneous source end arriving at the protocol adaptation module in real time, and when a data packet is detected to enter the receiving buffer, read the continuous binary byte sequence starting from the header address of the data packet; if the length of the continuous binary byte sequence is greater than or equal to the sampling window length L, then truncate the first L bytes as the preamble segment. (2) Establish a byte frequency statistics array of length 256, traverse the extracted preamble, count the number of times each byte value from 0 to 255 appears in the preamble, and generate a byte frequency distribution histogram. (3) Divide the occurrence count of each byte value in the byte frequency distribution histogram by the sampling window length L to calculate the probability of occurrence of each byte value i. Using the formula Calculate the Shannon information entropy value H of the leader fragment; (4) Based on the byte frequency distribution histogram, calculate the cumulative percentage of visible ASCII character bytes in the preamble segment. and the cumulative percentage of zero-value bytes And calculate the statistical variance of all byte values. To represent the degree of dispersion in the data distribution; (5) The calculated Shannon information entropy value H and the statistical variance The cumulative percentage of visible ASCII character bytes and the cumulative percentage of the zero-value bytes. The protocol fingerprint feature vector is generated by normalizing and combining the features in a preset order. .

3. The distributed real-time data acquisition and management system according to claim 2, characterized in that, In the protocol adaptation module, the protocol fingerprint feature vector is matched with a preset fingerprint feature library, and the corresponding decoding strategy is dynamically loaded according to the matching result to convert the data packets in the heterogeneous source raw data stream into standardized data containing several data records. The specific steps include: (1) Using the protocol fingerprint feature vector V as the query vector, traverse the reference vector set in the preset fingerprint feature database. The similarity score between the query vector and each reference vector is calculated using the cosine similarity formula. (2) Select the highest similarity score and its corresponding target reference vector, and Matching threshold with preset Compare; (3) If If the match is successful, the corresponding decoding dynamic link library or script plugin is called from the local plugin library according to the index ID of the target reference vector. The decoding dynamic link library or script plugin parses the payload content of the data packet according to the predefined field mapping rules and reassembles it into the standardized data in key-value pair format. (4) If If the match fails, a self-learning update mechanism is triggered, marking the preamble fragment and its associated data packet as an unknown protocol sample and uploading it to the data processing center. The data processing center uses a clustering algorithm to perform structural analysis on the unknown protocol sample to generate a new decoding strategy and a new reference vector, and then distributes the new decoding strategy and the new reference vector to the protocol adaptation module through the dynamic transmission network to update the local plugin library and the preset fingerprint feature library.

4. The distributed real-time data acquisition and management system according to claim 3, characterized in that, The specific implementation steps of the semantic value weighting module include: (1) Parse the key-value pair structure of each data record and extract the values ​​corresponding to the preset key fields; wherein, the preset key fields include log text field, numerical measurement field and source address identifier field; (2) Match the values ​​corresponding to the preset key fields with the multiple rule items contained in the preset business importance rules one by one; (3) When a rule item is successfully matched, the weight level corresponding to the rule item is obtained and assigned to the data record as the dynamic value weight label; if no match is found after traversing all rule items, a default low priority weight label is assigned to the data record.

5. The distributed real-time data acquisition and management system according to claim 4, characterized in that, The preset business importance rules include the following rule items: Keyword matching rule item: Determine whether the log text field contains the keywords Error, Alarm, or Critical. If it does, it has higher priority. Numerical threshold determination rule item: Determine whether the value of the numerical measurement field exceeds the preset safety alarm threshold. If it exceeds the threshold, it corresponds to high priority. Source address whitelist rule item: Determine whether the source address identifier field belongs to the preset core device IP list. If it does, it has the highest priority.

6. The distributed real-time data acquisition and management system according to claim 1, characterized in that, The specific implementation steps of the differential flow control module include: (1) Set a target idle rate threshold for the sending buffer, collect the current idle rate of the sending buffer in real time, calculate the difference between the target idle rate threshold and the current idle rate, and use the difference as a real-time congestion feedback signal. ; (2) The real-time congestion feedback signal Input the PID control model to calculate the congestion control coefficients reflecting the current congestion level using the PID control algorithm. ; (3) Based on the mapping relationship between the pre-stored congestion control coefficient and the dynamic value weight label, obtain the weight decay factor corresponding to each level of the transmission queue in the multi-level transmission queue; wherein, the weight decay factor is negatively correlated with the dynamic value weight label, that is, the higher the priority of the dynamic value weight label, the smaller the corresponding weight decay factor. (4) Using the congestion control coefficient The weight decay factors corresponding to each level of the transmission queue are dynamically calculated and updated to determine the token generation rate of each level of the transmission queue at the current moment; wherein, when the congestion level increases, leading to... When the priority level increases, the token generation rate of the low-priority queue decreases more significantly than that of the high-priority queue.

7. The distributed real-time data acquisition and management system according to claim 6, characterized in that, The congestion control coefficient, reflecting the current level of congestion, is calculated using a PID control algorithm. The calculation formula is: in, This is the proportional gain coefficient, used in response to the real-time congestion feedback signal. The current amplitude; The calculated congestion control coefficient It is normalized and restricted to the numerical range of [0, 1].

8. The distributed real-time data acquisition and management system according to claim 7, characterized in that, The token generation rate of each level of the transmission queue is dynamically calculated and updated at the current moment. The calculation formula is as follows: in, Let be the token generation rate of the k-th level transmission queue at time t; The preset maximum transmission rate for this queue; The weight attenuation factor corresponding to the k-th level transmission queue; the weight attenuation factor The value range is [0, 1], and for the transmission queue with the highest priority dynamic value weight label, its corresponding Set to a minimum value of 0 or close to 0 to ensure that even in As the number of tokens increases, the token generation rate of the queue remains at the baseline level.

9. The distributed real-time data acquisition and management system according to claim 1, characterized in that, The data processing center adopts a tiered storage architecture for hot and cold data, including an in-memory database cluster and a distributed file system. For data records with high-priority dynamic value weight tags, they are written to the in-memory database cluster first for real-time analysis. For data records with low-priority dynamic value weight tags, they are written to the distributed file system in batches for persistent archiving.

10. The distributed real-time data acquisition and management system according to claim 1, characterized in that, The distributed edge acquisition node adopts a dual-buffer mechanism, including a circular receiving buffer for temporarily storing the raw data streams from heterogeneous source ends and a sending buffer for temporarily storing standardized data.

Images