Extensible real-time data transmission structure

Abstract

The invention relates to the technical field of data transmission, and discloses an extensible real-time data transmission structure. The data acquisition module is used for collecting real-time data from equipment; the data security module is used for encrypting the transmission data to ensure the security and privacy of the data in the transmission process; the data transmission module is used for transmitting the collected data through a specified channel; and the data processing module is used for receiving and analyzing the transmitted data to realize real-time decision making. And the confidentiality and the tamper-proof capability of the data in the transmission process are ensured through a multi-level security mechanism of the data security module. The prior art only depends on symmetric encryption simple Hash verification, and attack and data leakage are difficult to defend in an open network environment. Through the combination of end-to-end encryption and key dynamic negotiation, data transmission has higher security, and the problems that in a traditional scheme, data is prone to being stolen, and potential safety hazards of malicious tampering cannot be effectively resisted are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and particularly to an extensible real-time data transmission structure. Background Art

[0002] Real-time data transmission is the process of transmitting data from a source system (such as a database, an application program, a sensor, etc.) to a target system in real time. The data can be transmitted immediately when it is generated for real-time business decision-making and operations. The real-time data transmission structure is a key component of this process, which allows the data to be transmitted and processed immediately after it is generated, and is crucial for application scenarios that require immediate response.

[0003] Looking at the existing security protection measures again. Traditional encryption and authentication methods are mostly based on symmetric encryption and hash algorithms. Although they have certain security, there are still great security risks when facing man-in-the-middle attacks and replay attacks. Especially in a complex open network environment, the data may be tampered with during the transmission process or stolen by criminals. The existing technologies generally lack an effective key management mechanism, and combined with simple hash verification, it is easy to be bypassed, and the security cannot be effectively guaranteed.

[0004] In the prior art, many network transmission schemes rely on fixed transmission paths. This approach often cannot adapt to the dynamically changing network conditions when the network conditions fluctuate greatly. For example, in the case of network congestion or high latency, the transmission path remains unchanged, resulting in an increase in the packet loss rate and latency. Moreover, the bandwidth utilization rate of the fixed path scheme is low and cannot be adaptively adjusted in different network environments.

[0005] In many systems, data cleaning is usually performed at the terminal or the server side. This process not only increases the transmission burden, but also results in too much invalid data, occupying bandwidth. There are often many outliers in the data, leading to low efficiency in subsequent calculations and processing. Especially when no appropriate preprocessing is performed in the data collection link, the quality of the transmitted raw data often cannot meet the requirements of subsequent analysis, affecting the overall system performance and data accuracy. Summary of the Invention

[0006] Aiming at the deficiencies of the prior art, the present invention provides an extensible real-time data transmission structure, which solves the problems of low efficiency caused by fixed path transmission, low efficiency of error code recovery mechanism, insufficient data security and data cleaning ability in the existing data transmission system.

[0007] To achieve the above object, the present invention is realized through the following technical solutions: An extensible real-time data transmission structure, including;

[0008] A data collection module, used to collect real-time data from devices;

[0009] The data security module is used to encrypt the transmitted data to ensure the security and privacy of the data during transmission;

[0010] The data transmission module is used to transmit the collected data through a specified channel;

[0011] The data processing module is used to receive and analyze the transmitted data to achieve real-time decision-making;

[0012] The integrity verification module is used to verify whether the transmitted data is complete and valid.

[0013] Preferably, the data acquisition module includes;

[0014] The data source interface unit is used to collect various types of data from devices in real time;

[0015] The data parsing unit is used to parse data in different formats and perform structured processing;

[0016] The data preprocessing unit is used to denoise, format the collected data, standardize the storage and transmission of data. The data storage structure includes data field definition, data entry definition, data set block, and data file. The data is defined and transmitted according to the BLOCK structure.

[0017] Preferably, the data security module includes;

[0018] The encryption unit is used to perform multiple encryptions on the transmitted data to avoid data leakage during data transmission;

[0019] The access control mechanism is used to manage users' access rights to data to ensure the security and privacy of data. The standardized definition of file formats can improve the consistency of data parsing and ensure that data parsers can efficiently access data content.

[0020] Preferably, the data transmission module includes;

[0021] The data transmission channel supports multiple communication protocols to achieve fast data transmission;

[0022] The intelligent routing unit is used to dynamically select the best data transmission path according to the current network situation.

[0023] Preferably, the data processing module includes;

[0024] The analysis unit is used to perform real-time analysis on data according to specific algorithms to generate decision support information;

[0025] The output report unit is used to generate and report the analysis results for users and the system to use.

[0026] Preferably, the integrity verification module verifies the data using a hash algorithm based on the transmitted data to ensure the integrity of the data during transmission.

[0027] Preferably, the data preprocessing unit can perform data standardization and outlier detection based on machine learning algorithms to improve data quality.

[0028] Preferably, the access control mechanism includes user authentication and role management to control access permissions to data.

[0029] Preferably, the intelligent routing unit has a real-time network monitoring function and can dynamically adjust the data transmission path according to network latency and bandwidth to achieve the best transmission performance.

[0030] Preferably, the analysis unit can generate a real-time feedback mechanism and timely feedback the analysis results to the data acquisition module to adjust the data acquisition strategy and improve the system response ability and the effectiveness of data processing.

[0031] The present invention provides a scalable real-time data transmission structure, which has the following beneficial effects:

[0032] 1. Through the multi-level security mechanism of the data security module, the present invention ensures the confidentiality and anti-tampering ability of data during transmission. The prior art only relies on symmetric encryption and simple hash verification, and it is difficult to defend against attacks and data leakage in an open network environment. Through the combination of end-to-end encryption + dynamic key negotiation, the data transmission of the present invention has higher security, and solves the problems of easy data theft and inability to effectively resist malicious tampering in traditional solutions.

[0033] 2. By adopting an intelligent routing mechanism based on network status awareness and combining adaptive traffic control and error code recovery strategies, the present invention realizes network dynamic optimization and improves the stability of data transmission. Compared with the defects of fixed-path transmission in the prior art, such as being easily affected by network congestion, high packet loss rate, and inefficient retransmission mechanism, the present invention reduces transmission latency, improves bandwidth utilization through network condition monitoring + real-time path adjustment + forward error correction technology, and makes data transmission more reliable and efficient.

[0034] 3. By adopting a technical solution that combines the data acquisition module and the data preprocessing unit, the present invention completes formatting, denoising, and outlier detection of data before entering the transmission link, ensuring data quality and consistency. Compared with the prior art that simply relies on terminals or servers for data cleaning, the present invention reduces the transmission burden of invalid data, reduces the waste of transmission bandwidth, and improves the real-time performance of data processing, solving the problems of messy transmitted data, excessive outliers, and large computational load.

[0035] 4. By adopting the technical solution of combining a machine learning model with a rule engine, the present invention achieves the technical effects of real-time analysis, anomaly detection, trend prediction of data, and dynamically adjusting the data acquisition strategy. Compared with the prior art technical solution that relies on fixed thresholds and has no intelligent analysis ability, it solves the deficiencies of system response lag, ineffective identification of abnormal data, and decision-making relying on manual adjustment, enables the system to have an adaptive optimization ability, and improves the data processing efficiency and decision-making accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the system framework diagram of the present invention;

[0037] Figure 2 It is the schematic diagram of the data acquisition module of the present invention;

[0038] Figure 3 It is the schematic diagram of the data security module of the present invention;

[0039] Figure 4 It is the schematic diagram of the data transmission module of the present invention;

[0040] Figure 5 It is the schematic diagram of the data processing module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0042] Please refer to the appended Figure 1 - appended Figure 5 , the embodiments of the present invention provide a scalable real-time data transmission structure, including;

[0043] A data acquisition module, used to collect real-time data from devices;

[0044] Specifically, the data acquisition module plays a key role in this embodiment. The main task of this module is to obtain real-time data from various data sources and perform preprocessing to ensure the accuracy and consistency of the data, providing a reliable data basis for subsequent transmission, storage, and analysis. Generally, the data acquisition module needs to have efficient data acquisition capabilities, and at the same time support multiple devices, protocols, and data formats to meet the requirements of different application scenarios.

[0045] Specifically, the data source interface unit is responsible for interacting with various data sources to achieve data collection and preliminary conversion. This unit supports multiple communication protocols to adapt to different types of data devices and provides a plug-in-based protocol adaptation mechanism, enabling users to expand or modify the data interface according to actual needs. For example, for some industrial devices using proprietary protocols, a custom adapter can be developed to make them compatible with this system.

[0046] In addition, to standardize the storage and transmission of data, this embodiment makes a standardized definition of the real-time data structure. The real-time data structure is described in a BLOCK (block) organization manner to ensure the consistency and parsability of data fields. The storage structure of data includes data field definitions, data entry definitions, data set blocks, data files, etc. The data is defined and transmitted according to the BLOCK structure, including the following parts;

[0047] Data field definition;

[0048] cid (category ID): 1 byte, offset 0x00, identifying the category of data.

[0049] sid (sequence ID): 1 byte, offset 0x01, identifying the position serial number of the data in the specified category.

[0050] length (data length): 1 byte, offset 0x02, indicating the length of the data block, 0 indicating variable length.

[0051] type (value type): 1 byte, offset 0x03, defining the type of data.

[0052] reserved: 4 bytes, offset 0x04, reserved bytes.

[0053] Data description information

[0054] name (name): 16 bytes, offset 0x08, string, defining the data item name.

[0055] unit (unit): 8 bytes, offset 0x18, string, data measurement unit.

[0056] comment (remark): 32 bytes, offset 0x20, string, providing additional explanations about the data item.

[0057] Specifically, the main function of the data parsing unit in this embodiment is to convert raw data in different formats into a unified data structure for subsequent processing. This unit supports the parsing of multiple data formats and can dynamically adjust the parsing strategy according to the characteristics of the data source.

[0058] In this embodiment, the data preprocessing unit is used to standardize, denoise, and perform anomaly detection on the parsed data to ensure that the data quality meets the transmission requirements. Specifically, the data preprocessing unit can perform operations such as mean filtering, median filtering, and moving window smoothing to eliminate high-frequency noise and improve data stability. For time series data, the exponentially weighted moving average method can be used to calculate the smoothed data value, and its calculation formula is as follows;

[0059] S t = αX t +(1 - α)S t-1 ;

[0060] Where:

[0061] S t is the current smoothed value;

[0062] X t is the current sampled data;

[0063] S t -1 is the smoothed value at the previous moment;

[0064] α is the smoothing coefficient, and its value range is 0 < α ≤ 1.

[0065] In terms of anomaly detection, the data preprocessing unit can detect using statistical methods or machine learning algorithms. Based on the three-sigma method, it is determined whether a certain data point exceeds the normal range. The specific calculation formula is as follows;

[0066] UCL = μ + 3σ, LCL = μ - 3σ;

[0067] Where:

[0068] UCL and LCL are the upper control limit and lower control limit of the data respectively;

[0069] μ is the data mean;

[0070] σ is the standard deviation.

[0071] When a certain data point exceeds the control limit, the data is considered abnormal. In another possible implementation, a neural network model based on the long short-term memory network can be used to predict whether there is an anomaly in the current data by training historical data patterns, further improving the stability and reliability of the data.

[0072] In this embodiment, the data acquisition module also considers the data time synchronization problem. Since the data may come from different devices and the timestamp accuracies are inconsistent, it will affect subsequent data analysis. Therefore, in some embodiments, the present invention uses the Network Time Protocol (NTP) or the IEEE 1588 Precision Time Protocol (PTP) for time synchronization to ensure that the timestamps of all acquired data are consistent.

[0073] The data acquisition module of the present invention ensures the real-time, stable, and efficient acquisition of data between different devices through multi-protocol adaptation, data parsing, data preprocessing, time synchronization, and caching mechanisms, providing a reliable data basis for subsequent data transmission and processing.

[0074] The data security module is used to encrypt the transmitted data to ensure the security and privacy of the data during transmission;

[0075] After the data is acquired, it will enter the transmission link, and the transmission process may face risks such as data leakage, tampering, forgery, and replay attacks. Therefore, to ensure the security of the data during transmission, a multi-level security mechanism of encryption, identity authentication, access control, and data integrity verification is required. The main task of the data security module is to provide end-to-end data protection, which not only encrypts the data during transmission but also involves multiple aspects such as identity verification, session security, and integrity verification.

[0076] During the data storage and management process, the standardized definition of the file format can improve the consistency of data parsing. In this embodiment, the file structure is stored in a block structure to ensure that the data parser can efficiently access the data content. The file format includes the following parts:

[0077] File identifier and version information

[0078] fileformatmark (file format identifier): 32 bytes, offset 0x0000, to identify the file format and verify the file type.

[0079] structversion (data structure version): 4 bytes, offset 0x0020, to specify the current file structure version number and ensure file compatibility.

[0080] dataareaoffset (data area offset): 4 bytes, offset 0x0024, to indicate the start position of the data area, enabling the parser to reposition to the data starting point.

[0081] The data security module can cooperate closely with the data acquisition module. After the data acquisition is completed, it encrypts the critical data. Specifically, the data security module includes an encryption unit and an access control unit. Among them, the encryption unit is mainly responsible for protecting the confidentiality of the data, and the access control unit is mainly used for identity authentication and permission management.

[0082] The encryption unit combines symmetric encryption and asymmetric encryption to balance security and transmission efficiency. To prevent data from being eavesdropped or tampered with, AES (Advanced Encryption Standard) can be used for encryption. Specifically, before data transmission, the data is first padded to make the data length meet the requirements of block encryption. The padding rules are as follows;

[0083] P = B - (L mod B);

[0084] Among them;

[0085] P is the number of padding bytes;

[0086] B is the encryption block size (usually 16 bytes);

[0087] L is the length of the original data.

[0088] When the data length is exactly an integer multiple of the block size, a complete block still needs to be padded to avoid data truncation errors during decryption.

[0089] The data security module can use the RSA algorithm for key exchange. Since the RSA algorithm is based on the mathematical problem of large prime factor decomposition, its security is relatively high. In some embodiments, RSA encryption uses a 2048-bit key. The public key is used to encrypt the data, and the private key is used to decrypt the data. Its mathematical representation is as follows;

[0090] C = M e mod N;

[0091] M = C d mod N;

[0092] Among them, C is the ciphertext, M is the plaintext, e and d are the exponents of the public key and the private key respectively, and N is the product of two large prime numbers. When the symmetric key needs to be encrypted during transmission, the RSA public key can be used for encryption, and the receiving party uses the private key for decryption, thus ensuring the security of key exchange.

[0093] A separate encryption mechanism cannot completely prevent data tampering. Therefore, the data security module also needs to have an integrity verification function. In this embodiment, the integrity verification uses SHA-256 (Secure Hash Algorithm) to generate a data digest. Its calculation formula is as follows;

[0094] H = SHA-256(M);

[0095] Wherein:

[0096] H is the hash value;

[0097] M is the input data.

[0098] Since SHA-256 has collision resistance, even if the input data changes slightly, its hash value will change significantly, thus effectively detecting whether the data has been tampered with.

[0099] In this embodiment, the access control unit is used to manage data access permissions and prevent unauthorized access. In some embodiments, the access control unit adopts role-based access control (RBAC), and different users or devices are assigned different access permissions. For example, in an Internet of Things system, some devices only have the permission to collect data, while the server has the permission to store and analyze data. In another possible implementation, the access control unit combines attribute-based access control (ABAC) to dynamically determine whether to allow access according to attributes such as data content, time, and geographical location. For example, a certain user can access a certain type of data during a specific time period, but cannot access it during other time periods;

[0100] To improve data security, the data transmission process needs to prevent replay attacks. In this embodiment, the data security module can prevent attackers from repeatedly sending historical data by means of a timestamp + unique session token. In some embodiments, the sender attaches the current timestamp T and the randomly generated session token Token to the data packet, and the data packet structure is as follows;

[0101] Packet = {T, Token, M, H};

[0102] Wherein:

[0103] M is the encrypted data;

[0104] H is the data hash value.

[0105] The receiving end verifies whether the timestamp is within the valid window and checks whether the session token has been used. If a duplicate or expired token is found, the data packet is rejected, thus effectively preventing replay attacks.

[0106] The data transmission module is used to transmit the collected data through a specified channel;

[0107] Specifically, in this embodiment, the data transmission module is used to achieve reliable and efficient transmission of data after the data acquisition module obtains and encrypts the data. Data transmission not only involves point-to-point data exchange, but also needs to be compatible with different communication protocols, adapt to different network environments, and perform intelligent routing selection in complex topological structures.

[0108] In this embodiment, the data transmission module includes a data transmission channel and an intelligent routing unit, which are used to transmit data based on different communication protocols and dynamically optimize the data transmission path according to the network environment.

[0109] The data transmission channel is used to transfer data between different devices or systems and supports multiple transmission protocols, such as TCP, UDP, MQTT, and AMQP. In high-reliability scenarios, a TCP-based long connection transmission mechanism is adopted and combined with data integrity verification to ensure the accurate transmission of data between different devices.

[0110] Based on data storage and transmission, data entries are defined in a BLOCK structure to ensure the consistency and scalability of data parsing:

[0111] Data entry definition;

[0112] datadefine (data entry definition): Offset 0x0080, number of occupied bytes is 64 × the number of marker field definitions. An ordered list of marker data field definitions.

[0113] Marker data body;

[0114] databody (data body): The offset is the value stored in DataAreaOffset of BLOCK1 area, and the length is the file size minus the data area offset. Store the actually collected data and write it in sequence column by column and row by row according to the marker data definition.

[0115] If data transmission needs to support message queues or event-driven architectures, the MQTT or AMQP protocol can be used. Specifically, the MQTT protocol is based on the publish / subscribe model and can be used for communication between low-power and low-bandwidth Internet of Things devices. The message structure of the MQTT protocol is as follows;

[0116] Message = {Topic, QoS, Payload};

[0117] Where:

[0118] Topic represents the message topic;

[0119] QoS is the message service quality level (0, 1, or 2)

[0120] Payload is the encrypted transmission data.

[0121] In scenarios with high real-time requirements, UDP-based low-latency data transmission can be adopted. The UDP protocol is suitable for the transmission of streaming data. In some embodiments, the structure of UDP packets is as follows;

[0122] UDP_Packet = {Source_Port, Dest_Port, Length, Checksum, Payload};

[0123] Where:

[0124] Source_Port and Dest_Port represent the source port and the destination port respectively;

[0125] Length represents the total length of the data packet;

[0126] Checksum is used for data integrity verification, and Payload is the encrypted data.

[0127] The intelligent routing unit is used to dynamically adjust the data transmission path according to the network conditions to improve the transmission efficiency and network adaptability. As an option, the intelligent routing unit can select the optimal data path based on factors such as bandwidth utilization rate, packet loss rate, latency, and network congestion status, and can use the Dijkstra shortest path algorithm to calculate the optimal transmission path. Its basic calculation method is as follows;

[0128] d(v) = min{d(u) + w(u, v)};

[0129] Where:

[0130] d(v) represents the shortest path length from node v to the source node;

[0131] w(u, v) represents the weight of the edge (u, v), that is, the network delay or bandwidth overhead;

[0132] A routing optimization algorithm based on reinforcement learning can be adopted to train the model through historical data to make the routing selection more intelligent;

[0133] During the data transmission process, problems such as network jitter and packet loss may be encountered. Therefore, a flow control and exception recovery mechanism needs to be equipped. In this embodiment, the data transmission module adopts adaptive flow control, that is, based on the acknowledgment message (ACK) feedback from the receiving end, dynamically adjusts the data sending rate. In some embodiments, data flow control can be performed based on the weighted fair queue (WFQ) algorithm. The specific calculation formula is as follows;

[0134]

[0135] Where:

[0136] Rate i represents the bandwidth allocation rate of queue i;

[0137] W i is the weight of queue i, and C is the total link bandwidth;

[0138] ∑W j Is the total weight of all queues.

[0139] In another possible implementation, the TCP congestion control algorithm can be adopted to dynamically adjust the transmission window size according to the network congestion situation.

[0140] In this embodiment, a data retransmission mechanism is adopted to ensure data integrity. Generally, if the sender does not receive the acknowledgment packet from the receiver within the set timeout period, retransmission will be triggered. In some embodiments, the Go-Back-N protocol can be used for bulk data retransmission, and the specific method is as follows;

[0141] Window s ize = min(CWND, RWND);

[0142] Where:

[0143] CWND is the congestion window size of the sender, which controls the amount of unacknowledged data that the sender can send, and is dynamically adjusted according to the network congestion situation;

[0144] RWND is the receiving window size of the receiver, which controls the amount of data that the receiver can receive, and depends on the buffer size of the receiver;

[0145] Window_size is the actual data transmission window size, which depends on the minimum value of the sender's congestion window and the receiver's receiving window.

[0146] When the acknowledgment sequence number exceeds the window range, the unacknowledged data will be retransmitted. In another possible implementation, if a data packet is lost during network transmission, forward error correction code (FEC) can be used for lost packet recovery. For example, Reed-Solomon coding is adopted, and its generating matrix G can be expressed as;

[0147] C = M·G;

[0148] Where:

[0149] M is the original data block matrix;

[0150] G is the generating matrix

[0151] C is the encoded data.

[0152] The receiver can recover some lost data blocks based on the error detection information, reduce the number of retransmissions, and improve the transmission efficiency.

[0153] The data transmission module ensures the efficient transmission of data in different network environments through multi-protocol support, intelligent routing, flow control, and exception recovery mechanisms, providing a reliable transmission guarantee for real-time data processing and analysis.

[0154] A data processing module, which is used to receive and analyze the transmitted data to achieve real-time decision-making;

[0155] Specifically, in this embodiment, the data processing module is mainly used to receive and analyze the encrypted data transmitted from the data transmission module. Generally, the task of the data processing module is not only data reception and storage, but more importantly, to extract valuable information from the transmitted data through real-time analysis and processing, so as to assist decision-making or trigger automated operations. The data processing module needs to have functions such as rapid data parsing, feature extraction, real-time analysis, and decision support. In addition, considering that the data provided by the data transmission module may have certain noise or incompleteness, the data processing module also needs to perform data cleaning, repair, and completion to ensure the accuracy and reliability of the analysis results.

[0156] In this embodiment, the data processing module includes a data analysis unit and a feedback control unit. The data analysis unit is used to perform real-time analysis and processing on the received data to generate decision support information. The feedback control unit then dynamically adjusts the data acquisition strategy, data transmission method, or triggers other response mechanisms in the system according to the analysis results.

[0157] The core function of the data analysis unit is to perform feature extraction and analysis based on real-time data. Specifically, the data analysis unit analyzes the data using machine learning algorithms according to the received data type and business requirements. For time series data, time series analysis methods such as the autoregressive integrated moving average model (ARIMA) can be used to predict future trends. The basic form of the ARIMA model is as follows;

[0158]

[0159] Where:

[0160] Y t represents the current value of the time series;

[0161] μ is the constant term, usually used to describe the mean of the sequence;

[0162] ∈ t is the white noise term, representing the random disturbance or error term;

[0163] φ i is the coefficient of the autoregressive term (AR), representing the relationship between the current value and its values at the past i time moments;

[0164] θ j is the coefficient of the moving average term (MA), representing the relationship between the current error and the errors at the past j time moments; p is the order of the autoregressive term, indicating how many past time moment observations are used to predict the current value;

[0165] q is the order of the moving average term, indicating how many past error terms are used to adjust the current prediction.

[0166] Specifically, in some embodiments, the data analysis unit extracts features and classifies the received data through a machine learning model. For a time series containing multiple sensor data, the data analysis unit can identify abnormal patterns through an SVM model and label them as potential faults or abnormal events. The basic form of the SVM model is as follows:

[0167] f(x) = sign(w T x + b);

[0168] Where:

[0169] f(x) is the classification function, which outputs the predicted value of the class;

[0170] x is the input feature vector, which contains various features for classification;

[0171] w is the weight vector of the model, indicating the importance of each feature in the classification decision

[0172] b is the bias term, which controls the position of the classification decision boundary;

[0173] w T w x + b represents the weighted sum of the input features and the weights, plus the bias term b, which determines the position of the classification decision boundary;

[0174] sign(·) is the sign function, which determines the classification result according to the sign of the weighted sum. If the result is positive, the class is 1; if it is negative, the class is -1.

[0175] By training the SVM model, the data analysis unit can identify patterns in the data and make judgments, thereby providing a basis for system decision-making.

[0176] The data analysis unit can also combine data clustering and data dimensionality reduction techniques to improve processing efficiency and analysis quality. Through clustering analysis, the system can group similar data and discover hidden patterns in the data; while dimensionality reduction techniques can project high-dimensional data into a low-dimensional space for subsequent analysis. The mathematical formula of PCA is;

[0177] Z = XW;

[0178] Where:

[0179] Z is the data matrix after dimensionality reduction;

[0180] X is the original data matrix;

[0181] W is the transformation matrix.

[0182] Through PCA, the data analysis unit can effectively reduce the amount of computation and improve the data processing speed.

[0183] The feedback control unit dynamically adjusts the data acquisition strategy and data transmission strategy according to the data analysis results. Specifically, during the data analysis process, if the data analysis unit detects an abnormality in the system, the feedback control unit will automatically adjust the data acquisition frequency, transmission frequency or trigger a system alarm according to the preset rules;

[0184] The feedback control unit can combine the feedback loop to adjust the system parameters based on the analysis results. For example, the data analysis unit can predict the probability of equipment failure through the LSTM model and feed this information back to the feedback control unit. The feedback control unit automatically adjusts the working state of the equipment or the data acquisition frequency according to the failure probability, thereby optimizing the operation efficiency of the system. Specifically, the feedback control unit adjusts the working state of the equipment through the following formula;

[0185] U t = KP t + D t ;

[0186] Where:

[0187] U t is the control signal;

[0188] K is the control gain;

[0189] P t is the current system state;

[0190] D t is the feedback information based on the data analysis results.

[0191] In this way, the feedback control unit can achieve dynamic adjustment based on the data analysis results, thus ensuring the flexibility and intelligence of the system operation.

[0192] The data processing module can not only process and analyze the transmitted data in real time, but also make dynamic adjustments according to the analysis results, enhancing the intelligent response ability and real-time processing ability of the system. This enables the entire system to still maintain an efficient and accurate response ability when processing large-scale data.

[0193] The integrity verification module is used to verify whether the transmitted data is complete and valid.

[0194] Specifically, in this embodiment, the integrity verification module is mainly used to ensure the integrity of data during transmission, prevent data loss or tampering. During data transmission, data will experience multiple hops or transfers, and there may be risks of packet loss, delay, or malicious tampering in the network. Therefore, ensuring data integrity is particularly important. The integrity verification module can ensure that the data received by the receiving party is consistent with the data sent by the sending party through data encryption and verification mechanisms, thereby providing data reliability and consistency guarantees.

[0195] In this embodiment, the integrity verification module verifies data based on hash algorithms, encryption verification, and segmented verification technologies. The hash algorithm is used to generate a verification value for the data to ensure that the data has not been tampered with during transmission; encryption verification further enhances data security; segmented verification is used to process large-scale data streams to ensure that the integrity of large data packets can still be efficiently verified.

[0196] The integrity verification module first uses the SHA-256 hash algorithm to perform integrity verification on the transmitted data. SHA-256 is a commonly used hash function that generates a 256-bit hash value and can effectively resist collision attacks, and is widely used in data verification. Specifically, the receiving end performs a SHA-256 operation on the received data to generate a verification value for the data, and compares it with the verification value transmitted by the sending end to determine whether the data has been tampered with. The calculation formula is as follows;

[0197] H = SHA-256(M);

[0198] Where:

[0199] H represents the calculated hash value (digest value) used to uniquely identify the input data;

[0200] M is the data message, that is, the input data for which the hash operation needs to be performed;

[0201] SHA-256(·) is a secure hash algorithm, and its output is a 256-bit (32-byte) hash value;

[0202] SHA-256 has collision resistance and unidirectionality, that is, the same input M will always obtain the same hash value H, but it is difficult to reverse M from H.

[0203] The integrity verification module can also use HMAC (Hash-based Message Authentication Code) technology to strengthen data integrity verification. HMAC ensures that the data is not only complete but also can verify the authenticity of its source by adding a key when calculating the hash value. Specifically, the calculation method of HMAC is as follows;

[0204] HMAC = SHA-256(K ⊕ opad || SHA-256(K ⊕ ipad || M));

[0205] Wherein:

[0206] HMAC is a hash-based message authentication code used to verify data integrity and authenticity.

[0207] K is the key, which is shared only by the two communicating parties to ensure the security of HMAC calculation.

[0208] opad is the outer padding constant.

[0209] iPad is the inner padding constant.

[0210] ⊕ represents the bitwise exclusive OR (XOR) operation.

[0211] M is the data message to be verified.

[0212] SHA-256(·) is the Secure Hash Algorithm 256-bit.

[0213] Specifically, HMAC verification not only ensures data integrity but also prevents replay attacks. Even if an attacker steals a valid data packet, they cannot replay it to the receiver without being detected.

[0214] To improve the efficiency of data integrity verification, especially during large-scale data stream transmission, the integrity verification module can adopt a segmented verification mechanism. This mechanism divides the data into multiple small chunks, calculates the hash value for each data block separately, and verifies the integrity of each data block sequentially at the receiving end. Specifically, the data transmission and verification process is as follows;

[0215] Data segmentation: Divide the large data packet into multiple data segments of a fixed size.

[0216] Data encryption and verification: Perform hash calculation and encryption on each data segment separately.

[0217] Data transmission: Transmit these data segments one by one through the data transmission channel.

[0218] Receiver verification: The receiving end performs hash verification on each received data segment and aggregates the hash values of each segment to verify the integrity of the data packet.

[0219] During this process, every time the receiving end receives a data segment, it can immediately perform integrity verification and aggregate the verification results. After the final data packet is transmitted, the receiving end can verify the integrity of all data segments. If a certain segment of data is lost or damaged, it can be immediately detected through the verification mechanism, thereby requesting retransmission and reducing the risk of data loss.

[0220] The integrity verification module uses multi-level technical means such as hash algorithms, HMAC, segmented verification, recursive verification, and serial number verification to ensure the integrity of data during transmission, preventing problems such as tampering, loss, and duplicate transmission. Through these technical means, the data reliability and security of the system can be effectively improved, providing strong guarantee for data transmission.

[0221] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An extensible real-time data transmission structure, characterized in that, including; a data acquisition module, which is used to collect real-time data from devices; a data security module, which is used to encrypt the transmitted data to ensure the security and privacy of the data during transmission; a data transmission module, which is used to transmit the collected data through a specified channel; a data processing module, which is used to receive and analyze the transmitted data to achieve real-time decision-making; an integrity verification module, which is used to verify whether the transmitted data is complete and valid.

2. The scalable real-time data transmission structure according to claim 1, characterized in that, The data acquisition module includes; a data source interface unit, which is used to collect various types of data from devices in real time; a data parsing unit, which is used to parse data in different formats and perform structured processing; a data preprocessing unit, which is used to denoise and format the collected data, standardize the storage and transmission of data. The storage structure of the data includes data field definition, data entry definition, data set block, and data file. The data is defined and transmitted according to the BLOCK structure.

3. An extensible real-time data transmission structure according to claim 1, characterized in that, The data security module includes; an encryption unit, which is used to perform multiple encryptions on the transmitted data to ensure the avoidance of data leakage during data transmission; an access control mechanism, which is used to manage the access rights of users to data to ensure the security and privacy of the data. The standardized definition of the file format can improve the consistency of data parsing and ensure that the data parser can efficiently access the data content.

4. An extensible real-time data transmission structure according to claim 1, characterized in that The data transmission module includes; a data transmission channel, which supports multiple communication protocols to achieve fast data transmission; an intelligent routing unit, which is used to dynamically select the best data transmission path according to the current network situation.

5. An extensible real-time data transmission structure according to claim 1, characterized in that, The data processing module includes; an analysis unit, which is used to perform real-time analysis on the data according to specific algorithms to generate decision support information; an output report unit, which is used to generate and report the analysis results for users and the system to use.

6. An extensible real-time data transmission structure according to claim 1, characterized in that The integrity verification module verifies the data using a hash algorithm according to the transmitted data to ensure the integrity of the data during transmission.

7. An extensible real-time data transmission structure according to claim 2, characterized in that, The data preprocessing unit can perform data standardization and outlier detection based on machine learning algorithms to improve data quality.

8. An extensible real-time data transmission structure according to claim 3, characterized in that, The access control mechanism includes user authentication and role management to control the access rights to data.

9. An extensible real-time data transmission structure according to claim 4, characterized in that The intelligent routing unit has a real-time network monitoring function and can dynamically adjust the data transmission path according to network latency and bandwidth to achieve the best transmission performance.

10. An extensible real-time data transmission structure according to claim 5, characterized in that, The analysis unit can generate a real-time feedback mechanism to timely feedback the analysis results to the data acquisition module to adjust the data acquisition strategy and improve the system response ability and the effectiveness of data processing.

Images