Data transmission system

Abstract

The invention discloses a data transmission system, and particularly relates to the technical field of data transmission, which comprises a data arrangement module for calling data, a control center for providing a computing platform and data transmission, a transmission detection unit for monitoring the data transmission and a receiving end for receiving the data. According to the invention, the network flow is monitored in real time, the data packet is captured and analyzed in time, missing data can be automatically retransmitted under the condition that the data packet is lost or delayed, reliable delivery of the data is ensured, the data packet is verified in the transmission process, the data packet is supplemented in time when the data packet is lost, and verification is carried out when data transmission is carried out. Problems can be found immediately, so that the overall transmission time is shortened, verification and retransmission can be carried out in parallel with data transmission, the delay time perceived by a user is shortened, the bandwidth is utilized to the maximum extent, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a data transmission system. Background Art

[0002] When data is transmitted, file integrity verification is usually performed at the receiving end. This is because the receiving end is responsible for receiving and processing data. The receiving end needs to confirm whether the received data is consistent with the data sent by the sending end to ensure the integrity of the data. Common error detection mechanisms (such as checksums, hash functions, etc.) are usually applied at the receiving end to check whether the data packet is damaged or lost during transmission. The receiving end performs a one-time verification after receiving the entire data file. China Patent Publication (Announcement) No.: CN209844972U, a data transmission system, comprising: an unmanned water device, comprising a data acquisition module, a transmission control module and a first hydroacoustic communication module, the data acquisition module is used to detect and collect water environment data of the water area to be tested, and the water environment data is sent to the outside through the transmission control module and the first hydroacoustic communication module in turn; an on-water relay buoy, comprising a main control module and a second hydroacoustic communication module; a near-ground buoy, comprising a data control processing module, a third hydroacoustic communication module and a first mobile communication module; a ground station, comprising a host computer and a second mobile communication module, the host computer is used to perform data analysis on the water environment data and generate detection results, and store and display the water environment data and the detection results. Compared with the related art, the data transmission system of the utility model transmits water environment data in real time to perform real-time pollution detection on the water area to be tested, and at the same time, provides pollution alarm prompts in real time.

[0003] However, in the above technical solution, the receiving end performs a one-time verification after receiving the entire data file, and cannot immediately discover problems during data transmission, such as packet loss and delay. The overall transmission time is long, and verification and retransmission cannot be performed in parallel with data transmission, which increases the user-perceived delay time, fails to maximize the use of bandwidth, and has a poor user experience. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problems that the above-mentioned receiving end performs a one-time verification after receiving the entire data file, cannot immediately discover problems during data transmission, such as packet loss, delay, long overall transmission time, verification and retransmission cannot be performed in parallel with data transmission, increase the user-perceived delay time, cannot maximize the use of bandwidth, and has poor user experience, and a data transmission system is proposed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A data transmission system, comprising:

[0007] A data sorting module for retrieving data, a control center for providing a computing platform and data transmission, a transmission detection unit for monitoring data transmission, and a receiving end for receiving data;

[0008] The data sorting module is composed of a data reading module for identifying the data stored in the database, a data selection module for checking and confirming the data, a data cleaning module for removing duplicates, a data copying module for copying data, a data unification module for converting the data format, a data encoding module for data grouping encoding, and a data compression module;

[0009] The transmission monitoring unit is composed of a data packet capture module for capturing data packets, an analysis module for checking the total number of data packets, a sequence number check module for checking the group coding of data packets, a timestamp analysis module for checking the timestamps between data packets, a determination module for obtaining the results of data transmission packet loss and damage, a threshold module for setting the standard data comparison difference, and a retransmission module for correcting the transmitted data.

[0010] The present invention is further configured as follows: the data reading module is connected to the data selection module by signal, and the data selection module is used to obtain the data file read by the data reading module and select the data file;

[0011] The data selection module is signal-connected to the data cleaning module, and the data cleaning module is used to remove duplicate, incomplete or erroneous data selected by the data module.

[0012] Through the above technical solution: the data reading module is responsible for reading data files from a specified source (such as a database, file system, etc.), and then outputs the read data files. The data selection module receives data files from the data reading module, allowing users to select specific data files for further processing. The data cleaning module cleans the data and removes duplicate data. Check and repair incomplete data, correct erroneous data, and the data cleaning module inputs the selected data files from the data selection module, and then outputs the cleaned data.

[0013] The present invention is further configured as follows: the data cleaning module is signal-connected to a data copying module, and the data copying module is used to copy the cleaned data;

[0014] The data copy module is signal-connected to the data unification module, and the data unification module is used to receive data transmitted by the data copy module and standardize the data format to make all fields, data types and codes consistent.

[0015] Through the above technical solution: the data copy module receives the cleaned data from the data cleaning module and is responsible for copying the data for subsequent processing. The data copy module is connected to the data unification module and passes the copied data to the data unification module. The data unification module receives the data from the data copy module and standardizes the data format to ensure that all fields, data types and encodings are consistent. The standardized data is ready for further analysis or storage.

[0016] The present invention is further configured as follows: the data unification module is signal-connected to the data encoding module, and the data encoding module is used to receive data transmitted by the data unification module and group and number the data;

[0017] The data encoding module is signal-connected to the data compression module, and the data compression module is used to receive data transmitted by the data encoding module and compress the data into multiple data packets according to the grouping.

[0018] Through the above technical solution: the data encoding module receives data from the data unification module, groups and numbers the data to provide identification for subsequent processing; the data compression module receives data from the data encoding module, compresses the data according to the data grouping, and converts it into multiple data packets to optimize storage and transmission efficiency.

[0019] The present invention is further configured as follows: the data packet capture module is connected to the analysis module by signal, the analysis module is used to obtain the data packets obtained by the data packet capture module in the network using the network packet capture tool, and check the total number of data packets and the number of lost data packets;

[0020] The analysis module is signal-connected to the sequence number checking module, and the sequence number checking module is used to receive data transmitted by the analysis module and check whether the group numbers of the data packets are arranged in sequence;

[0021] The sequence number checking module is connected to the time stamp analysis module by signal. The time stamp analysis module is used to receive data transmitted by the sequence number checking module, check the time stamps between data packets, and analyze the transmission delay.

[0022] Through the above technical solution: the data packet capture module uses the network packet capture tool to obtain data packets in the network, and the captured data packets are used for subsequent analysis. The analysis module receives data packets from the data packet capture module, checks the total number of data packets and the number of lost data packets, and the sequence number check module checks whether the group numbers of the data packets are arranged in sequence to ensure the correctness of the data packet sequence. The timestamp analysis module receives data from the sequence number check module and checks the timestamps between data packets to analyze the transmission delay.

[0023] The present invention is further configured as follows: the timestamp analysis module is signal-connected to the determination module, the determination module is used to receive data transmitted by the timestamp analysis module and compare the data with standard data;

[0024] The determination module is connected to the threshold module signal, and the determination module is used to call the threshold module, compare the threshold with the time difference between two data packets, the two timestamps are t1 and t2, and calculate the time difference between them:

[0025] Delta t=t2-t1

[0026] Wherein, Delta t represents the time difference;

[0027] The determination module is connected to the retransmission module and the sequence number check module by signal. The retransmission module is used to receive the result transmitted by the determination module. When the time difference is outside the threshold range, the retransmission module retrieves the data inside the sequence number check module to obtain the group number of the missing data.

[0028] The retransmission module is connected to the data encoding module by signal, and the data encoding module is used to receive the group number transmitted by the retransmission module and retransmit the data of the group number;

[0029] The determination module is signal-connected to a receiving end, and the receiving end is used to receive data transmitted by the determination module.

[0030] Through the above technical solution: the timestamp analysis module calculates the time difference between data packets and outputs the time difference data. The judgment module receives the time difference data of the timestamp analysis module, compares the time difference with the standard data, calls the threshold module to obtain the threshold, and judges whether the time difference is within the acceptable range. The formula is: Delta t=t2-t1. Delta t is connected to the threshold module, the threshold is obtained for comparison, and whether to retransmit is determined based on the comparison result. The threshold module provides a time difference threshold for judgment. The retransmission module receives the result of the judgment module. If the time difference exceeds the threshold range, the missing data packet number is obtained from the sequence number check module, and the data encoding module is called to retransmit the missing data packet.

[0031] The beneficial effects of the present invention are as follows: by cleaning the original data, duplicate, erroneous or incomplete data can be removed, ensuring that subsequent analysis is based on high-quality data; sequence number checking can ensure that data packets are received in order, which helps maintain data integrity and avoid information loss; each module focuses on a specific task, making the overall architecture clearer, easier to maintain and expand; during data processing, through replication and standardization, the burden of subsequent processing is reduced and efficiency is improved; through compression technology, the storage requirements and transmission time of data can be significantly reduced, which is particularly important in the case of limited network bandwidth; data is grouped and numbered, specific data packets can be quickly identified and located, and subsequent processing procedures can be optimized; and the time difference between data packets can be monitored in real time. , it can promptly detect network delays or transmission problems, so as to take measures to adjust or retransmit, judge the data transmission quality according to the set threshold, automate the processing flow, improve the self-healing ability of the system, monitor network traffic in real time, capture and analyze data packets in time, enhance the transparency and monitorability of the system, and automatically retransmit the missing data in the event of data packet loss or delay to ensure reliable data delivery, verify the data packets during transmission, and replenish them in time when data packets are lost, reducing the delay time perceived by users, and verify during data transmission, which can immediately detect problems and reduce the overall transmission time. Verification and retransmission can be carried out in parallel with data transmission to maximize the use of bandwidth and improve user experience.

[0032] In the present invention, the transmission module is responsible for transmitting the encoded and compressed data to the abnormality detection module. The abnormality detection module receives the compressed data, performs decoding and decompression operations, and then compares it with the threshold data to determine whether there is an abnormal signal (outside the normal temperature range). The threshold module checks the data according to the set threshold. When the data is greater than or equal to the set threshold, the threshold module transmits the data to the early warning module. After receiving the abnormal data, the early warning module issues an alarm to notify relevant personnel or systems. The second preprocessing module receives the data transmitted by the threshold module or the early warning module according to the situation. It performs secondary enhancement and noise reduction on the received data to prepare for further analysis and processing. The second feature extraction module receives the data processed by the second preprocessing module. It extracts feature data such as temperature distribution and hot spot areas from the data to provide input for the subsequent analysis and recognition modules. The analysis and recognition module receives the data extracted by the second feature extraction module, and uses support vector machines, artificial neural networks or convolutional neural networks and other technologies to perform feature classification and comprehensive analysis to identify the data that can be used. The user interface module receives the result data generated by the analysis and identification module. It is responsible for converting the data into charts and other forms, and displaying it in real time through an external display device so that users can monitor the data status and analysis results. It can realize real-time monitoring and analysis of complex data streams, and provide timely anomaly detection and early warning functions. The data received by the storage module will be connected to the time synchronization module to ensure that each data entry has an accurate time stamp, so that the time sequence of events can be accurately restored during subsequent analysis or backtracking of data. The backup module is responsible for regularly or in real time backing up the data in the storage module. The connection between the storage module and the backup module ensures the security and availability of the data. If data loss or damage occurs, the backup module can quickly restore the data. The backup module usually uploads the backup data to the cloud or other remote storage devices through a network connection to prevent single point failure of local data, ensuring the coordinated operation of data storage, time synchronization and backup functions in the system, and improving the reliability of the system and data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of a data transmission system in the present invention. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0035] Example 1

[0036] like Figure 1 As shown, a data transmission system comprises:

[0037] A data sorting module for retrieving data, a control center for providing a computing platform and data transmission, a transmission detection unit for monitoring data transmission, and a receiving end for receiving data;

[0038] The data sorting module is composed of a data reading module for identifying files saved in the database, a data selection module for checking and confirming data, a data cleaning module for removing duplicates, a data copying module for copying data, a data unification module for converting data formats, a data encoding module for data grouping encoding, and a data compression module;

[0039] The transmission monitoring unit consists of a data packet capture module for capturing data packets, an analysis module for checking the total number of data packets, a sequence number check module for checking the group coding of data packets, a timestamp analysis module for checking the timestamps between data packets, a determination module for obtaining the results of data transmission packet loss and damage, a threshold module for setting the standard data comparison difference, and a retransmission module for correcting the transmitted data.

[0040] The data reading module is connected to the data selection module by signal, and the data selection module is used to obtain the data file read by the data reading module and select the data file;

[0041] The data selection module is signal-connected to the data cleaning module, and the data cleaning module is used to remove duplicate, incomplete or erroneous data from the data selected by the data module.

[0042] The data cleaning module is connected to the data copy module by signal, and the data copy module is used to copy the cleaned data;

[0043] The data copy module is connected to the data unification module by signal. The data unification module is used to receive the data transmitted by the data copy module and standardize the data format to make all fields, data types and codes consistent.

[0044] The data unification module is connected to the data encoding module by signal, and the data encoding module is used to receive the data transmitted by the data unification module and group and number the data;

[0045] The data encoding module is signal-connected to the data compression module. The data compression module is used to receive the data transmitted by the data encoding module and compress the data into multiple data packets according to the grouping.

[0046] The data packet capture module is connected to the analysis module signal, and the analysis module is used to obtain the data packets captured by the data packet capture module in the network using the network packet capture tool, and check the total number of data packets and the number of lost data packets;

[0047] The analysis module is connected to the sequence number checking module by signal, and the sequence number checking module is used to receive the data transmitted by the analysis module and check whether the group numbers of the data packets are arranged in sequence;

[0048] The sequence number check module is connected to the time stamp analysis module signal. The time stamp analysis module is used to receive the data transmitted by the sequence number check module, check the time stamps between the data packets, and analyze the transmission delay.

[0049] The timestamp analysis module is connected to the determination module signal, and the determination module is used to receive the data transmitted by the timestamp analysis module and compare the data with the standard data;

[0050] The judgment module is connected to the threshold module signal. The judgment module is used to call the threshold module, compare the threshold with the time difference of two data packets. The two timestamps are t1 and t2, and calculate the time difference between them:

[0051] Delta t=t2-t1

[0052] Wherein, Delta t represents the time difference;

[0053] The determination module is connected to the retransmission module and the sequence number check module by signal. The retransmission module is used to receive the result transmitted by the determination module. When the time difference is outside the threshold range, the retransmission module retrieves the data inside the sequence number check module to obtain the group number of the missing data.

[0054] The retransmission module is connected to the data encoding module by signal, and the data encoding module is used to receive the group number transmitted by the retransmission module and retransmit the data of the group number;

[0055] The determination module is signal-connected to the receiving end, and the receiving end is used to receive data transmitted by the determination module.

[0056] In the above embodiment, the data reading module is responsible for reading data files from a specified source (such as a database or a file system). The module outputs the read data files to provide a basis for subsequent data processing. After receiving the data files from the data reading module, the data selection module allows the user to select specific data files for further processing. This process ensures that the user can focus on the required data set. After the data selection is completed, the data cleaning module processes the selected data files. The main tasks of this module include removing duplicate data, checking and repairing incomplete data, and correcting erroneous data. The cleaned data helps to improve data quality and facilitate subsequent analysis. The data cleaning module passes the cleaned data to the data copy module. This module is responsible for copying the cleaned data for subsequent processing. This process provides a reliable backup for data unification and subsequent analysis. The data copy module is connected to the data unification module. The data unification module receives the copied data and standardizes its format to ensure that all fields, data types and encodings are consistent. This standardization step prepares for further analysis or storage. Next, the data encoding module receives the data from the data unification module. The task of this module is to group and number the data to provide a unique identifier for subsequent processing, which is convenient for tracking and management. The data output by the data encoding module is then passed to the data compression module, which compresses the data according to the data grouping and converts it into multiple data packets to improve the efficiency of storage and transmission. The data packet capture module uses the network packet capture tool to obtain these data packets in the network. The analysis module receives the data packets from the data packet capture module and checks the total number of data packets and the number of lost data packets. The sequence number check module is responsible for checking whether the group numbers of the data packets are arranged in sequence to ensure that the order of the data packets is correct. The timestamp analysis module receives the data from the sequence number check module and checks the timestamps between the data packets. It calculates the time difference between the data packets and outputs the time difference data to analyze the transmission delay. The judgment module receives the time difference data from the timestamp analysis module and compares it with the standard data.It calls the threshold module to obtain the preset time difference threshold and determines whether the time difference is within an acceptable range. The formula is: Delta t=t2-t1, which is used to calculate the difference between two timestamps. The threshold module provides a time difference threshold for judgment so that the judgment module can make a comparison. Finally, the retransmission module receives the result of the judgment module. If the time difference exceeds the threshold range, the retransmission module will obtain the missing data packet number from the sequence number check module and call the data encoding module to resend the missing data packet. It monitors network traffic in real time, captures and analyzes data packets in a timely manner, and enhances the transparency and monitorability of the system. In the event of data packet loss or delay, it can automatically retransmit missing data to ensure reliable data delivery. It verifies data packets during transmission and supplements them in time when data packets are lost, reducing the delay time perceived by users. It verifies during data transmission and can immediately detect problems, thereby reducing the overall transmission time. Verification and retransmission can be performed in parallel with data transmission to maximize bandwidth utilization and improve user experience.

[0057] Working principle: The data reading module is responsible for reading data files from a specified source (such as a database or file system). The module outputs the read data files to provide a basis for subsequent data processing. After receiving the data files from the data reading module, the data selection module allows the user to select specific data files for further processing. This process ensures that users can focus on the required data sets. After the data selection is completed, the data cleaning module processes the selected data files. The main tasks of this module include removing duplicate data, checking and repairing incomplete data, and correcting erroneous data. The cleaned data helps to improve data quality and facilitate subsequent analysis. The data cleaning module passes the cleaned data to the data copy module. This module is responsible for copying the cleaned data for subsequent processing. This process provides a reliable backup for data unification and subsequent analysis. The data copy module is connected to the data unification module. The data unification module receives the copied data and standardizes its format to ensure that all fields, data types and encodings are consistent. This standardization step prepares for further analysis or storage. Next, the data encoding module receives the data from the data unification module. The task of this module is to group and number the data to provide a unique identifier for subsequent processing, which is convenient for tracking and management. The data output by the data encoding module is then passed to the data compression module, which compresses the data according to the data grouping and converts it into multiple data packets to improve the efficiency of storage and transmission. The data packet capture module uses the network packet capture tool to obtain these data packets in the network. The analysis module receives the data packets from the data packet capture module and checks the total number of data packets and the number of lost data packets. The sequence number check module is responsible for checking whether the group numbers of the data packets are arranged in sequence to ensure that the order of the data packets is correct. The timestamp analysis module receives the data from the sequence number check module and checks the timestamps between the data packets. It calculates the time difference between the data packets and outputs the time difference data to analyze the transmission delay. The judgment module receives the time difference data from the timestamp analysis module and compares it with the standard data. It calls the threshold module to obtain the preset time difference threshold and determines whether the time difference is within the acceptable range. The formula is: Delta t = t2-t1, which is used to calculate the difference between the two timestamps. The threshold module provides the time difference threshold for judgment so that the determination module can make comparisons. Finally, the retransmission module receives the result of the determination module. If the time difference exceeds the threshold range, the retransmission module will obtain the missing data packet number from the sequence number check module and call the data encoding module to resend the missing data packet.

[0058] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data transmission system, characterized in that: include: A data sorting module for retrieving data, a control center for providing a computing platform and data transmission, a transmission detection unit for monitoring data transmission, and a receiving end for receiving data; The data sorting module is composed of a data reading module for identifying files stored in the database, a data selection module for checking and confirming data, a data cleaning module for removing duplicate items, a data copying module for copying data, a data unification module for converting data formats, a data encoding module for data grouping encoding, and a data compression module; The transmission monitoring unit is composed of a data packet capture module for capturing data packets, an analysis module for checking the total number of data packets, a sequence number check module for checking the group coding of data packets, a timestamp analysis module for checking the timestamps between data packets, a determination module for obtaining the results of data transmission packet loss and damage, a threshold module for setting the standard data comparison difference, and a retransmission module for correcting the transmitted data.

2. A data transmission system according to claim 1, characterized in that: The data reading module is connected to the data selection module by signal, and the data selection module is used to obtain the data file read by the data reading module and select the data file; The data selection module is signal-connected to the data cleaning module, and the data cleaning module is used to remove duplicate, incomplete or erroneous data selected by the data module.

3. A data transmission system according to claim 2, characterized in that: The data cleaning module is signal-connected to the data copying module, and the data copying module is used to copy the cleaned data; The data copy module is signal-connected to the data unification module, and the data unification module is used to receive data transmitted by the data copy module and standardize the data format to make all fields, data types and codes consistent.

4. A data transmission system according to claim 1, characterized in that: The data unification module is connected to the data encoding module by signal, and the data encoding module is used to receive the data transmitted by the data unification module and group and number the data; The data encoding module is signal-connected to the data compression module, and the data compression module is used to receive data transmitted by the data encoding module and compress the data into multiple data packets according to the grouping.

5. A data transmission system according to claim 1, characterized in that: The data packet capture module is connected to the analysis module by signal, and the analysis module is used to obtain the data packets captured by the data packet capture module in the network using the network packet capture tool, and check the total number of data packets and the number of lost data packets; The analysis module is signal-connected to the sequence number checking module, and the sequence number checking module is used to receive data transmitted by the analysis module and check whether the group numbers of the data packets are arranged in sequence; The sequence number checking module is connected to the time stamp analysis module by signal. The time stamp analysis module is used to receive data transmitted by the sequence number checking module, check the time stamps between data packets, and analyze the transmission delay.

6. A data transmission system according to claim 1, characterized in that: The timestamp analysis module is connected to the determination module by signal, and the determination module is used to receive data transmitted by the timestamp analysis module and compare the data with standard data; The determination module is connected to the threshold module signal, and the determination module is used to call the threshold module, compare the threshold with the time difference between two data packets, the two timestamps are t1 and t2, and calculate the time difference between them: Delta t=t2-t1 Wherein, Delta t represents the time difference; The determination module is connected to the retransmission module and the sequence number check module by signal. The retransmission module is used to receive the result transmitted by the determination module. When the time difference is outside the threshold range, the retransmission module retrieves the data inside the sequence number check module to obtain the group number of the missing data. The retransmission module is connected to the data encoding module by signal, and the data encoding module is used to receive the group number transmitted by the retransmission module and retransmit the data of the group number; The determination module is signal-connected to a receiving end, and the receiving end is used to receive data transmitted by the determination module.

Images