Data transmission method and system based on low-delay optimization and computer readable storage medium

Abstract

The invention discloses a data transmission method based on low-delay optimization, which comprises the following steps of data preprocessing, data classification, partitioning, compression and encryption preprocessing operation, transmission scheduling, multi-path selection, priority scheduling and flow shaping function realization, network transmission, actual data packet sending and receiving processing and basic transmission control. According to the data transmission method and system based on low-delay optimization and the computer readable storage medium, through real-time network awareness and machine learning technologies, the system can dynamically adjust a transmission strategy and adapt to diversified network environments, and the data transmission efficiency is improved. And a multi-dimensional priority model and an intelligent scheduling algorithm ensure that key data obtains an optimal transmission path and resources.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data transmission method, system, and computer-readable storage medium based on low latency optimization. Background Technology

[0002] In today's digital age, data has become a core production factor, and efficient, low-latency data transmission technology is a key infrastructure supporting various real-time applications. With the rapid development of emerging technologies such as 5G, the Internet of Things, the Industrial Internet, autonomous driving, and telemedicine, the requirements for data transmission latency are becoming increasingly stringent. Traditional data transmission methods often struggle to meet the stringent latency requirements when faced with large-scale, high-concurrency real-time data transmission demands, making low-latency optimization technology a current research hotspot in the field of network communication.

[0003] Currently, mainstream data transmission technologies mainly include traditional transmission methods based on the TCP / IP protocol stack, real-time transmission protocols based on UDP, and various proprietary protocols. While TCP provides reliable transmission guarantees, its congestion control mechanisms and retransmission strategies often introduce high latency; UDP, although with lower latency, lacks reliability guarantees. In recent years, newer protocols such as QUIC and HTTP / 3 have attempted to combine the advantages of both, but further optimization is still needed in specific scenarios. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, one of the objectives of this invention is to provide a data transmission method, system and computer-readable storage medium based on low latency optimization.

[0005] One of the objectives of this invention is achieved through the following technical solution: A data transmission method based on low latency optimization, characterized by comprising the following steps: Step 1: Data preprocessing, responsible for data classification, segmentation, compression, and encryption preprocessing operations; Step 2: Transmission scheduling, implementing multi-path selection, priority scheduling, and traffic shaping functions; Step 3: Network transmission, handling the actual sending and receiving of data packets and basic transmission control; Step 4: Quality control, monitoring network status, adjusting transmission parameters, and ensuring service quality.

[0006] Furthermore, in step one, the large data block is divided into multiple sub-blocks to support parallel transmission and independent retransmission.

[0007] Furthermore, in step two, the priorities include business priority, timeliness priority, dependency priority, and resource priority. Business priority is the level of business importance specified by the application layer, timeliness priority is calculated based on the urgency of data validity, dependency priority considers the dependency relationship between data blocks, and resource priority is dynamically adjusted according to the current system resource load.

[0008] A low-latency optimized data transmission system includes: a client module, a transmission engine module, a network awareness module, a policy management module, and a security module.

[0009] Furthermore, the client module is deployed at both the data sending and receiving ends to implement the user interface and basic communication functions; the transmission engine is the core processing unit of the system, implementing various optimization algorithms; the network awareness module is used to monitor the network status in real time and provide a basis for decision-making; the policy management module is used to store and update various transmission policy parameters; and the security module is used to ensure the security and privacy of data transmission.

[0010] Furthermore, the network awareness module is responsible for collecting and analyzing network status information. Its key sub-modules include a probe engine, a passive monitor, a prediction model, and a topology discovery model. The probe engine is used to actively send probe packets to measure network parameters. The passive monitor is used to analyze the transmission characteristics of service data packets to infer network status. The prediction model predicts the trend of network parameter changes based on time series analysis. The topology discovery model is used to identify available transmission paths and their characteristics.

[0011] Furthermore, the transmission engine is the core processing unit of the system. Its main sub-modules include the scheduler, path manager, congestion controller, error processor, and buffer manager. The scheduler is used to implement data chunking and priority scheduling, the path manager is used to maintain path status and perform path selection, the congestion controller is used to implement congestion detection and rate control, the error processor is used to handle data loss and error recovery, and the buffer manager is used to optimize the use of send and receive buffers.

[0012] Furthermore, the policy management module provides a configurable policy framework, which mainly includes a QoS policy library, an optimization rule library, an adaptive learner, and a policy decision-maker. The QoS policy library is used to define the service quality requirements for different service types, the optimization rule library is used to store optimization parameter combinations for various scenarios, the adaptive learner is used to optimize policy parameters based on historical data, and the policy decision-maker is used to select the optimal policy based on the current context.

[0013] A computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the aforementioned low-latency optimized data transmission method.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. The present invention provides a data transmission method, system, and computer-readable storage medium based on low latency optimization. Through real-time network sensing and machine learning technology, the system can dynamically adjust the transmission strategy to adapt to diverse network environments. 2. This invention provides a data transmission method, system, and computer-readable storage medium based on low-latency optimization, a multi-dimensional priority model, and an intelligent scheduling algorithm to ensure that critical data obtains the optimal transmission path and resources; 3. This invention provides a data transmission method, system, and computer-readable storage medium based on low latency optimization. It breaks through the traditional protocol stack layering limitations, achieves vertical optimization from the application layer to the network layer, significantly improves efficiency, and combines the advantages of FEC and selective ARQ to achieve a better balance between reliability and latency. The optimized encryption and authentication mechanisms minimize performance overhead while ensuring security.

[0015] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0017] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0018] It should be noted that when a component is described as "fixed to" another component, it can be directly on the other component or may have a component in between. When a component is considered "connected to" another component, it can be directly connected to the other component or may have a component in between. When a component is considered "set on" another component, it can be directly set on the other component or may have a component in between. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0020] Please see Figure 1 The present invention provides a technical solution: A data transmission method based on low latency optimization, characterized by comprising the following steps: Step 1: Data preprocessing, responsible for data classification, segmentation, compression and encryption preprocessing operations, dividing large data blocks into multiple sub-blocks, supporting parallel transmission and independent retransmission; Step 2: Transmission scheduling, which implements multi-path selection, priority scheduling and traffic shaping functions. Priorities include business priority, timeliness priority, dependency priority and resource priority. Business priority is the level of business importance specified by the application layer. Timeliness priority is calculated based on the urgency of data validity period. Dependency priority considers the dependency relationship between data blocks. Resource priority is dynamically adjusted according to the current system resource load. Step 3: Network transmission, handling the actual sending and receiving of data packets and basic transmission control; Step 4: Quality control, monitoring network status, adjusting transmission parameters, and ensuring service quality.

[0021] A low-latency optimized data transmission system includes: a client module, a transmission engine module, a network awareness module, a policy management module, and a security module; The client module is deployed at both the data sending and receiving ends, implementing the user interface and basic communication functions; the transmission engine is the core processing unit of the system, implementing various optimization algorithms; the network awareness module is used to monitor the network status in real time, providing a basis for decision-making; the policy management module is used to store and update various transmission policy parameters; and the security module is used to ensure the security and privacy of data transmission. The network awareness module is responsible for collecting and analyzing network status information. Its key sub-modules include a probe engine, a passive monitor, a prediction model, and a topology discovery model. The probe engine is used to actively send probe packets to measure network parameters. The passive monitor is used to analyze the transmission characteristics of service data packets to infer network status. The prediction model predicts the trend of network parameter changes based on time series analysis. The topology discovery model is used to identify available transmission paths and their characteristics. The transmission engine is the core processing unit of the system. Its main sub-modules include the scheduler, path manager, congestion controller, error processor, and buffer manager. The scheduler is used to implement data chunking and priority scheduling, the path manager is used to maintain path status and perform path selection, the congestion controller is used to implement congestion detection and rate control, the error processor is used to handle data loss and error recovery, and the buffer manager is used to optimize the use of send and receive buffers. The policy management module provides a configurable policy framework, which mainly includes a QoS policy library, an optimization rule library, an adaptive learner, and a policy decision-maker. The QoS policy library is used to define the service quality requirements for different service types, the optimization rule library is used to store optimization parameter combinations for various scenarios, the adaptive learner is used to optimize policy parameters based on historical data, and the policy decision-maker is used to select the optimal policy based on the current context.

[0022] A computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the aforementioned low-latency optimized data transmission method.

[0023] The end-to-end workflow of the system can be divided into the following stages: The initialization phase mainly involves establishing physical connections, exchanging capability parameters, negotiating security parameters and QoS requirements, initial network probing, and path establishment. During the data transmission phase, application layer data is submitted to the transmission system, data is preprocessed, a transmission path is selected according to the current strategy, data packets are sent and received, and quality is monitored and dynamically adjusted. During the connection maintenance phase, regular network status checks, path quality reassessments, adaptive adjustments to transmission parameters, and anomaly detection and recovery are performed. During the termination phase, data streams are shut down in an orderly manner, network resources are released, and transmission statistics are collected and reported.

[0024] End-to-end latency can be broken down into the following components and optimization strategies: To address latency, zero-copy technology is used to reduce data movement, SIMD instructions are used to accelerate data processing, and critical path code is deeply optimized. Queuing delay, priority-based scheduling algorithm, dynamic traffic shaping, and intelligent buffer management; To reduce transmission latency, data compression reduces the amount of data transmitted; edge node deployment shortens physical distances; and protocol header compression is also helpful. To mitigate propagation delays, a better network path is selected, utilizing high-speed physical links, retransmission delays are reduced, forward error correction technology is employed, a fast retransmission mechanism is implemented, and multi-path redundant transmission is employed.

[0025] Protocol stack optimization, header compression and simplification, optimization measures include: static field compression, identifying and eliminating redundant fields in the headers of each layer of the protocol stack; dynamic field differential encoding, transmitting only the difference for changing fields such as sequence numbers; header compression protocol adoption, using standard technologies such as ROHC (RobustHeaderCompression); and aggregated transmission, merging multiple small data packets into one large packet for transmission.

[0026] Cross-layer optimization design breaks away from the strict layering of traditional protocol stacks, enabling cross-layer information sharing and collaborative optimization. The application layer collaborates with the transport layer, with the application indicating data priority and deadlines, and the transport layer providing feedback on network conditions to guide application adjustments. The transport layer collaborates with the network layer to share congestion status information and jointly make path selection decisions. The physical layer collaborates with the application layer to adaptively adjust the coding rate based on wireless signal quality and use channel state prediction to guide transmission scheduling.

[0027] Performance testing and comparison: The test environment and equipment are configured as follows: Server: Dual-socket Xeon 6248 processors, 128GB RAM; Client: Core i7 1185G7, 32GB RAM; Network equipment: Commercial-grade switches and routers.

[0028] Latency performance test results for small data packets (256 bytes): Throughput test, large data stream (1GB file) transmission test: Performance under complex scenario testing with mixed traffic (small packets + large flows): In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0029] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0030] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0031] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0032] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0033] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A data transmission method based on low-latency optimization, characterized in that, Includes the following steps: Step 1: Data preprocessing, responsible for data classification, segmentation, compression, and encryption preprocessing operations; Step 2: Transmission scheduling, implementing multi-path selection, priority scheduling, and traffic shaping functions; Step 3: Network transmission, handling the actual sending and receiving of data packets and basic transmission control; Step 4: Quality control, monitoring network status, adjusting transmission parameters, and ensuring service quality.

2. The data transmission method based on low-latency optimization as described in claim 1, characterized in that: In step one, the large data block is divided into multiple sub-blocks, supporting parallel transmission and independent retransmission.

3. The data transmission method based on low-latency optimization as described in claim 1, characterized in that: In step two, priorities include business priority, timeliness priority, dependency priority, and resource priority. Business priority is the level of business importance specified by the application layer. Timeliness priority is calculated based on the urgency of data validity. Dependency priority considers the dependencies between data blocks. Resource priority is dynamically adjusted according to the current system resource load.

4. A data transmission system based on low-latency optimization, characterized in that, include: The module consists of a client module, a transmission engine module, a network awareness module, a policy management module, and a security module.

5. A data transmission system based on low-latency optimization as described in claim 4, characterized in that: The client module is deployed at both the data sending and receiving ends, implementing the user interface and basic communication functions; the transmission engine is the core processing unit of the system, implementing various optimization algorithms; the network awareness module is used to monitor the network status in real time, providing a basis for decision-making; the policy management module is used to store and update various transmission policy parameters; and the security module is used to ensure the security and privacy of data transmission.

6. A data transmission system based on low-latency optimization as described in claim 4, characterized in that: The network awareness module is responsible for collecting and analyzing network status information. Its key sub-modules include a probe engine, a passive monitor, a prediction model, and a topology discovery model. The probe engine is used to actively send probe packets to measure network parameters. The passive monitor is used to analyze the transmission characteristics of service data packets to infer network status. The prediction model predicts the trend of network parameter changes based on time series analysis. The topology discovery model is used to identify available transmission paths and their characteristics.

7. A data transmission system based on low-latency optimization as described in claim 4, characterized in that: The transmission engine is the core processing unit of the system. Its main sub-modules include the scheduler, path manager, congestion controller, error processor, and buffer manager. The scheduler is used to implement data chunking and priority scheduling, the path manager is used to maintain path status and perform path selection, the congestion controller is used to implement congestion detection and rate control, the error processor is used to handle data loss and error recovery, and the buffer manager is used to optimize the use of send and receive buffers.

8. A data transmission system based on low-latency optimization as described in claim 4, characterized in that: The policy management module provides a configurable policy framework, which mainly includes a QoS policy library, an optimization rule library, an adaptive learner, and a policy decision-maker. The QoS policy library is used to define the service quality requirements for different service types, the optimization rule library is used to store optimization parameter combinations for various scenarios, the adaptive learner is used to optimize policy parameters based on historical data, and the policy decision-maker is used to select the optimal policy based on the current context.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the low-latency optimized data transmission method as described in any one of claims 1 to 8.

Images