Distributed data transmission method, system, equipment and medium

Abstract

The invention discloses a distributed data transmission method, system and device and a medium, and the method comprises the steps: dynamically segmenting a file into N data blocks according to the attribute of the file, and distributing a unique identifier to each data block; calculating the shortest path between the nodes through an improved Dijkstra algorithm, and generating a dynamic routing table in combination with real-time bandwidth monitoring data; each node synchronously transmits a data block based on a routing table, and an asynchronous I / O model is adopted to reduce waiting time; a receiving end passes CRC32 + SHA-256 dual verification, and missing data blocks trigger an automatic retransmission mechanism; compared with the traditional method, the bandwidth utilization rate is improved by more than 40%, and the end-to-end delay is reduced by 35%; breakpoint resume and error verification are supported, and the data integrity rate is greater than or equal to 99.99%; the method can be adapted to thousand-level node scales, and meets the requirements of high-concurrency scenes such as industrial internet and the like.

Description

Technical Field

[0001] The present invention relates to the field of data transmission technology, and in particular to a distributed data transmission method, system, device and medium. Background Art

[0002] With the popularization of cloud computing and the Internet of Things, distributed system data transmission faces the following challenges: Transmission efficiency bottleneck: Traditional TCP / IP protocols suffer from high handshake delays and insufficient bandwidth utilization during cross-node transmission. Load imbalance: Fixed routing strategies can easily lead to overload of some nodes, causing transmission link congestion; Weak fault tolerance: Single point failures can easily lead to transmission interruptions, and there is a lack of automatic retransmission mechanism.

[0003] Existing technologies mostly use static sharding or single hash routing, which cannot dynamically adjust the transmission strategy according to the real-time network status. An adaptive distributed transmission solution is urgently needed. Summary of the Invention

[0004] The object of the present invention is to provide a distributed data transmission method, system, device and medium to address the above-mentioned problems in the prior art, thereby solving all or one of the above-mentioned problems in the prior art.

[0005] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows: In one aspect, the present invention provides a distributed data transmission method, comprising: S1. Data sharding: Dynamically split the file into N data blocks based on its attributes, and assign a unique identifier to each data block; S2, routing optimization: The shortest path between nodes is calculated using the improved Dijkstra algorithm, and a dynamic routing table is generated based on real-time bandwidth monitoring data; S3, parallel transmission: Each node transmits data blocks synchronously based on the routing table, using an asynchronous I / O model to reduce waiting time; S4. Integrity verification: The receiving end uses CRC32+SHA-256 double verification, and missing data blocks trigger the automatic retransmission mechanism.

[0006] As an improved solution, the sharding strategy in S1 satisfies: S=min(B / M, L / K); Among them, S is the optimal number of shards, B is the file size, M is the average bandwidth, L is the number of nodes, and K is the redundancy coefficient.

[0007] As an improved solution, the S2 routing table generation includes: Real-time bandwidth weighting factor: BW w=1 / (RTT×σ); where RTT is the round-trip delay and σ is the jitter standard deviation; Node load factor: L c =(CPU util +MEM used ) / 2, dynamically adjust path priority.

[0008] As an improved solution, it also includes: During the transmission process, forward error correction coding is used, and m check blocks are added to each data block to meet the following requirements: m=⌈log(1+SNR)N*(1−P e )⌉ Among them, P e is the bit error rate, and SNR is the signal-to-noise ratio.

[0009] In another aspect, the present invention further provides a distributed data transmission system, comprising: Smart sharding engine: used to execute dynamic sharding algorithm; Adaptive routing module: used for built-in improved Dijkstra algorithm and real-time topology database; FEC error correction unit: used to generate redundant check data; Heartbeat monitoring component: used to periodically update node status and link quality data.

[0010] As an improved solution, the distributed data transmission system supports a hybrid transmission mode, enabling dedicated channels for high-priority data, enabling multi-path parallel transmission for large files, and enabling UDP+FEC combined transmission for real-time streaming media.

[0011] As an improved solution, the distributed data transmission system has a built-in blockchain evidence storage module for recording data block hash values, transmission path fingerprints and operation log hash chains.

[0012] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the distributed data transmission method are implemented.

[0013] On the other hand, the present invention further provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; wherein: The memory is used to store computer programs; The processor is configured to execute the steps of the distributed data transmission method by running the program stored in the memory.

[0014] The beneficial effects of the technical solution of the present invention are: 1. Compared with traditional methods, the distributed data transmission method described in the present invention improves bandwidth utilization by more than 40% and reduces end-to-end latency by 35%. It supports breakpoint resumption and error checking, and has a data integrity rate of ≥ 99.99%. It can be adapted to thousands of nodes and meet the needs of high-concurrency scenarios such as the Industrial Internet.

[0015] 2. The distributed data transmission system described in the present invention can implement the method logic of the distributed data transmission method described in the present invention.

[0016] 3. The computer-readable storage medium described in the present invention can realize the cooperation of the guidance system module, thereby realizing the distributed data transmission method described in the present invention, and effectively improving the operability of the method.

[0017] 4. The computer device described in the present invention can store and execute the computer-readable storage medium, thereby implementing the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 1 is a flow chart of the distributed data transmission method according to embodiment 1 of the present invention; Figure 2 2 is a schematic diagram of the architecture of the distributed data transmission system according to Embodiment 2 of the present invention; Figure 3 is a schematic structural diagram of the computer device described in Example 4 of the present invention; The symbols in the accompanying drawings are described as follows: 1501. Processor; 1502. Communication interface; 1503. Memory; 1504. Communication bus. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0021] In the description of the present invention, it should be noted that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of the present invention.

[0022] The terms "first," "second," and the like in the specification and claims herein and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device. Example 1

[0023] This embodiment provides a distributed data transmission method, such as Figure 1 Shown, including: S1. Data sharding: Dynamically split the file into N data blocks based on its attributes, and assign a unique identifier to each data block; S2, routing optimization: The shortest path between nodes is calculated using the improved Dijkstra algorithm, and a dynamic routing table is generated based on real-time bandwidth monitoring data; S3, parallel transmission: Each node transmits data blocks synchronously based on the routing table, using an asynchronous I / O model to reduce waiting time; S4. Integrity verification: The receiving end uses CRC32+SHA-256 double verification, and missing data blocks trigger the automatic retransmission mechanism.

[0024] As an improved solution, the sharding strategy in S1 satisfies: S=min(B / M, L / K); Among them, S is the optimal number of shards, B is the file size, M is the average bandwidth, L is the number of nodes, and K is the redundancy coefficient.

[0025] As an improved solution, the S2 routing table generation includes: Real-time bandwidth weighting factor: BW w =1 / (RTT×σ); where RTT is the round-trip delay and σ is the jitter standard deviation; Node load factor: L c =(CPU util +MEM used ) / 2, dynamically adjust the path priority.

[0026] As an improved solution, it also includes: During the transmission process, forward error correction coding is used, and m check blocks are added to each data block to meet the following requirements: m=⌈log(1+SNR)N*(1−P e )⌉ Among them, P e is the bit error rate, and SNR is the signal-to-noise ratio.

[0027] It should be noted that the examples herein are only for explaining the present invention and are not intended to limit the scope of protection of the present invention. Example 2

[0028] This embodiment is based on the same inventive concept as the distributed data transmission method described in Example 1, and provides a distributed data transmission system, such as Figure 2 Shown, including: Smart sharding engine: used to execute dynamic sharding algorithm; Adaptive routing module: used for built-in improved Dijkstra algorithm and real-time topology database; FEC error correction unit: used to generate redundant check data; Heartbeat monitoring component: used to periodically update node status and link quality data.

[0029] As an improved solution, the distributed data transmission system supports a hybrid transmission mode, enabling dedicated channels for high-priority data, enabling multi-path parallel transmission for large files, and enabling UDP+FEC combined transmission for real-time streaming media.

[0030] As an improved solution, the distributed data transmission system has a built-in blockchain evidence storage module for recording data block hash values, transmission path fingerprints and operation log hash chains. Example 3

[0031] This embodiment provides a computer-readable storage medium, including: The storage medium is used to store computer software instructions used to implement the distributed data transmission method described in the above-mentioned embodiment 1, which includes a program for executing the above-mentioned program set for the distributed data transmission method; specifically, the executable program can be built into the distributed data transmission system described in embodiment 2, so that the distributed data transmission method described in embodiment 1 can be implemented by executing the built-in executable program.

[0032] In addition, the computer-readable storage medium of this embodiment may adopt any combination of one or more computer-readable storage media, wherein the computer-readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof. Example 4

[0033] This embodiment provides an electronic device, such as Figure 3 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0034] Memory 1503, used for storing computer programs; The processor 1501 is configured to implement the steps of the distributed data transmission method described in the above-mentioned embodiment 1 when executing the computer program stored in the memory 1503 .

[0035] As an embodiment of the present invention, the communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0036] As an embodiment of the present invention, the communication interface is used for communication between the above-mentioned terminal and other devices.

[0037] As one embodiment of the present invention, the memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located remote from the processor.

[0038] As an embodiment of the present invention, the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0039] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0040] It should also be understood that in the embodiments herein, the term "and / or" merely describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" could represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0041] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0042] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific logical process of the method described above can refer to the corresponding working processes of the systems, devices and units in the aforementioned method embodiments, and will not be repeated here.

[0043] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices, or units, or can be an electrical, mechanical, or other form of connection.

[0044] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments herein.

[0045] In addition, the functional units in the various embodiments herein may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0046] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0047] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A distributed data transmission method, characterized in that: The following steps are involved: S1. Data sharding: Dynamically split the file into N data blocks based on its attributes, and assign a unique identifier to each data block; S2, routing optimization: The shortest path between nodes is calculated using the improved Dijkstra algorithm, and a dynamic routing table is generated based on real-time bandwidth monitoring data; S3, parallel transmission: Each node transmits data blocks synchronously based on the routing table, using an asynchronous I / O model to reduce waiting time; S4. Integrity verification: The receiving end uses CRC32+SHA-256 double verification, and missing data blocks trigger the automatic retransmission mechanism.

2. The distributed data transmission method according to claim 1, wherein: The sharding strategy in S1 satisfies: S=min(B / M, L / K); Among them, S is the optimal number of shards, B is the file size, M is the average bandwidth, L is the number of nodes, and K is the redundancy coefficient.

3. The distributed data transmission method according to claim 1 or 2, characterized in that: The S2 routing table generation includes: Real-time bandwidth weighting factor: BW w =1 / (RTT×σ); where RTT is the round-trip delay and σ is the jitter standard deviation; Node load factor: L c =(CPU util +MEM used ) / 2, dynamically adjust the path priority.

4. The distributed data transmission method according to claim 1 or 2, characterized in that: Also includes: During the transmission process, forward error correction coding is used, and m check blocks are added to each data block to meet the following requirements: m=⌈log(1+SNR)N*(1−P e )⌉ Among them, P e is the bit error rate, and SNR is the signal-to-noise ratio.

5. A distributed data transmission system based on the distributed data transmission method according to claim 1 or 2, characterized in that: include: Smart sharding engine: used to execute dynamic sharding algorithm; Adaptive routing module: used for built-in improved Dijkstra algorithm and real-time topology database.

6. A distributed data transmission system based on the distributed data transmission method according to claim 1 or 2, characterized in that: Also includes: FEC error correction unit: used to generate redundant check data; Heartbeat monitoring component: used to periodically update node status and link quality data.

7. The distributed data transmission system according to claim 6, characterized in that: The distributed data transmission system supports a mixed transmission mode, enabling dedicated channels for high-priority data, enabling multi-path parallel transmission for large files, and enabling UDP+FEC combined transmission for real-time streaming media.

8. The distributed data transmission system according to claim 7, characterized in that: The distributed data transmission system has a built-in blockchain evidence storage module for recording data block hash values, transmission path fingerprints and operation log hash chains.

9. A computer-readable storage medium, characterized in that A computer program is stored, and when the program is executed, the steps of the distributed data transmission method according to any one of claims 1 to 4 are implemented.

10. A computer device comprising a processor, a communication interface, a memory and a bus, characterized in that: The processor executes the functions of the distributed data transmission system described in any one of claims 5-8; the memory stores a dynamic routing table and a sharding strategy configuration file.

Images