Distributed data transmission method, apparatus, device, medium and product

Abstract

The application discloses a distributed data transmission method and device, equipment, medium and product, relates to the technical field of data transmission, and the distributed data transmission method comprises the steps of determining a plurality of computing nodes in a distributed computing terminal; constructing a transmission matrix according to the transmission delay, bandwidth and to-be-transmitted data volume between the computing nodes, wherein the computing nodes comprise source nodes and target nodes; performing path optimization on the path between the source nodes and the target nodes according to the transmission matrix to obtain a first path; dividing all the computing nodes into a plurality of levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel; and transmitting preset task data to the target nodes through the multi-level aggregation structure. The application improves the data transmission efficiency of the distributed computing terminal.

Description

Technical Field

[0001] The present application relates to the field of data transmission technology, and in particular to distributed data transmission methods, devices, equipment, media and products. Background Art

[0002] In modern computing environments, distributed computing systems are widely used to process large-scale data sets and perform complex computational tasks. Such systems typically consist of multiple heterogeneous devices, including but not limited to high-performance computing units (HPCUs), such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), and FPGAs (Field-Programmable Gate Arrays), interconnected via a network to work collaboratively. The key advantage of distributed computing systems is their ability to leverage the parallel processing capabilities of multiple devices, significantly improving computational efficiency and speed. However, in practical applications, the performance of distributed computing systems is often limited by data transmission bottlenecks. In particular, in the fields of deep learning hardware acceleration and high-performance computing architectures, data transmission latency and bandwidth usage become significant factors affecting overall performance. For example, in large-scale computing tasks (such as deep learning training and scientific simulations), frequent data exchange between nodes can lead to communication bottlenecks, significantly reducing system throughput and responsiveness. Most traditional systems use static routing or simple load balancing strategies, which are unable to dynamically adjust data transmission paths. The high volume and frequency of data transmission lead to high bandwidth usage and uncontrollable latency, a problem that becomes more pronounced as task complexity and the number of nodes increase. Furthermore, the single data aggregation method lacks hierarchical optimization, concentrating node communication pressure on a single aggregation point. This easily creates a transmission bottleneck, leading to low transmission efficiency between distributed computing terminals and limiting system scalability and overall performance. Traditional methods struggle to meet the requirements of high-performance computing, especially in the face of growing data volumes and complex computing demands. There is an urgent need for a method that can improve the data transmission efficiency of distributed computing terminals. Summary of the Invention

[0003] The main purpose of this application is to provide a distributed data transmission method, device, equipment, medium and product, aiming to improve the data transmission efficiency of distributed computing terminals.

[0004] To achieve the above objectives, the present application proposes a distributed data transmission method, which is applied to distributed computing terminals, comprising:

[0005] Determining multiple computing nodes in a distributed computing terminal;

[0006] Constructing a transmission matrix based on transmission delay, bandwidth, and amount of data to be transmitted between computing nodes, where computing nodes include source nodes and destination nodes;

[0007] Optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0008] Divide all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0009] The preset task data is transmitted to the target node through a multi-level aggregation structure.

[0010] In one embodiment, the step of performing path optimization on a path between a source node and a target node according to a transmission matrix to obtain a first path includes:

[0011] For any source node in the transmission matrix, all feasible paths from the source node to the target node are determined, and the shortest path that meets the minimum cost condition is found among all feasible paths, and the shortest path is used as the first path, where the cost between any two adjacent nodes in the shortest path is the lowest.

[0012] In one embodiment, after the step of optimizing the path between the source node and the target node according to the transmission matrix to obtain the first path, the following steps are included:

[0013] When the network state changes frequently, determining whether the cost of the first path is updated;

[0014] If the cost of the first path is updated, the step of optimizing the path between the source node and the target node according to the transmission matrix is ​​re-executed to find the latest shortest path, and the latest shortest path is updated as the first path.

[0015] In one embodiment, the step of dividing all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure includes:

[0016] The computing nodes whose distance is less than a preset distance threshold, whose delay is less than a preset delay threshold and which have no dependency on other computing nodes are regarded as first-level computing nodes;

[0017] For any level after the first level, the computing node closest to the previous level and with a delay less than the preset delay index is selected according to the first path as the computing node of the current level, resulting in a multi-level aggregation structure, where the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node;

[0018] The step of dividing all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure includes:

[0019] If the delay of any first computing node is greater than or equal to the preset delay index, the first computing node is replaced with a preset standby node.

[0020] In one embodiment, the step of transmitting the preset task data to the target node via the multi-level aggregation structure includes:

[0021] Input the task data into the first-level computing nodes in the multi-level aggregation structure, aggregate the task data through the first-level computing nodes, and obtain the aggregation results;

[0022] For any computing node of the level after the first-level computing node, the aggregation result of the previous level is aggregated again to obtain the aggregation result of the current level, until it is aggregated to the target node in the multi-level aggregation structure, wherein the aggregation result of the current level is stored in the computing node of the current level that is closest to the next level.

[0023] In one embodiment, the step of constructing a transmission matrix based on the transmission delay, bandwidth, and amount of data to be transmitted between computing nodes includes:

[0024] Determine all computing node pairs, where a computing node pair consists of two computing nodes;

[0025] For each computing node pair, detect the network status to obtain the transmission delay and bandwidth between the computing node pairs;

[0026] Determine the amount of data to be transmitted between pairs of computing nodes based on the task data;

[0027] The ratio of the amount of data to be transmitted to the bandwidth is calculated, and the product of the ratio and the transmission delay is used as the transmission cost of each computing node pair. The transmission cost is used as an element of the transmission matrix to obtain the transmission matrix.

[0028] In addition, to achieve the above-mentioned purpose, the present application also proposes a distributed data transmission device, which is provided in a distributed computing terminal. The distributed data transmission device includes:

[0029] Determine a computing node module, and determine multiple computing nodes in a distributed computing terminal;

[0030] A transmission matrix construction module is used to construct a transmission matrix based on the transmission delay, bandwidth, and amount of data to be transmitted between computing nodes, where the computing nodes include source nodes and destination nodes;

[0031] A path optimization module, which optimizes the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0032] an aggregation structure construction module, which divides all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0033] The aggregation transmission module transmits the preset task data to the target node through a multi-level aggregation structure.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a distributed data transmission device, which includes: a memory, a processor, and a computer program stored in the memory and runnable on the processor, and the computer program is configured to implement the steps of the distributed data transmission method as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a medium, which is a computer-readable storage medium. A computer program is stored on the medium. When the computer program is executed by a processor, the steps of the distributed data transmission method as described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a product, which is a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps of the distributed data transmission method as described above.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] The present application is applied to a distributed computing terminal, determines multiple computing nodes in the distributed computing terminal, and constructs a transmission matrix based on the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes. The matrix comprehensively reflects the transmission cost information in the network and provides key data support for path optimization; according to the transmission matrix, the path between the source node and the target node is optimized to obtain a first path, which minimizes the transmission cost while ensuring the quality of data transmission, thereby improving the efficiency of data transmission; according to the first path, all computing nodes are divided into multiple levels to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel, and this structure not only improves the parallelism of data processing, but also optimizes the hierarchical structure of data transmission, further improving the overall computing efficiency; the preset task data is transmitted to the target node through the multi-level aggregation structure. This process makes full use of the optimized path and hierarchical structure to achieve high-speed and low-latency transmission of data, effectively solving the defect of low transmission efficiency of distributed computing terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is a flow chart of the first embodiment of the distributed data transmission method of the present application;

[0042] Figure 2 A schematic diagram of a distributed data transmission method for this application;

[0043] Figure 3 This is another flowchart of the distributed data transmission method of this application;

[0044] Figure 4 This is a schematic diagram of the module structure of the distributed data transmission device according to an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the distributed data transmission method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or terminal system capable of implementing the above functions. The following uses the system as an example to illustrate this embodiment and the following embodiments.

[0050] Based on this, this embodiment provides a distributed data transmission method, which is applied to distributed computing terminals.

[0051] In this embodiment, refer to Figure 1 , Figure 1 This is a flow chart of the distributed data transmission method of the present application. The distributed data transmission method includes steps S10 to S50:

[0052] Step S10, determining multiple computing nodes in the distributed computing terminal;

[0053] Step S20, constructing a transmission matrix based on transmission delay, bandwidth, and amount of data to be transmitted between computing nodes, wherein the computing nodes include source nodes and target nodes;

[0054] It should be noted that a distributed computing terminal refers to a computing system composed of multiple computing nodes connected via a network. These computing nodes can collaborate to complete tasks and improve computing efficiency. A computing node is the basic unit in a distributed computing terminal and can be a server, personal computer, or other computing device. Each node has data processing capabilities. During the data transmission process, the source node generates data and prepares it for transmission. The destination node is the node that receives the data during the data transmission process.

[0055] In a distributed computing system, all nodes involved in the computation must first be identified. These nodes include at least one source node (the node that generates data) and at least one destination node (the node that receives data). These nodes are connected via a network and together form a distributed computing terminal.

[0056] It should be noted that the transmission matrix is ​​a mathematical model used to describe key information such as transmission delay, bandwidth, and amount of data to be transmitted between the source node and the destination node.

[0057] The transmission matrix is ​​constructed based on the transmission delay, bandwidth, and amount of data to be transmitted between the source and destination nodes. Transmission delay refers to the time required for data to travel from the source node to the destination node; bandwidth represents the network's transmission capacity, or the amount of data that can be transmitted per second; and the amount of data to be transmitted refers to the amount of data that the source node needs to send to the destination node. This information is determined through real-time monitoring of network status and computing tasks. A matrix is ​​then constructed that includes transmission delay, bandwidth, and amount of data to be transmitted. This matrix serves as the basis for path optimization.

[0058] For example, the calculation method of the transmission matrix is ​​expressed as formula (1)

[0059]

[0060] Among them, M trans (i,j) is the transmission matrix, Latency i,j is the transmission delay from computing node i to computing node j, Bandwidth i,j is the bandwidth from computing node i to computing node j, Data_Size i,j is the amount of data that needs to be transmitted from computing node i to computing node j.

[0061] Step S30, optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0062] It should be noted that path optimization is to adjust the data transmission path through an algorithm to reduce transmission delay, improve bandwidth utilization, etc. The first path is the optimal data transmission path obtained after path optimization.

[0063] The goal of path optimization is to find the best transmission path from the source node to the destination node to minimize the total transmission overhead of the system. This usually involves constructing an objective function that takes into account multiple factors such as transmission delay, bandwidth utilization, and the amount of data to be transmitted. A dynamic programming algorithm (such as Dijkstra or Floyd-Warshall) is used to calculate the shortest path between nodes, while taking into account that the data for each task must be transmitted completely and the path cannot exceed the available bandwidth limit. If the network status changes frequently, a dynamic optimization method based on the ant colony algorithm can also be used to continuously adjust the transmission path. Through path optimization, we obtain the best path from the source node to the destination node, which is the first path.

[0064] Step S40: Divide all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0065] It's important to note that a multi-level aggregation structure is a data aggregation strategy that divides computing nodes into multiple levels and aggregates data level by level to reduce data transmission. Parallel processing is when multiple computing nodes process data simultaneously to increase data processing speed.

[0066] The multi-level aggregation structure is designed to further reduce data transmission and improve system scalability. It divides all computing nodes into multiple tiers, completing data aggregation step by step. Within each tier, nodes independently and in parallel process data to reduce the total amount of backhaul. The results of each level of aggregation serve as input for the next level, until the top level of aggregation completes to achieve a global result. Aggregation nodes are selected based on factors such as distance and latency, with nodes with shorter distances and lower latency being prioritized as first-level aggregation nodes. Furthermore, the positions of aggregation nodes are dynamically adjusted based on the real-time transmission matrix to respond to changes in network status.

[0067] Step S50: transmitting the preset task data to the target node through the multi-level aggregation structure.

[0068] It should be noted that task data specifically refers to the various information involved in executing distributed computing tasks. This task data can include raw input data, intermediate computational results, final output results, and more. In distributed computing, task data often needs to be transmitted and shared across multiple computing nodes for collaborative computing and processing. During transmission between distributed computing terminals, task data can be in a variety of formats, including text, images, audio, and video, depending on the task requirements and the nature of the data.

[0069] After determining the optimal transmission path and multi-level aggregation structure, the pre-set task data is transmitted according to these paths and structures. During the data transmission process, it passes through multiple aggregation nodes, where data aggregation and compression are performed at each level. Each level of aggregation reduces the total amount of data, thus preventing excessive data transmission at the next level. During the transmission process, data is processed and aggregated by computing nodes within each level according to the hierarchical relationship of the multi-level aggregation structure, ultimately reaching the destination node.

[0070] For example, assuming there are source nodes A, B, C, D, and target node E, the transmission matrix M trans Expressed as formula (2), the node data volume is: A: 100MB, B: 200MB, C: 150MB, D: 50MB.

[0071]

[0072] Use Dijkstra algorithm to optimize the path and get the first path as A→B, B→D, C→D, D→E. Divide all computing nodes into multiple levels according to the first path. Figure 2 The first-level computing nodes are A and C, the second-level computing nodes are B and D, and the third-level computing node is the target node E. Therefore, the data is aggregated to B through A, and B obtains data A+B. At the same time, it is aggregated to D through C, and D obtains data C+D. Finally, the data is aggregated to E through B and D of the second-level computing nodes, and E obtains data A+B+C+D. Through multi-level aggregation and path optimization, the total transmission volume is reduced by 40%.

[0073] For example, for GPU nodes G1 and G2 and CPU nodes C1 and C2, real-time latency monitoring of the dynamic changes in the transmission matrix is ​​performed. GPU nodes prioritize computationally intensive aggregation, while CPU nodes handle lightweight transmission. The transmission matrix is ​​adjusted in real time to optimize the path. After dynamic path optimization, latency is reduced by 30% and the total transmission volume is reduced by 25%.

[0074] This embodiment is applied to a distributed computing terminal, determines multiple computing nodes in the distributed computing terminal, and constructs a transmission matrix based on the transmission delay, bandwidth, and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes. The matrix comprehensively reflects the transmission cost information in the network and provides key data support for path optimization; the path between the source node and the target node is optimized according to the transmission matrix to obtain a first path, which minimizes the transmission cost while ensuring the quality of data transmission, thereby improving the efficiency of data transmission; all computing nodes are divided into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel. This structure not only improves the parallelism of data processing, but also optimizes the hierarchical structure of data transmission, further improving the overall computing efficiency; the preset task data is transmitted to the target node through the multi-level aggregation structure. This process fully utilizes the optimized path and hierarchical structure to achieve high-speed, low-latency data transmission, effectively solving the defect of low transmission efficiency of the distributed computing terminal.

[0075] In a feasible implementation, step S50 may include steps T10 to T20:

[0076] Step T10, applying a preset lightweight compression algorithm to compress the data to obtain compressed data;

[0077] It's important to note that lightweight compression algorithms are data processing techniques designed to reduce data size while maintaining data integrity or maintaining acceptable loss. They are typically faster and more efficient than traditional compression algorithms and are suitable for environments with strict real-time or resource-intensive requirements. Compressing data involves converting original data into a smaller form by applying a compression algorithm. Compressed data retains all or most of the original data's information while taking up less storage space or transmission bandwidth.

[0078] Apply a pre-defined lightweight compression algorithm to the original data. This process typically involves analyzing data characteristics such as repetitiveness, redundancy, and predictability, and leveraging these characteristics to reduce the overall data size. The advantage of lightweight compression algorithms lies in their efficiency and low resource consumption, enabling them to significantly reduce data size without sacrificing significant data quality. By applying this algorithm, we obtain compressed data that, while smaller in size, still contains the core information of the original data.

[0079] Step T20: Transmit the compressed data to the target node through the multi-level aggregation structure.

[0080] Compressed data is transmitted to the destination node through a multi-level aggregation structure. During this process, the compressed data is divided into multiple data blocks or packets and transmitted according to the hierarchical relationship of the multi-level aggregation structure. The computing nodes at each layer receive the data from the previous layer, perform necessary processing (such as further compression and format conversion), and then pass the processed data to the next layer. This layer-by-layer transmission and processing method ensures data reliability and integrity while improving transmission efficiency. The multi-level aggregation structure plays a key role in this process. It not only provides an efficient data transmission channel, but also reduces data redundancy and complexity through layer-by-layer aggregation. During the transmission process, the computing nodes at each layer can optimize and process the data, ensuring that the data that ultimately reaches the destination node is efficient and accurate. Ultimately, the compressed data is successfully transmitted to the destination node. At the destination node, this data can be further processed, analyzed, or stored.

[0081] This embodiment uses a lightweight compression algorithm and a multi-level aggregation structure, which not only improves the efficiency and reliability of data transmission but also reduces resource consumption and costs. This gives the method of this embodiment a significant advantage when processing large-scale data sets or real-time data streams.

[0082] In a feasible implementation, step S50 may include steps R10 to R40:

[0083] Step R10, determining all computing node pairs, wherein a computing node pair consists of two computing nodes;

[0084] It should be noted that a computing node pair is a pair of two computing nodes in a distributed computing environment, and there may be a need for data transmission between the two nodes.

[0085] Identify all compute node pairs. In a distributed computing environment, there are multiple compute nodes, each of which may transmit data to other nodes. Therefore, these nodes need to be paired to form compute node pairs. This step forms the basis for constructing the transmission matrix.

[0086] Step R20: for each computing node pair, detect the network status to obtain the transmission delay and bandwidth between the computing node pairs;

[0087] It's important to note that network status refers to various parameters and metrics that describe the current state of a network, including but not limited to transmission latency and bandwidth. Transmission latency is the time required for data to travel across a network, typically due to factors such as network distance, congestion, and device performance. Bandwidth, the amount of data a network can transmit per unit time, is a key indicator of network transmission capacity.

[0088] For each pair of computing nodes, the network status is monitored to obtain the transmission delay and bandwidth between them. Network status monitoring is a complex process that involves measuring and analyzing multiple network parameters. In this embodiment, we focus on transmission delay and bandwidth. Transmission delay reflects the speed at which data is transmitted within the network, while bandwidth determines the amount of data that can be transmitted per unit time. These two parameters together determine the efficiency and cost of data transmission.

[0089] Step R30, determining the amount of data to be transmitted between the computing node pairs based on the task data;

[0090] It should be noted that the amount of data to be transmitted is the amount of data that needs to be transmitted from the source node to the target node in a distributed computing task.

[0091] The amount of data to be transferred between pairs of compute nodes is determined based on the task data. Task data refers to the data that needs to be processed in a distributed computing task. When constructing the transfer matrix, the amount of data to be transferred between each pair of compute nodes must be considered. This amount of data may vary depending on the task.

[0092] Step R40 , calculating the ratio of the amount of data to be transmitted to the bandwidth, taking the product of the ratio and the transmission delay as the transmission cost of each computing node pair, and taking the transmission cost as an element of the transmission matrix to obtain the transmission matrix.

[0093] It's important to note that transmission cost is the total cost of data transmission, calculated after taking into account transmission latency and bandwidth. It's typically measured in terms of time or resource consumption. The transmission matrix is ​​a two-dimensional matrix that represents the transmission costs between pairs of computing nodes in a distributed computing terminal.

[0094] Calculate the ratio of the amount of data to be transmitted to the bandwidth, and multiply this ratio by the transmission delay as the transmission cost for each pair of computing nodes. This step is the core of constructing the transmission matrix. The transmission cost reflects the resources and time required for data transmission. By calculating the ratio of the amount of data to be transmitted to the bandwidth, we can determine the time required for data transmission (in ideal conditions, ignoring transmission delay). This time is then added to the transmission delay to obtain the total transmission cost. This cost takes into account both the data transmission speed and the additional delay caused by factors such as network congestion. Using the transmission cost as an element of the transmission matrix, we obtain the transmission matrix. The transmission matrix is ​​a two-dimensional matrix, in which each element represents the transmission cost for a pair of computing nodes. This matrix provides critical data support for subsequent path optimization. By comparing the transmission costs of different paths, the optimal transmission path can be selected to improve the efficiency and reliability of data transmission.

[0095] This embodiment constructs a transmission matrix by identifying computing node pairs, detecting network status, determining the amount of data to be transmitted, and calculating transmission costs. This matrix provides data support for subsequent path optimization and helps achieve efficient data transmission in distributed computing terminals.

[0096] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction, and no further details will be given later. On this basis, the steps of step S30 also include step A10:

[0097] Step A10: For any source node in the transmission matrix, determine all feasible paths from the source node to the target node, find the shortest path that meets the minimum cost condition among all feasible paths, and use the shortest path as the first path, where the cost between any two adjacent nodes in the shortest path is the lowest.

[0098] It should be noted that a feasible path is one or more paths from a source node to a destination node in the transmission matrix. The nodes and connections along these paths are all valid, meaning the transmission costs between them are known. The minimum cost condition is the criterion for selecting the path with the lowest transmission cost among multiple feasible paths as the optimal path. The shortest path is the path from the source node to the destination node that passes through the fewest nodes or has the lowest total transmission cost, while satisfying the minimum cost condition. Adjacent nodes are two adjacent nodes on a path.

[0099] This embodiment aims to find the shortest path with the lowest cost from any source node in the transmission matrix to the destination node. This step is a critical part of the path optimization problem in a distributed computing environment, determining the efficiency and cost of data transmission. First, we need to determine all feasible paths from the source node to the destination node. This typically involves traversing the transmission matrix to find all paths from the source node to the destination node. These paths may contain different combinations of nodes and connections, but they all need to satisfy the condition of known transmission costs. Next, among all feasible paths, we need to find the shortest path that satisfies the minimum cost condition. This step is typically achieved by comparing the total transmission cost of each path. The total transmission cost is the sum of the transmission costs between all adjacent nodes on the path. To find the path with the lowest cost, we can use graph theory algorithms such as Dijkstra's algorithm, Bellman-Ford algorithm, or Floyd-Warshall algorithm. These algorithms can efficiently calculate the shortest paths from the source node to all other nodes and provide the corresponding transmission costs. After finding the shortest path with the lowest cost, we select it as the first path. This path not only meets the minimum cost requirement, but also has the lowest cost between any two adjacent nodes. This means that every step of the data transmission on this path is optimized, ensuring maximum overall transmission efficiency.

[0100] Furthermore, a dynamic programming algorithm can be used to solve the shortest path from the source node to the target node. The core of the dynamic programming algorithm is to decompose the complex problem of solving the shortest path into multiple sub-problems through the state transition equation to gradually construct the global optimal solution. In this example, we regard each node in the network as a state and the path from the source node to the target node as a state sequence. By defining the state transition equation, we can calculate the length of the shortest path from the source node to each intermediate node and gradually expand it to the target node. This process involves a lot of calculations, but thanks to the memoization search feature of the dynamic programming algorithm, we can find the shortest path from the source node to the target node by backtracking and use this path as the first path.

[0101] This example successfully finds an optimal solution to data transmission in a distributed computing environment by identifying feasible paths, calculating the total transmission cost, and comparing and selecting the shortest path with the lowest cost. This solution not only improves data transmission efficiency but also reduces transmission costs, providing strong support for optimizing distributed computing environments.

[0102] In a feasible implementation manner, step S30 further includes steps A20 to A30:

[0103] Step A20, when the network state changes frequently, determining whether the cost of the first path is updated;

[0104] In step A30, if the cost of the first path is updated, the step of optimizing the path between the source node and the target node according to the transmission matrix is ​​re-executed to find the latest shortest path, and the latest shortest path is updated as the first path.

[0105] It's important to note that network status refers to the collection of parameters such as the connection status between nodes in the network, transmission speed, latency, and packet loss rate, which together determine the network's performance and stability. Cost updates refer to changes in the transmission cost along a path due to changes in network status (such as node failures, link congestion, and decreased transmission speeds), necessitating recalculation of the path cost.

[0106] In a network environment, network conditions change dynamically, which may cause the previously determined first path to no longer be the optimal path. Therefore, this embodiment proposes a dynamic path optimization mechanism to adapt to changing network conditions. First, the system needs to monitor changes in network conditions and determine whether these changes may result in an update to the cost of the first path. This typically involves continuous monitoring of network conditions and real-time monitoring of parameters that may affect path costs, such as latency and packet loss rate. If the system detects that the cost of the first path may have updated, it triggers a path optimization step. This step is similar to the previous path optimization process, but now uses an updated transmission matrix (reflecting changes in network conditions). Based on the new transmission matrix, the system recalculates the costs of all feasible paths between the source and destination nodes and finds the shortest path with the lowest cost. After finding the latest shortest path, the system updates it as the first path. This means that subsequent data transmission will follow this new shortest path to ensure optimal transmission efficiency and cost. It should be noted that due to the frequent changes in network conditions, this path optimization process may need to be performed frequently. Therefore, the system needs to have efficient path calculation and optimization capabilities to quickly respond to changes in network conditions and ensure stable and efficient data transmission.

[0107] Furthermore, the ant colony algorithm (ACO) can be introduced for dynamic path optimization. The ACO simulates the foraging behavior of ants to find the optimal path. In the algorithm, each ant represents a potential path option. They choose their next move based on the "pheromone" on their current path (i.e., the quality of the path, which can be calculated based on the elements in the transmission matrix). As ants move through the network, the pheromone on the path accumulates, guiding more ants to choose more optimal paths. This process represents a positive feedback mechanism, allowing the algorithm to gradually approach the global optimal solution. In the ACO, we define a pheromone update rule and an ant movement rule. The pheromone update rule determines how the quality of the path is updated, while the ant movement rule determines how the ants choose their next move. After multiple iterations, the algorithm converges to a path with the highest pheromone concentration, which is the new optimal path. Notably, the ACO also incorporates heuristic information to guide the ants' search direction. This heuristic information, calculated based on the elements in the transmission matrix, reflects the expected cost from the current node to the target node. By combining heuristic information and pheromones, we can better guide ants to move toward the optimal path.

[0108] The embodiment achieves dynamic optimization of data transmission paths in a network environment by monitoring changes in network status and re-performing the path optimization steps when necessary. This mechanism not only improves the efficiency and stability of data transmission, but also enhances the network's adaptability to dynamic changes.

[0109] In a feasible implementation manner, step S30 further includes steps B10 to B20:

[0110] Step B10: Construct an objective function that minimizes transmission cost based on the transmission matrix

[0111] It’s important to note that an objective function is a mathematical expression or function used to evaluate the pros and cons of different solutions to an optimization problem. In this case, the objective function aims to minimize the transmission cost.

[0112] An objective function is constructed based on the transmission matrix. The transmission matrix is ​​a prerequisite, as it contains information about the cost of transmitting data between all pairs of nodes in the network. These costs can be specific numerical values ​​for performance indicators such as latency, bandwidth consumption, and packet loss rate. The objective function is constructed based on this cost information to quantify the total cost of transmitting data from the source node to the destination node. The objective function can be linear or nonlinear, depending on how the transmission cost is calculated and the complexity of the optimization problem. This objective function is used to evaluate the transmission costs of different paths. This typically involves extracting and calculating the relevant elements in the transmission matrix. For each possible path, the total transmission cost is calculated based on the corresponding cost information in the transmission matrix. This process may require iterating over multiple elements of the transmission matrix and performing appropriate mathematical operations.

[0113] Step B20 : Evaluate the transmission costs of the paths according to the objective function, and select the path with the lowest transmission cost as the first path.

[0114] It should be noted that the transmission cost is calculated based on the elements in the transmission matrix and represents the total cost or price required to transmit data from the source node to the destination node. The first path is the optimal path selected from multiple possible paths based on a specific criterion (such as the lowest transmission cost).

[0115] The path with the lowest transmission cost is selected as the first path. After evaluating the transmission costs of all possible paths, we can compare these costs and find the one with the lowest cost. The path corresponding to this minimum cost is the first path we are looking for. The selection of the first path is based on a quantitative comparison of transmission costs, and therefore represents a solution to the optimization problem, representing the path with the lowest cost for transmitting data from the source node to the destination node.

[0116] For example, the objective function is calculated as:

[0117] min∑ (i,j) M trans (i,j) formula (3)

[0118] Among them, i and j are positive integers, M trans (i, j) is the cost of transmitting from source node i to target node j. The minimum cost of the entire transmission path can be calculated using formula (3).

[0119] By constructing a transmission cost minimization objective function based on the transmission matrix, this embodiment accurately quantifies and compares the transmission costs of different paths from the source node to the destination node. This method not only considers multiple performance indicators such as latency, bandwidth consumption, and packet loss rate, but also ensures that the first path selected has the lowest transmission cost by traversing the transmission matrix and calculating the total cost of each path. This beneficial effect not only improves data transmission efficiency but also reduces overall costs, providing strong support for network optimization and resource allocation.

[0120] Based on the first or second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first or second embodiment can be referred to the above introduction and will not be described in detail. Steps C10 to C30 are also included after step S40:

[0121] Step C10: select computing nodes whose distance is less than a preset distance threshold, whose delay is less than a preset delay threshold, and which have no dependency on other computing nodes as first-level computing nodes;

[0122] It should be noted that the preset distance threshold is a critical value used to screen the distance between computing nodes when building a multi-level aggregation structure. Only computing nodes with a distance less than the threshold are considered potential hierarchical connection nodes. The preset delay threshold is a critical value used to evaluate the data transmission delay between computing nodes when building a multi-level aggregation structure. Only computing nodes with a delay less than the threshold are considered valid connection nodes. No dependency means that there is no order or dependency in data or calculation between computing nodes, that is, the calculation or data transmission of one node does not depend on the completion of another node. The first-level computing nodes are the set of computing nodes that are first selected when building a multi-level aggregation structure, which meet the preset distance and delay thresholds and have no dependencies. They constitute the top layer of the multi-level aggregation structure.

[0123] Based on preset distance and latency thresholds, we select compute nodes that meet the criteria as first-tier compute nodes. These nodes must be closer than the preset distance threshold, have a latency lower than the preset latency threshold, and have no dependencies. These nodes form the top layer of the multi-level aggregation structure, providing the foundation for building subsequent layers.

[0124] Step C20: For any level after the first level, select the computing node closest to the previous level and with a delay less than a preset delay index according to the first path as the computing node of the current level, thereby obtaining a multi-level aggregation structure, wherein the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node;

[0125] It should be noted that when constructing a multi-level aggregation structure, each subsequent level, except the first, is constructed based on the selection logic of the previous level. The target node is the computing node that serves as the end point of data transmission or calculation in the multi-level aggregation structure.

[0126] For subsequent layers, we follow the first-path selection logic and select the closest compute node from the previous layer with latency below a preset latency threshold as the compute node for the next layer. This process is recursive until the final layer, where the target node, is selected. This ensures that connections between each layer are based on the optimal path, thereby optimizing data transmission and computation efficiency.

[0127] Step C30: If the delay of any first computing node is greater than or equal to a preset delay index, the first computing node is replaced with a preset standby node.

[0128] It should be noted that in this embodiment, the multi-level aggregation structure is a network structure composed of multiple hierarchical computing nodes connected by optimal paths, designed to optimize data transmission and computing efficiency. Position updating involves adjusting the hierarchical position of a computing node within the structure to optimize overall performance if its latency does not meet the requirements during the construction of the multi-level aggregation structure.

[0129] If, while building a multi-level aggregation structure, we discover that the latency of a primary compute node is greater than or equal to a preset latency metric, we need to update the node's position within the structure. This typically means moving the node to a level closer to the target node or replacing it with another node that meets the criteria. This process is dynamic and aims to continuously optimize the performance of the multi-level aggregation structure.

[0130] In this embodiment, preset distance and latency thresholds form the basis for selecting compute nodes. The first path selection logic is key to connecting compute nodes at different levels. The location update operation is an important means of optimizing the performance of the multi-level aggregation structure. These steps together constitute the core of this embodiment, aiming to build an efficient and flexible multi-level aggregation structure to optimize data transmission and computation efficiency.

[0131] In one possible embodiment, referring to Figure 3 , step S50 further includes steps D10 to D20:

[0132] Step D10: inputting the task data into the first-level computing nodes in the multi-level aggregation structure, aggregating the task data through the first-level computing nodes to obtain an aggregation result;

[0133] It's important to note that first-level compute nodes are the first compute nodes to receive and process data within a multi-level aggregation structure. These nodes are typically located at the top level of the aggregation structure and are responsible for initial data aggregation operations. The initial aggregation result is the intermediate result obtained by aggregating the input data at the first-level compute nodes. This result serves as input for subsequent aggregation levels.

[0134] Data is transmitted to the first-tier computing nodes for preliminary aggregation. After receiving the data, these nodes process it according to pre-set aggregation logic (such as summation and averaging) to obtain preliminary aggregation results. This result forms the basis for subsequent aggregation.

[0135] Step D20: For any computing node of the first level after the computing node of the first level, the aggregation result of the previous level is aggregated again to obtain the aggregation result of the current level, until it is aggregated to the target node in the multi-level aggregation structure, wherein the aggregation result of the current level is stored in the computing node of the current level that is closest to the next level.

[0136] It should be noted that the target node is the computing node responsible for receiving and storing the final aggregation results in the multi-level aggregation structure. This node is usually the last or designated layer of the aggregation structure. The aggregation result of the current level is the intermediate result obtained on each level of computing node based on the aggregation result of the previous level and the aggregation logic of the current level node. This result will serve as the input for the aggregation of the next level. The computing node closest to the next level is the node with the closest network connection to the computing node of the next level in the multi-level aggregation structure (such as close physical distance and low network latency). It is used to store the aggregation results of the current level for rapid transmission to the next level.

[0137] Initial aggregation results are further aggregated through a multi-level aggregation structure. At each level, compute nodes combine the aggregation results of the previous level and the aggregation logic of the current level nodes to produce the current level's aggregation results. This result reflects the node's further understanding and processing of the data. Importantly, to optimize data transmission efficiency and reduce latency, the current level's aggregation results are stored in the compute nodes closest to the next level. This ensures that data can be transmitted to the next level for computation as quickly as possible, thereby improving the efficiency of the entire aggregation process.

[0138] Finally, after multiple levels of aggregation, the final aggregated results are transmitted to the target node. This target node is the final or designated layer of the multi-level aggregation structure and is responsible for receiving and storing the final aggregated results. This result represents a comprehensive understanding and processing of the input data and can be used for subsequent data analysis, decision support, and other tasks.

[0139] For example, the calculation method of the final aggregation result is expressed as formula (4)

[0140] G(l)=∑ i∈Level(l) Data i Formula (4)

[0141] Among them, G(l) is the total aggregation result, i and l are positive integers, Level(l) refers to the lth level, i∈Level(l) refers to the i computing node of the lth level, Data i Refers to the data processed by the i computing node.

[0142] This embodiment provides an efficient data transmission and aggregation mechanism through a multi-level aggregation structure, and also reduces data processing latency and costs by optimizing data storage and transmission paths. This makes the method of this embodiment have significant advantages when processing large-scale data sets.

[0143] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the distributed data transmission method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0144] This application also provides a distributed data transmission device, please refer to Figure 4 , the distributed data transmission device includes:

[0145] Determine computing node module 10, determine multiple computing nodes in the distributed computing terminal;

[0146] A transmission matrix construction module 20 constructs a transmission matrix according to transmission delay, bandwidth, and amount of data to be transmitted between computing nodes, wherein the computing nodes include source nodes and target nodes;

[0147] A path optimization module 30 performs path optimization on a path between a source node and a target node according to a transmission matrix to obtain a first path;

[0148] An aggregation structure construction module 40 divides all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0149] The aggregation transmission module 50 transmits the preset task data to the target node through a multi-level aggregation structure.

[0150] The distributed data transmission device provided in this application utilizes the distributed data transmission method of the aforementioned embodiment to improve the data transmission efficiency of distributed computing terminals. Compared to the prior art, the distributed data transmission device provided in this application has the same beneficial effects as the distributed data transmission method provided in the aforementioned embodiment. Other technical features of the distributed data transmission device are the same as those disclosed in the aforementioned embodiment and are not further described here.

[0151] The present application provides a distributed data transmission device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed data transmission method in the above-mentioned embodiment one.

[0152] Reference below Figure 5 , which shows a schematic diagram of the structure of a distributed data transmission device suitable for implementing the embodiments of the present application. The distributed data transmission device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The distributed data transmission device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0153] like Figure 5As shown, the distributed data transmission device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the distributed data transmission device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. The communication device 1009 can allow the distributed data transmission device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a distributed data transmission device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have alternatively.

[0154] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0155] The distributed data transmission device provided in this application utilizes the distributed data transmission method of the aforementioned embodiment to improve the data transmission efficiency of distributed computing terminals. Compared to the prior art, the distributed data transmission device provided in this application has the same beneficial effects as the distributed data transmission method provided in the aforementioned embodiment. Other technical features of the distributed data transmission device are the same as those disclosed in the aforementioned embodiment and are not further described here.

[0156] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0157] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0158] The present application provides a medium, which is a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the distributed data transmission method in the above embodiment.

[0159] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0160] The computer-readable storage medium may be included in the distributed data transmission device; or it may exist independently without being assembled into the distributed data transmission device.

[0161] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the distributed data transmission device, the distributed data transmission device:

[0162] Determining multiple computing nodes in a distributed computing terminal;

[0163] Constructing a transmission matrix based on transmission delay, bandwidth, and amount of data to be transmitted between computing nodes, where computing nodes include source nodes and destination nodes;

[0164] Optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0165] Divide all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0166] The preset task data is transmitted to the target node through a multi-level aggregation structure.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0168] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0169] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0170] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described distributed data transmission method, thereby improving the data transmission efficiency of distributed computing terminals. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the distributed data transmission method provided in the above-described embodiments, and are not further elaborated here.

[0171] The present application also provides a product, which is a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the distributed data transmission method as described above are implemented.

[0172] The computer program product provided in this application can improve the data transmission efficiency of distributed computing terminals. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the distributed data transmission method provided in the above embodiment, and will not be repeated here.

[0173] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A distributed data transmission method, characterized in that: Applied to a distributed computing terminal, the distributed data transmission method includes: Determining a plurality of computing nodes in the distributed computing terminal; Constructing a transmission matrix based on transmission delay, bandwidth, and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes; Optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path; The step of performing path optimization on the path between the source node and the target node according to the transmission matrix to obtain a first path includes: For any source node in the transmission matrix, determine all feasible paths from the source node to the target node, find the shortest path that satisfies a minimum cost condition among all the feasible paths, and use the shortest path as the first path, wherein the cost between any two adjacent nodes in the shortest path is the lowest; Dividing all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel; The step of dividing all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure includes: The computing nodes whose distance is less than a preset distance threshold, whose delay is less than a preset delay threshold and which have no dependency on other computing nodes are regarded as first-level computing nodes; For any level after the first level, the computing node closest to the previous level and with a delay less than a preset delay index is selected according to the first path as the computing node of the current level, thereby obtaining a multi-level aggregation structure, wherein the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node; Inputting the task data into a first-level computing node in a multi-level aggregation structure, and aggregating the task data through the first-level computing node to obtain an aggregation result; For any computing node of a level after the first-level computing node, the aggregation result of the previous level is aggregated again to obtain the aggregation result of the current level, until it is aggregated to the target node in the multi-level aggregation structure, wherein the aggregation result of the current level is stored in the computing node of the current level that is closest to the next level.

2. The distributed data transmission method according to claim 1, wherein: After the step of optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path, the following steps are included: When the network state changes frequently, determining whether the cost of the first path is updated; If the cost of the first path is updated, the step of optimizing the path between the source node and the target node according to the transmission matrix is ​​re-executed to find the latest shortest path, and the latest shortest path is updated as the first path.

3. The distributed data transmission method according to claim 1, wherein: After the step of dividing all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, the following steps are included: If the delay of any first computing node is greater than or equal to a preset delay index, the first computing node is replaced with a preset standby node.

4. The distributed data transmission method according to claim 1, wherein: The step of constructing a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes comprises: Determine all computing node pairs, wherein the computing node pairs consist of two computing nodes; For each computing node pair, detecting a network status to obtain a transmission delay and a bandwidth between the computing node pair; determining, based on the task data, an amount of data to be transmitted between the computing node pairs; The ratio of the amount of data to be transmitted to the bandwidth is calculated, the product of the ratio and the transmission delay is used as the transmission cost of each computing node pair, and the transmission cost is used as an element of the transmission matrix to obtain the transmission matrix.

5. A distributed data transmission device, characterized in that: Set up in a distributed computing terminal, the distributed data transmission device includes: Determine a computing node module to determine multiple computing nodes in the distributed computing terminal; A transmission matrix construction module is configured to construct a transmission matrix according to transmission delay, bandwidth, and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes; A path optimization module is configured to optimize the path between the source node and the target node according to the transmission matrix to obtain a first path, wherein optimizing the path between the source node and the target node according to the transmission matrix to obtain the first path includes: For any source node in the transmission matrix, determine all feasible paths from the source node to the target node, find the shortest path that satisfies a minimum cost condition among all the feasible paths, and use the shortest path as the first path, wherein the cost between any two adjacent nodes in the shortest path is the lowest; An aggregation structure construction module divides all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel, wherein dividing all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure includes: The computing nodes whose distance is less than a preset distance threshold, whose delay is less than a preset delay threshold and which have no dependency on other computing nodes are regarded as first-level computing nodes; For any level after the first level, the computing node closest to the previous level and with a delay less than a preset delay index is selected according to the first path as the computing node of the current level, thereby obtaining a multi-level aggregation structure, wherein the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node; an aggregation transmission module, which inputs the task data into the first-level computing nodes in the multi-level aggregation structure, aggregates the task data through the first-level computing nodes, and obtains an aggregation result; For any computing node of a level after the first-level computing node, the aggregation result of the previous level is aggregated again to obtain the aggregation result of the current level, until it is aggregated to the target node in the multi-level aggregation structure, wherein the aggregation result of the current level is stored in the computing node of the current level that is closest to the next level.

6. A distributed data transmission device, characterized in that: The distributed data transmission device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the distributed data transmission method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the distributed data transmission method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the distributed data transmission method according to any one of claims 1 to 4 are implemented.

Images