Distributed data transmission method, device, equipment, medium and product

Abstract

The invention discloses a distributed data transmission method, device and equipment, a medium and a product, and relates to the technical field of data transmission, the distributed data transmission method comprises the following steps: determining a plurality of computing nodes in a distributed computing terminal; a transmission matrix is constructed according to the transmission delay, the bandwidth and the to-be-transmitted data volume between computational nodes, and the computational nodes comprise a source node and a target node; performing path optimization on a path between the source node and the target node according to the transmission matrix to obtain a first path; all the computing nodes are divided into multiple levels according to the first path, a multi-level aggregation structure is obtained, and the computing nodes in each level process data in parallel; and transmitting the preset task data to the target node through the multi-stage aggregation structure. The data transmission efficiency of the distributed computing terminal is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to distributed data transmission methods, devices, equipment, media and products. Background Art

[0002] In modern computing environments, distributed computing systems are widely used to process large-scale data sets and perform complex computing tasks. Such systems are usually composed of multiple heterogeneous devices, including but not limited to CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array) and other high-performance computing units, which are interconnected through a network to work together. The key advantage of distributed computing systems is that they can utilize the parallel processing capabilities of multiple devices, thereby significantly improving computing efficiency and processing speed. However, in practical applications, the performance of distributed computing systems is often limited by the bottleneck of data transmission. Especially in the field of deep learning hardware acceleration and high-performance computing architecture, data transmission latency and bandwidth occupancy have become important factors affecting overall performance. For example, in large-scale computing tasks (such as deep learning training and scientific simulation), frequent data exchange between nodes will lead to communication bottlenecks, which will significantly reduce the throughput and response speed of the system. Most traditional systems use static routing or simple load balancing strategies, and cannot dynamically adjust the data transmission path. Due to the large amount of data and frequent transmission, the bandwidth is occupied and the delay is uncontrollable. This problem is more prominent when the task complexity and the number of nodes increase. In addition, the data aggregation method is single and lacks hierarchical optimization, which makes the node communication pressure concentrated on a single aggregation point, easily forming a transmission bottleneck, resulting in low transmission efficiency between distributed computing terminals, limiting the scalability and overall performance of the system. In particular, in the face of growing data volumes and complex computing needs, traditional methods are difficult to meet the requirements of high-performance computing, and there is an urgent need for a method that can improve the data transmission efficiency of distributed computing terminals. Summary of the invention

[0003] The main purpose of this application is to provide a distributed data transmission method, device, equipment, medium and product, aiming to improve the data transmission efficiency of distributed computing terminals.

[0004] To achieve the above objectives, the present application proposes a distributed data transmission method, which is applied to a distributed computing terminal, comprising:

[0005] Determining a plurality of computing nodes in a distributed computing terminal;

[0006] Constructing a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between computing nodes, wherein the computing nodes include source nodes and target nodes;

[0007] Optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0008] According to the first path, all computing nodes are divided into multiple levels to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0009] The preset task data is transmitted to the target node through a multi-level aggregation structure.

[0010] In one embodiment, the step of performing path optimization on the path between the source node and the target node according to the transmission matrix to obtain the first path includes:

[0011] For any source node in the transmission matrix, all feasible paths from the source node to the target node are determined, the shortest path that meets the minimum cost condition is found among all feasible paths, and the shortest path is taken as the first path, wherein the cost between any two adjacent nodes in the shortest path is the lowest.

[0012] In one embodiment, after the step of optimizing the path between the source node and the target node according to the transmission matrix to obtain the first path, the following steps are included:

[0013] When the network state changes frequently, determining whether the cost of the first path is updated;

[0014] If the cost of the first path is updated, the step of optimizing the path between the source node and the target node according to the transmission matrix is ​​re-executed to find the latest shortest path, and the latest shortest path is updated as the first path.

[0015] In one embodiment, the step of dividing all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure includes:

[0016] The computing nodes whose distance is less than a preset distance threshold, whose delay is less than a preset delay threshold and which have no dependency on other computing nodes are regarded as the first-level computing nodes;

[0017] For any level after the first level, a computing node that is closest to the previous level and whose delay is less than a preset delay index is selected according to the first path as the computing node of this level, and a multi-level aggregation structure is obtained, in which the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node;

[0018] The step of dividing all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure includes:

[0019] If the delay of any first computing node is greater than or equal to the preset delay index, the first computing node is replaced with a preset standby node.

[0020] In one embodiment, the step of transmitting the preset task data to the target node through the multi-level aggregation structure includes:

[0021] Inputting the task data into the first-level computing nodes in the multi-level aggregation structure, aggregating the task data through the first-level computing nodes to obtain an aggregation result;

[0022] For any computing node of the first level after the computing node of the first level, the aggregation result of the previous level is aggregated again to obtain the aggregation result of the current level, until it is aggregated to the target node in the multi-level aggregation structure, wherein the aggregation result of the current level is stored in the computing node of the current level that is closest to the next level.

[0023] In one embodiment, the step of constructing a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes includes:

[0024] Determine all computing node pairs, wherein a computing node pair consists of two computing nodes;

[0025] For each computing node pair, detect the network status to obtain the transmission delay and bandwidth between the computing node pairs;

[0026] Determine the amount of data to be transmitted between computing node pairs based on the task data;

[0027] The ratio of the amount of data to be transmitted to the bandwidth is calculated, the product of the ratio and the transmission delay is used as the transmission cost of each computing node pair, and the transmission cost is used as an element of the transmission matrix to obtain the transmission matrix.

[0028] In addition, to achieve the above-mentioned purpose, the present application also proposes a distributed data transmission device, which is arranged in a distributed computing terminal, and the distributed data transmission device includes:

[0029] Determine a computing node module, and determine a plurality of computing nodes in a distributed computing terminal;

[0030] A transmission matrix construction module is used to construct a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include the source node and the target node;

[0031] A path optimization module optimizes the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0032] An aggregation structure construction module divides all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0033] The aggregation transmission module transmits the preset task data to the target node through a multi-level aggregation structure.

[0034] In addition, to achieve the above objectives, the present application also proposes a distributed data transmission device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the distributed data transmission method as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the distributed data transmission method as described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a product, which is a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the distributed data transmission method as described above are implemented.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] The present application is applied to a distributed computing terminal, determines multiple computing nodes in the distributed computing terminal, and constructs a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes, and the matrix comprehensively reflects the transmission cost information in the network, providing key data support for path optimization; optimizes the path between the source node and the target node according to the transmission matrix to obtain a first path, which minimizes the transmission cost while ensuring the quality of data transmission, thereby improving the efficiency of data transmission; divides all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel, and this structure not only improves the parallelism of data processing, but also optimizes the hierarchical structure of data transmission, further improving the overall computing efficiency; transmits the preset task data to the target node through the multi-level aggregation structure, and this process makes full use of the optimized path and hierarchical structure to achieve high-speed and low-latency data transmission, effectively solving the defect of low transmission efficiency of distributed computing terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of the first embodiment of the distributed data transmission method of the present application;

[0042] Figure 2 A schematic diagram of a distributed data transmission method for this application;

[0043] Figure 3 This is another flow chart of the distributed data transmission method of the present application;

[0044] Figure 4 This is a schematic diagram of the module structure of a distributed data transmission device according to an embodiment of the present application;

[0045] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the distributed data transmission method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a terminal system, etc. The following takes the system as an example to illustrate this embodiment and the following embodiments.

[0050] Based on this, this embodiment provides a distributed data transmission method, which is applied to distributed computing terminals.

[0051] In this embodiment, refer to Figure 1 , Figure 1 This is a flow chart of the distributed data transmission method of the present application. The distributed data transmission method includes steps S10 to S50:

[0052] Step S10, determining a plurality of computing nodes in a distributed computing terminal;

[0053] Step S20, constructing a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes;

[0054] It should be noted that a distributed computing terminal refers to a computing system composed of multiple computing nodes connected through a network. These computing nodes can work together to complete tasks and improve computing efficiency. The computing node is the basic unit in the distributed computing terminal. It can be a server, personal computer or other computing device. Each node has data processing capabilities. The source node is the node that generates data and prepares to send it during the data transmission process. The target node is the node that receives data during the data transmission process.

[0055] In a distributed computing system, first of all, it is necessary to determine all nodes involved in the calculation, which include at least one source node (node ​​that generates data) and at least one target node (node ​​that receives data). These nodes are connected through a network and together form a distributed computing terminal.

[0056] It should be noted that the transmission matrix is ​​a mathematical model used to describe key information such as transmission delay, bandwidth and amount of data to be transmitted between source nodes and target nodes.

[0057] The construction of the transmission matrix is ​​based on the transmission delay, bandwidth and amount of data to be transmitted between the source node and the target node. Transmission delay refers to the time required for data to be transmitted from the source node to the target node; bandwidth represents the capacity of network transmission, that is, the amount of data that can be transmitted per second; and the amount of data to be transmitted is the amount of data that the source node needs to send to the target node. This information is determined by real-time monitoring of network status and computing tasks, and then constructed into a matrix containing transmission delay, bandwidth and amount of data to be transmitted. This matrix will serve as the basis for path optimization.

[0058] Exemplarily, the calculation method of the transmission matrix is ​​expressed as formula (1)

[0059]

[0060] Among them, M trans (i,j) is the transmission matrix, Latency i,j is the transmission delay from computing node i to computing node j, Bandwidth i,j is the bandwidth from computing node i to computing node j, Data_Size i,j is the amount of data that needs to be transmitted from computing node i to computing node j.

[0061] Step S30, optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0062] It should be noted that path optimization is to adjust the data transmission path through an algorithm to reduce transmission delay, improve bandwidth utilization, etc. The first path is the best data transmission path obtained after path optimization.

[0063] The purpose of path optimization is to find the best transmission path from the source node to the target node to minimize the total transmission overhead of the system. This usually involves the construction of an objective function that takes into account multiple factors such as transmission delay, bandwidth utilization, and the amount of data to be transmitted. Use a dynamic programming algorithm (such as Dijkstra or Floyd-Warshall) to calculate the shortest path between nodes, while considering that the data of each task must be fully transmitted and the path cannot exceed the available bandwidth limit. If the network status changes frequently, a dynamic optimization method based on an ant colony algorithm can also be used to continuously adjust the transmission path. Through path optimization, we obtain an optimal path from the source node to the target node, namely the first path.

[0064] Step S40, dividing all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0065] It should be noted that the multi-level aggregation structure is a data aggregation strategy that divides computing nodes into multiple levels and aggregates data level by level to reduce the amount of data transmission. Parallel processing is the simultaneous processing of data by multiple computing nodes to increase data processing speed.

[0066] The multi-level aggregation structure is designed to further reduce the amount of data transmission and improve the scalability of the system. It divides all computing nodes into multiple levels and completes data aggregation step by step. In each level, nodes process data independently and in parallel to reduce the total amount of backhaul. The results of each level of aggregation will be used as the input of the next level of aggregation until the global result is obtained after the top-level aggregation is completed. The selection of aggregation nodes is based on factors such as distance and latency, and nodes with shorter distances and lower latency are preferred as first-level aggregation nodes. At the same time, the position of the aggregation nodes is dynamically adjusted according to the real-time transmission matrix to cope with changes in network status.

[0067] Step S50: transmitting the preset task data to the target node through the multi-level aggregation structure.

[0068] It should be noted that task data specifically refers to various information involved in the execution of distributed computing tasks. These task data can be original input data, intermediate calculation results, final output results, etc. In distributed computing, task data usually needs to be transmitted and shared between multiple computing nodes for collaborative computing and processing. During the transmission process of distributed computing terminals, task data can be in a variety of formats, including text data, image data, audio data, video data, etc., depending on the requirements of the task and the nature of the data.

[0069] After determining the best transmission path and multi-level aggregation structure, the preset task data will be transmitted according to these paths and structures. During the transmission process, the data will pass through multiple aggregation nodes, and the data will be aggregated and compressed step by step. Each level of aggregation will reduce the total amount of data, thereby avoiding excessive data transmission in the next level. During the transmission process, the data will be processed and aggregated in the computing nodes in each level in turn according to the hierarchical relationship of the multi-level aggregation structure, and finally reach the target node.

[0070] For example, assuming there are source nodes A, B, C, D, and target node E, the transmission matrix M trans Expressed as formula (2), the node data volume is: A: 100MB, B: 200MB, C: 150MB, D: 50MB.

[0071]

[0072] Use Dijkstra algorithm to optimize the path, and get the first path as A→B, B→D, C→D, D→E. Divide all computing nodes into multiple levels according to the first path. Figure 2 The first-level computing nodes are A and C, the second-level computing nodes are B and D, and the third-level computing node is the target node E. Therefore, the data is aggregated to B through A, and B obtains data A+B. At the same time, it is aggregated to D through C, and D obtains data C+D. Finally, the data is aggregated to E through B and D of the second-level computing nodes, and E obtains data A+B+C+D. Through multi-level aggregation and path optimization, the total transmission volume is reduced by 40%.

[0073] For example, for GPU nodes G1, G2, CPU nodes C1, C2, real-time delay monitoring of dynamic changes in the transmission matrix. GPU nodes prioritize computationally intensive aggregation, CPU nodes handle lightweight transmission, and real-time adjustment of the transmission matrix to optimize the path. After dynamically optimizing the path, the delay is reduced by 30%, and the total transmission volume is reduced by 25%.

[0074] This embodiment is applied to a distributed computing terminal, and multiple computing nodes in the distributed computing terminal are determined. A transmission matrix is ​​constructed according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes. The matrix fully reflects the transmission cost information in the network and provides key data support for path optimization. The path between the source node and the target node is optimized according to the transmission matrix to obtain a first path, which minimizes the transmission cost while ensuring the quality of data transmission, thereby improving the efficiency of data transmission. All computing nodes are divided into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel. This structure not only improves the parallelism of data processing, but also optimizes the hierarchical structure of data transmission, thereby further improving the overall computing efficiency. The preset task data is transmitted to the target node through the multi-level aggregation structure. This process makes full use of the optimized path and hierarchical structure to achieve high-speed and low-latency data transmission, thereby effectively solving the defect of low transmission efficiency of the distributed computing terminal.

[0075] In a feasible implementation, step S50 may include steps T10 to T20:

[0076] Step T10, compressing the data using a preset lightweight compression algorithm to obtain compressed data;

[0077] It should be noted that a lightweight compression algorithm is a data processing technology designed to reduce the size of data while maintaining data integrity or acceptable loss. It is usually faster and more efficient than traditional compression algorithms and is suitable for environments with strict requirements on real-time performance or resource consumption. Compressing data is to convert original data into a smaller data form by applying a compression algorithm. Compressed data retains all or most of the information of the original data, but takes up less storage space or transmission bandwidth.

[0078] Apply a preset lightweight compression algorithm to compress the raw data. This process usually involves analyzing the characteristics of the data, such as repetitiveness, redundancy, predictability, etc., and using these characteristics to reduce the overall size of the data. The advantage of lightweight compression algorithms is their efficiency and low resource consumption. They can significantly reduce the size of data without sacrificing too much data quality. By applying this algorithm, we get compressed data that is smaller in size but still contains the core information of the original data.

[0079] Step T20, transmitting the compressed data to the target node through the multi-level aggregation structure.

[0080] The compressed data is transmitted to the target node through a multi-level aggregation structure. In this process, the compressed data is divided into multiple data blocks or data packets and transmitted according to the hierarchical relationship of the multi-level aggregation structure. The computing nodes at each layer receive the data from the previous layer, perform necessary processing (such as further compression, format conversion, etc.), and then pass the processed data to the next layer. This layer-by-layer transmission and processing method ensures the reliability and integrity of the data while improving the transmission efficiency. The multi-level aggregation structure plays a key role in this process. It not only provides an efficient data transmission channel, but also reduces the redundancy and complexity of the data through layer-by-layer aggregation. During the transmission process, the computing nodes at each layer can optimize and process the data to ensure that the data that finally reaches the target node is efficient and accurate. Finally, the compressed data is successfully transmitted to the target node. At the target node, the data can be further processed, analyzed or stored.

[0081] Since this embodiment adopts a lightweight compression algorithm and a multi-level aggregation structure, the entire process not only improves the efficiency and reliability of data transmission, but also reduces resource consumption and costs. This makes the method of this embodiment have significant advantages when processing large-scale data sets or real-time data streams.

[0082] In a feasible implementation, step S50 may include steps R10 to R40:

[0083] Step R10, determining all computing node pairs, wherein a computing node pair consists of two computing nodes;

[0084] It should be noted that a computing node pair is a pair of two computing nodes in a distributed computing environment, and there may be a need for data transmission between the two nodes.

[0085] Determine all computing node pairs. In a distributed computing environment, there are multiple computing nodes, and each node may transmit data to other nodes. Therefore, it is necessary to pair these nodes in pairs to form computing node pairs. This step is the basis for building a transmission matrix.

[0086] Step R20, for each computing node pair, detecting the network status to obtain the transmission delay and bandwidth between the computing node pairs;

[0087] It should be noted that the network status refers to various parameters and indicators that describe the current operation status of the network, including but not limited to transmission delay, bandwidth, etc. Transmission delay is the time required for data to be transmitted in the network, usually due to factors such as network distance, network congestion, and device performance. Bandwidth is the amount of data that can be transmitted per unit time on the network and is an important indicator for measuring network transmission capacity.

[0088] For each pair of computing nodes, the network status is detected to obtain the transmission delay and bandwidth between them. Network status detection is a complex process involving the measurement and analysis of multiple network parameters. In this embodiment, the transmission delay and bandwidth are mainly concerned. The transmission delay reflects the speed at which data is transmitted in the network, while the bandwidth determines the amount of data that can be transmitted in the network per unit time. These two parameters together determine the efficiency and cost of data transmission.

[0089] Step R30, determining the amount of data to be transmitted between the computing node pairs according to the task data;

[0090] It should be noted that the amount of data to be transmitted is the amount of data that needs to be transmitted from the source node to the target node in a distributed computing task.

[0091] The amount of data to be transmitted between computing node pairs is determined based on the task data. Task data refers to the data that needs to be processed in the distributed computing task. When constructing the transmission matrix, the amount of data that needs to be transmitted between each computing node pair needs to be considered. This amount of data may vary depending on the task.

[0092] Step R40, calculate the ratio of the amount of data to be transmitted to the bandwidth, use the product of the ratio and the transmission delay as the transmission cost of each computing node pair, use the transmission cost as an element of the transmission matrix, and obtain the transmission matrix.

[0093] It should be noted that the transmission cost is the total cost required for data transmission after considering the transmission delay and bandwidth, which is usually measured in terms of time or resource consumption. The transmission matrix is ​​a two-dimensional matrix used to represent the transmission cost between each pair of computing nodes in a distributed computing terminal.

[0094] Calculate the ratio of the amount of data to be transmitted to the bandwidth, and use the product of this ratio and the transmission delay as the transmission cost of each computing node pair. This step is the core of the transmission matrix construction. The transmission cost reflects the resources and time consumption required for data transmission. By calculating the ratio of the amount of data to be transmitted to the bandwidth, the time required for data transmission can be obtained (in an ideal case, that is, without considering the transmission delay). Then, add this time to the transmission delay to get the total transmission cost. This cost takes into account both the speed of data transmission and the additional delay caused by factors such as network congestion. Use the transmission cost as an element of the transmission matrix to obtain the transmission matrix. The transmission matrix is ​​a two-dimensional matrix, in which each element represents the transmission cost of a computing node pair. This matrix provides key data support for subsequent path optimization. By comparing the transmission costs of different paths, the optimal transmission path can be selected to improve the efficiency and reliability of data transmission.

[0095] This embodiment constructs a transmission matrix by determining computing node pairs, detecting network status, determining the amount of data to be transmitted, calculating transmission costs, etc. This matrix provides data support for subsequent path optimization and helps to achieve efficient data transmission in distributed computing terminals.

[0096] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction, and will not be repeated hereafter. On this basis, the step S30 also includes step A10:

[0097] Step A10, for any source node in the transmission matrix, determine all feasible paths from the source node to the target node, find the shortest path that meets the minimum cost condition among all feasible paths, and use the shortest path as the first path, wherein the cost between any two adjacent nodes in the shortest path is the lowest.

[0098] It should be noted that a feasible path is one or more paths from a source node to a target node in a transmission matrix, and the nodes and connections on these paths are valid, that is, the transmission costs between them are known. The minimum cost condition is to select the path with the minimum transmission cost as the criterion for the optimal path among multiple feasible paths. The shortest path is the path with the least number of nodes or the minimum total transmission cost from a source node to a target node under the premise of meeting the minimum cost condition. Adjacent nodes are two nodes arranged adjacent to each other on a path.

[0099] This embodiment aims to find the shortest path with the lowest cost from any source node in the transmission matrix to the target node. This step is an important part of the path optimization problem in a distributed computing environment, which determines the efficiency and cost of data transmission. First, we need to determine all feasible paths from the source node to the target node. This usually involves traversing the transmission matrix to find all paths that can reach the target node from the source node. These paths may contain different node combinations and connections, but they all need to meet the condition that the transmission cost is known. Then, among all feasible paths, we need to find the shortest path that meets the minimum cost condition. This step is usually achieved by comparing the total transmission cost of each path. The total transmission cost is the sum of the transmission costs between all adjacent nodes on the path. In order to find the path with the lowest cost, we can use graph theory algorithms such as Dijkstra's algorithm, Bellman-Ford algorithm, or Floyd-Warshall algorithm. These algorithms can efficiently calculate the shortest path from the source node to all other nodes and give the corresponding transmission cost. After finding the shortest path with the lowest cost, we use it as the first path. This path not only meets the condition of the lowest cost, but also has the lowest cost between any two adjacent nodes on it. This means that on this path, every step of data transmission is optimal, thus ensuring the maximization of overall transmission efficiency.

[0100] Furthermore, a dynamic programming algorithm can be used to solve the shortest path from the source node to the target node. The core of the dynamic programming algorithm is to decompose the complex problem of solving the shortest path into multiple sub-problems through the state transition equation to gradually build a global optimal solution. In this case, we regard each node in the network as a state and the path from the source node to the target node as a state sequence. By defining the state transition equation, we can calculate the shortest path length from the source node to each intermediate node and gradually expand it to the target node. This process involves a lot of calculations, but thanks to the memoization search feature of the dynamic programming algorithm, we can find the shortest path from the source node to the target node by backtracking and use this path as the first path.

[0101] This embodiment successfully finds the optimal solution for the data transmission problem in the distributed computing environment by determining feasible paths, calculating the total transmission cost, comparing and selecting the shortest path with the lowest cost, etc. This solution not only improves the efficiency of data transmission, but also reduces the transmission cost, providing strong support for the optimization of the distributed computing environment.

[0102] In a feasible implementation manner, step S30 further includes steps A20 to A30:

[0103] Step A20, when the network status changes frequently, determining whether the cost of the first path is updated;

[0104] Step A30: if the cost of the first path is updated, re-execute the step of optimizing the path between the source node and the target node according to the transmission matrix to find the latest shortest path, and update the latest shortest path as the first path.

[0105] It should be noted that the network state refers to the connection state, transmission speed, delay, packet loss rate and other parameters between nodes in the network, which together determine the performance and stability of the network. Cost update refers to the change in the transmission cost on the path due to changes in the network state (such as node failure, link congestion, transmission speed reduction, etc.), which requires the recalculation of the path cost.

[0106] In a network environment, the network state changes dynamically, which may cause the previously determined first path to no longer be the optimal path. Therefore, this embodiment proposes a dynamic path optimization mechanism to adapt to changes in network state. First, the system needs to monitor changes in network state and determine whether these changes may cause the cost of the first path to be updated. This usually involves continuous monitoring of the network state and real-time detection of parameters that may affect the path cost (such as delay, packet loss rate, etc.). If it is detected that the cost of the first path may have been updated, the system needs to trigger a path optimization step. This step is similar to the previous path optimization process, but the updated transmission matrix (reflecting the change in network state) is used at this time. The system will recalculate the cost of all feasible paths between the source node and the target node based on the new transmission matrix, and find the shortest path with the lowest cost. After finding the latest shortest path, the system needs to update it to the first path. This means that subsequent data transmission will be carried out along this new shortest path to ensure the optimization of transmission efficiency and cost. It should be noted that due to the frequent changes in network state, this path optimization process may need to be performed frequently. Therefore, the system needs to have efficient path calculation and optimization capabilities to quickly respond to changes in network state and ensure the stability and efficiency of data transmission.

[0107] Furthermore, the ant colony algorithm can be introduced to dynamically optimize the path. The ant colony algorithm finds the optimal path by simulating the foraging behavior of ants. In the algorithm, each ant represents a potential path selection plan, and they choose the next direction based on the "pheromone" on the current path (that is, the quality of the path, which can be calculated based on the elements in the transmission matrix). As the ants move in the network, the pheromones on the path will continue to accumulate, thereby guiding more ants to choose a better path. This process reflects the positive feedback mechanism, allowing the algorithm to gradually approach the global optimal solution. In the ant colony algorithm, we define the pheromone update rule and the ant movement rule. The pheromone update rule determines the update method of the path quality, while the ant movement rule determines how the ants choose the next direction. Through multiple iterations, the algorithm will converge to a path with the highest pheromone concentration, which is the new optimal path. It is worth noting that in the ant colony algorithm, we also introduce heuristic information to guide the search direction of the ants. Heuristic information can be calculated based on the elements in the transmission matrix, which reflects the expected cost from the current node to the target node. By combining heuristic information and pheromones, we can better guide ants to move toward the optimal path.

[0108] The embodiment realizes dynamic optimization of the data transmission path in the network environment by monitoring the changes of the network status and re-executing the path optimization step when necessary. This mechanism not only improves the efficiency and stability of data transmission, but also enhances the adaptability of the network to dynamic changes.

[0109] In a feasible implementation manner, step S30 further includes steps B10 to B20:

[0110] Step B10: construct an objective function that minimizes the transmission cost based on the transmission matrix

[0111] It should be noted that the objective function is a mathematical expression or function used to evaluate the pros and cons of different solutions in an optimization problem. In this case, the objective function aims to minimize the transmission cost.

[0112] Construct an objective function based on the transmission matrix. The transmission matrix is ​​a prerequisite, which contains the cost information of transmitting data between all pairs of nodes in the network. These costs can be specific values ​​of performance indicators such as latency, bandwidth consumption, packet loss rate, etc. The objective function is constructed based on this cost information to quantify the total cost of transmitting data from the source node to the destination node. The objective function can be linear or nonlinear, depending on how the transmission cost is calculated and the complexity of the optimization problem. Use this objective function to evaluate the transmission cost of different paths. This usually involves extracting and calculating the relevant elements in the transmission matrix. For each possible path, we calculate the total transmission cost of the path based on the corresponding cost information in the transmission matrix. This process may require traversing multiple elements of the transmission matrix and performing corresponding mathematical operations.

[0113] Step B20, evaluating the transmission costs of the paths according to the objective function, and selecting the path with the lowest transmission cost as the first path.

[0114] It should be noted that the transmission cost is calculated based on the elements in the transmission matrix, which represents the total cost or price required to transmit data from the source node to the target node. The first path is the optimal path selected from multiple possible paths based on a specific criterion (such as the lowest transmission cost).

[0115] The path with the lowest transmission cost is selected as the first path. After evaluating the transmission costs of all possible paths, we can compare these cost values ​​and find the smallest one. The path corresponding to this minimum cost is the first path we want to find. The selection of the first path is based on a quantitative comparison of transmission costs, so it is a solution to the optimization problem and represents the lowest cost path for transmitting data from the source node to the destination node.

[0116] Exemplarily, the objective function is calculated as:

[0117] min∑ (i,j) M trans (i,j) Formula (III)

[0118] Among them, i and j are positive integers, M trans (i, j) is the cost of transmitting from source node i to target node j. The minimum cost of the entire transmission path can be calculated using formula (3).

[0119] By constructing a transmission cost minimization objective function based on the transmission matrix, this embodiment can accurately quantify and compare the transmission costs of different paths from the source node to the target node. This method not only takes into account multiple performance indicators such as delay time, bandwidth consumption, and packet loss rate, but also ensures that the first path selected has the lowest transmission cost by traversing the transmission matrix and calculating the total cost of each path. This beneficial effect not only improves the efficiency of data transmission, but also reduces the overall cost, providing strong support for network optimization and resource allocation.

[0120] Based on the first or second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first or second embodiment can be referred to the above introduction, and will not be described in detail later. Step S40 also includes steps C10 to C30:

[0121] Step C10, taking a computing node whose distance is less than a preset distance threshold, whose delay is less than a preset delay threshold and which has no dependency on other computing nodes as a first-level computing node;

[0122] It should be noted that the preset distance threshold is a critical value used to screen the distance between computing nodes when building a multi-level aggregation structure. Only computing nodes with a distance less than the threshold are considered potential hierarchical connection nodes. The preset delay threshold is a critical value used to evaluate the data transmission delay between computing nodes when building a multi-level aggregation structure. Only computing nodes with a delay less than the threshold are considered valid connection nodes. No dependency means that there is no order or dependency in data or calculation between computing nodes, that is, the calculation or data transmission of one node does not depend on the completion of another node. The first-level computing nodes are the set of computing nodes that are first selected when building a multi-level aggregation structure, which meet the preset distance and delay thresholds and have no dependencies. They constitute the top layer of the multi-level aggregation structure.

[0123] According to the preset distance and delay thresholds, the computing nodes that meet the conditions are selected as the first-level computing nodes. The distance between these nodes is less than the preset distance threshold, the delay is less than the preset delay threshold, and there is no dependency between them. These nodes constitute the top layer of the multi-level aggregation structure, providing a foundation for the construction of subsequent layers.

[0124] Step C20: for any level after the first level, a computing node which is closest to the previous level and whose delay is less than a preset delay index is selected according to the first path as a computing node of the current level, and a multi-level aggregation structure is obtained, wherein the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node;

[0125] It should be noted that the subsequent levels are when constructing a multi-level aggregation structure. Except for the first level, each level is constructed based on the selection logic of the previous level. The target node is a computing node that serves as the end point of data transmission or calculation in a multi-level aggregation structure.

[0126] For subsequent levels, we follow the selection logic of the first path and select the computing node with the closest distance and delay less than the preset delay index from the previous level as the computing node of the next level. This process is recursive until the last level, that is, the target node, is selected. In this process, we ensure that the connection between each level is based on the optimal path, thereby optimizing the efficiency of data transmission and calculation.

[0127] Step C30: If the delay of any first computing node is greater than or equal to a preset delay index, the first computing node is replaced with a preset standby node.

[0128] It should be noted that in this embodiment, the multi-level aggregation structure is a network structure formed by connecting multiple hierarchical computing nodes through an optimal path, aiming to optimize the efficiency of data transmission and computing. In the process of building a multi-level aggregation structure, if the delay of a computing node does not meet the requirements, the hierarchical position of the computing node in the structure is adjusted to optimize the overall performance.

[0129] If, in the process of building a multi-level aggregation structure, we find that the latency of a first computing node is greater than or equal to the preset latency index, we need to update the position of this node in the structure. This usually means that we need to move this node to a level closer to the target node, or reselect other nodes that meet the conditions to replace it. This process is dynamic and aims to continuously optimize the performance of the multi-level aggregation structure.

[0130] In this embodiment, the preset distance and delay thresholds are the basis for screening computing nodes, the selection logic of the first path is the key to connecting computing nodes at different levels, and the operation of updating the position is an important means to optimize the performance of the multi-level aggregation structure. These steps together constitute the core content of this embodiment, which aims to build an efficient and flexible multi-level aggregation structure to optimize the efficiency of data transmission and calculation.

[0131] In one possible implementation, refer to Figure 3 , step S50 further includes steps D10 to D20:

[0132] Step D10, inputting the task data into the first-level computing node in the multi-level aggregation structure, aggregating the task data through the first-level computing node to obtain an aggregation result;

[0133] It should be noted that the first-level computing nodes are the computing nodes that first receive and process data in the multi-level aggregation structure. These nodes are usually located at the top level of the aggregation structure and are responsible for preliminary data aggregation operations. The preliminary aggregation result is the intermediate result obtained by aggregating the input data on the first-level computing nodes. This result is the input for subsequent aggregation levels.

[0134] The data is transmitted to the first-level computing nodes for preliminary aggregation. After receiving the data, these nodes process the data according to the preset aggregation logic (such as summation, average, etc.) to obtain the preliminary aggregation result. This result is the basis for subsequent level aggregation.

[0135] Step D20, for any level of computing nodes after the first level computing nodes, the aggregation results of the previous level are aggregated again to obtain the aggregation results of the current level, until they are aggregated to the target node in the multi-level aggregation structure, wherein the aggregation results of the current level are stored in the computing node of the current level that is closest to the next level.

[0136] It should be noted that the target node is a computing node in a multi-level aggregation structure that is responsible for receiving and storing the final aggregation results. This node is usually the last layer or designated layer of the aggregation structure. The aggregation result of the current level is the intermediate result obtained on each level of computing nodes based on the aggregation result of the previous level and the aggregation logic of the current level node. This result will be used as the input for the aggregation of the next level. The computing node closest to the next level is the node with the closest network connection to the computing node of the next level in the multi-level aggregation structure (such as short physical distance and low network latency). It is used to store the aggregation results of the current level for rapid transmission to the next level.

[0137] The preliminary aggregation results are further aggregated through a multi-level aggregation structure. At each level, the computing node will obtain the aggregation results of the current level based on the aggregation results of the previous level and the aggregation logic of the nodes at this level. This result is the result of further understanding and processing of the data by the nodes at this level. Importantly, in order to optimize data transmission efficiency and reduce latency, the aggregation results of the current level are stored in the computing nodes closest to the next level. This operation ensures that the data can be transmitted to the next level for calculation as quickly as possible, thereby improving the efficiency of the entire aggregation process.

[0138] Finally, after multiple levels of aggregation operations, the final aggregation result is transmitted to the target node. This target node is the last or designated layer of the multi-level aggregation structure, which is responsible for receiving and storing the final aggregation result. This result is a comprehensive understanding and processing result of the input data, which can be used for subsequent data analysis, decision support and other tasks.

[0139] Exemplarily, the calculation method of the final aggregation result is expressed as formula (IV)

[0140] G(l)=∑ i∈Level(l) Data i Formula (IV)

[0141] Among them, G(l) is the total aggregation result, i and l are positive integers, Level(l) refers to the lth level, i∈Level(l) refers to the i computing node of the lth level, Data i Refers to the data processed by the i computing node.

[0142] In this embodiment, a multi-level aggregation structure is used to provide an efficient data transmission and aggregation mechanism, and data processing delay and cost are reduced by optimizing data storage and transmission paths, which makes the method of this embodiment have significant advantages when processing large-scale data sets.

[0143] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the distributed data transmission method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0144] This application also provides a distributed data transmission device, please refer to Figure 4 , the distributed data transmission device comprises:

[0145] Determine computing node module 10, determine multiple computing nodes in the distributed computing terminal;

[0146] A transmission matrix construction module 20 is used to construct a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes;

[0147] A path optimization module 30 performs path optimization on a path between a source node and a target node according to a transmission matrix to obtain a first path;

[0148] The aggregation structure construction module 40 divides all computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel;

[0149] The aggregation transmission module 50 transmits the preset task data to the target node through a multi-level aggregation structure.

[0150] The distributed data transmission device provided by the present application adopts the distributed data transmission method in the above embodiment, which can improve the data transmission efficiency of the distributed computing terminal. Compared with the prior art, the beneficial effects of the distributed data transmission device provided by the present application are the same as the beneficial effects of the distributed data transmission method provided by the above embodiment, and other technical features in the distributed data transmission device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0151] The present application provides a distributed data transmission device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the distributed data transmission method in the above-mentioned embodiment one.

[0152] Reference below Figure 5 , which shows a schematic diagram of the structure of a distributed data transmission device suitable for implementing the embodiments of the present application. The distributed data transmission device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The distributed data transmission device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0153] like Figure 5As shown, the distributed data transmission device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the distributed data transmission device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the distributed data transmission device to communicate with other devices wirelessly or by wire to exchange data. Although the distributed data transmission device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0154] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0155] The distributed data transmission device provided by the present application adopts the distributed data transmission method in the above embodiment, which can improve the data transmission efficiency of the distributed computing terminal. Compared with the prior art, the beneficial effects of the distributed data transmission device provided by the present application are the same as the beneficial effects of the distributed data transmission method provided by the above embodiment, and other technical features in the distributed data transmission device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0156] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0157] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0158] The present application provides a medium, which is a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the distributed data transmission method in the above-mentioned embodiment.

[0159] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0160] The computer-readable storage medium may be included in the distributed data transmission device; or may exist independently without being assembled into the distributed data transmission device.

[0161] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the distributed data transmission device, the distributed data transmission device:

[0162] Determining a plurality of computing nodes in a distributed computing terminal;

[0163] Constructing a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between computing nodes, wherein the computing nodes include source nodes and target nodes;

[0164] Optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path;

[0165] According to the first path, all computing nodes are divided into multiple levels to obtain a multi-level aggregation structure, wherein computing nodes in each level process data in parallel;

[0166] The preset task data is transmitted to the target node through a multi-level aggregation structure.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0170] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned distributed data transmission method, and can improve the data transmission efficiency of distributed computing terminals. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the distributed data transmission method provided by the above-mentioned embodiment, and will not be repeated here.

[0171] The present application also provides a product, which is a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the distributed data transmission method as described above are implemented.

[0172] The computer program product provided by the present application can improve the data transmission efficiency of distributed computing terminals. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the distributed data transmission method provided by the above embodiment, which will not be repeated here.

[0173] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A distributed data transmission method, characterized in that: Applied to a distributed computing terminal, the distributed data transmission method includes: Determining a plurality of computing nodes in the distributed computing terminal; Constructing a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes; Optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path; Dividing all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel; The preset task data is transmitted to the target node through the multi-level aggregation structure.

2. The distributed data transmission method according to claim 1, characterized in that: The step of optimizing the path between the source node and the target node according to the transmission matrix to obtain a first path comprises: For any source node in the transmission matrix, all feasible paths from the source node to the target node are determined, the shortest path that meets the minimum cost condition is found among all the feasible paths, and the shortest path is used as the first path, wherein the cost between any two adjacent nodes in the shortest path is the lowest.

3. The distributed data transmission method according to claim 1, characterized in that: After the step of optimizing the path between the source node and the target node according to the transmission matrix to obtain the first path, the following steps are included: When the network state changes frequently, determining whether the cost of the first path is updated; If the cost of the first path is updated, the step of optimizing the path between the source node and the target node according to the transmission matrix is ​​re-executed to find the latest shortest path, and the latest shortest path is updated as the first path.

4. The distributed data transmission method according to claim 1, characterized in that: The step of dividing all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure includes: The computing nodes whose distance is less than a preset distance threshold, whose delay is less than a preset delay threshold and which have no dependency on other computing nodes are regarded as the first-level computing nodes; For any level after the first level, a computing node that is closest to the previous level and has a delay less than a preset delay index is selected according to the first path as a computing node of the current level, to obtain a multi-level aggregation structure, wherein the computing node of the last level is the target node, and any computing node before the computing node of the last level is the source node; After the step of dividing all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, the following steps are included: If the delay of any first computing node is greater than or equal to a preset delay index, the first computing node is replaced with a preset standby node.

5. The distributed data transmission method according to claim 4, characterized in that: The step of transmitting the preset task data to the target node through the multi-level aggregation structure includes: Inputting the task data into a first-level computing node in a multi-level aggregation structure, and aggregating the task data through the first-level computing node to obtain an aggregation result; For any level of computing nodes after the first level computing nodes, the aggregation results of the previous level are aggregated again to obtain the aggregation results of the current level, until they are aggregated to the target node in the multi-level aggregation structure, wherein the aggregation results of the current level are stored in the computing node of the current level that is closest to the next level.

6. The distributed data transmission method according to claim 1, characterized in that: The step of constructing a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes comprises: Determine all computing node pairs, wherein the computing node pairs consist of two of the computing nodes; For each computing node pair, detecting a network status to obtain a transmission delay and a bandwidth between the computing node pairs; Determining the amount of data to be transmitted between the computing node pairs according to the task data; The ratio of the amount of data to be transmitted to the bandwidth is calculated, the product of the ratio and the transmission delay is used as the transmission cost of each computing node pair, and the transmission cost is used as an element of the transmission matrix to obtain the transmission matrix.

7. A distributed data transmission device, characterized in that: Set up in a distributed computing terminal, the distributed data transmission device includes: Determine a computing node module to determine a plurality of computing nodes in the distributed computing terminal; A transmission matrix construction module is used to construct a transmission matrix according to the transmission delay, bandwidth and amount of data to be transmitted between the computing nodes, wherein the computing nodes include source nodes and target nodes; A path optimization module, which optimizes the path between the source node and the target node according to the transmission matrix to obtain a first path; an aggregation structure construction module, which divides all the computing nodes into multiple levels according to the first path to obtain a multi-level aggregation structure, wherein the computing nodes in each level process data in parallel; The aggregation transmission module transmits the preset task data to the target node through the multi-level aggregation structure.

8. A distributed data transmission device, characterized in that: The distributed data transmission device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the distributed data transmission method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the distributed data transmission method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the distributed data transmission method according to any one of claims 1 to 6 are implemented.

Images