A Distributed Data Transmission Method and Apparatus Based on Multi-Agent Collaboration

Abstract

This invention provides a method and apparatus for transmitting distributed datasets based on multi-agent collaboration. The method includes: constructing a global topology graph based on the fragmentation information of the target data fragments held by the target agent and the data source agent, and the physical location of the agents; and performing decentralized transmission negotiation based on the global topology graph and the resource information of the data source agent to generate and execute a transmission plan. The method provided by this invention, by constructing a global topology graph using the fragmentation information of the target data fragments held by the target agent and the data source agent, and the physical location of the agents, and by performing decentralized transmission negotiation based on the global topology graph and the resource information of the data source agent to generate and execute a transmission plan, avoids the performance bottleneck of a single control node when handling a large number of nodes, achieves global load balancing and resource optimization, and significantly improves the overall transmission efficiency and robustness of distributed datasets.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a method and apparatus for transmitting distributed datasets based on multi-agent collaboration. Background Technology

[0002] Modern scientific computing, big data analytics, and artificial intelligence model training often require handling distributed datasets that are logically unified but physically dispersed. For example, a large bioinformatics dataset may be generated and stored collaboratively by multiple sequencing centers around the world; training data for an artificial intelligence model may be scattered across multiple edge data centers. How to efficiently and reliably aggregate this dispersed data into a centralized computing cluster or cloud is a significant challenge facing current data-intensive applications. Traditional transmission schemes typically rely on a central control node to orchestrate the transmission tasks of all data sources.

[0003] However, when the number of data sources is huge, the central control node not only becomes a performance and decision-making bottleneck, but also constitutes a single point of failure. Once it fails, the entire transmission task will be interrupted. Summary of the Invention

[0004] This invention provides a distributed dataset transmission method and apparatus based on multi-agent collaboration, which solves the defects of existing technologies that rely on a central control node to uniformly arrange transmission tasks, leading to performance and decision-making bottlenecks and single points of failure when the number of data sources is huge, and achieves highly robust and efficient distributed dataset transmission.

[0005] This invention provides a distributed dataset transmission method based on multi-agent collaboration, comprising: Determine the transmitting agent corresponding to the dataset to be transmitted, wherein the transmitting agent includes a data source agent and a destination agent; Based on the target agent, the fragmentation information of the target data fragment held by the data source agent, and the physical location of the agent, a global topology graph is constructed. Based on the global topology graph and the data source agent's own resource information, a decentralized transmission negotiation is performed to generate and execute the transmission plan for the dataset to be transmitted.

[0006] According to the present invention, a distributed dataset transmission method based on multi-agent collaboration is provided, wherein the method involves performing decentralized transmission negotiation based on the global topology graph and the resource information of the data source agent, generating and executing a transmission plan for the dataset to be transmitted, comprising: Based on the target agent, a tender for the transmission task of the dataset to be transmitted is broadcast to the data source agent. Based on the data source agent's response to the transmission task bidding, the agent calculates the transmission bidding score according to its own resource information and transmission cost, and returns the transmission bidding score to the target agent; Based on the target agent and the received transmission bid score, the execution agent is selected from the transmission agents; Based on the executing agent and the destination agent, the transmission plan is generated and executed.

[0007] According to the present invention, a distributed dataset transmission method based on multi-agent collaboration is provided, wherein the transmitting agent further includes an intermediate agent adjacent to the data source agent; The step of calculating the transmission bid score based on the self-resource information and transmission cost includes: Based on the intermediate agent's own resource information and aggregation cost, and based on the forwarding cost between the intermediate agent and the target agent, the aggregated transmission bid score is calculated; the aggregation cost includes the transmission cost between the intermediate agent and at least one neighboring data source agent. The aggregated transmission bid score is used as the transmission bid score; Accordingly, when the intermediate agent acts as the executing agent, generating and executing the transmission plan based on the executing agent and the destination agent includes: Determine an aggregation transmission plan between the intermediate agent and the at least one neighboring data source agent, and determine a forwarding transmission plan between the intermediate agent and the destination agent; The aggregated transmission plan and the forwarding transmission plan are used as the transmission plan, and the transmission plan is executed.

[0008] According to the present invention, a distributed dataset transmission method based on multi-agent cooperation is provided, wherein the step of selecting an execution agent from the transmission agents based on the received transmission bid score by the target agent includes: In the case where there are multiple data replicas corresponding to the same target data fragment with multiple transmission bid scores, the data source agent corresponding to the highest transmission bid score of the same target data fragment is selected as the execution agent.

[0009] According to the present invention, a distributed dataset transmission method based on multi-agent cooperation is provided, wherein the step of selecting an execution agent from the transmission agents based on the received transmission bid score by the target agent includes: All data source agents corresponding to the same target data shards are used as the execution agents; if any data source agent in all the data source agents completes the data transmission, the remaining data source agents in all the data source agents are removed from the list of execution agents.

[0010] According to the present invention, a distributed dataset transmission method based on multi-agent cooperation is provided, the method further comprising: If a failure is detected in any data source agent, a backup agent that holds the same data copy as the failed agent is identified. The backup agent is used as the executing agent.

[0011] According to the present invention, a distributed dataset transmission method based on multi-agent collaboration is provided, wherein the self-resource information includes agent CPU load information, agent storage status, and agent network bandwidth. The transmission cost includes network cost and transmission path cost.

[0012] The present invention also provides a distributed dataset transmission device based on multi-agent cooperation, comprising: The agent activation unit determines the transmitting agent corresponding to the dataset to be transmitted, wherein the transmitting agent includes a data source agent and a destination agent; The topology graph construction unit constructs a global topology graph based on the target agent, the fragmentation information of the target data fragment held by the data source agent, and the physical location of the agent. The negotiation transmission unit performs decentralized transmission negotiation based on the global topology graph and the data source agent's own resource information, and generates and executes the transmission plan for the dataset to be transmitted.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the distributed dataset transmission method based on multi-agent cooperation as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed dataset transmission method based on multi-agent cooperation as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the distributed dataset transmission method based on multi-agent collaboration as described above.

[0016] The distributed dataset transmission method and apparatus based on multi-agent collaboration provided by this invention constructs a global topology map by using the fragmentation information of the target data fragments held by the target agent and the data source agent, as well as the physical location of the agents. Based on the global topology map and the resource information of the data source agent, decentralized transmission negotiation is performed to generate and execute the transmission plan of the dataset to be transmitted. This avoids the performance bottleneck of a single control node when handling a large number of nodes, and automatically avoids nodes with excessive load or network congestion, selecting the data source with the best state for transmission. It achieves global load balancing and resource optimization, significantly improving the overall transmission efficiency and robustness of the distributed dataset. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the distributed dataset transmission method based on multi-agent collaboration provided by the present invention; Figure 2 This is a flowchart illustrating the decentralized transmission negotiation method provided by the present invention. Figure 3 This is a schematic diagram of the data aggregation and transmission process provided by the present invention; Figure 4 This is a schematic diagram of the process of intelligent agents performing transmission negotiation provided by the present invention; Figure 5 This is a schematic diagram of the backup intelligent agent activation process provided by the present invention; Figure 6 This is the second flowchart of the distributed dataset transmission method based on multi-agent collaboration provided by the present invention; Figure 7 This is a schematic diagram of the structure of the distributed dataset transmission device based on multi-agent collaboration provided by the present invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] To address the aforementioned issues, this invention provides a distributed dataset transmission method based on multi-agent collaboration to avoid single-point bottlenecks and achieve highly robust and efficient distributed dataset transmission. Figure 1 This is one of the flowcharts illustrating the distributed dataset transmission method based on multi-agent collaboration provided by this invention, such as... Figure 1 As shown, the method includes: Step 110: Determine the transmitting agent corresponding to the dataset to be transmitted, wherein the transmitting agent includes the data source agent and the destination agent.

[0021] Here, the dataset to be transmitted refers to a collective term for data files that logically belong to the same set but may be physically scattered across different network nodes, potentially containing a massive number of file fragments or data blocks. The transmission agent refers to a software process deployed on a network node, possessing independent environmental awareness, autonomous decision-making, communication interaction, and task execution capabilities; it constitutes the basic unit of the distributed transmission network. Specifically, the data source agent refers to the transmission agent located on the node holding all or part of the data fragments of the dataset to be transmitted, its role similar to that of a data supplier; the destination agent refers to the transmission agent located on the final data aggregation or receiving node, its role similar to that of a data demander. It can be understood that by identifying these agents, a peer-to-peer collaborative network covering the entire data link is constructed.

[0022] Specifically, first, the system can receive user-submitted transmission tasks, which include a manifest file defining the dataset to be transmitted and the final destination information of the data. Then, by parsing the manifest file, it identifies the network addresses of all source and destination nodes containing data fragments from that dataset. Next, on these identified source and destination nodes, transmission agents are initialized, activated, or deployed, respectively. This process can be accomplished through a temporary bootstrapping mechanism, such as setting up one or more bootstrapping agents. Once the transmission agents are activated and connections are established on each node, a self-managing agent network is formed, logically connecting physically dispersed nodes and preparing them for collaborative work.

[0023] It should be noted that by deploying autonomous transmission agents on each of the involved nodes, the traditional centralized control logic is devolved to the edge nodes, realizing the fusion of the transmission control plane and the data forwarding plane. This not only avoids the performance bottleneck of a single control node when dealing with a large number of nodes, but also provides architectural support for the system's high scalability and robustness.

[0024] Step 120: Based on the target agent, the fragmentation information of the target data fragment held by the data source agent, and the physical location of the agent, construct a global topology graph.

[0025] Here, target data fragments refer to the individual data blocks or file segments that make up the dataset to be transmitted. Fragment information includes metadata such as the unique identifier of the data fragment, data size, checksum hash value, and specific location in the storage medium. The agent's physical location here represents the network address, network topology coordinates, or subnet region of the node where the transmitting agent resides, reflecting the network distance between nodes. Furthermore, the global topology map here refers to a logical mapping view reflecting the connection relationships, link quality status, and data fragment distribution among all participating agent nodes. It is understandable that the global topology map is not a static, pre-defined map, but a dynamically perceived, real-time panoramic view of the status.

[0026] Specifically, after each transport agent is activated, it first scans its local nodes to identify the target data fragment information it holds. Subsequently, each transport agent discovers and communicates with the others through a pre-defined peer-to-peer communication protocol, exchanging its fragment information and its physical location. During this process, the data source agent broadcasts or unicasts its fragment list to the destination agent and other neighboring agents. The destination agent and other agents in the network collect this dispersed information, summarize it, and construct a global topology map that includes data distribution and network distances—that is, a global topology map showing which nodes hold copies of which fragments and the logical proximity relationships between nodes.

[0027] Understandably, the process of constructing a global topology graph can discover redundant copies of data, meaning that the same target data shard may exist simultaneously at multiple different data source agents.

[0028] It should be noted that through the active perception and information interaction of intelligent agents, a global topology map containing data distribution and network status is constructed, which can break the blind spot of data physical location perception in traditional transmission and provide accurate decision-making basis for subsequent path optimization.

[0029] Step 130: Based on the global topology graph and the data source agent's own resource information, perform decentralized transmission negotiation to generate and execute the transmission plan for the dataset to be transmitted.

[0030] Here, "self-resource information" refers to the current computing, storage, and network status parameters of the node where the agent resides, specifically including CPU utilization, disk I / O read / write load, memory usage, and remaining network egress bandwidth. These indicators directly determine the node's ability to process transmission tasks. Here, "transmission plan" refers to a dynamic execution scheme determined through negotiation among the agents, including specific data flow paths, sender-receiver pairings, and transmission timing.

[0031] Specifically, a negotiation mechanism similar to the Contract Network protocol can be adopted. The target agent, based on a global topology graph, publishes task requirements for the various target data shards. Then, the data source agents holding the corresponding shards, as potential responders, evaluate their own resources. For example, if a data source agent has low CPU load and sufficient network bandwidth, it calculates a high bid score and provides feedback; conversely, if node resources are scarce, it provides a lower score or does not participate in the bidding. After collecting feedback from all parties, the target agent selects the data source agent with the best conditions, such as the highest bid score, to establish a transmission contract, thereby generating a transmission plan for that shard and immediately beginning data transmission. During the execution of the transmission plan, the data stream flows directly from the selected data source agent to the target agent.

[0032] It should be noted that the decentralized transmission negotiation here refers to a process in which each intelligent agent reaches a consensus on transmission intention through a specific interaction protocol, without relying on instructions from a central server.

[0033] The method provided in this invention constructs a global topology map by using the fragmentation information of the target data fragments held by the target agent and the data source agent, as well as the physical location of the agents. Based on the global topology map and the resource information of the data source agent, it performs decentralized transmission negotiation, generates and executes a transmission plan for the dataset to be transmitted. This avoids the performance bottleneck of a single control node when handling a large number of nodes, and automatically avoids nodes with excessive load or network congestion, selecting the data source with the best state for transmission. This achieves global load balancing and resource optimization, significantly improving the overall transmission efficiency and robustness of the distributed dataset.

[0034] Based on any of the above embodiments Figure 2 This is a flowchart illustrating the decentralized transmission negotiation method provided by the present invention, as shown below. Figure 2 As shown, the method includes: Step 210: Broadcast the tender for the transmission task of the dataset to be transmitted to the data source agent based on the target agent.

[0035] Specifically, a task bidding information containing the data shard ID can be generated by the target agent. Then, using the established agent communication network, the bidding information is sent to all connected transmission agents in the network in the form of broadcast or multicast, or directed to potential data source agents known to hold copies of the data shard according to the global topology graph.

[0036] Step 220: Based on the data source agent's response to the transmission task bidding, the agent calculates the transmission bid score according to its own resource information and transmission cost, and returns the transmission bid score to the destination agent.

[0037] Here, transmission cost refers to the expected time or network resource cost incurred during data transmission, which is typically positively correlated with network round-trip time, network hop count, or link congestion. Additionally, the transmission bid score here refers to a numerical indicator calculated by potential data providers to quantify their suitability for undertaking the transmission task; a higher score indicates better overall conditions for the sender.

[0038] Specifically, after receiving the bidding information, each data source agent in the transmission agent network first searches its local storage index to confirm whether it holds the requested data fragment. If it does, it triggers a self-evaluation process, collecting its own resource information in real time, such as the current CPU load percentage, disk I / O queue depth, and real-time throughput of the network interface. Simultaneously, it estimates the transmission cost with the target agent, such as the RTT (Round Trip Time), using network probing data. Next, it uses a pre-defined weighted algorithm to comprehensively calculate resource sufficiency and transmission cost. For example, a higher bid score is calculated for situations with idle CPU, smooth disk read / write, and low network latency; while a lower score is calculated for situations with high system load or high latency at the network edge. Finally, the calculated transmission bid score is encapsulated in a response message and sent back to the target agent.

[0039] It should be noted that each data source node can obtain a transmission bid score based on its own real-time status. The quantified bid score reflects the current node's ability and willingness to undertake tasks, avoiding blindly pulling data from any replica node and preventing transmission blockage caused by selecting a busy node. This demonstrates the agent's autonomous decision-making ability in environmental perception.

[0040] Step 230: Based on the received transmission bid score, the target agent selects the execution agent from the transmission agents.

[0041] Here, the executing agent refers to the transmission agent that wins out among many competing data source agents and is ultimately designated to be responsible for actually sending data fragments.

[0042] Specifically, after issuing a tender, the target agent can open a short waiting window to collect transmission bid scores from agents of different data sources. Once the window closes or a sufficient number of responses have been collected, all received scores are sorted and compared. Following a merit-based selection principle, the data source agent with the highest transmission bid score is locked and marked as the executing agent for this transmission task. If scores are tied, further selection can be made based on a random strategy or historical stability metrics.

[0043] It should be noted that by comparing bid scores, dynamic optimization of transmission sources is achieved, ensuring that each data transmission task is assigned to the node with the best overall performance in the network topology at the current moment, thereby achieving global load balancing.

[0044] Step 240: Based on the executing agent and the destination agent, generate and execute the transmission plan.

[0045] Specifically, the target agent can send a confirmation command to the selected execution agent, informing it that it has been selected. Upon receiving the confirmation, the execution agent generates a read and send task for that data fragment, thus forming a specific transmission plan. Subsequently, the execution agent begins reading data from its local disk and streams the data fragment to the target agent through the established data transmission channel. Simultaneously, the target agent can send rejection notifications to other unselected bidders, or other bidders can automatically release their reserved resources if they do not receive confirmation within the timeout period.

[0046] The method provided in this invention allows each data source node to evaluate its transmission bid score based on its real-time status, avoiding blindly pulling data from arbitrary replica nodes and preventing transmission congestion caused by selecting busy nodes. This demonstrates the agent's autonomous decision-making ability in terms of environmental perception. Furthermore, the explicit confirmation mechanism ensures that the establishment of the transmission link is based on mutual consensus, avoiding resource conflicts and enabling efficient data flow along the optimal path, significantly improving the overall transmission efficiency of large-scale datasets.

[0047] It should be noted that the data transmission path is usually a fixed source-destination point-to-point pattern, which cannot dynamically construct a better transmission topology based on real-time network conditions. To address this issue, based on any of the above embodiments, the transmission agent also includes an intermediate agent adjacent to the data source agent; The step of calculating the transmission bid score based on the self-resource information and transmission cost includes: Based on the intermediate agent's own resource information and aggregation cost, and based on the forwarding cost between the intermediate agent and the target agent, the aggregated transmission bid score is calculated; the aggregation cost includes the transmission cost between the intermediate agent and at least one neighboring data source agent. The aggregated transmission bid score is used as the transmission bid score; Accordingly, when the intermediate agent acts as the executing agent, generating and executing the transmission plan based on the executing agent and the destination agent includes: Determine an aggregation transmission plan between the intermediate agent and the at least one neighboring data source agent, and determine a forwarding transmission plan between the intermediate agent and the destination agent; The aggregated transmission plan and the forwarding transmission plan are used as the transmission plan, and the transmission plan is executed.

[0048] Here, an intermediate agent refers to a transmission agent located between the data source agent and the destination agent in the network topology. Although it does not possess the data fragments to be transmitted, it has the ability to assist in data forwarding. It is understood that intermediate agents are typically deployed on nodes with ample network bandwidth or idle computing resources.

[0049] Furthermore, the aggregation cost here refers to the network overhead incurred in transmitting data fragments from a nearby data source agent to an intermediate agent, which typically depends on the local area network latency or link quality between the two. The forwarding cost, on the other hand, refers to the network overhead incurred in sending data from the intermediate agent to the destination agent. Therefore, the aggregation transmission bid score is a quantitative indicator derived from a comprehensive evaluation of the overall efficiency of this source-intermediate-destination polygonal path.

[0050] Specifically, when the target agent broadcasts a transmission task tender, the intermediate agent, although not holding the data, discovers that a neighboring data source agent holds the data by querying the global topology. At this point, the intermediate agent does not immediately give up but initiates proxy bidding logic. In detail, the intermediate agent can detect its own resource information and combine this with the aggregation cost of pulling data from neighboring source nodes and the forwarding cost of sending data to the target node to calculate an aggregated transmission bid score. For example, if the source node's network egress is congested and direct transmission costs are extremely high, while the intermediate node is in the same high-speed LAN as the source node and has a very large egress bandwidth, the calculated aggregated transmission bid score may be much higher than the bid score for direct transmission from the source node. Finally, the intermediate agent returns this aggregated transmission bid score as a bidding response to the target agent.

[0051] It should be noted that by introducing an intermediary agent's proxy bidding mechanism, the limitations of traditional point-to-point transmission are broken. This allows the system to discover potential high-performance nodes in the network, enabling it to automatically identify and utilize non-directly connected optimized paths to bypass the network bottleneck of the source node and accelerate traffic by using idle high-performance nodes in the network.

[0052] Accordingly, when the intermediate agent acts as the executing agent, the process of generating and executing the transmission plan includes: determining an aggregated transmission plan between the intermediate agent and at least one neighboring data source agent, and determining a forwarding transmission plan between the intermediate agent and the destination agent. The aggregated transmission plan and the forwarding transmission plan are then used as the transmission plan, and the transmission plan is executed.

[0053] Here, the aggregation transmission plan refers to the set of operational instructions that guides data flow from a nearby data source to an intermediate agent, typically involving high-speed data retrieval within a local area network. The forwarding transmission plan refers to the set of operational instructions that guides data flow from an intermediate agent to a destination agent, typically involving high-speed distribution over a wide area network.

[0054] Specifically, once the target agent selects an intermediate agent as the executing agent, the intermediate agent immediately generates a two-stage execution plan. First, an aggregation transmission plan is executed, whereby the intermediate agent initiates data requests to one or more nearby data source agents, rapidly fetching data fragments into local memory or cache. Simultaneously or immediately afterward, a forwarding transmission plan is executed, utilizing the intermediate agent's superior network egress resources to rapidly push the cached data to the target agent. Here, these two processes can be executed in a pipelined parallel manner, i.e., fetching and forwarding simultaneously to reduce buffering latency.

[0055] In one embodiment, Figure 3 This is a schematic diagram of the data aggregation and transmission process provided by the present invention, as shown below. Figure 3As shown, it includes source node D, source node E, aggregation node F, and destination node A. Here, the source node refers to the data source agent, the aggregation node refers to the intermediate agent, and the destination node refers to the destination agent.

[0056] In detail, assume that source nodes D and E both hold the data needed by destination node A, but source nodes D and E have limited network egress bandwidth. Meanwhile, neighboring node F has a high-speed network connection. Therefore, during the task negotiation phase, node F can play a special role. When bidding to node A, node F's proposed solution is not to directly provide data, but rather, "I can collect data from Agent D and Agent E, then package it and send it to you at high speed." If Agent A evaluates and determines that the overall cost of this aggregation solution is lower than direct transmission from D and E respectively, it will choose Agent F's solution.

[0057] When the transmission plan is executed, node F will first pull data from nodes D and E, cache and aggregate it locally, and then use its high-speed link to transmit the aggregated big data stream to node A in one go with high efficiency, effectively avoiding the bottleneck of the source end exit.

[0058] The method provided in this invention constructs a combined aggregation-forward transmission mode, which in essence logically creates a virtual transmission channel that is superior to physical direct connection. This not only effectively solves the long-tail transmission problem caused by poor network configuration or high load of some data source nodes, but also bypasses local network congestion through the traffic aggregation capability of intermediate nodes, realizing the shaping and acceleration of data streams, and significantly improving the overall transmission performance in harsh network environments.

[0059] It should be noted that traditional tools are completely unaware of the physical distribution of the dataset. That is, they cannot utilize redundant copies or erasure coding shards for opportunistic optimization. For example, when a data source node has a poor network connection, they cannot intelligently switch to another node with the same data copy and a better network connection. To address this issue, based on any of the above embodiments, step 230 includes: In the case where there are multiple data replicas corresponding to the same target data fragment with multiple transmission bid scores, the data source agent corresponding to the highest transmission bid score of the same target data fragment is selected as the execution agent.

[0060] Specifically, after collecting tender responses for a specific target data fragment, the received transmission bid scores can be grouped according to the fragment's unique identifier. If multiple bid scores from different network addresses are found for the same fragment identifier, it indicates that the fragment has redundant copies in the network. In this case, these scores are numerically compared and sorted, and the data source agent associated with the highest transmission bid score is selected as the sole executing agent, while other bids with lower scores are ignored or rejected.

[0061] In one embodiment, Figure 4 This is a schematic diagram of the process of intelligent agents performing transmission negotiation provided by the present invention, as shown below. Figure 4 As shown, the process includes: target agent A, source agent B, and source agent C. Here, the source agent refers to the data source agent.

[0062] In detail, if the transmitting agent on the destination node A needs to obtain a data fragment Block-X, it can broadcast a task tender to the agent network (source agent B and source agent C), declaring that it needs Block-X.

[0063] Both agents (Agent B and Agent C) on source nodes B and C have copies of Block-X. Source agents B and C exchange information via a peer-to-peer communication protocol and are aware of each other's existence and network topology. Upon receiving a tender, source agents B and C each evaluate their own conditions. Agent B finds that its CPU load is low and its network path to destination node A is of good quality, with low transmission latency and high bandwidth. Therefore, it calculates a high tender score, for example, 95 / 100, and sends it to Agent A. Meanwhile, Agent C finds that its node has busy disk I / O and a poor network path. Therefore, it calculates a lower tender score, for example, 60 / 100, and also sends it to Agent A.

[0064] After Agent A collects all bids, it selects Agent B, which has the highest score, as the executor for this transmission. Agent A can send a confirmation of winning the bid to Agent B and a notification of rejection to Agent C, in which case Agent B will act as the executing agent for transmitting the data fragment destined for that location.

[0065] The method provided in this invention fully utilizes the redundancy of distributed datasets and determines the preferred execution agent through an optimization mechanism. This not only solves the problem of blind data source selection but also ensures that each data transmission runs on the optimal path in the current network environment, effectively avoiding nodes that have data replicas but are currently overloaded or congested, thereby maximizing global transmission efficiency.

[0066] Based on any of the above embodiments, step 230 includes: All data source agents corresponding to the same target data shards are used as the execution agents; if any data source agent in all the data source agents completes the data transmission, the remaining data source agents in all the data source agents are removed from the list of execution agents.

[0067] Specifically, to address the uncertainties caused by network fluctuations or node performance jitter, when determining the execution target, instead of a single selection, a winning bid confirmation can be simultaneously sent to all data source agents holding the same shard replica. This triggers multiple source nodes to simultaneously send the same data stream to the destination node. During transmission, the reception progress of each data stream is continuously monitored. Once the first complete data shard is successfully received and verified, meaning any data source agent has completed transmission, a cancellation command can be immediately sent from the destination agent to other data source agents currently transmitting that shard, interrupting the remaining redundant transmission tasks.

[0068] The steps provided in this embodiment of the invention employ an aggressive transmission strategy that trades bandwidth for time. By using a concurrent contention mechanism, the long-tail effect caused by sudden node failures or network jitter is eliminated. This ensures that the acquisition time of data fragments depends on the shortest path among all available paths, greatly reducing the overall latency of data transmission. It is particularly suitable for computing scenarios with extremely high timeliness requirements.

[0069] It should be noted that during large-scale transmission, the downtime of individual data source nodes or network interruptions is common. Traditional solutions are slow to respond to such failures, usually requiring manual intervention to reconfigure the task, resulting in prolonged transmission interruptions and severely impacting the overall timeliness of the operation. To address this issue, based on any of the above embodiments, the method includes: If a failure is detected in any data source agent, a backup agent that holds the same data copy as the failed agent is identified. The backup agent is used as the executing agent.

[0070] Specifically, during the execution of the transmission plan, the destination agent can continuously monitor the status of the data source agent currently performing the transmission task through a heartbeat monitoring mechanism, such as periodically sending Ping packets or heartbeat signaling, or through a data stream timeout detection mechanism, such as if the expected data packet is not received within a preset time. Once it is determined that the current executing agent is disconnected or has malfunctioned, a fault recovery process is immediately triggered. Here, a fault refers to an abnormal state in which data transmission is interrupted due to the physical node where the data source agent resides crashing, network disconnection, or unresponsive transmission process.

[0071] At this point, the pre-built global topology graph can be retrieved or previous bidding records can be viewed to check if other data source agents holding the same data shard ID exist. If so, these nodes are locked as candidate backup agents.

[0072] Then, a transmission start command can be immediately sent to the designated backup agent, or a round of rapid renegotiation for that specific data fragment can be initiated, requesting the backup agent to take over the remaining transmission tasks. After receiving the command, the backup agent reads the local copy of the data and sends the data stream to the destination agent from the point of interruption or at the beginning, thereby achieving a seamless switchover of the transmission tasks.

[0073] In one embodiment, Figure 5 This is a schematic diagram of the backup intelligent agent activation process provided by the present invention, such as... Figure 5 As shown, during the process of Agent B transmitting data to Agent A, node B suddenly crashed, and Agent B lost contact.

[0074] In detail, Agent A detects Agent B's loss of connection through a heartbeat mechanism or by continuously not receiving data packets. Other agents adjacent to Agent B in the network also detect this simultaneously. Agent A immediately initiates a new task tender for Block-X. At this time, Agent C, which holds a copy of Block-X, will reflect the failure of Agent B in its bid. Since there are no competitors, Agent C's bid is quickly accepted by Agent A. Finally, Agent A confirms that Agent C has won the bid, and Agent C immediately begins transmitting Block-X. The entire process is completed within seconds, and the transmission task is almost seamless from interruption to recovery, requiring no manual intervention.

[0075] The method provided in this invention realizes the self-healing capability of a distributed transmission system through real-time fault detection and automatic switching mechanisms. It can automatically bypass faulty nodes and restore services using redundant copies of data without human intervention, greatly improving the success rate and reliability of large-scale data transmission tasks and ensuring business continuity.

[0076] Based on any of the above embodiments, the self-resource information includes agent CPU load information, agent storage status, and agent network bandwidth. The transmission cost includes network cost and transmission path cost.

[0077] Here, the agent's CPU load information refers to the node's processor utilization or average load, reflecting the node's computational workload. The agent's storage status refers to the disk I / O read / write queue depth, throughput, or disk utilization, directly affecting the speed of reading data fragments. The agent's network bandwidth refers to the node's network interface card's real-time uplink rate or remaining available bandwidth.

[0078] Here, network cost typically refers to the fees charged for the transmission network used between agent nodes. Transmission path cost refers to the number of hops or logical network distance the data flows through; a higher number of hops generally indicates greater uncertainty.

[0079] Specifically, before calculating the transmission bid score, each data source agent can start a background monitoring thread to periodically read the kernel statistics of the operating system, such as reading the / proc file system or calling system APIs, to obtain real-time CPU load, disk I / O status and network card traffic data, thereby obtaining the agent's own resource information.

[0080] Additionally, network costs can be calculated using preset charging rules, and path costs can be estimated using tools such as Traceroute. Finally, these collected multi-dimensional parameters are substituted into a preset weighted scoring formula. For example, network bandwidth and disk read speed can be given positive weights, while CPU load, network cost, and path hop count can be given negative weights, thereby calculating a bid score that comprehensively reflects the node's transmission capabilities.

[0081] The method provided in this invention clarifies the key physical indicators that determine transmission quality. Through multi-dimensional resource evaluation and cost calculation, it ensures that the bidding results can truly reflect the actual carrying capacity of the nodes and the network environment. This not only avoids assigning tasks to nodes with computational overload or storage bottlenecks, but also tends to select paths with low network latency and sufficient bandwidth, thereby ensuring the efficiency and stability of data transmission at the physical level.

[0082] Based on any of the above embodiments Figure 6 This is the second flowchart of the distributed dataset transmission method based on multi-agent collaboration provided by the present invention, as shown below. Figure 6 As shown, the method includes: vaginal node, source node A, source node B, source node C, and destination node D.

[0083] The guiding agent typically resides on the user side or a separate management node. Its responsibilities include receiving transmission tasks submitted by the user, containing a list of distributed datasets, parsing the list, and initializing, activating, or deploying the transmission agents on all involved source and destination nodes, thereby waking up the entire agent network. After initialization, the guiding agent can exit and no longer participate in subsequent transmission decisions.

[0084] Furthermore, transport agents are typically deployed on each data source node and destination node. Each transport agent is an autonomous software process with perception, decision-making, and communication capabilities. Together, they form a peer-to-peer data transmission network, collaboratively completing the entire data transmission task.

[0085] Specifically, the first step is to execute task publishing and agent network construction. In this phase, a user-submitted transmission task is received, defining a logically distributed dataset and its final destination. Then, one or more guiding agents can activate or deploy transmission agents on all relevant data source and destination nodes based on this list, building a full-link agent communication network.

[0086] Then, distributed topology awareness and resource assessment are performed. At this stage, after the agent network is built, each transport agent begins to autonomously perceive its environment. The transport agents not only identify the data fragment information they hold locally, but also jointly construct a global topology map based on the distributed dataset through peer-to-peer communication protocols, including which fragments have redundant copies and the physical proximity relationships of each node. Simultaneously, each agent assesses the resource status of its local nodes, such as storage I / O load, CPU utilization, and network egress bandwidth.

[0087] Next, decentralized task negotiation and path planning are performed. At this stage, based on the perceived global topology and local resource state, the transmitting agents autonomously allocate tasks and plan paths through a decentralized negotiation protocol, such as the Contract Network protocol. The destination node's agent, acting as a "buyer," publishes data requests, while the source node's agent, acting as a "seller," submits "bids." They comprehensively consider their own load, network costs, and transmission paths, ultimately forming a dynamic, many-to-many data flow transmission plan. This plan may include non-directly connected aggregation-forwarding paths.

[0088] Furthermore, data stream aggregation and offset transmission are performed. During transmission, the agent designated as the aggregator in negotiation first pulls data from multiple nearby source nodes, aggregates it into a larger data stream, and then sends it to the destination at high speed to improve network transmission efficiency. For data fragments with redundant replicas, the destination agent can initiate offset transmission, that is, simultaneously request data from two or more source agents holding the replica, adopt the data replica that arrives first, and then cancel other requests, thereby effectively reducing long-tail latency.

[0089] Furthermore, during transmission, agents monitor each other through a heartbeat mechanism. When a source agent loses connection due to a node failure, its neighboring agents quickly detect the failure. The agent network immediately triggers a task renegotiation, automatically finding a backup node holding the same data copy to take over the transmission task, or adjusting the overall transmission plan to bypass the faulty area, thereby achieving rapid self-healing without human intervention.

[0090] The method provided in this invention aims to solve the challenge of efficiently aggregating large-scale, physically dispersed datasets. The method includes: deploying transmission agents at the data source and destination nodes to construct a communication network; the agents collaboratively perceiving the global topology and redundant copies of data fragments through a peer-to-peer communication protocol; through a decentralized negotiation protocol on a contract network, the agents autonomously decide to form a transmission plan, which may include aggregated transmission paths executed by intermediate agents; for redundant data, offsetting transmissions can be initiated to reduce latency. The system achieves rapid fault detection through heartbeat monitoring between agents and automatically triggers task renegotiation for self-healing. Therefore, by adopting a decentralized architecture, single-point bottlenecks are avoided, and the method offers advantages such as high scalability, high robustness, and high efficiency, making it suitable for scientific computing, big data analysis, and other scenarios in wide area network environments.

[0091] Based on any of the above embodiments Figure 7 This is a schematic diagram of the structure of the distributed dataset transmission device based on multi-agent collaboration provided by the present invention, as shown below. Figure 7 As shown, the device includes: The agent activation unit 710 determines the transmitting agent corresponding to the dataset to be transmitted, wherein the transmitting agent includes a data source agent and a destination agent. The topology graph construction unit 720 constructs a global topology graph based on the target agent, the fragmentation information of the target data fragment held by the data source agent, and the physical location of the agent. The negotiation transmission unit 730 performs decentralized transmission negotiation based on the global topology graph and the data source agent's own resource information, and generates and executes the transmission plan for the dataset to be transmitted.

[0092] The apparatus provided in this invention constructs a global topology map by using the fragmentation information of the target data fragments held by the target agent and the data source agent, as well as the physical location of the agents. Based on the global topology map and the resource information of the data source agent, it performs decentralized transmission negotiation, generates and executes a transmission plan for the dataset to be transmitted. This avoids the performance bottleneck of a single control node when handling a large number of nodes, and automatically avoids nodes with excessive load or network congestion, selecting the data source with the best state for transmission. This achieves global load balancing and resource optimization, significantly improving the overall transmission efficiency and robustness of the distributed dataset.

[0093] Based on any of the above embodiments, the negotiation transmission unit is specifically used for: Based on the target agent, a tender for the transmission task of the dataset to be transmitted is broadcast to the data source agent. Based on the data source agent's response to the transmission task bidding, the agent calculates the transmission bidding score according to its own resource information and transmission cost, and returns the transmission bidding score to the target agent; Based on the target agent and the received transmission bid score, the execution agent is selected from the transmission agents; Based on the executing agent and the destination agent, the transmission plan is generated and executed.

[0094] Based on any of the above embodiments, the transmitting agent further includes an intermediate agent adjacent to the data source agent; The negotiation transmission unit is also specifically used for: Based on the intermediate agent's own resource information and aggregation cost, and based on the forwarding cost between the intermediate agent and the target agent, the aggregated transmission bid score is calculated; the aggregation cost includes the transmission cost between the intermediate agent and at least one neighboring data source agent. The aggregated transmission bid score is used as the transmission bid score; Accordingly, when the intermediate agent acts as the executing agent, the negotiation transmission unit is further specifically used for: Determine an aggregation transmission plan between the intermediate agent and the at least one neighboring data source agent, and determine a forwarding transmission plan between the intermediate agent and the destination agent; The aggregated transmission plan and the forwarding transmission plan are used as the transmission plan, and the transmission plan is executed.

[0095] Based on any of the above embodiments, the negotiation transmission unit is further specifically used for: In the case where there are multiple data replicas corresponding to the same target data fragment with multiple transmission bid scores, the data source agent corresponding to the highest transmission bid score of the same target data fragment is selected as the execution agent.

[0096] Based on any of the above embodiments, the negotiation transmission unit is further specifically used for: All data source agents corresponding to the same target data shards are used as the execution agents; if any data source agent in all the data source agents completes the data transmission, the remaining data source agents in all the data source agents are removed from the list of execution agents.

[0097] Based on any of the above embodiments, the negotiation transmission unit is further specifically used for: If a failure is detected in any data source agent, a backup agent that holds the same data copy as the failed agent is identified. The backup agent is used as the executing agent.

[0098] Based on any of the above embodiments, the self-resource information includes agent CPU load information, agent storage status, and agent network bandwidth; The transmission cost includes network cost and transmission path cost.

[0099] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a distributed dataset transmission method based on multi-agent collaboration. The method includes: determining the transmitting agent corresponding to the dataset to be transmitted, the transmitting agent including a data source agent and a destination agent; constructing a global topology map based on the destination agent, as well as the fragmentation information of the target data fragment held by the data source agent and the agent's physical location; and performing decentralized transmission negotiation based on the global topology map and the data source agent's own resource information to generate and execute a transmission plan for the dataset to be transmitted.

[0100] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the distributed dataset transmission method based on multi-agent collaboration provided by the above methods. The method includes: determining the transmission agent corresponding to the dataset to be transmitted, the transmission agent including a data source agent and a destination agent; constructing a global topology graph based on the destination agent, the fragmentation information of the target data fragment held by the data source agent, and the physical location of the agent; and performing decentralized transmission negotiation based on the global topology graph and the data source agent's own resource information to generate and execute a transmission plan for the dataset to be transmitted.

[0102] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the distributed dataset transmission method based on multi-agent collaboration provided by the above methods. The method includes: determining the transmission agent corresponding to the dataset to be transmitted, the transmission agent including a data source agent and a destination agent; constructing a global topology graph based on the destination agent, as well as the fragmentation information of the target data fragment held by the data source agent and the physical location of the agent; and performing decentralized transmission negotiation based on the global topology graph and the data source agent's own resource information to generate and execute a transmission plan for the dataset to be transmitted.

[0103] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed data set transmission method based on multi-agent collaboration, characterized in that, The method comprises: determining a transmission agent corresponding to a data set to be transmitted, the transmission agent comprising a data source agent and a destination agent; constructing a global topology graph based on the destination agent and shard information and physical location of a target data shard held by the data source agent; performing decentralized transmission negotiation based on the global topology graph and self-resource information of the data source agent, generating and executing a transmission plan of the data set to be transmitted.

2. The method according to claim 1, wherein, The method of performing decentralized transmission negotiation based on the global topology graph and self-resource information of the data source agent, generating and executing a transmission plan of the data set to be transmitted comprises: broadcasting a transmission task tender of the data set to be transmitted by the destination agent to the data source agent; calculating a transmission bid score according to the self-resource information and transmission cost, and returning the transmission bid score to the destination agent in response to the transmission task tender by the data source agent; selecting an execution agent from the transmission agent according to the received transmission bid score by the destination agent; generating and executing the transmission plan based on the execution agent and the destination agent. 3.The method of claim 2, wherein, The transmission agent further comprises an intermediate agent adjacent to the data source agent. The method of calculating a transmission bid score according to the self-resource information and transmission cost comprises: calculating an aggregated transmission bid score based on self-resource information and aggregation cost of the intermediate agent, and based on forwarding cost between the intermediate agent and the destination agent; the aggregation cost comprises transmission cost between the intermediate agent and at least one adjacent data source agent; taking the aggregated transmission bid score as the transmission bid score; Accordingly, in the case that the intermediate agent is the execution agent, the method of generating and executing the transmission plan based on the execution agent and the destination agent comprises: determining an aggregated transmission plan between the intermediate agent and the at least one adjacent data source agent, and determining a forwarding transmission plan between the intermediate agent and the destination agent; taking the aggregated transmission plan and the forwarding transmission plan as the transmission plan, and executing the transmission plan. 4.The multi-agent collaboration based distributed data set transmission method of claim 2, wherein, The method of selecting an execution agent from the transmission agent according to the received transmission bid score by the destination agent comprises: in the case that there are multiple data replicas corresponding to the same target data shard and multiple transmission bid scores, selecting a data source agent corresponding to the highest transmission bid score as the execution agent.

5. The multi-agent collaboration based distributed data set transmission method of claim 2, wherein, The method of selecting an execution agent from the transmission agent according to the received transmission bid score by the destination agent comprises: taking all data source agents corresponding to the same target data shard as the execution agent; in the case that any data source agent among the all data source agents completes data transmission, canceling the remaining data source agents among the all data source agents as execution agents.

6. The multi-agent collaboration based distributed data set transmission method according to any one of claims 2 to 5, characterized in that, The method further comprises: If a failure is detected in any data source agent, a backup agent that holds the same data copy as the failed agent is identified. The backup agent is used as the executing agent.

7. The multi-agent collaboration based distributed data set transmission method according to any one of claims 2 to 5, characterized in that, The self-resource information includes agent CPU load information, agent storage status, and agent network bandwidth. The transmission cost includes network cost and transmission path cost.

8. A distributed data set transmission device based on multi-agent cooperation, characterized in that, include: The agent activation unit determines the transmitting agent corresponding to the dataset to be transmitted, wherein the transmitting agent includes a data source agent and a destination agent; The topology graph construction unit constructs a global topology graph based on the target agent, the fragmentation information of the target data fragment held by the data source agent, and the physical location of the agent. The negotiation transmission unit performs decentralized transmission negotiation based on the global topology graph and the data source agent's own resource information, and generates and executes the transmission plan for the dataset to be transmitted.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the distributed dataset transmission method based on multi-agent cooperation as described in any one of claims 1 to 7. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the distributed dataset transmission method based on multi-agent cooperation as described in any one of claims 1 to 7.

Images