A data transmission system and method

Abstract

The application provides a data transmission system and method, belonging to the technical field of data transmission, the system comprising a service layer interface, a message middleware and a distributed intelligent agent network; the message middleware communicates with the service layer interface through an application programming interface, and communicates with each intelligent agent in the distributed intelligent agent network through a standard protocol, and provides event publishing and task subscription services for each intelligent agent in the distributed intelligent agent network through an event bus; the distributed intelligent agent network comprises a decision module, and different decision intelligent agent modules in the decision module are used for executing different steps in a transmission strategy generation process of a data transmission task. The application sets the message middleware between the service layer interface and the distributed intelligent agent network, and realizes loose coupling cooperation between intelligent agents through a message driving mechanism and a standardized protocol of the message middleware, thereby solving the single-point performance bottleneck problem of centralized architecture design and the problem of non-uniform data exchange interfaces.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and in particular to a data transmission system and method. BACKGROUND

[0002] With the rapid development and wide application of artificial intelligence, big data and cloud computing technologies, data has become the core production material and strategic basic resource in modern information society. Efficient and reliable data transmission systems constitute the key infrastructure supporting various data-driven applications.

[0003] With the development and spillover of artificial intelligence technologies, the industry is gradually exploring the application of artificial intelligence technologies to realize intelligent scheduling of high-throughput transmission. However, the current mainstream open source and commercial data transmission systems and intelligent scheduling systems generally adopt centralized architecture design, with all decision-making logic concentrated in a single central node for execution, which makes the central node a performance bottleneck of the data transmission system or intelligent scheduling system, and there is a high risk of single point failure, making it difficult to guarantee the reliability of the data transmission system or intelligent scheduling system. In addition, the data exchange interfaces between functional components in the centralized architecture lack unified standardized design, which makes the context information of the artificial intelligence model in the data transmission system or intelligent scheduling system unable to be efficiently transmitted and shared, seriously restricting the overall collaborative decision-making capability of the system. SUMMARY

[0004] The present application provides a data transmission system and method, aiming to solve the problem of low overall collaborative decision-making capability caused by the single-point performance bottleneck of the centralized architecture design and the non-uniform data exchange interface.

[0005] In a first aspect, the present application provides a data transmission system, comprising a service layer interface, a message middleware and a distributed intelligent agent network;

[0006] The service layer interface is configured to receive a data transmission task and return a processing report of the data transmission task.

[0007] The message middleware communicates with the service layer interface through an application programming interface, and communicates with each intelligent agent in the distributed intelligent agent network through a standard protocol, and provides event publishing and task subscription services for each intelligent agent in the distributed intelligent agent network through an event bus.

[0008] The distributed intelligent agent network comprises a decision module, different decision intelligent agent modules in the decision module are configured to execute different steps in the transmission strategy generation process of the data transmission task, and the decision module forwards the transmission strategy to a data transmission node through the message middleware.

[0009] As an embodiment, the distributed agent network further comprises an execution module, the execution module comprising execution agent modules deployed on each data transmission node, each execution agent module executing data transmission on the data transmission node where the execution agent module is located.

[0010] As an embodiment, the distributed agent network further comprises a monitoring and optimization module, different monitoring and optimization agents within the monitoring and optimization module performing different monitoring or optimization operations on all agents within the distributed agent network.

[0011] As an embodiment, the decision module comprises an analysis agent module and a scheduling agent module;

[0012] The analysis agent module is configured to analyze the network state of the distributed agent network, the characteristics of the data transmission task, and the running state of each data transmission node, and forward the analysis results to the scheduling agent module through the message middleware;

[0013] The scheduling agent module is configured to formulate a transmission strategy for the data transmission task based on the analysis results, and forward the transmission strategy to the data transmission nodes through the message middleware.

[0014] As an embodiment, any execution agent module comprises a data sharding agent and a transmission channel management agent;

[0015] The data sharding agent is configured to divide the data of the data transmission node where the execution agent module is located into multiple data shards;

[0016] The transmission channel management agent is configured to establish multiple parallel transmission channels, and transmit the multiple data shards in parallel to the next data transmission node through the multiple transmission channels.

[0017] As an embodiment, the analysis agent module comprises a data characteristic analysis agent, a network state analysis agent, and a resource utilization analysis agent;

[0018] The data characteristic analysis agent is configured to analyze the characteristics of the data transmission task, and forward the characteristics to the scheduling agent module through the message middleware;

[0019] The network state analysis agent is configured to collect and analyze the network state of the distributed agent network, and forward the network state to the scheduling agent module through the message middleware;

[0020] The resource utilization analysis agent is configured to analyze the running state of each data transmission node, and forward the running state to the scheduling agent module through the message middleware.

[0021] As an embodiment, the scheduling agent module comprises a priority scheduling agent and a transmission strategy generation agent;

[0022] The priority scheduling agent is configured to schedule tasks based on the priorities and deadlines of all current data transmission tasks, and determine a task scheduling result;

[0023] The transmission strategy generation agent is configured to generate a transmission strategy of the data transmission task based on the analysis result and the task scheduling result, and forward the transmission strategy to the data transmission node through the message middleware.

[0024] As an embodiment, the monitoring and optimization module comprises an index collection and analysis agent and a performance prediction and optimization agent;

[0025] The index collection and analysis agent is configured to collect transmission performance indexes and resource utilization conditions of each data transmission node in real time;

[0026] The performance prediction and optimization agent is configured to optimize the historical data based decision module.

[0027] In a second aspect, the present application further provides a data transmission method based on the decision module, the data transmission method comprising:

[0028] In response to receiving the data transmission task from the message middleware, generating a transmission strategy of the data transmission task as a decision result;

[0029] Publishing a decision result event to the message middleware for the message middleware to forward the transmission strategy to the data transmission node.

[0030] In a third aspect, the present application further provides a data transmission method based on the execution module, the data transmission method comprising:

[0031] In response to receiving the data to be transmitted from the message middleware, splitting the data to be transmitted into a plurality of data fragments;

[0032] Establishing a plurality of parallel transmission channels;

[0033] Publishing a transmission start event to the message middleware;

[0034] In response to receiving a transmission start notification from the message middleware, transmitting the plurality of data fragments to the next data transmission node in parallel through the plurality of transmission channels;

[0035] In response to completion of transmission of the data to be transmitted, publishing a transmission completion event to the message middleware;

[0036] Receiving a transmission completion notification from the message middleware. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0038] Figure 1 is one of the structural schematic diagrams of the data transmission system provided by the application;

[0039] Figure 2 is the second structural schematic diagram of the data transmission system provided by the application;

[0040] Figure 3 is one of the flow schematic diagrams of the transmission task submission stage and the decision stage in the data transmission method provided by the application;

[0041] Figure 4 is one of the flow schematic diagrams of the transmission execution stage and the completion and feedback stage in the data transmission method provided by the application;

[0042] Figure 5 is the structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the application clearer, the technical solutions in the application will be clearly and completely described below in combination with the accompanying drawings in the application. Obviously, the described embodiments are some embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the application.

[0044] It should be noted that in the description of the application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0045] The application takes message middleware as the underlying communication framework, and realizes efficient and reliable transmission of large-scale data through multi-agent collaborative decision and adaptive optimization, thereby solving the above technical problems.

[0046] The data transmission system and method provided by the application will be described below in combination with Figures 1 to 5

[0047] Figure 1 ​Fig. 1 is a structural schematic diagram of a data transmission system provided by the present application. Figure 2 Fig. 2 is another structural schematic diagram of a data transmission system provided by the present application.

[0048] As shown in Figure 1 and Figure 2 , the data transmission system provided by the present application comprises a service layer interface, a message middleware and a distributed intelligent agent network.

[0049] The service layer interface provides services for upper-layer applications and users. The service layer interface comprises a transmission task application programming interface (API), a monitoring and reporting API and a policy configuration API.

[0050] The transmission task API provides a data transmission task submission and management interface, for receiving data transmission tasks submitted by upper-layer applications and users (for example, data producers and data consumers). The monitoring and reporting API provides a state monitoring and reporting function of the data transmission system (for example, returning a processing report of a data transmission task to an upper-layer application or user). The policy configuration API allows a user to customize transmission policies and parameters.

[0051] The message middleware communicates with the service layer interface through an API interface, and communicates with each intelligent agent in the distributed intelligent agent network through a standard protocol, and provides an event publishing and task subscription service for each intelligent agent in the distributed intelligent agent network through an event bus.

[0052] The message middleware, as a bottom-layer communication framework of the system, provides an efficient and reliable context transmission mechanism.

[0053] In a possible implementation manner, the message middleware is an MCP (Message Channel Protocol, a communication protocol based on a message channel) middleware. The MCP middleware, as an efficient and reliable communication mechanism, is centered on a model context message, supports asynchronous communication, reliable transmission and protocol evolution, and provides a solid foundation for constructing a high-performance intelligent distributed system. As shown in Figure 2 , the MCP middleware comprises the following functions:

[0054] Message routing service: responsible for message routing and distribution between the service layer interface and the distributed intelligent agent network.

[0055] Service registration center: manages various services and resources in the data transmission system.

[0056] Event bus: provides event publishing and task subscription services for individual agents within the distributed agent network, a memory-based publish / subscribe system suitable for high real-time requirement scenarios.

[0057] Message storage: ensures message persistence and reliable delivery.

[0058] Unified resource interface: provides a unified abstraction and access mechanism for system resources, and communicates with individual agents within the distributed agent network through standard protocols.

[0059] Unified method call interface: provides a standardized method call mechanism across services and nodes.

[0060] Automatic discovery mechanism: provides automatic discovery and registration of services, resources and capabilities, and enables dynamic perception and adaptation of resources and capabilities.

[0061] The distributed agent network, as the intelligent decision-making core of the data transmission system, is composed of multiple specialized agents.

[0062] In one possible implementation, the distributed agent network includes a decision module, which adopts a distributed network structure and includes multiple decision agent modules with different functions. Different decision agent modules within the decision module are used to perform different steps in the transmission strategy generation process of the data transmission task, and the decision module forwards the transmission strategy to the data transmission node through the message middleware.

[0063] As can be understood, the data transmission system also includes a data transmission resource pool that provides actual data transmission resources. The data transmission resource pool works together with the decision module and the execution module to realize resource deployment. The data transmission resource pool includes a transmission node pool, network link resources and a storage resource pool. The transmission node pool is a collection of server nodes for data transmission, providing multi-regional and multi-level transmission nodes, which can be physical servers, virtual machines or containers, etc. The network link resources are network links connecting various transmission nodes, including public networks, private lines, SDN and other network resources. The storage resource pool is used to provide temporary storage and data caching functions, including hard disks, solid state disks, network storage, etc.

[0064] The embodiments of the present application set a message middleware between a service layer interface and a distributed intelligent agent network, the intelligent agents realize loose coupling cooperation through a message driving mechanism and a standardized protocol of the message middleware, and context data is efficiently delivered through the message middleware, so that each intelligent agent can make a decision based on global information. Compared with a centralized architecture design, the present application solves the single-point performance bottleneck problem of the centralized architecture design, simultaneously solves the problem of non-uniform data exchange interface, and significantly improves decision accuracy. Moreover, the loose coupling architecture of the message middleware enables the system to be seamlessly expanded, easily cope with the growth of data size and node number, and each component communicates through a standardized interface, supports flexible deployment and dynamic expansion in a heterogeneous environment. In addition, under the loose coupling architecture, each component in the data transmission system can be independently upgraded and expanded, the influence of single-point failure is limited, and the overall reliability of the system is significantly improved.

[0065] In a possible implementation, the decision module includes an analysis intelligent agent module and a scheduling intelligent agent module.

[0066] The analysis intelligent agent module is configured to analyze the network state of the distributed intelligent agent network, the characteristics of the data transmission task, and the running state of each data transmission node, and forward the analysis result to the scheduling intelligent agent module through the message middleware.

[0067] In a possible implementation, the analysis intelligent agent module can sample the data of the data transmission task, extract part of the data for analysis, and reduce the calculation complexity.

[0068] The scheduling intelligent agent module is configured to formulate a transmission strategy for the data transmission task based on the analysis result, and forward the transmission strategy to the data transmission node through the message middleware.

[0069] The embodiments of the present application decompose the transmission strategy generation process into analysis and decision through the analysis intelligent agent module and the scheduling intelligent agent module, and realize different functions through different intelligent agents. Compared with a scheme of realizing the transmission strategy generation by a single intelligent agent, the disintegration of functions can improve processing efficiency. Moreover, the embodiments of the present application combine intelligent analysis and adaptive optimization of the transmission strategy, so that the data transmission efficiency is significantly improved compared with a traditional scheme, and support efficient transmission of PB-level (data size of 100 million billion bytes) data.

[0070] In a possible implementation, the analysis intelligent agent module includes a data characteristic analysis intelligent agent, a network state analysis intelligent agent, and a resource utilization analysis intelligent agent.

[0071] The data characteristic analysis intelligent agent is configured to analyze the characteristics of the data transmission task, and forward the characteristics to the scheduling intelligent agent module through the message middleware. The characteristics include the data size, type (including structured and unstructured) and structure of the data transmission task.

[0072] In one possible implementation, the data feature analysis agent can obtain data types and other features by parsing the metadata information of the data.

[0073] In one possible implementation, the data feature analysis agent can use machine learning algorithms such as classification and clustering to identify data types and features. For example, for image data, a convolutional neural network can be used for feature extraction; for text data, natural language processing techniques can be used.

[0074] The network state analysis agent is used to collect and analyze the network state of the distributed agent network, and forward the network state to the scheduling agent module through the message middleware. The network state includes network topology, bandwidth, and delay, etc.

[0075] In one possible implementation, the network state analysis agent can use tools such as Ping, Traceroute, and iPerf to measure network performance indicators.

[0076] In one possible implementation, the network state analysis agent can use tools such as Wireshark and tcpdump to capture and analyze network traffic.

[0077] In one possible implementation, the TCP parameter optimization interface provided by the message middleware can dynamically adjust the TCP window size, retransmission timeout, and congestion control parameters of each agent in the execution module (see the following description) according to the real-time network state obtained by the network state analysis agent, greatly improving transmission efficiency.

[0078] The resource utilization analysis agent is used to analyze the running state of each data transmission node (including CPU usage, memory occupancy, disk input / output (I / O), network I / O, etc.), and forward the running state to the scheduling agent module through the message middleware. The running state includes the load and availability of the data transmission node.

[0079] In one possible embodiment, the resource utilization analysis agent can use tools such as Nagios and Zabbix to monitor system resource usage.

[0080] In one possible implementation, when the system runs in a container environment, the resource utilization analysis agent can use tools such as cAdvisor and Kubernetes Metrics Server to monitor container resource usage.

[0081] The embodiments of the present application analyze the system according to data characteristics, network states, resource utilization and other multi-dimensions, provide a data basis for automatic optimization of transmission strategies, and enable the transmission strategies to adapt to complex and changeable network environments.

[0082] In a possible implementation, the scheduling agent module includes a priority scheduling agent and a transmission strategy generation agent.

[0083] The priority scheduling agent is configured to schedule tasks based on the priorities and deadlines of all current data transmission tasks, and determine a task scheduling result.

[0084] In a possible implementation, the priority scheduling agent can use a priority queue to manage tasks to be transmitted, and schedule the tasks in a priority order.

[0085] In a possible implementation, the priority scheduling agent can use a deadline scheduling algorithm such as Earliest Deadline First (EDF).

[0086] The transmission strategy generation agent is configured to generate a transmission strategy for data transmission tasks based on the analysis result and the task scheduling result, and forward the transmission strategy to data transmission nodes through a message middleware. In one aspect, the transmission strategy generation agent can intelligently select an optimal transmission path to avoid congested areas and significantly reduce end-to-end delay based on a network state perception and path selection interface in a multi-path network environment.

[0087] In a possible implementation, the transmission strategy generation agent uses a reinforcement learning model to generate a transmission strategy, and learns to select an optimal transmission strategy in different states.

[0088] The embodiments of the present application use a distributed agent to implement task scheduling and strategy generation, and generate a transmission strategy based on the analysis result and the task scheduling result according to real-time states, so that the system resource utilization rate is greatly improved, resource waste and overload are avoided, and the efficiency and accuracy of strategy generation are improved.

[0089] In a possible implementation, the scheduling agent module further includes a resource allocation optimization agent, which is responsible for globally optimizing and configuring resources such as transmission nodes, network links and storage resources. Through intelligent resource scheduling and optimization, hardware resource costs and energy consumption are effectively reduced.

[0090] In a possible implementation, the resource allocation optimization agent can use a linear programming model to optimize resource allocation, for example, to maximize transmission throughput or minimize transmission cost.

[0091] In a possible implementation, the resource allocation optimization intelligent agent can use an integer programming model to handle the discrete constraint of resource allocation.

[0092] In a possible implementation, a heuristic algorithm (such as a greedy algorithm or a simulated annealing algorithm) can be used for resource allocation.

[0093] In a possible implementation, the distributed intelligent agent network further includes an execution module, the execution module being configured to execute a specific data transmission task. The execution module includes an execution intelligent agent module deployed on each data transmission node, and each execution intelligent agent module executes data transmission on the data transmission node where the execution intelligent agent module is located.

[0094] By deploying the execution intelligent agent module on the data transmission node, the embodiments of the present application can make autonomous decisions and execution according to the received context data, and make internal transmission decisions based on global information, so that the data transmission on each data transmission node can be optimized in real time, and the overall efficiency of data transmission is improved.

[0095] In a possible implementation, any execution intelligent agent module includes a data sharding intelligent agent and a transmission channel management intelligent agent.

[0096] The data sharding intelligent agent is configured to split the data of the data transmission node where the execution intelligent agent module is located into multiple data shards. It can be understood that the data sharding intelligent agent formulates a data splitting strategy in real time according to the task undertaken by the data transmission node, the resource occupation condition, and the like.

[0097] In a possible implementation, the data sharding intelligent agent can split the data into shards of a fixed size.

[0098] In a possible implementation, the data sharding intelligent agent can split the data based on the data content, for example, according to file boundaries or record boundaries.

[0099] The transmission channel management intelligent agent is configured to establish multiple parallel transmission channels, and transmit the multiple data shards in parallel to the next data transmission node through the multiple transmission channels.

[0100] In a possible implementation, the transmission channel management intelligent agent can use a multi-thread or multi-process technology to create parallel transmission channels.

[0101] The embodiments of the present application implement data sharding and multi-channel parallel transmission through distributed intelligent agents, greatly improve the reliability of data transmission, and greatly reduce the transmission failure rate.

[0102] In a possible implementation, any of the executing agent modules further comprises an error retry processing agent, which is responsible for detecting and processing transmission errors to achieve automatic recovery. On the basis of data sharding and multi-channel parallel transmission, the data transmission node further improves the error handling capability of the data transmission node itself through the error retry processing agent, thereby further improving the transmission success rate.

[0103] In the existing centralized data transmission architecture, the scheduling algorithm and optimization strategy are usually fixed in a preset manner and built into the system core module, lacking dynamic adjustment capability and environmental adaptation capability, and being unable to effectively cope with complex and variable network environments and transmission requirements.

[0104] Based on the above considerations, in a possible implementation, the distributed agent network further comprises a monitoring and optimization module, which is responsible for collecting system running state indicators and transmission indicators and continuously optimizing transmission strategies. Specifically, different monitoring and optimization agents in the monitoring and optimization module perform different monitoring or optimization operations on all agents in the distributed agent network.

[0105] The embodiments of the present application can dynamically adapt to environmental changes through continuous learning of the distributed agents, and the system performance will be continuously improved over time, realizing self-evolution. Moreover, the present application realizes autonomous learning and optimization through the monitoring and optimization module, reduces manual intervention, and significantly reduces operation and maintenance costs.

[0106] In a possible implementation, the monitoring and optimization module comprises an indicator collection and analysis agent and a performance prediction and optimization agent.

[0107] The indicator collection and analysis agent is configured to collect transmission performance indicators and resource utilization conditions of each data transmission node in real time.

[0108] In a possible implementation, the indicator collection and analysis agent can obtain performance counter data provided by the operating system to realize indicator collection.

[0109] In a possible implementation, the indicator collection and analysis agent can use tools such as Ping, Traceroute, iPerf, etc. to measure network performance indicators.

[0110] In a possible implementation, the indicator collection and analysis agent can extract performance indicators by analyzing system logs.

[0111] The performance prediction and optimization agent is configured to optimize the decision module based on historical data, for example, to predict system performance and provide optimization suggestions.

[0112] The embodiments of the present application realize the collection of indicators and the analysis and optimization of decision performance through distributed agents, respectively, thereby improving the efficiency of monitoring and optimization.

[0113] In a possible implementation, the monitoring and optimization module further includes an exception detection intelligent agent, which is configured to detect the state of each intelligent agent in the distributed intelligent agent network, and trigger a coping strategy corresponding to an exception when any intelligent agent has an exception.

[0114] The embodiment of the application improves the supervision of the distributed intelligent agent network through the exception identification and feedback mechanism, and improves the operation and maintenance quality.

[0115] In a possible implementation, the monitoring and optimization module further includes a model training and updating intelligent agent, which is responsible for training and updating each intelligent agent in the distributed intelligent agent network based on historical data periodically or aperiodically.

[0116] The embodiment of the application realizes the automatic updating mechanism of the data transmission system through the model training and updating intelligent agent, and reduces the manual operation and maintenance cost.

[0117] Based on the above, the application further provides a data transmission method. The data transmission method can be mutually referred to the data transmission system. It should be noted that the data transmission method provided by the embodiment of the application is realized based on the data transmission system.

[0118] The embodiment of the application takes the data transmission system as an execution subject to describe the data transmission method.

[0119] As an example, as shown in Figure 3 The data transmission method includes:

[0120] S310: a transmission task submission stage, which receives the data transmission task of the upper layer application and the user (data source) through the service layer interface, and distributes the data transmission task to the distributed intelligent agent network through the event bus of the message middleware.

[0121] Specifically, as shown in Figure 3 After the data source submits the data transmission task to the service layer interface, the service layer interface generates a unique task identifier corresponding to the data transmission task, and publishes a task event to the message middleware. The message middleware transmits the task information to the analysis intelligent agent module of the decision module.

[0122] S320: a decision stage, in which the decision module generates a transmission strategy of the data transmission task, and realizes intelligent decision.

[0123] In a possible implementation, for the decision module, this stage includes:

[0124] Q1: in response to receiving the data transmission task from the message middleware, generating a transmission strategy of the data transmission task as a decision result;

[0125] Q2: Publish the decision result event to the message middleware, so that the message middleware can forward the transmission strategy to the data transmission node.

[0126] This application embodiment uses message middleware to monitor the decision-making progress, thereby improving the monitoring of the entire data transmission process.

[0127] In one possible implementation, the decision-making phase includes an analysis phase and a strategy generation phase:

[0128] During the analysis phase, the analysis agent module subscribes to and receives data transmission tasks through the event bus of the message middleware. It obtains system metrics through the unified resource interface of the message middleware and collaboratively analyzes data characteristics, network conditions, and resource availability through the various agents within the analysis agent module.

[0129] Specifically, such as Figure 3 As shown, the data feature analysis agent analyzes the characteristics of the data transmission task (e.g., data size, type, structure, etc.), the network state analysis agent collects and analyzes the network state of the distributed agent network (e.g., network topology, bandwidth, latency, etc.), and the resource utilization analysis agent analyzes the current operating status of each data transmission node (e.g., load, availability, etc.). The data feature analysis agent, network state analysis agent, and resource utilization analysis agent publish analysis result events to the message middleware, which then transmits the analysis result information to the scheduling agent module.

[0130] During the strategy generation phase, the scheduling agent module generates a transmission strategy based on the analysis results by using the unified method call interface of the message middleware, dynamically optimizes resource allocation, and obtains the transmission strategy for the data transmission task.

[0131] Specifically, such as Figure 3 As shown, the resource allocation optimization agent performs global optimization of resource allocation. The priority scheduling agent schedules tasks based on the priority and deadline of all current data transmission tasks, determining the scheduling result. The transmission policy generation agent generates transmission policies for data transmission tasks using a reinforcement learning model based on the analysis results and task scheduling results. Both the priority scheduling agent and the transmission policy generation agent publish their decision results to the message middleware, transmitting the task scheduling results and transmission policies to the message middleware. The message middleware then transmits the transmission policies and scheduling decisions to the execution module.

[0132] S330: Transmission execution phase. In this phase, each agent in the execution module optimizes TCP parameters through the parameter optimization interface of the message middleware, performs data fragmentation, and transmits the data to the data receiver in parallel through multiple channels.

[0133] In a possible implementation, for the execution module, the stage includes the following steps:

[0134] P1: in response to receiving the data to be transmitted from the message middleware, splitting the data to be transmitted into a plurality of data fragments;

[0135] P2: establishing a plurality of parallel transmission channels;

[0136] P3: publishing a transmission start event to the message middleware;

[0137] P4: in response to receiving a transmission start notification from the message middleware, transmitting the plurality of data fragments to the next data transmission node in parallel through the plurality of transmission channels;

[0138] P5: in response to completion of transmission of the data to be transmitted, publishing a transmission completion event to the message middleware;

[0139] P6: receiving a transmission completion notification from the message middleware.

[0140] In the embodiment of the application, the execution module realizes monitoring of the data transmission progress through interaction with the message middleware, thereby improving controllability of data transmission.

[0141] Specifically, as shown in Figure 4 , the data fragment agent splits the data of the data transmission node where the execution agent module is located into a plurality of data fragments. The transmission channel management agent establishes a plurality of parallel transmission channels. Subsequently, the transmission channel management agent publishes a transmission start event to the message middleware, and the message middleware delivers a transmission start notification to the transmission channel management agent. Subsequently, the transmission channel management agent transmits the plurality of data fragments to the next data transmission node in parallel through the plurality of transmission channels. During the multi-channel parallel transmission process, the transmission channel management agent regularly publishes a transmission status event (the transmission status includes transmission performance indicators, resource utilization status, etc.) to the message middleware, and the message middleware delivers the transmission status information to the monitoring and optimization module after receiving the transmission status.

[0142] It should be noted that the next data transmission node is the next data transmission node of the data transmission node in the transmission strategy, which can be a data receiver (as shown in Figure 4 ), or can not be a data receiver, i.e., a data transmission node in the middle of the transmission path.

[0143] In a possible implementation, during the stage, the monitoring and optimization module monitors transmission performance and continuously optimizes the transmission strategy, and monitors abnormal situations and triggers countermeasures.

[0144] Specifically, as shown in Figure 4As shown, the monitoring and optimization module determines whether there is an abnormal processing according to the transmission state information, if there is, the monitoring and optimization module publishes an abnormal processing suggestion event, i.e. a coping strategy of the transmission abnormality, to the message middleware. The message middleware delivers the abnormal processing instruction to the transmission channel management agent, and the transmission channel management agent executes a fault recovery operation after receiving the abnormal processing instruction.

[0145] Meanwhile, the monitoring and optimization module proposes an optimization suggestion of the transmission strategy according to the transmission state information, and publishes an optimization suggestion event to the message middleware. The message middleware delivers the optimization suggestion to the scheduling agent module, and the scheduling agent module adjusts the transmission strategy according to the optimization suggestion. Then, the scheduling agent module publishes an updated strategy event to the message middleware, and delivers the new transmission strategy to the message middleware. The message middleware delivers the updated strategy to the execution module, and the execution module applies the new transmission strategy for data transmission, forming a closed-loop optimization.

[0146] After the execution module of each data transmission node completes the data transmission to the next data transmission node, the execution module publishes a transmission completion event to the message middleware, and the message middleware delivers a transmission completion notification to the monitoring and optimization module.

[0147] S340: a completion and feedback stage, in which each agent of the execution module publishes a processing result of the data transmission task to the message middleware, and the message middleware returns a processing report of the data transmission task to the data source.

[0148] Specifically, as shown in the figure, Figure 4 When the transmission channel management agent of the execution module of the last data transmission node in the transmission strategy sends the data to the data receiving party, the transmission channel management agent sends information of completing the data transmission to the data receiving party, and then performs data integrity verification by itself. After the verification is passed, the transmission channel management agent publishes a transmission result event to the message middleware, and delivers the processing result of the data transmission task. The message middleware delivers the processing result of the data transmission task to the service layer interface, and then the service layer interface returns a processing report of the data transmission task to the data source.

[0149] In a possible implementation, the data transmission method further includes a monitoring and optimization stage, which is after the completion and feedback stage. In the stage, the monitoring and optimization module collects performance data of the whole data transmission process, determines an updated strategy of the scheduling agent module according to the performance data, and publishes an updated strategy event to the message middleware. Then, the message middleware feeds back the updated strategy to the scheduling agent module.

[0150] Figure 5 is a structural schematic diagram of an electronic device provided by the present application, as shown in the figure, Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete communications with each other through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute the data transmission method of one of the service layer interface, the message middleware, or each agent in the distributed agent network.

[0151] In addition, the logical instruction in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or in other words the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0152] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data transmission method of one of the service layer interface, the message middleware, or each agent in the distributed agent network.

[0153] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the data transmission method of one of the service layer interface, the message middleware, or each agent in the distributed agent network.

[0154] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement it without creative labor.

[0155] Those skilled in the art can clearly understand the implementation of the embodiments by the description of the above embodiments, and the embodiments can be implemented by means of software and necessary universal hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a read-only memory (ROM) / random access memory (RAM), a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some parts of the embodiment.

[0156] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features thereof; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data transmission system, characterized in that, This includes service layer interfaces, message middleware, and distributed intelligent agent networks; The service layer interface is used to receive data transmission tasks and return processing reports for the data transmission tasks; The message middleware communicates with the service layer interface through an application programming interface, communicates with each agent in the distributed agent network through a standard protocol, and provides event publishing and task subscription services to each agent in the distributed agent network through an event bus. The distributed intelligent agent network includes a decision module, in which different decision intelligent agent modules are used to perform different steps in the transmission strategy generation process of the data transmission task. The decision module forwards the transmission strategy to the data transmission node through the message middleware. The decision-making module includes an analysis agent module and a scheduling agent module; The analysis agent module is used to analyze the network status of the distributed agent network, the characteristics of the data transmission task, and the operating status of each data transmission node, and forwards the analysis results to the scheduling agent module through the message middleware. The scheduling agent module is used to formulate a transmission strategy for the data transmission task based on the analysis results, and forward the transmission strategy to the data transmission node through the message middleware.

2. The data transmission system according to claim 1, characterized in that, The distributed intelligent agent network further includes an execution module, which includes execution intelligent agent modules deployed on each data transmission node. Each execution intelligent agent module performs data transmission on the data transmission node where the execution intelligent agent module is located.

3. The data transmission system according to claim 1 or 2, characterized in that, The distributed agent network also includes a monitoring and optimization module, in which different monitoring and optimization agents perform different monitoring or optimization operations on all agents in the distributed agent network.

4. The data transmission system according to claim 2, characterized in that, Any of the aforementioned execution agent modules includes a data sharding agent and a transmission channel management agent; The data sharding agent is used to divide the data of the data transmission node where the execution agent module is located into multiple data shards; The transmission channel management agent is used to establish multiple parallel transmission channels and transmit the multiple data fragments in parallel to the next data transmission node through the multiple transmission channels.

5. The data transmission system according to claim 2, characterized in that, The analytical agent module includes a data feature analysis agent, a network status analysis agent, and a resource utilization analysis agent; The data feature analysis agent is used to analyze the features of the data transmission task and forward the features to the scheduling agent module through the message middleware. The network state analysis agent is used to collect and analyze the network state of the distributed agent network, and forward the network state to the scheduling agent module through the message middleware; The resource utilization analysis agent is used to analyze the current operating status of each data transmission node and forward the operating status to the scheduling agent module through the message middleware.

6. The data transmission system according to claim 2, characterized in that, The scheduling agent module includes a priority scheduling agent and a transmission strategy generation agent; The priority scheduling agent is used to schedule tasks based on the priority and deadline of all current data transmission tasks and to determine the task scheduling result. The transmission strategy generating agent is used to generate the transmission strategy for the data transmission task based on the analysis results and the task scheduling results, and forwards the transmission strategy to the data transmission node through the message middleware.

7. The data transmission system according to claim 3, characterized in that, The monitoring and optimization module includes an indicator collection and analysis agent and a performance prediction and optimization agent; The indicator collection and analysis agent is used to collect the transmission performance indicators and resource utilization status of each data transmission node in real time. The performance prediction and optimization agent is used to optimize the decision-making module based on historical data.

8. A data transmission method based on the data transmission system according to any one of claims 1 to 7, characterized in that, Based on the decision module, the data transmission method includes: In response to receiving a data transmission task from the message middleware, a transmission strategy for the data transmission task is generated as a decision result; The decision result event is published to the message middleware, so that the message middleware can forward the transmission strategy to the data transmission node.

9. A data transmission method based on the data transmission system according to any one of claims 1 to 7, characterized in that, Based on the execution module, the data transmission method includes: In response to receiving data to be transmitted from the message middleware, the data to be transmitted is divided into multiple data fragments; Establish multiple parallel transmission channels; Publish a transmission start event to the message middleware; In response to receiving a transmission start notification from the message middleware, the multiple data fragments are transmitted in parallel to the next data transmission node through multiple transmission channels; In response to the completion of the data transmission to be transmitted, a transmission completion event is published to the message middleware; Receive a transmission completion notification from the message middleware.

Images