Data transmission system and method

Abstract

The invention provides a data transmission system and method, and belongs to the technical field of data transmission, and the system comprises a service layer interface, message middleware and a distributed agent network. The message middleware communicates with the service layer interface through an application programming interface, communicates with each agent in the distributed agent network through a standard protocol, and provides event publishing and task subscription services for each agent in the distributed agent network through an event bus; the distributed agent network comprises a decision module, and different decision agent modules in the decision module are used for executing different steps in the transmission strategy generation process of the data transmission task. According to the invention, the message-oriented middleware is arranged between the service layer interface and the distributed intelligent agent network, and the intelligent agents realize loose coupling cooperation through a message-driven mechanism and a standardized protocol of the message-oriented middleware, so that the problem of single-point performance bottleneck of a centralized architecture design and the problem of non-uniform data exchange interfaces are solved.

Description

Technical Field

[0001] The present application relates to the field of data transmission technology, and in particular to a data transmission system and method. Background Art

[0002] With the rapid development and widespread application of artificial intelligence, big data, and cloud computing technologies, data has become a core means of production and a strategic foundational resource in the modern information society. Efficient and reliable data transmission systems constitute the critical infrastructure supporting various data-driven applications.

[0003] With the development of artificial intelligence technology and the spillover of its capabilities, the industry is gradually exploring the application of artificial intelligence technology to achieve intelligent scheduling of high-throughput transmission. However, the current mainstream open source and commercial data transmission systems and intelligent scheduling systems generally adopt a centralized architecture design. All decision-making logic is concentrated on a single central node for execution, causing the central node to become a performance bottleneck for the data transmission system or intelligent scheduling system, and there is a high risk of single point failure. The reliability of the data transmission system or intelligent scheduling system is difficult to guarantee; in addition, the data exchange interface between functional components in the centralized architecture lacks a unified and standardized design, resulting in the inability to efficiently transmit and share the context information of the artificial intelligence model in the data transmission system or intelligent scheduling system, which seriously restricts the overall collaborative decision-making ability of the system. Summary of the Invention

[0004] The present application provides a data transmission system and method, which aims to solve the problems of single-point performance bottlenecks in centralized architecture design, which make it difficult to ensure reliability, and low overall collaborative decision-making capabilities caused by inconsistent data exchange interfaces.

[0005] In a first aspect, the present application provides a data transmission system, including a service layer interface, a message middleware, and a distributed agent network; The service layer interface is used to receive data transmission tasks and return data transmission task processing reports; The message middleware communicates with the service layer interface through the application programming interface. The message middleware communicates with each agent in the distributed agent network through the standard protocol and provides event publishing and task subscription services for each agent in the distributed agent network through the event bus. The distributed agent network includes a decision module. Different decision agent modules in the decision module are used to execute different steps in the transmission strategy generation process of the data transmission task. The decision module forwards the transmission strategy to the data transmission node through the message middleware.

[0006] As an embodiment, the distributed intelligent agent network further includes an execution module, which includes an execution intelligent agent module deployed on each data transmission node, and each execution intelligent agent module executes data transmission on the data transmission node where the execution intelligent agent module is located.

[0007] As an embodiment, the distributed agent network further includes a monitoring and optimization module, and different monitoring and optimization agents within the monitoring and optimization module perform different monitoring or optimization operations on all agents within the distributed agent network.

[0008] As an embodiment, the decision module includes an analysis agent module and a scheduling agent module; The analysis agent module is used to analyze the network status of the distributed agent network, the characteristics of the data transmission task, and the operating status of each data transmission node, and forward the analysis results to the scheduling agent module through the message middleware; The scheduling agent module is used to formulate a transmission strategy for the data transmission task based on the analysis results, and forward the transmission strategy to the data transmission node through the message middleware.

[0009] As an embodiment, any execution agent module includes a data sharding agent and a transmission channel management agent; The data sharding agent is used to split the data of the data transmission node where the execution agent module is located into multiple data shards; The transmission channel management agent is used to establish multiple parallel transmission channels and transmit multiple data fragments in parallel to the next data transmission node through multiple transmission channels.

[0010] As an embodiment, the analysis agent module includes a data feature analysis agent, a network status analysis agent, and a resource utilization analysis agent; The data feature analysis agent is used to analyze the features of the data transmission task and forward the features to the scheduling agent module through the message middleware; The network status analysis agent is used to collect and analyze the network status of the distributed agent network, and forward the network status to the scheduling agent module through the message middleware; The resource utilization analysis agent is used to analyze the current operating status of each data transmission node and forward the operating status to the scheduling agent module through the message middleware.

[0011] As an embodiment, the scheduling agent module includes a priority scheduling agent and a transmission strategy generation agent; The priority scheduling agent is used to schedule tasks based on the priorities and deadlines of all current data transmission tasks and determine the task scheduling results; The transmission strategy generation agent is used to generate the transmission strategy of the data transmission task based on the analysis results and task scheduling results, and forward the transmission strategy to the data transmission node through the message middleware.

[0012] As an embodiment, the monitoring and optimization module includes an indicator collection and analysis agent and a performance prediction and optimization agent; The indicator collection and analysis agent is used to collect the transmission performance indicators and resource utilization status of each data transmission node in real time; The performance prediction optimization agent is used to optimize the decision-making module based on historical data.

[0013] In a second aspect, the present application further provides a data transmission method. Based on the decision module, the data transmission method includes: In response to receiving a data transmission task from the message middleware, generating a transmission strategy for the data transmission task as a decision result; Publish decision result events to the message middleware so that the message middleware can forward the transmission strategy to the data transmission node.

[0014] In a third aspect, the present application further provides a data transmission method. Based on the execution module, the data transmission method includes: In response to receiving the data to be transmitted from the message middleware, dividing the data to be transmitted into a plurality of data fragments; Establish multiple parallel transmission channels; Publish the transmission start event to the message middleware; In response to receiving a transmission start notification from the message middleware, transmitting the multiple data shards in parallel to the next data transmission node through the multiple transmission channels; In response to the completion of the transmission of the data to be transmitted, a transmission completion event is published to the message middleware; Receives transfer completion notifications from the messaging middleware. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 This is one of the structural diagrams of the data transmission system provided by this application; Figure 2 This is the second structural diagram of the data transmission system provided by this application; Figure 3This is one of the flow charts of the transmission task submission phase and the decision-making phase in the data transmission method provided by this application; Figure 4 This is one of the flow charts of the transmission execution phase and the completion and feedback phase in the data transmission method provided by this application; Figure 5 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0019] This application uses message middleware as the underlying communication framework, and through multi-agent collaborative decision-making and adaptive optimization, it achieves efficient and reliable transmission of large-scale data, solving the above-mentioned technical difficulties.

[0020] The following combination Figures 1 to 5 Describe the data transmission system and method provided by this application.

[0021] Figure 1 This is one of the structural diagrams of the data transmission system provided in this application. Figure 2 This is the second structural diagram of the data transmission system provided by this application.

[0022] like Figure 1 and Figure 2 As shown, the data transmission system provided by this application includes a service layer interface, a message middleware and a distributed intelligent agent network.

[0023] The service layer interface provides services to upper-layer applications and users. The service layer interface includes the transmission task application programming interface (API), the monitoring and reporting API, and the policy configuration API.

[0024] The Transfer Task API provides an interface for submitting and managing data transfer tasks, receiving data transfer tasks submitted by upper-layer applications and users (e.g., data producers and data consumers). The Monitoring and Reporting API provides status monitoring and reporting capabilities for the data transfer system (e.g., returning data transfer task processing reports to upper-layer applications and users). The Policy Configuration API allows users to customize transfer policies and parameters.

[0025] The message middleware communicates with the service layer interface through the API interface. The message middleware communicates with each agent in the distributed agent network through the standard protocol, and provides event publishing and task subscription services for each agent in the distributed agent network through the event bus.

[0026] As the underlying communication framework of the system, the message middleware provides an efficient and reliable context transfer mechanism.

[0027] In one possible implementation, the message middleware is MCP (Message Channel Protocol, a communication protocol based on message channels). As an efficient and reliable communication mechanism, MCP middleware is centered around model context messages and supports features such as asynchronous communication, reliable transmission, and protocol evolution, providing a solid foundation for building high-performance intelligent distributed systems. Figure 2 As shown, the MCP middleware includes the following functions: Message routing service: responsible for message routing and distribution between the service layer interface and the distributed intelligent agent network.

[0028] Service Registry: manages various services and resources in the data transmission system.

[0029] Event Bus: Provides event publishing and task subscription services for each agent in the distributed agent network. It is a memory-based publish / subscribe system suitable for scenarios with high real-time requirements.

[0030] Message storage: ensures the persistence and reliable delivery of messages.

[0031] Unified Resource Interface: Provides a unified abstraction and access mechanism for system resources, and communicates with each agent in the distributed agent network through standard protocols.

[0032] Unified method call interface: provides a standardized method call mechanism across services and nodes.

[0033] Automatic discovery mechanism: Provides automatic discovery and registration of services, resources, and capabilities, enabling dynamic perception and adaptation of resources and capabilities.

[0034] The distributed intelligent agent network, as the intelligent decision-making core of the data transmission system, is composed of multiple specialized intelligent agents.

[0035] In one possible implementation, a distributed agent network includes a decision module. This module employs a distributed network structure and comprises multiple decision agent modules with different functions. Different decision agent modules within the decision module are responsible for executing different steps in generating a transmission strategy for a data transmission task. The decision module then forwards the strategy to the data transmission node via a message-based middleware.

[0036] It can be understood that the data transmission system also includes a data transmission resource pool, which provides actual data transmission resources. The data transmission resource pool works together with the decision-making module and the execution module to realize resource deployment. The data transmission resource pool includes a transmission node pool, network link resources and a storage resource pool. The transmission node pool is a collection of server nodes for data transmission, providing multi-regional and multi-level transmission nodes, which can use physical servers, virtual machines or containers. Network link resources are network links connecting various transmission nodes, including public networks, dedicated lines, SDN and other network resources. The storage resource pool is used to provide temporary storage and data caching functions, including hard disks, solid-state drives, network storage, etc.

[0037] The embodiment of the present application sets up a message middleware between the service layer interface and the distributed intelligent agent network, and realizes loosely coupled collaboration between intelligent agents through the message-driven mechanism and standardized protocol of the message middleware, and efficiently transmits context data through the message middleware, so that each intelligent agent can make decisions based on global information. Compared with the centralized architecture design, the present application solves the single-point performance bottleneck problem of the centralized architecture design, and solves the problem of inconsistent data exchange interfaces, and the decision-making accuracy is significantly improved. In addition, the loosely coupled architecture of the message middleware allows the system to be seamlessly expanded, easily cope with the growth of data scale and number of nodes, and each component communicates through a standardized interface to support flexible deployment and dynamic expansion in heterogeneous environments. In addition, under the loosely coupled architecture, each component in the data transmission system can be independently upgraded and expanded, the impact of single-point failures is limited, and the overall reliability of the system is significantly improved.

[0038] In one possible implementation, the decision module includes an analysis agent module and a scheduling agent module.

[0039] The analysis agent module is used to analyze the network status of the distributed agent network, the characteristics of the data transmission tasks and the operating status of each data transmission node, and forward the analysis results to the scheduling agent module through the message middleware.

[0040] In one possible implementation, the analysis agent module can sample the data of the data transmission task, extract part of the data for analysis, and reduce the computational complexity.

[0041] The scheduling agent module is used to formulate a transmission strategy for the data transmission task based on the analysis results, and forward the transmission strategy to the data transmission node through the message middleware.

[0042] The embodiment of the present application decomposes the transmission strategy generation process into two parts: analysis and decision-making, using an analysis agent module and a scheduling agent module. Different agents implement different functions. Compared to solutions that implement transmission strategy generation using a single agent, this functional decomposition improves processing efficiency. Furthermore, the embodiment of the present application combines intelligent analysis with adaptive optimization of transmission strategies, significantly improving data transmission efficiency compared to traditional solutions and supporting the efficient transmission of petabyte-level (petabyte) data.

[0043] In one possible implementation, the analysis agent module includes a data feature analysis agent, a network status analysis agent, and a resource utilization analysis agent.

[0044] The data feature analysis agent analyzes the characteristics of data transmission tasks and forwards them to the scheduling agent module through the message middleware. The characteristics include the data size, type (including structured and unstructured), and structure of the data transmission tasks.

[0045] In one possible implementation, the data feature analysis agent can obtain features such as data type by parsing metadata information of the data.

[0046] In one possible implementation, the data feature analysis agent can use machine learning algorithms such as classification and clustering to identify data types and features. For example, for image data, convolutional neural networks can be used for feature extraction; for text data, natural language processing techniques can be used.

[0047] The network status analysis agent is used to collect and analyze the network status of the distributed agent network and forward the network status to the scheduling agent module through the message middleware. The network status includes network topology, bandwidth, and latency.

[0048] In one possible implementation, the network status analysis agent may use tools such as Ping, Traceroute, iPerf, etc. to measure network performance indicators.

[0049] In one possible implementation, the network status analysis agent may use tools such as Wireshark and tcpdump to capture and analyze network traffic.

[0050] In one possible implementation, the TCP (Transmission Control Protocol) parameter optimization interface provided by the message middleware can analyze the real-time network status obtained by the intelligent agent based on the network status, and dynamically adjust the TCP window size, retransmission timeout, congestion control and other parameters of each intelligent agent in the execution module (see the following description), thereby significantly improving transmission efficiency.

[0051] The resource utilization analysis agent analyzes the current operating status of each data transmission node (including CPU usage, memory utilization, disk input / output (I / O), network I / O, etc.) and forwards this status to the scheduling agent module via the message middleware. This status includes the load and availability of the data transmission node.

[0052] In a possible embodiment, the resource utilization analysis agent may use tools such as Nagios, Zabbix, etc. to monitor system resource usage.

[0053] In one possible implementation, when the system runs in a container environment, the resource utilization analysis agent can use tools such as cAdvisor and Kubernetes Metrics Server to monitor container resource usage.

[0054] The embodiment of the present application analyzes the system based on multiple dimensions such as data characteristics, network status, and resource utilization, providing a data basis for automatically optimizing the transmission strategy so that the transmission strategy can adapt to complex and changing network environments.

[0055] In one possible implementation, the scheduling agent module includes a priority scheduling agent and a transmission strategy generation agent; The priority scheduling agent is used to schedule tasks based on the priorities and deadlines of all current data transmission tasks and determine the task scheduling results.

[0056] In one possible implementation, the priority scheduling agent may use a priority queue to manage tasks to be transmitted and schedule them in order of priority.

[0057] In one possible implementation, the priority scheduling agent may use a deadline scheduling algorithm such as Earliest Deadline First (EDF).

[0058] The transmission strategy generation agent generates transmission strategies for data transmission tasks based on analysis and task scheduling results, and forwards these strategies to data transmission nodes via the messaging middleware. In particular, the transmission strategy generation agent, based on network state awareness and path selection interfaces, intelligently selects the optimal transmission path in multi-path networks, avoiding congested areas and significantly reducing end-to-end latency.

[0059] In one possible implementation, the transmission strategy generation agent uses a reinforcement learning model to generate transmission strategies and learn to select the optimal transmission strategy under different states.

[0060] The embodiment of the present application uses a distributed intelligent agent to implement the two functions of task scheduling and strategy generation, and generates a transmission strategy based on the analysis results and task scheduling results according to the real-time status, so that the system resource utilization rate is greatly improved, resource waste and overload are avoided, and the efficiency and accuracy of strategy generation are improved.

[0061] In one possible implementation, the scheduling agent module also includes a resource allocation optimization agent, which is responsible for globally optimizing resource allocation, such as allocating transmission nodes, network links, and storage resources. Through intelligent resource scheduling and optimization, hardware resource costs and energy consumption can be effectively reduced.

[0062] In one possible implementation, the resource allocation optimization agent may use a linear programming model to optimize resource allocation, such as maximizing transmission throughput or minimizing transmission cost.

[0063] In one possible implementation, the resource allocation optimization agent can use an integer programming model to handle discrete constraints on resource allocation.

[0064] In a possible implementation, a heuristic algorithm (such as a greedy algorithm or a simulated annealing algorithm) may be used for resource allocation.

[0065] In one possible implementation, the distributed agent network further includes an execution module configured to execute specific data transmission tasks. The execution module includes an execution agent module deployed on each data transmission node, each execution agent module executing data transmission on the data transmission node where the execution agent module is located.

[0066] By deploying an execution agent module on the data transmission node, the embodiment of the present application can make autonomous decisions and executions based on the received context data, and can make internal transmission decisions based on global information, so that the data transmission on each data transmission node can be optimized in real time, thereby improving the overall efficiency of data transmission.

[0067] In one possible implementation, any execution agent module includes a data sharding agent and a transmission channel management agent.

[0068] The data sharding agent is used to split the data of the data transmission node where the execution agent module is located into multiple data shards. It can be understood that the data sharding agent formulates the data segmentation strategy in real time based on the tasks undertaken by the data transmission node and the resource usage.

[0069] In one possible implementation, the data sharding agent may split the data into shards of fixed size.

[0070] In one possible implementation, the data sharding agent may perform sharding based on data content, such as splitting according to file boundaries or record boundaries.

[0071] The transmission channel management agent is used to establish multiple parallel transmission channels and transmit multiple data fragments in parallel to the next data transmission node through multiple transmission channels.

[0072] In a possible implementation, the transmission channel management agent may use multi-threading or multi-process technology to create parallel transmission channels.

[0073] The embodiments of the present application implement data sharding and multi-channel parallel transmission through distributed intelligent entities, which greatly improves the reliability of data transmission and greatly reduces the transmission failure rate.

[0074] In one possible implementation, each execution agent module also includes an error retry processing agent, which is responsible for detecting and handling transmission errors and enabling automatic recovery. Based on data sharding and multi-channel parallel transmission, the data transmission node further enhances its own error handling capabilities through the error retry processing agent, further improving the transmission success rate.

[0075] In the existing centralized data transmission architecture, scheduling algorithms and optimization strategies are usually fixed in a preset manner and built into the core modules of the system. They lack dynamic adjustment capabilities and environmental adaptability, and cannot effectively cope with complex and changing network environments and transmission requirements.

[0076] Based on the above considerations, in one possible implementation, the distributed agent network also includes a monitoring and optimization module. This module is responsible for collecting system operational status and transmission metrics and continuously optimizing transmission strategies. Specifically, different monitoring and optimization agents within the monitoring and optimization module perform different monitoring or optimization operations on all agents in the distributed agent network.

[0077] The embodiments of the present application can dynamically adapt to environmental changes through continuous learning of distributed intelligent agents. The system performance will continue to improve with running time and achieve self-evolution. In addition, the present application realizes autonomous learning and optimization through monitoring and optimization modules, reduces manual intervention, and significantly reduces operation and maintenance costs.

[0078] In one possible implementation, the monitoring and optimization module includes an indicator collection and analysis agent and a performance prediction and optimization agent.

[0079] The indicator collection and analysis agent is used to collect the transmission performance indicators and resource utilization status of each data transmission node in real time.

[0080] In one possible implementation, the indicator collection and analysis agent can obtain performance counter data provided by the operating system to implement indicator collection.

[0081] In one possible implementation, the indicator collection and analysis agent can use tools such as Ping, Traceroute, iPerf, etc. to measure network performance indicators.

[0082] In one possible implementation, the indicator collection and analysis agent can extract performance indicators by analyzing system logs.

[0083] The performance prediction and optimization agent is used to optimize the decision-making module based on historical data, such as predicting system performance and providing optimization suggestions.

[0084] The embodiments of the present application respectively realize the collection of indicators and the analysis and optimization of decision-making performance through distributed intelligent agents, thereby improving the efficiency of monitoring and optimization.

[0085] In one possible implementation, the monitoring and optimization module also includes an anomaly detection agent, which is used to detect the status of each agent in the distributed agent network and trigger a response strategy corresponding to the anomaly when an anomaly occurs in any agent.

[0086] The embodiments of the present application enhance the supervision of the distributed intelligent network and improve the quality of operation and maintenance through anomaly identification and feedback mechanisms.

[0087] In a possible implementation, the monitoring and optimization module further includes a model training and updating agent, which is responsible for regularly or irregularly training and updating each agent in the distributed agent network based on historical data.

[0088] The embodiment of the present application implements an automatic update mechanism for the data transmission system by updating the intelligent agent through model training, thereby reducing manual operation and maintenance costs.

[0089] Based on the above, the present application also provides a data transmission method. The data transmission method and the above-mentioned data transmission system can be referenced in correspondence with each other. It should be noted that the data transmission method provided in the embodiment of the present application is implemented based on the data transmission system.

[0090] The embodiment of the present application describes the data transmission method by taking a data transmission system as an execution subject as an example.

[0091] As an example, Figure 3 As shown, the data transmission method includes: S310: Transmission task submission phase, in which the data transmission tasks from upper-layer applications and users (data sources) are received through the service layer interface, and the data transmission tasks are distributed to the distributed intelligent agent network through the event bus of the message middleware.

[0092] Specifically, if Figure 3 As shown in Figure 1, after the data source submits a data transmission task to the service layer interface, the service layer interface generates a unique task identifier corresponding to the data transmission task and publishes a task event to the message middleware. The message middleware passes the task information to the analysis agent module of the decision module.

[0093] S320: Decision-making stage. In this stage, the decision-making module generates a transmission strategy for the data transmission task to achieve intelligent decision-making.

[0094] In one possible implementation, for the decision module, this stage includes: Q1: In response to receiving a data transmission task from the message middleware, a transmission strategy for the data transmission task is generated as a decision result; Q2: Publish the decision result event to the message middleware so that the message middleware can forward the transmission policy to the data transmission node.

[0095] The embodiment of the present application implements the monitoring of decision progress through message middleware, thereby improving the monitoring of the entire data transmission process.

[0096] In one possible implementation, the decision phase includes an analysis phase and a strategy generation phase: During the analysis phase, the analysis agent module subscribes to and accepts data transmission tasks through the message middleware's event bus. It obtains system metrics through the message middleware's unified resource interface, and the various agents within the analysis agent module collaboratively analyze data characteristics, network conditions, and resource availability.

[0097] Specifically, if Figure 3As shown in the figure, the data feature analysis agent analyzes the characteristics of data transmission tasks (such as data size, type, and structure), the network status analysis agent collects and analyzes the network status of the distributed agent network (such as network topology, bandwidth, and latency), and the resource utilization analysis agent analyzes the current operating status of each data transmission node (such as load and availability). The data feature analysis agent, network status analysis agent, and resource utilization analysis agent publish analysis result events to the message middleware, which then passes the analysis result information to the scheduling agent module.

[0098] In the strategy generation stage, the scheduling agent module generates a transmission strategy based on the analysis results using the unified method call interface of the message middleware, dynamically optimizes resource allocation, and obtains the transmission strategy for the data transmission task.

[0099] Specifically, if Figure 3 As shown in the figure, the resource allocation optimization agent performs global optimization of resource allocation. The priority scheduling agent schedules tasks based on the priorities and deadlines of all current data transmission tasks and determines the task scheduling results. The transmission strategy generation agent uses a reinforcement learning model to generate transmission strategies for data transmission tasks based on the analysis results and task scheduling results. The priority scheduling agent and the transmission strategy generation agent each publish their decision results to the message middleware, transmitting the task scheduling results and transmission strategy to the message middleware. The message middleware then transmits the transmission strategy and scheduling decisions to the execution module.

[0100] S330: Transmission execution phase. In this phase, each intelligent agent in the execution module optimizes the TCP parameters through the parameter optimization interface of the message middleware, segments the data and transmits it to the data receiver in parallel through multiple channels.

[0101] In one possible implementation, for the execution module, this stage includes the following steps: P1: In response to receiving the data to be transmitted from the message middleware, split the data to be transmitted into multiple data fragments; P2: Establish multiple parallel transmission channels; P3: Publishes a transmission start event to the message middleware; P4: In response to receiving a transmission start notification from the message middleware, multiple data shards are transmitted in parallel to the next data transmission node through multiple transmission channels; P5: In response to the completion of the data to be transmitted, a transmission completion event is published to the message middleware; P6: Receives the transmission completion notification from the messaging middleware.

[0102] In the embodiment of the present application, the execution module monitors the progress of data transmission through interaction with the message middleware, thereby improving the controllability of data transmission.

[0103] Specifically, if Figure 4 As shown, the data sharding agent splits the data at the data transmission node where the execution agent module resides into multiple data shards. The transmission channel management agent establishes multiple parallel transmission channels. Subsequently, the transmission channel management agent publishes a transmission start event to the message middleware, which in turn transmits a transmission start notification to the transmission channel management agent. The transmission channel management agent then transmits the multiple data shards in parallel to the next data transmission node via multiple transmission channels. During the multi-channel parallel transmission process, the transmission channel management agent periodically publishes transmission status events (including transmission performance indicators and resource utilization) to the message middleware. Upon receiving the transmission status information, the message middleware passes it to the monitoring and optimization module.

[0104] It should be noted that the next data transmission node is the next data transmission node of the data transmission node in the transmission strategy, which can be the data receiver (such as Figure 4 As shown), it may not be the data receiver, that is, the data transmission node in the middle of the transmission path.

[0105] In one possible implementation, at this stage, the monitoring and optimization module monitors the transmission performance and continuously optimizes the transmission strategy, and detects abnormal situations and triggers response strategies.

[0106] Specifically, if Figure 4 As shown, the monitoring and optimization module determines whether exception handling is required based on transmission status information. If so, it publishes an exception handling suggestion event to the message middleware, indicating a response strategy for the transmission anomaly. The message middleware then passes the exception handling instructions to the transmission channel management agent, which then performs fault recovery operations upon receiving them.

[0107] At the same time, the monitoring and optimization module proposes optimization suggestions for the transmission strategy based on the transmission status information, and publishes optimization suggestion events to the message middleware. The message middleware passes the optimization suggestions to the scheduling agent module, and the scheduling agent module adjusts the transmission strategy according to the optimization suggestions. Then the scheduling agent module publishes an update strategy event to the message middleware, and transmits the new transmission strategy to the message middleware. The message middleware passes the updated strategy to the execution module, and the execution module applies the new transmission strategy for data transmission, forming a closed-loop optimization.

[0108] After the execution module of each data transmission node completes the data transmission to the next data transmission node, it publishes a transmission completion event to the message middleware, and the message middleware passes the transmission completion notification to the monitoring and optimization module.

[0109] S340: Completion and feedback phase, in which each agent of the execution module publishes the processing result of the data transmission task to the message middleware, and the message middleware returns the processing report of the data transmission task to the data source.

[0110] Specifically, if Figure 4 As shown, after the transmission channel management agent of the execution module of the last data transmission node in the transmission strategy sends data to the data recipient, it sends a message that the data transmission is completed to the data recipient. Then the transmission channel management agent itself performs data integrity verification. After the verification is passed, it publishes a transmission result event to the message middleware and passes the processing result of the data transmission task. The message middleware passes the processing result of the data transmission task to the service layer interface, and then the service layer interface returns a processing report of the data transmission task to the data source.

[0111] In one possible implementation, the data transmission method also includes a monitoring and optimization phase, which occurs after the completion and feedback phase. During this phase, the monitoring and optimization module collects performance data from the entire data transmission process, determines an update strategy for the scheduling agent module based on the performance data, and publishes an update strategy event to the message middleware. The message middleware then feeds the updated strategy back to the scheduling agent module.

[0112] Figure 5 This is a schematic diagram of the structure of the electronic device provided by this application. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may invoke logic instructions in the memory 530 to execute the data transmission method of the aforementioned service layer interface, message middleware, or each agent in the distributed agent network.

[0113] In addition, the logical instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program code.

[0114] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data transmission method of one of the above-mentioned service layer interface, message middleware or each intelligent agent in the distributed intelligent agent network.

[0115] On the other hand, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the data transmission method of one of the above-mentioned service layer interface, message middleware or each intelligent agent in the distributed intelligent agent network.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0117] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a read-only memory (ROM) / random access memory (RAM), a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data transmission system, characterized in that: Includes service layer interface, message middleware and distributed intelligent agent network; The service layer interface is used to receive data transmission tasks and return processing reports of the data transmission tasks; The message middleware communicates with the service layer interface through an application programming interface, communicates with each agent in the distributed agent network through a standard protocol, and provides event publishing and task subscription services for each agent in the distributed agent network through an event bus; The distributed agent network includes a decision module, and different decision agent modules within the decision module are used to execute different steps in the transmission strategy generation process of the data transmission task. The decision module forwards the transmission strategy to the data transmission node through the message middleware.

2. The data transmission system according to claim 1, characterized in that The distributed agent network further includes an execution module, which includes an execution agent module deployed on each data transmission node, and each of the execution agent modules executes data transmission on the data transmission node where the execution agent module is located.

3. The data transmission system according to claim 1 or 2, characterized in that: The distributed agent network further comprises a monitoring and optimization module, wherein different monitoring and optimization agents within the monitoring and optimization module perform different monitoring or optimization operations on all agents within the distributed agent network.

4. The data transmission system according to claim 2, characterized in that The decision-making module includes an analysis agent module and a scheduling agent module; The analysis agent module is used to analyze the network status of the distributed agent network, the characteristics of the data transmission task and the operating status of each of the data transmission nodes, and forward the analysis results to the scheduling agent module through the message middleware; The scheduling agent module is used to formulate a transmission strategy for the data transmission task based on the analysis result, and forward the transmission strategy to the data transmission node through the message middleware.

5. The data transmission system according to claim 4, characterized in that Any of the execution agent modules includes a data sharding agent and a transmission channel management agent; The data sharding agent is used to split the data of the data transmission node where the execution agent module is located into multiple data shards; The transmission channel management agent is used to establish multiple parallel transmission channels, and transmit the multiple data slices in parallel to the next data transmission node through the multiple transmission channels.

6. The data transmission system according to claim 4, characterized in that The analysis agent module includes a data feature analysis agent, a network status analysis agent and a resource utilization analysis agent; The data feature analysis agent is used to analyze the features of the data transmission task and forward the features to the scheduling agent module through the message middleware; The network status analysis agent is used to collect and analyze the network status of the distributed agent network, and forward the network status to the scheduling agent module through the message middleware; The resource utilization analysis agent is used to analyze the current operating status of each data transmission node and forward the operating status to the scheduling agent module through the message middleware.

7. The data transmission system according to claim 4, characterized in that The scheduling agent module includes a priority scheduling agent and a transmission strategy generation agent; The priority scheduling agent is used to schedule tasks based on the priorities and deadlines of all current data transmission tasks and determine the task scheduling results; The transmission strategy generating agent is used to generate a transmission strategy for the data transmission task based on the analysis result and the task scheduling result, and forward the transmission strategy to the data transmission node through the message middleware.

8. The data transmission system according to claim 3, characterized in that The monitoring and optimization module includes an indicator collection and analysis agent and a performance prediction and optimization agent; The indicator collection and analysis agent is used to collect the transmission performance indicators and resource utilization status of each data transmission node in real time; The performance prediction optimization agent is used to optimize the decision module based on historical data.

9. A data transmission method, characterized in that: Based on the decision module, the data transmission method includes: In response to receiving a data transmission task from the message middleware, generating a transmission strategy for the data transmission task as a decision result; Publish a decision result event to the message middleware so that the message middleware forwards the transmission strategy to the data transmission node.

10. A data transmission method, characterized in that: Based on the execution module, the data transmission method includes: In response to receiving the data to be transmitted from the message middleware, dividing the data to be transmitted into a plurality of data fragments; Establish multiple parallel transmission channels; Publishing a transmission start event to the message middleware; In response to receiving a transmission start notification from the message middleware, transmitting the plurality of data fragments in parallel to a next data transmission node through the plurality of transmission channels; In response to completion of the transmission of the data to be transmitted, publishing a transmission completion event to the message middleware; Receive a transmission completion notification from the message middleware.

Images