Data transmission method and electronic device

Abstract

The application discloses a data transmission method and electronic equipment. The method comprises the following steps: obtaining a transmission task sent by a scheduling engine, the scheduling engine comprising an engine in communication with a target intelligent agent node, the transmission task comprising a channel selection strategy, and the transmission task being used for triggering the target intelligent agent node to transmit to-be-transmitted data to a target platform; based on the channel selection strategy, analyzing the characteristics of the to-be-transmitted data, and selecting a target transmission channel of the to-be-transmitted data from a plurality of candidate transmission channels; and transmitting the to-be-transmitted data to the target platform through the target transmission channel. The method can improve the flexibility of data transmission, adapt to changes in data characteristics, and improve the reliability of data transmission.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a data transmission method and electronic device. Background Technology

[0002] In cross-platform data interaction scenarios, data synchronization is a key step in enabling collaborative work across different platforms. It usually requires certain technical means to achieve data transfer and consistency maintenance between different platforms.

[0003] Currently, cross-platform data synchronization largely relies on middleware or gateways for data forwarding and conversion. Specifically, the source platform sends data to the middleware or gateway, which then performs format conversion, protocol adaptation, and other processing before forwarding the data to the target platform. However, when the data volume is large or the communication channel is not ideal, delays can easily occur, affecting the real-time performance of data synchronization. Furthermore, middleware or gateways struggle to flexibly adapt to all different platform architectures, leading to reduced reliability of data transmission. Summary of the Invention

[0004] This invention provides a data transmission method and an electronic device to address the issues of improving the flexibility and reliability of data transmission.

[0005] According to one aspect of the present invention, a data transmission method is provided, comprising: The transmission task sent by the scheduling engine is obtained. The scheduling engine includes an engine that communicates with the target agent node. The transmission task includes a channel selection strategy and is used to trigger the target agent node to transmit the data to be transmitted to the target platform. Based on the channel selection strategy, the characteristics of the data to be transmitted are analyzed, and the target transmission channel for the data to be transmitted is selected from multiple candidate transmission channels. The data to be transmitted is transmitted to the target platform through the target transmission channel.

[0006] According to another aspect of the present invention, a data transmission method is provided, comprising: A transmission task is sent to the target intelligent agent node. The transmission task includes a channel selection strategy. The transmission task is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform. Obtain the set indicators under multiple detection dimensions during the process of the target intelligent agent node executing the transmission task.

[0007] According to another aspect of the present invention, a data transmission apparatus is provided, comprising: The first acquisition module is used to acquire the transmission task sent by the scheduling engine. The scheduling engine includes an engine that communicates with the target intelligent agent node. The transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform. The selection module is used to analyze the characteristics of the data to be transmitted based on the channel selection strategy, and select the target transmission channel of the data to be transmitted from multiple candidate transmission channels. The transmission module is used to transmit the data to be transmitted to the target platform through the target transmission channel.

[0008] According to another aspect of the present invention, a data transmission apparatus is provided, comprising: The sending module is used to send a transmission task to the target intelligent agent node. The transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform. The second acquisition module is used to acquire set indicators under multiple detection dimensions during the process of the target intelligent agent node executing the transmission task.

[0009] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method described in any embodiment of the present invention.

[0010] The technical solution of this invention involves acquiring a transmission task sent by a scheduling engine, which includes an engine communicating with the target intelligent agent node. The transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit data to a target platform. Based on the channel selection strategy, the characteristics of the data to be transmitted are analyzed, and a target transmission channel is selected from multiple candidate transmission channels. The data is then transmitted to the target platform through the target transmission channel. By analyzing the characteristics of the data to be transmitted and selecting a target transmission channel, data transmission becomes more flexible, can utilize the characteristics of different transmitted data, improves the reliability of data transmission, and eliminates the need for middleware or gateways in the communication. Communication between the target intelligent agent node and the target platform improves the real-time performance of data communication and reduces transmission latency.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a data transmission method provided in an embodiment of the present invention; Figure 2 This is a flowchart of a data transmission method provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of a data transmission method provided in Embodiment 3 of the present invention; Figure 4 This is a flowchart of another data transmission method provided in an embodiment of the present invention; Figure 5 This is a timing diagram of a data transmission method provided in an embodiment of the present invention; Figure 6 This is a flowchart of another data transmission method provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a data transmission device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of another data transmission device provided in an embodiment of the present invention; Figure 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] The communication nodes include a scheduling engine, target agent nodes, and agent nodes in the target platform. The scheduling engine sends a transmission task to the target agent node located in the source platform. After receiving the task, the target agent node transmits data to the agent node located in the target platform. The target platform includes an agent cluster, which includes at least one agent node.

[0017] The scheduling engine can be an engine that generates and sends transmission tasks. It can communicate with target agent nodes and is responsible for task distribution, policy calculation, protocol maintenance, and status monitoring. The source platform can be a platform that transmits data to be transmitted. Target agent nodes can be agent nodes located within the source platform. Target agent nodes include agent clusters, each containing at least one agent node. Target agent nodes can be responsible for data acquisition, preliminary cleaning, format conversion, and data sharding (for large datasets), supporting Application Programming Interface (API) calls, direct database connections, and log parsing. They can be deployed locally on the source platform or on adjacent nodes to reduce data acquisition latency. The target platform can be a platform that receives transmitted data. Devices inherited by the target platform can be, for example, terminals or the cloud. Agent nodes within the target platform can be deployed on various target platforms, responsible for data reception, integrity verification, data sharding and reassembly, and business adaptation; supporting multi-format data parsing and transactional writing (ensuring data consistency on the target platform), and deployed locally on the target platform or at the access layer. Target agent nodes and agent nodes within the target platform can transmit data bidirectionally.

[0018] Example 1 Figure 1This is a flowchart illustrating a data transmission method provided in an embodiment of the present invention. This embodiment is applicable to data transmission scenarios. The method can be executed by a data transmission device, which can be implemented in hardware and / or software and can be configured in an electronic device. The method can be executed by a target intelligent agent node. Figure 1 As shown, the method includes: S110. Obtain the transmission task sent by the scheduling engine. The scheduling engine includes an engine that communicates with the target intelligent agent node. The transmission task includes a channel selection strategy. The transmission task is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform.

[0019] In this embodiment, the transmission task can be a task that triggers the source platform to transmit data to the target platform. The transmission task can be generated by the scheduling engine and sent to the target intelligent agent node with which it communicates. The channel selection strategy can be a strategy for selecting the transmission method (such as the transmission channel). The selection criteria for the channel selection strategy can be, for example, the data characteristics of the data to be transmitted and / or the quality of the transmission channel.

[0020] Specifically, the target agent node receives the transmission task sent by the scheduling engine, which indicates the channel selection strategy. After receiving the transmission task, the target agent node begins preparing to transmit the data to the target platform.

[0021] Optionally, the selection of the transmission channel in the channel selection strategy is associated with at least one of the following: the total size of the data to be transmitted; the priority of the data to be transmitted; and network quality.

[0022] In this embodiment, priority can be the priority of the data to be transmitted. Priority can be dynamically determined, for example, based on the priority of the service corresponding to the data to be transmitted. Network quality can be the network quality of the target platform used to transmit the data to be transmitted. Network quality includes, but is not limited to, bandwidth utilization.

[0023] Optionally, the priority is determined based on the following parameters: data importance coefficient, load coefficient of the target platform, total data size, data volume benchmark threshold, and timeliness requirement coefficient.

[0024] In this embodiment, the data importance coefficient can be the importance coefficient of the data to be transmitted. For example, the data importance coefficient can be set to 1.0, 0.6, and 0.3 according to high importance, medium importance, and low importance, respectively.

[0025] The load factor can be a coefficient that reflects the network load of the target platform. The load factor can be related to one or more of the following: CPU utilization, memory utilization, and network bandwidth utilization of the target platform. The load factor can be expressed as a percentage.

[0026] The data volume baseline threshold can be a preset baseline threshold for the data to be transmitted. For example, the data volume baseline threshold could be 1024MB. The timeliness requirement coefficient, also known as the timeliness requirement coefficient, can be a coefficient reflecting the timeliness requirement of the data to be transmitted. For example, the timeliness requirement coefficient can be divided into real-time, near-real-time, and non-real-time according to timeliness requirements, and the corresponding timeliness requirement coefficients can be 1.0, 0.5, and 0.2, respectively.

[0027] For example, the priority can be obtained by weighted summation of the data importance coefficient, the load coefficient, the negative of the ratio of the total data size to the baseline data volume threshold, and the timeliness requirement coefficient.

[0028] Optionally, the transmission task includes a control instruction, which includes one or more of the following fields: the identification information of the control instruction; the type of the control instruction; the identification information of the target platform corresponding to the control instruction; a set of parameters related to the control instruction; the timestamp of the control instruction; and first signature information.

[0029] In this embodiment, the control command can be an instruction to control the target intelligent agent node. The control command can be issued by the scheduling engine and received by the target intelligent agent node that communicates with the scheduling engine. The first signature information can be the signature information in the control command. The first signature information can be used to verify the integrity and authenticity of the data.

[0030] Specifically, the transmission task includes control instructions, which contain at least one of the following fields: identification information of the control instructions, used to uniquely identify the control instructions; the type of the control instructions, with possible values ​​such as synchronous configuration update, task allocation, or priority adjustment; identification information of the target platform corresponding to the control instructions, used to indicate the target platform; a set of parameters related to the control instructions; relevant parameters for determining priority; the frequency of data synchronization; the timestamp of the control instructions, used to identify the time when the instructions were generated; and first signature information, etc.

[0031] S120. Based on the channel selection strategy, analyze the characteristics of the data to be transmitted, and select the target transmission channel for the data to be transmitted from multiple candidate transmission channels.

[0032] In this embodiment, the target transmission channel can be a transmission channel selected based on a channel selection strategy. Different target transmission channels can correspond to various different transmission methods.

[0033] Specifically, based on the channel selection strategy, the method for analyzing the data to be transmitted is determined. After analyzing the data, a suitable transmission channel is selected as the target transmission channel. The channel selection strategy describes the conditions under which the characteristics of the data to be transmitted should be met, and it analyzes the characteristics of the data to be transmitted to achieve the selection of the transmission channel.

[0034] S130. Transmit the data to be transmitted to the target platform through the target transmission channel.

[0035] Specifically, using a defined target transmission channel, the target intelligent agent node transmits the data to be transmitted to the target platform.

[0036] Optionally, the data transmission protocol between the target agent node and the agent node corresponding to the target platform includes one or more of the following fields: data identification information; identification information of the transmission task; identification information of the source platform; identification information of the target platform; business data; data generation timestamp; encryption value; total data size; priority; and transmission mode.

[0037] In this embodiment, the transmission mode can be the transmission mode of the data to be transmitted. Different target transmission channels correspond to different transmission modes, and the selected transmission channel can be characterized by the transmission model.

[0038] Specifically, the data transmission protocol between the target agent node and the corresponding agent node of the target platform is a bidirectional transmission protocol. This protocol includes one or more of the following fields: data identification information, a globally unique data identifier used to uniquely identify the data to be transmitted; transmission task identification information, which can be represented as the identifier of the transmission task assigned by the scheduling engine; source platform identification information, used to uniquely identify the target agent node of the source platform; target platform identification information, used to identify the agent node of the target platform receiving the data; business data, the business data ontology (structured data is JavaScript Object Notation (JSON) objects, large files (i.e., files larger than a certain threshold) are binary fragments); data generation timestamp, identifying the data generation time (milliseconds, used for timeliness verification); encryption value, using, for example, Message-Digest Algorithm 5 (MD5); total data size; priority; and transmission mode.

[0039] The technical solution of this invention involves acquiring a transmission task sent by a scheduling engine, which includes an engine communicating with the target intelligent agent node. The transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit data to a target platform. Based on the channel selection strategy, the characteristics of the data to be transmitted are analyzed, and a target transmission channel is selected from multiple candidate transmission channels. The data is then transmitted to the target platform through the target transmission channel. By analyzing the characteristics of the data to be transmitted and selecting a target transmission channel, data transmission becomes more flexible, can utilize the characteristics of different transmitted data, improves the reliability of data transmission, and eliminates the need for middleware or gateways in the communication. Communication between the target intelligent agent node and the target platform improves the real-time performance of data communication and reduces transmission latency.

[0040] Example 2 Figure 2 This is a flowchart of a data transmission method provided in Embodiment 2 of the present invention. This embodiment is an optimization based on any of the above embodiments, and mainly includes a detailed description of the processes of determining a target transmission channel, transmitting data through a direct connection channel, transmitting data through a message queue channel, and transmitting data through a fragmented storage channel. It should be noted that technical details not described in detail in this embodiment can be found in any of the above embodiments. Figure 2 As shown, the method includes: S210. Obtain the transmission task sent by the scheduling engine, wherein the scheduling engine includes an engine that communicates with the target intelligent agent node, the transmission task includes a channel selection strategy, and the transmission task is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform.

[0041] S220. If the total size of the data to be transmitted corresponding to the transmission task is less than the first threshold, or the priority of the data to be transmitted is greater than the second threshold, then a direct connection channel is selected as the target transmission channel from multiple candidate transmission channels; otherwise, if the total size of the data is greater than the third threshold, then a fragmented storage channel is selected as the target transmission channel from multiple candidate transmission channels; otherwise, a message queue channel is selected as the target transmission channel from multiple candidate transmission channels.

[0042] In this embodiment, the first threshold can be a preset threshold, which is a threshold of the total size of the data to be transmitted that satisfies the requirement of selecting a direct connection channel as the target transmission channel. The first threshold can be, for example, 10MB. The second threshold can be a preset lower limit of the priority threshold.

[0043] When the total size of the data to be transmitted for a transmission task is less than the first threshold (i.e., small data), or the priority is greater than the second threshold (i.e., high priority), a direct connection channel is selected as the target transmission channel from multiple candidate transmission channels. The direct connection channel can be a transmission channel based on the Transmission Control Protocol (TCP) and Transport Layer Security (TLS) 1.3 encryption protocol. Direct connection channels can be used for, for example, low-latency transmission of high-priority data, or transmission of small data packets (<1MB), high-priority data.

[0044] The third threshold can be a lower limit for the total data size that satisfies the requirement of selecting a fragmented storage channel as the target transmission channel. If the total data size corresponding to the transmission task is less than the first threshold, or the priority of the data to be transmitted is greater than the second threshold, the following judgment is made: when the total data size is greater than the third threshold, a fragmented storage channel is selected as the target transmission channel from multiple candidate transmission channels. In other cases, a message queue channel can be used, such as when the packet loss rate is greater than the corresponding threshold.

[0045] Sharded storage channels can be used for temporary storage of sharded data and synchronization of status logs. They can be used for persistent transmission of large datasets (e.g., larger than 100MB). Message queue channels can be used for transmitting ordinary data. They support asynchronous scheduling and can also handle medium-sized datasets (e.g., 1MB-100MB).

[0046] Specifically, if the total size of the data to be transmitted is less than the first threshold, or the priority of the data to be transmitted is greater than the second threshold, the transmission channel is determined to be a direct connection channel; if the total size of the data to be transmitted is greater than the third threshold, the transmission channel is determined to be a fragmented storage channel; in addition, the transmission channel is determined to be a message queue channel.

[0047] S230: Initiate a direct connection request to the target platform; obtain confirmation information and session key transmitted by the target platform; transmit the data to be transmitted to the target platform based on the session key, and end the operation.

[0048] Specifically, the target agent node proactively initiates a direct connection request to the target platform. Upon receiving the request, the target platform transmits an acknowledgment message indicating that it has successfully received the connection request from the target agent node, along with a session key. The session key can be represented as a string generated based on an encryption algorithm. After receiving the acknowledgment message and the session key, the target agent node encrypts and transmits the data to be transmitted according to the session key.

[0049] S240. Publish the data to be transmitted to a message queue, whereby the target platform can retrieve the data to be transmitted as needed after receiving a push notification from the message queue; retrieve persistent confirmation information returned by the message queue; retrieve a first notification transmitted by the target platform, which is sent after the target platform has retrieved the data to be transmitted from the message queue, and end the operation.

[0050] In this embodiment, the first notification may be a notification sent by the target platform after it has acquired the data to be transmitted. The first notification is sent by the target platform to the target intelligent agent node, instructing the target platform to complete the data reception.

[0051] Specifically, the target agent node publishes the data to be transmitted to a message queue, which persistently stores the data and returns a persistent confirmation message. The message queue then proactively sends a push notification to the target platform. The target platform retrieves the data to be transmitted as needed, processes it, and upon completion, sends a first notification to the target agent node. Upon receiving the first notification, the target agent node terminates the data transmission.

[0052] S250. The data to be transmitted is fragmented to obtain fragmented data; each fragmented data and its corresponding Uniform Resource Locator (URL) are transmitted to the fragmented storage node; the URL is transmitted to the target platform; a second notification is obtained from the target platform, which is transmitted after the target platform has obtained each fragmented data from the fragmented storage node based on the URL, and the operation ends.

[0053] In this embodiment, the second notification may be a notification indicating the completion of downloading the fragmented data from the fragmented storage node. The second notification may be sent by the target platform to the target agent node.

[0054] Specifically, the target agent node fragments the data to be transmitted and then transmits the fragmented data and its corresponding Uniform Resource Locator (URL) to the fragmented storage node. The target agent node then sends the URL to the target platform. Upon receiving the URL, the target platform downloads the corresponding fragmented data from the fragmented storage node based on the URL's indication. After the download is complete, the target platform sends a second notification to the target agent node. Upon receiving the second notification, the target agent node terminates the data transmission.

[0055] Optionally, during the process of fragmenting the data to be transmitted, the fragment size is the maximum value between the minimum fragment size and the base fragment size, and the base fragment size is determined based on the total data size, network quality factor, and number of available threads.

[0056] In this embodiment, the network quality factor can be a numerical value reflecting the current network quality. The network quality factor can be determined by data such as bandwidth, packet loss rate, and latency.

[0057] Specifically, the size of the data fragment to be transmitted can be the maximum of the minimum fragment size and the base fragment size. The minimum fragment size can be a preset value, while the base fragment size can be determined based on the total data size, network quality factor, and number of available threads. For example, the calculation process for the base fragment size can be as follows: first, calculate the product of the base fragment size and the network quality factor; then, calculate the ratio of this product to the number of available threads; and finally, determine the base fragment size based on this ratio.

[0058] Optionally, the network quality factor is associated with one or more of the following: current bandwidth; current latency; packet loss rate.

[0059] Specifically, the network quality factor can be determined by the current bandwidth, current latency, and packet loss rate.

[0060] Optionally, the network quality factor determination operation includes: taking the ratio of the current bandwidth to the bandwidth upper limit as a first value; determining the minimum value between the current delay and the set delay, and determining the difference between 1 and the minimum value as a second value; determining the ratio of the packet loss rate to the packet loss rate upper limit, and determining the difference between the packet loss rate upper limit and the ratio as a third value; and determining the network quality factor as the product of the first value, the second value, and the third value.

[0061] In this embodiment, the first value can be a value reflecting the current bandwidth utilization. This first value can be obtained from the ratio of the current bandwidth to the bandwidth limit. The second value can be a value reflecting the impact of the current latency on network quality. The third value can be a value reflecting the impact of the current packet loss rate on network quality.

[0062] Specifically, first, the ratio of the current bandwidth to a preset bandwidth limit is calculated to obtain a first value; second, the ratio of the current latency to the maximum allowable latency (i.e., the latency limit) is calculated, and this ratio is subtracted from 1 to obtain the impact of the current latency on network quality; then, the minimum value between the current packet loss rate and the packet loss rate limit is determined, and this minimum value is subtracted from 1 to obtain the impact of the current packet loss rate on network quality; finally, the first, second, and third values ​​are multiplied together, and the product is the network quality factor. This invention does not restrict the calculation order of the first, second, and third values.

[0063] Optionally, the number of available threads may be associated with one or more of the following: the number of CPU cores; CPU load.

[0064] In this embodiment, the number of CPU cores can be the number of processor cores. The number of CPU cores can represent the upper limit of computing power. CPU load can be the CPU utilization rate. CPU load can represent the current CPU busyness level of the entire system.

[0065] Optionally, the operation of determining the number of available threads includes: determining a first difference between the upper limit of the load and the CPU load; determining the product of the first difference and the number of CPU cores; determining a second difference between the product and the number of reserved cores; and determining the maximum value between the second difference and the set number of threads as the number of available threads.

[0066] In this embodiment, the first difference can be the difference between the maximum load and the CPU load. This first difference reflects the current available CPU load. The second difference can be the difference between the product and the number of reserved cores. This second difference reflects the CPU's current actual remaining computing power.

[0067] Specifically, the process for determining the number of available threads is as follows: First, determine the first difference between the upper limit of the load and the CPU load; second, calculate the product of the first difference and the number of CPU cores; then, calculate the second difference between the product and the number of reserved cores; finally, determine the maximum value between the second difference and the set number of threads as the number of available threads.

[0068] The technical solution of this invention involves obtaining a transmission task sent by a scheduling engine. If the total size of the data to be transmitted corresponding to the transmission task is less than a first threshold, or the priority of the data to be transmitted is greater than a second threshold, a direct connection channel is selected as the target transmission channel from multiple candidate transmission channels. Otherwise, if the total size of the data is greater than a third threshold, a fragmented storage channel is selected as the target transmission channel from multiple candidate transmission channels; otherwise, a message queue channel is selected as the target transmission channel from multiple candidate transmission channels. By intelligently selecting the target transmission channel, the optimal transmission path is automatically selected based on data volume, priority, and network quality, taking into account the requirements of low latency, high reliability, and large data volume transmission. A direct connection request is initiated to the target platform; confirmation information and session keys transmitted by the target platform are obtained; the data to be transmitted is transmitted to the target platform based on the session key, and data is transmitted through the direct connection channel to ensure the reliability of data transmission. The data to be transmitted is published to a message queue, which is used by the target platform to retrieve the data as needed after receiving a push notification from the message queue. The system also retrieves persistent confirmation information returned by the message queue and a first notification transmitted by the target platform, which is sent after the target platform has retrieved the data to be transmitted from the message queue. Data transmission via the message queue channel improves the reliability of ordinary data transmission. The data to be transmitted is then fragmented to obtain fragmented data. Each fragmented data and its corresponding Uniform Resource Locator (URL) are transmitted to a fragmented storage node. The URL is then transmitted to the target platform. Finally, a second notification transmitted by the target platform is retrieved, which is sent after the target platform has retrieved each fragmented data from the fragmented storage node based on the URL. Data transmission via the fragmented storage channel ensures the reliability of large-volume data transmission.

[0069] In another embodiment, the present invention further includes: obtaining an exception handling instruction; performing one or more of the following operations corresponding to the exception handling instruction: switching the transmission channel; reducing the transmission rate of the data to be transmitted; retransmitting the data to be transmitted; and scheduling the transmission task to other intelligent agent nodes.

[0070] In this embodiment, the exception handling instruction can be a transmission instruction that handles exceptions that occur during data transmission. The exception handling instruction can be issued by the scheduling engine. The scheduling engine monitors the nodes and transmission channels throughout the entire communication process in real time, detects exceptions, and promptly sends exception handling instructions to instruct the target intelligent agent node to handle the exceptions according to the instructions.

[0071] Specifically, the target intelligent agent node receives an anomaly handling instruction and performs one or more of the following operations according to the instructions in the anomaly handling instruction: switching the transmission channel, reducing the transmission rate of the data to be transmitted, retransmitting the data to be transmitted, and / or scheduling the transmission task to other intelligent agent nodes.

[0072] Optionally, the data reporting protocol used by the target intelligent agent node includes one or more of the following fields: data reporting identification information; identification information of the target intelligent agent node; type of platform corresponding to the target intelligent agent node; status information set of the platform corresponding to the target intelligent agent node; network status information; task queue status; information related to the last synchronization; timestamp of data reporting; and second signature information.

[0073] In this embodiment, the second signature information may be the signature information in the data reporting protocol. The second signature information is used to verify the integrity and authenticity of the data.

[0074] Specifically, the data reporting protocol used by the target agent node or agent node on the target platform to send information to the scheduling engine may include one or more of the following fields: data reporting identification information, used to uniquely identify the data reporting information; target agent node identification information, used to uniquely identify the target agent node reporting the data; the type of the corresponding platform, which may be a target platform or a source platform; a set of status information of the corresponding platform; network status information, including but not limited to CPU utilization, memory utilization, network status information, network latency and / or network bandwidth, etc.; task queue status; information related to the last synchronization, including but not limited to the timestamp of the last synchronization, the unique identifier of the data synchronized in the last synchronization, the result of the last synchronization (indicated as success or failure), and the error information when the last synchronization failed (empty when successful); the timestamp of data reporting, used to record the generation time of the reported data; and second signature information.

[0075] Example 3 Figure 3 This is a flowchart of a data transmission method provided in Embodiment 3 of the present invention. This embodiment is applicable to data transmission scenarios. The method can be executed by a data transmission device, which can be implemented in hardware and / or software and can be configured in an electronic device. The method can be executed by a scheduling engine. For details not covered in this embodiment, please refer to the above embodiments. Figure 3 As shown, the method includes: S310. Send a transmission task to the target intelligent agent node. The transmission task includes a channel selection strategy. The transmission task is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform.

[0076] Specifically, the scheduling engine determines the data to be transmitted and sends a transmission task to the target agent node. The transmission task instructs the target agent node to select a transmission channel from the channel selection strategy to transmit the data to be transmitted.

[0077] S320. Obtain the set indicators under multiple detection dimensions during the process of the target intelligent agent node executing the transmission task.

[0078] Specifically, the scheduling engine monitors all nodes and transmission channels during the data transmission process, obtains set indicators under multiple detection dimensions, and determines whether any abnormalities occur during the data transmission process. If an abnormality occurs, it is then processed.

[0079] Optionally, the multiple detection dimensions include one or more of the following: node status dimension; channel performance dimension; data transmission dimension.

[0080] In this embodiment, the node state dimension can be a dimension that reflects the state of the communication node. The set indicators under the node state dimension include, but are not limited to: CPU utilization, memory usage, thread load, data acquisition / processing throughput (times / second), and abnormal log volume of the target agent node and the agent nodes in the target platform.

[0081] Channel performance can be a dimension that reflects the status of the transmission channel. The metrics for channel performance include, but are not limited to: for direct-connect channels, connection survival rate, average latency, packet loss rate, retransmission count, and encrypted handshake success rate; for message queue channels, queue backlog (number of messages), message enqueue / dequeue rate, and persistence failure count; and for sharded storage channels, sharded upload / download rate (MB / s), available storage node space, and sharded verification failure rate.

[0082] The data transmission dimension can reflect the state of the data transmission process. The metrics set under the data transmission dimension include, but are not limited to: single-task end-to-end time, which can be the total time from task issuance to target confirmation, and can be broken down into the time spent at each stage of "source processing → transmission channel transmission → target platform processing"; data integrity metrics: MD5 checksum success / failure count, fragment reassembly success rate, and amount of duplicate data; task status distribution, which can be the number and percentage of tasks pending, processing, successful, failed, and timed out.

[0083] Optionally, after obtaining the set indicators under multiple detection dimensions during the execution of the transmission task by the target intelligent agent node, the method further includes: analyzing the set indicators through a dynamic decision model, generating corresponding exception handling instructions in the event of an anomaly, and transmitting the exception handling instructions; wherein the anomaly includes one or more of the following: transmission channel anomaly; data integrity anomaly; node function anomaly; task timeout anomaly.

[0084] In this embodiment, the dynamic decision-making model can be a model that determines whether an anomaly has occurred, and if so, generates a response instruction, i.e., an anomaly handling instruction. If an anomaly occurs, the dynamic decision-making model can generate an anomaly handling instruction. Anomaly handling instructions can be instructions on how to handle the anomaly. Anomalies requiring handling include: transmission channel anomalies, data integrity anomalies, node function anomalies, and / or task timeout anomalies. Transmission channel anomalies can be anomalies occurring in the transmission channel. Transmission channel anomalies can include: direct connection channel anomalies, such as packet loss rate >10% or latency >100ms for 3 seconds; or message queue congestion, such as queue backlog exceeding a threshold (default 5000 messages). Data integrity anomalies can be anomalies caused by incomplete transmitted data. Data integrity anomalies can include: MD5 checksum failure, reported by the target platform; fragment reassembly failure, such as the target platform discovering missing fragments and requesting the missing fragments from the target agent node via the data transmission protocol. Node function anomalies can be anomalies occurring in the target agent node or the target platform's agent node. Node functionality anomalies can include: node offline, meaning the engine detects three consecutive heartbeat timeouts for a node (which can manifest as 5-second intervals); or, scheduling engine malfunction. Task timeout anomalies can be caused by a task completion time exceeding a preset threshold. For example, a task completion time exceeding 5 minutes could be a typical example.

[0085] Specifically, the scheduling engine monitors and acquires set indicators under multiple detection dimensions in real time, inputs the set indicators into the dynamic decision model, determines whether an anomaly has occurred, generates an anomaly handling instruction for the anomaly, and then sends the anomaly handling instruction to the target intelligent agent or target platform.

[0086] The technical solution of this invention involves sending a transmission task to the target intelligent agent node. This transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit data to the target platform. The solution also involves acquiring set indicators under multiple detection dimensions during the execution of the transmission task by the target intelligent agent node. By sending the transmission task to the target intelligent agent, the system instructs the start of data transmission, achieving efficient end-to-end data transmission, improving synchronization real-time performance and efficiency. Furthermore, by detecting whether any anomalies occur during data transmission through multiple detection dimensions, the system ensures the stability and traceability of data synchronization.

[0087] In another embodiment, Figure 4 This is a flowchart of another data transmission method provided in an embodiment of the present invention. The scheduling engine issues a channel selection strategy, and the target intelligent agent node determines the target transmission channel to transmit data.

[0088] In another embodiment, Figure 5This is a timing diagram of a data transmission method provided in an embodiment of the present invention. The scheduling engine issues transmission tasks, the target intelligent agent node determines the target transmission channel and transmits data to the target platform, which can deeply intervene in the entire transmission process and flexibly respond to the data transmission needs of different scenarios.

[0089] In another embodiment, Figure 6 This is a flowchart of another data transmission method provided in an embodiment of the present invention. The scheduling engine issues a transmission task, the target intelligent agent node determines the target transmission channel and transmits data to the target platform. During this process, if a transmission abnormality occurs, the transmission channel is switched to continue transmitting data.

[0090] The present invention will be described below by way of example, where “source agent” represents “target agent node in source platform”, “target agent” represents “agent node in target platform”, “penetrating scheduling engine” represents “scheduling engine”, and “control command protocol” represents “control command”: Currently, cross-platform data synchronization largely relies on middleware or gateways for data forwarding and conversion. For example, data synchronization between Platform 1 and Platform 2 is achieved through a third-party data exchange gateway. Platform 1 uploads data to the gateway according to the format specified by the gateway, and the gateway converts the data into a format that Platform 2 can recognize before pushing it to Platform 2.

[0091] Alternatively, data synchronization can be achieved using Extract Transform Load (ETL) tools. This involves extracting data from the source platform, transforming it, and then loading it onto the target platform. These tools typically require pre-configuration of data extraction rules, transformation logic, and loading strategies, and execute synchronization tasks periodically. For example, data synchronization between an enterprise resource planning (ERP) system and a customer relationship management (CRM) system can be achieved by using an ETL tool to extract customer data from the ERP system every morning, clean and convert its format, and then load it into the CRM system.

[0092] In addition, current attempts to introduce simple distributed scheduling lack integration with a penetration architecture, and problems such as intermediate node dependency and static scheduling strategies still exist, which cannot meet the needs of large-scale, high real-time cross-platform synchronization.

[0093] The technical solutions centered on middleware or gateways have the following drawbacks: the middleware or gateway becomes a bottleneck for data transmission, and when the data volume is large, delays are likely to occur, affecting the real-time performance of data synchronization; the middleware or gateway is difficult to fully adapt to all different platform architectures, and problems such as data format incompatibility and data loss are likely to occur, resulting in reduced accuracy and reliability of data synchronization.

[0094] The disadvantages of ETL-based technical solutions are: synchronous tasks are usually executed periodically, resulting in poor real-time performance, which cannot meet the needs of business scenarios with high real-time requirements; the configuration and maintenance of ETL tools are relatively complex, and when the data structure of the source or target platform changes, the extraction rules and transformation logic need to be reconfigured, resulting in insufficient flexibility.

[0095] The shortcomings of existing distributed scheduling technologies include: high coupling between scheduling nodes and platform architecture, making it impossible to directly interact with the underlying architecture; scheduling strategies are mostly statically configured, unable to be dynamically adjusted according to platform load and data characteristics; and a lack of data sharding and transmission protocols for large data volumes, making it difficult to support efficient synchronization of data larger than 10G.

[0096] The main technical problems addressed by this invention include: eliminating bottlenecks in intermediate links and improving the real-time performance of cross-platform data synchronization by combining distributed task scheduling with a penetration architecture; enhancing platform adaptability and resolving compatibility issues between different platforms by deploying Agent nodes at the source and target ends; designing a formula-based dynamic scheduling strategy to improve the flexibility of data synchronization and adapt to changes in platform load and data characteristics; defining data transmission protocols and fragmentation protocols, and using Message-Digest Algorithm 5 (MD5) verification to ensure data integrity; and implementing a fragmentation mechanism and parallel synchronization for large data volumes to improve the efficiency of large data synchronization.

[0097] This invention provides a cross-platform data synchronization method based on penetration scheduling. By constructing a penetration scheduling engine and a distributed task scheduling framework, and deploying Agent nodes at the source and target ends, efficient, accurate and flexible data synchronization is achieved.

[0098] The penetrating scheduling engine can include a distributed scheduling module, a protocol management module, and a monitoring module. It is deployed on a cross-platform data platform and is responsible for task distribution, policy calculation, protocol maintenance, and status monitoring.

[0099] In one example, the present invention provides a data transmission process, including: 1. Task Triggering: Precise Control Through a Penetrating Scheduling Engine As the central hub of the architecture, the penetrating scheduling engine possesses control capabilities that penetrate to the terminal Agent layer. It can directly issue transmission tasks containing channel selection strategies to the source Agent (i.e., the target intelligent agent node). The "penetrating" aspect here is reflected in the fact that the scheduling engine can not only specify transmission targets and rules, but also perceive the resource status (such as bandwidth and load coefficient) and network environment of the source (i.e., source platform) and target (i.e., target platform) in real time. Based on this dynamic data, it optimizes the channel selection strategy to ensure the accuracy of the initial task allocation.

[0100] 2. Multi-channel collaborative transmission (adapting to different scenarios as needed): Table 1 Confirmation of Transmission Channel (1) Direct Connection Channel (TCP): Low-latency real-time transmission This system is suitable for small batches of high-real-time data (such as command signals and status messages). In terms of process, the source agent proactively initiates a TCP connection request (Synchronize Sequence Numbers, SYN) to the target agent; the target agent returns an acknowledgment (ACK) and a session key (ACK+SessionKey), achieving encrypted transmission through key negotiation; after the source agent encrypts and transmits the data, the target agent sends back a "receive acknowledgment" (i.e., confirmation message), completing the closed loop. The penetration-based scheduling engine can monitor the connection status of this channel in real time, and immediately triggers a channel switching mechanism once packet loss or latency exceeding limits is detected. (2) Message queue channel: high concurrency asynchronous decoupling This is suitable for scenarios with high peak traffic and relatively low real-time requirements (such as log reporting and batch commands). The source agent publishes data (including metadata such as timestamps and data identifiers) to the message queue. The message queue persistently stores the data and returns a "persistence confirmation" (i.e., persistent confirmation information) to ensure that the data is not lost. Subsequently, the queue actively pushes notifications to the target agent. The target agent pulls data as needed, processes it, and sends a "processing completion notification" (i.e., the first notification) to the source agent upon completion. The penetrating scheduling engine can penetrate to the message queue layer, monitor the queue backlog, and dynamically adjust the source sending rate when congestion occurs to avoid overload. (3) Fragmented storage channel: High-efficiency transmission of large data volumes Suitable for large files exceeding 100MB (such as video clips and data packet backups). The source agent fragments the data and uploads it to the fragmented storage. The storage node returns a "storage confirmation" and a fragmented Uniform Resource Locator (URL). The source agent sends the URL to the target agent, which downloads and reassembles the data from the fragmented storage based on the URL. Upon completion, it sends a "download complete confirmation" (i.e., the second notification). The penetrating scheduling engine can penetrate to the storage layer during this process, monitor the fragmented upload / download progress, support the issuance of breakpoint resume commands, and improve the reliability of large file transfers. (4) Task closed loop: penetrating feedback and full-link traceability After the target agent completes data reception or processing, it submits a "task completion report" to the penetration scheduling engine. The report includes not only the transmission status but also key metrics collected by the engine through penetration (such as the time spent at each stage, encryption verification results, and channel resource utilization). Based on this data, the scheduling engine generates a full-link log, enabling traceability from task issuance to result feedback. At the same time, it continuously optimizes the channel selection strategy by analyzing historical data.

[0101] In summary, this architecture, through the deep management capabilities of the penetrating scheduling engine and the scenario adaptability of the three channels, achieves a closed-loop transmission system of "real-time monitoring - dynamic adjustment - full-link traceability," balancing transmission efficiency, security, and scalability.

[0102] In one example, a data transfer protocol is provided, including: 1. Unified data transmission protocol for Agent nodes For example, the control command protocol (penetrating scheduling engine → Agent) includes: Table 2 Control Command Protocol Wherein, data_size is the total size of the data to be transmitted, command_type is the type of control command, target_platform is the identification information of the target platform corresponding to the control command, parameters is the set of parameters related to the control command, timestamp is the timestamp of the control command, and signature is the first signature information.

[0103] For example, the data reporting protocol (Agent → Penetrating Scheduling Engine) includes: Table 3 Data Reporting Agreement Among them, report_id is the identification information of data reporting, agent_id is the identification information of the target intelligent agent node, platform_type is the type of the corresponding platform, status is the set of status information of the corresponding platform, status.resource_usage.cpu, status.resource_usage.memory, status.network_status, status.network_status.latency, and status.network_status.bandwidth are the corresponding network status information, last_sync, last_sync.data_id, last_sync.timestamp, last_sync.result, and last_sync.error_info are the information related to the last synchronization, timestamp is the timestamp of data reporting, and signature is the second signature information.

[0104] An example, a data transmission protocol (bidirectional), includes: All source agents and target agents use a unified data protocol field to define a unified data standard.

[0105] Protocol field definitions (to ensure the standardization of penetration-based interactions).

[0106] Table 4 Data Transmission Protocols Among them, data_id is the data identification information.

[0107] In one example, a smart channel selection mechanism is provided, including: Intelligent channel selection is used to select between three channels—direct connection channel (TCP), message queue channel, and fragmented storage channel—when data is transmitted from the source agent to the target agent.

[0108] In one example, a priority calculation method is provided, including: Priority calculation formula (quantitatively adapting to load and data characteristics): Priority = (I × W_I) + [(1 - L) × W_L] - (S / S0 × W_S) + (U × W_U) in: I: Data importance coefficient (high = 1.0, medium = 0.6, low = 0.3); L: Target load factor The load factor of the target platform (L = max (C%, M%, B%) / 100, range 0-1), where C% is CPU utilization, M% is memory utilization, and B% is target network bandwidth utilization. S: Data size (MB), i.e., total data size; S0: Baseline threshold (1024MB). U: Timeliness requirement coefficient (real-time = 1.0, near-real-time = 0.5, non-real-time = 0.2); W_I / W_L / W_S / W_U: Weight parameters (default 0.4 / 0.3 / 0.1 / 0.2, dynamic configuration is supported).

[0109] Example: Task priority based on high importance (I=1.0), target load 30% (L=0.3), data size 500MB (S=500), and real-time requirement (U=1.0): Priority = (1.0×0.4) + [(1-0.3)×0.3] - (500 / 1024×0.1) + (1.0×0.2)≈ 0.4 + 0.21 - 0.05 + 0.2 = 0.76 In one example, a dynamic sharding algorithm is provided to implement large data sharding and transmission, as detailed below: (1) Fragmentation Protocol Definition When data_size > 100MB, the penetrating scheduling engine starts fragmentation processing, and the fragmentation data protocol field is used.

[0110] Table 5 Fragmentation Protocol (2) Range of values ​​for the network quality factor: Ideal network: 0.9-1.0 (gigabit bandwidth, <10ms latency); Medium network: 0.5-0.7 (100 Mbps bandwidth, 30-50 ms latency); Poor network performance: 0.1-0.3 (mobile network, >100ms latency); (3) Algorithm advantages: Dynamic adaptability: Real-time response to network fluctuations and system load changes; Resource optimization: Avoid excessive consumption of CPU / bandwidth resources; Maximize transmission efficiency: Increase fragmentation to reduce overhead when the network is good, and decrease fragmentation to improve success rate when the network is poor; Anti-shake design: Ensures that the slices are not too small, avoiding excessive overhead; Parallel optimization: Match the number of shards to the available threads to maximize parallel transmission efficiency.

[0111] In one example, a full-link monitoring and anomaly handling mechanism is provided, including: (1) End-to-end monitoring system The monitoring module collects and analyzes key metrics through real-time interaction with the source agent, target agent, and various transmission channels (direct connection channel, message queue, sharded storage), forming a visualized monitoring view. Core monitoring dimensions include: a. Node status monitoring: Source / Target Agent Status: CPU utilization, memory usage, thread load, data acquisition / processing throughput (times / second), and abnormal log volume; The penetrating scheduling engine itself: task distribution success rate, policy calculation time, and protocol parsing error rate.

[0112] b. Channel performance monitoring: Direct connection channel (TCP): connection survival rate, average latency, packet loss rate, retransmission count, and encrypted handshake success rate; Message queue channel: queue backlog (number of messages), message enqueue / dequeue rate, number of persistence failures; Sharded storage channel: Sharded upload / download speed (MB / s), available space on storage nodes, and shard verification failure rate.

[0113] c. Data transmission monitoring: Single-task end-to-end time: The total time from task issuance to target confirmation, broken down into the time consumed in each stage of "source processing → channel transmission → target processing"; Data integrity metrics: MD5 checksum success / failure count, fragment reassembly success rate, and amount of duplicate data (based on data_id for deduplication). Task status distribution: the number and percentage of tasks in pending, processing, successful, failed, and timed out.

[0114] Monitoring data is uploaded in real time via a data reporting protocol (Agent → Engine). The engine updates the monitoring dashboard every 100ms and supports threshold alarm configuration (such as triggering alarms when CPU usage exceeds 80% or when the queue backlog exceeds 1000 records). (2) Exception handling strategy For anomalies detected during monitoring, the engine triggers a processing mechanism based on preset rules and a dynamic decision-making model to ensure that the impact of the anomalies is minimized. The core anomaly types and processing methods are as follows: a. Transmission channel error: Direct connection channel (TCP) failure: When the packet loss rate > 10% or the latency > 100ms for 3 seconds, the engine sends a "channel switching command" (via the control command protocol) to the source agent, automatically switches to the message queue channel, and records the original channel failure log. It can be manually switched back after the network recovers. Message queue congestion: When the queue backlog exceeds the threshold (default 5000 messages), the engine dynamically reduces the sending rate of the source agent to the queue (by adjusting the parameters.sync_frequency parameter) and prioritizes the dequeueing of high-priority tasks.

[0115] b. Data integrity exception: MD5 verification failed: After the target agent reports a verification failure, the engine sends a "retransmission command" to the source agent, specifying the data_id and corresponding fragment to be retransmitted; if it fails 3 times in a row, the data is marked as "pending manual processing" and an alarm is triggered; Shard reassembly failure: The target agent discovers a missing shard (based on the comparison between sequence and total_shards), and requests the shard_id of the missing shard from the source agent through the data transmission protocol. The source agent pulls the shard from the shard storage and resends it, supporting breakpoint resumption.

[0116] c. Node failure / abnormality: Agent Node Offline: If the engine detects that an Agent has experienced three consecutive heartbeat timeouts (with a 5-second interval), it will automatically reschedule the node's tasks to other available Agents within the same cluster (based on load balancing strategies) and attempt to restart the offline node (via remote commands). Engine module failure: When the distributed scheduling module fails, it automatically switches to the backup scheduling node (based on the master-slave backup mechanism) to ensure that task distribution is not interrupted, and records the failure time and recovery process.

[0117] d. Task timeout exception: For tasks that exceed the preset timeout threshold (configurable, default 5 minutes), the engine automatically identifies them as "timeout tasks" and analyzes the reasons for the timeout: if the source processing is slow, the thread priority of the task on the source agent is increased; if the transmission is slow, a better channel is switched; if the target processing is slow, the low-priority task is temporarily suspended from being sent to the target.

[0118] (3) Anomaly recovery and tracing After exception handling is completed, the engine achieves closed-loop management through the following mechanism: a. Automatically generate anomaly handling reports, including anomaly type, occurrence time, handling measures, recovery time, and scope of impact (such as a list of data_ids involved). b. Train a dynamic decision-making model based on historical anomaly data to optimize anomaly thresholds (such as packet loss alarm thresholds under different network environments) and processing strategies (such as triggering conditions for automatic channel switching). c. Supports abnormal data backtracking, and locates problematic nodes through full-link logs (related to task_id, data_id, and shard_id), providing a basis for platform operation and maintenance.

[0119] The end-to-end monitoring and anomaly handling mechanism significantly improves the stability of cross-platform data synchronization through a closed loop of "real-time perception - intelligent decision-making - automatic recovery - continuous optimization," increasing system availability to over 99.9% and reducing automatic fault recovery time to less than 10 seconds.

[0120] This invention constructs a scheduling engine that can penetrate the underlying architecture of various platforms, directly interacting with the source and target Agent nodes. It eliminates middleware / gateways and other intermediate links, achieving efficient end-to-end data transmission and improving synchronization real-time performance and efficiency. This is the core difference from existing technologies that rely on intermediate nodes. A unified data interaction protocol system is established: defining control command protocols, data reporting protocols, and bidirectional data transmission protocols, standardizing data formats, field meanings, and interaction rules, resolving cross-platform data format incompatibility issues, ensuring the standardization and accuracy of data transmission, and ensuring data integrity through an MD5 verification mechanism.

[0121] 3. Dynamic priority scheduling strategy: Based on data importance, target load, data size and timeliness requirements, task priority is calculated through quantitative formulas to achieve dynamic adjustment of synchronization order and frequency, adapt to platform load fluctuations and data characteristic changes, and break through the limitations of existing static scheduling.

[0122] 4. Intelligent multi-channel collaborative mechanism: The design incorporates dynamic switching logic for direct connection channel (TCP), message queue channel, and fragmented storage channel. Based on data volume, priority, and network quality, the optimal transmission path is automatically selected to balance the requirements of low latency, high reliability, and large data volume transmission.

[0123] 5. Dynamic Fragmentation Transmission Algorithm: For large data volumes of 100MB or more, the algorithm dynamically calculates the fragment size by combining network quality factors (bandwidth, latency, packet loss rate) and the number of available threads in the system, thereby achieving matching of fragment quantity with resources, supporting parallel transmission and breakpoint resumption, and improving the efficiency and reliability of big data synchronization.

[0124] 6. End-to-end monitoring and anomaly handling mechanism: The scheduling engine monitors transmission status, resource usage, and network quality in real time, and sets anomaly handling strategies such as channel switching and breakpoint resumption to ensure the stability and traceability of data synchronization.

[0125] The beneficial effects of this invention are at least as follows: 1. Significantly improved real-time performance: The penetrating architecture eliminates the forwarding latency of intermediate nodes, and data is transmitted directly from the source agent to the target agent. In scenarios with small data and high priority, the latency can be reduced to 10-50ms (direct connection channel), which is more than 60% lower than the latency of existing middleware solutions; the synchronization time for 10GB-level big data can be as short as 6 minutes (sharded storage channel), which is better than the transmission efficiency of more than 10 minutes of existing technologies.

[0126] 2. Enhanced compatibility and reliability: The unified protocol system supports multi-format data parsing (JSON / XML / custom formats) and multi-platform adaptation (cloud platform, edge node, IoT device), solving the problem of "incomplete platform adaptation" in existing technologies; MD5 verification and fragmentation verification mechanisms can promptly detect data tampering or loss, and the data synchronization accuracy is improved to over 99.9%.

[0127] 3. Superior flexibility and adaptability: The dynamic priority scheduling strategy can adjust the synchronization order in real time according to the platform load (CPU / memory / bandwidth) and data importance, making it more adaptable to business fluctuations than the "fixed periodic synchronization" of ETL tools; the intelligent channel switching mechanism can automatically switch to a more reliable path (such as switching from TCP to message queue) when network quality deteriorates, ensuring transmission continuity.

[0128] 4. Breakthrough in large data transmission capabilities: The dynamic fragmentation algorithm supports the transmission of ultra-large files of 10G and above. Through parallel transmission and dynamic fragmentation size adjustment, it improves efficiency by more than 50% compared with existing solutions that lack fragmentation protocols. It also supports breakpoint resumption, solving the pain point of needing to retransmit after large data transmission is interrupted.

[0129] 5. Lower maintenance costs: The source / target Agent nodes are loosely coupled with the platform. When the source / target platform architecture or data structure changes, only the Agent configuration needs to be updated to adapt. There is no need to modify the intermediate links, which reduces the maintenance cost by 40%-60% compared with the existing middleware / gateway solutions.

[0130] Example 4 Figure 7 This is a schematic diagram of the structure of a data transmission device provided in an embodiment of the present invention. Figure 7 As shown, the device includes: The first acquisition module 410 is used to acquire a transmission task sent by the scheduling engine. The scheduling engine includes an engine that communicates with the target intelligent agent node. The transmission task includes a channel selection strategy. The transmission task is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform. The selection module 420 is used to analyze the characteristics of the data to be transmitted based on the channel selection strategy, and select the target transmission channel of the data to be transmitted from multiple candidate transmission channels. The transmission module 430 is used to transmit the data to be transmitted to the target platform through the target transmission channel.

[0131] The technical solution of this invention involves a first acquisition module acquiring a transmission task sent by a scheduling engine. The scheduling engine includes an engine that communicates with the target intelligent agent node. The transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit data to a target platform. A selection module analyzes the characteristics of the data to be transmitted based on the channel selection strategy and selects a target transmission channel from multiple candidate transmission channels. The transmission module transmits the data to be transmitted to the target platform through the target transmission channel. By analyzing the characteristics of the data to be transmitted and selecting a target transmission channel, data transmission becomes more flexible, can utilize the characteristics of different transmitted data, improves the reliability of data transmission, and eliminates the need for middleware or gateways in communication. Communication between the target intelligent agent node and the target platform improves the real-time performance of data communication and reduces transmission latency.

[0132] In one embodiment, the selection of the transmission channel in the channel selection strategy is associated with at least one of the following: The total size of the data to be transmitted; the priority of the data to be transmitted; and the network quality.

[0133] In one embodiment, the priority is determined based on the following parameters: Data importance coefficient, target platform load coefficient, total data size, data volume benchmark threshold, and timeliness requirement coefficient.

[0134] In one embodiment, module 420 is selected and is specifically used for: If the total size of the data to be transmitted corresponding to the transmission task is less than the first threshold, or the priority of the data to be transmitted is greater than the second threshold, then a direct connection channel is selected as the target transmission channel from multiple candidate transmission channels; otherwise, if the total size of the data is greater than the third threshold, then a fragmented storage channel is selected as the target transmission channel from multiple candidate transmission channels; otherwise, a message queue channel is selected as the target transmission channel from multiple candidate transmission channels.

[0135] In one embodiment, the transmission module 430 is specifically used for: Initiate a direct connection request to the target platform; Obtain the confirmation information and session key transmitted by the target platform; The data to be transmitted is transmitted to the target platform based on the session key.

[0136] In one embodiment, the transmission module 430 is specifically used for: The data to be transmitted is published to a message queue, which is used by the target platform to retrieve the data to be transmitted as needed after receiving a push notification sent by the message queue. Retrieve persistent confirmation information returned by the message queue; Obtain a first notification transmitted by the target platform, which is sent by the target platform after it has retrieved the data to be transmitted from the message queue.

[0137] In one embodiment, the transmission module 430 is specifically used for: The data to be transmitted is fragmented to obtain fragmented data; Each of the fragmented data and the corresponding Uniform Resource Locator (URL) is transmitted to the fragmented storage node; Transmit the Uniform Resource Locator to the target platform; The second notification transmitted by the target platform is obtained after the target platform has obtained the data of each of the shards from the sharded storage nodes based on the Uniform Resource Locator.

[0138] In one embodiment, during the fragmentation of the data to be transmitted, the fragment size is the maximum value between the minimum fragment size and the base fragment size, and the base fragment size is determined based on the total data size, network quality factor, and number of available threads.

[0139] In one embodiment, the network quality factor is associated with one or more of the following: Current bandwidth; current latency; packet loss rate.

[0140] In one embodiment, the network quality factor determination operation includes: The ratio of the current bandwidth to the bandwidth limit is used as the first value; Determine the minimum value between the current delay and the set delay, and determine the difference between 1 and the minimum value as a second value; Determine the ratio of the packet loss rate to the upper limit of the packet loss rate, and determine the difference between the upper limit of the packet loss rate and the ratio as a third value; The product of the first value, the second value, and the third value is determined as the network quality factor.

[0141] In one embodiment, the number of available threads is associated with one or more of the following: CPU core count; CPU load.

[0142] In one embodiment, the operation of determining the number of available threads includes: Determine the first difference between the upper limit of the load and the CPU load; Determine the product of the first difference and the number of CPU cores; Determine the second difference between the product and the number of reserved cores; The maximum value between the second difference and the set number of threads is determined as the number of available threads.

[0143] In one embodiment, the transmission task includes control instructions, which contain one or more of the following fields: The control command's identification information; the control command's type; the target platform's identification information corresponding to the control command; the parameter set related to the control command; the control command's timestamp; and the first signature information.

[0144] In one embodiment, the data reporting protocol used by the target agent node and the agent nodes in the target platform includes one or more of the following fields: The data reporting identifier information; the identifier information of the target intelligent agent node; the type of the corresponding platform; the status information set of the corresponding platform; network status information; task queue status; information related to the last synchronization; the timestamp of the data reporting; and the second signature information.

[0145] In one embodiment, the data transmission protocol between the target agent node and the agent node corresponding to the target platform includes one or more of the following fields: Data identification information; identification information of the transmission task; identification information of the source platform; identification information of the target platform; business data; data generation timestamp; encryption value; total data size; priority; transmission mode.

[0146] In one embodiment, the device further includes: The exception handling instruction acquisition module is used to acquire exception handling instructions; The execution module is used to perform one or more of the following operations corresponding to the exception handling instructions: Switch the transmission channel; reduce the transmission rate of the data to be transmitted; retransmit the data to be transmitted; schedule the transmission task to other intelligent agent nodes.

[0147] The data transmission device provided in this embodiment of the invention can execute a data transmission method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0148] Example 5 Figure 8 This is a schematic diagram of another data transmission device provided in an embodiment of the present invention. Figure 8 As shown, the device includes: The sending module 510 is used to send a transmission task to the target intelligent agent node. The transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform. The second acquisition module 520 is used to acquire set indicators under multiple detection dimensions during the process of the target intelligent agent node executing the transmission task.

[0149] The technical solution of this invention involves sending a transmission task to the target intelligent agent node via a sending module. The transmission task includes a channel selection strategy and is used to trigger the target intelligent agent node to transmit data to the target platform. A second acquisition module acquires set indicators under multiple detection dimensions during the execution of the transmission task by the target intelligent agent node. By sending the transmission task to the target intelligent agent, the system instructs the start of data transmission, achieving efficient end-to-end data transmission, improving synchronization real-time performance and efficiency. By detecting whether any abnormalities occur during data transmission through multiple detection dimensions, the system ensures the stability and traceability of data synchronization.

[0150] In one embodiment, the multiple detection dimensions are one or more of the following: Node status dimension; channel performance dimension; data transmission dimension.

[0151] In one embodiment, after obtaining the set indicators under multiple detection dimensions during the execution of the transmission task by the target intelligent agent node, the method further includes: The set indicators are analyzed by a dynamic decision-making model. In the event of an anomaly, a corresponding anomaly handling instruction is generated and transmitted. The anomalies include one or more of the following: Transmission channel error; data integrity error; node function error; task timeout error.

[0152] The data transmission device provided in this embodiment of the invention can execute a data transmission method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0153] Example 6 Figure 9This is a structural block diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 9 The diagram illustrates a schematic representation of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0154] like Figure 9 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 to enable the at least one processor 11 to perform the method provided by the present invention.

[0155] The processor 11 can perform various appropriate actions and processes based on a computer program stored in the read-only memory (ROM) 12 or a computer program loaded from the storage unit 18 into the random access memory (RAM) 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0156] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0157] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the methods provided in this invention.

[0158] In some embodiments, the methods provided herein may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the methods by any other suitable means (e.g., by means of firmware).

[0159] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0161] In the context of this invention, a computer-readable storage medium stores computer instructions that are used to cause a processor to execute and implement the method provided by this invention.

[0162] The present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the method provided according to embodiments of the present invention. A computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0164] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0165] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0166] This invention also provides a computer program product, including a computer program that, when executed by a processor, can implement the methods provided in any embodiment of this application.

[0167] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data transmission method, characterized in that, The method is applied to a target agent node in a source platform, wherein the source platform includes an agent cluster, and the agent cluster includes at least one agent node. The transmission task sent by the scheduling engine is obtained. The scheduling engine includes an engine that communicates with the target agent node. The transmission task includes a channel selection strategy and is used to trigger the target agent node to transmit the data to be transmitted to the target platform. Based on the channel selection strategy, the characteristics of the data to be transmitted are analyzed, and the target transmission channel for the data to be transmitted is selected from multiple candidate transmission channels. The data to be transmitted is transmitted to the target platform through the target transmission channel.

2. The method according to claim 1, characterized in that, The selection of the transmission channel in the channel selection strategy is associated with at least one of the following: The total size of the data to be transmitted; the priority of the data to be transmitted; and the network quality.

3. The method according to claim 2, characterized in that, The priority is determined based on the following parameters: Data importance coefficient, target platform load coefficient, total data size, data volume benchmark threshold, and timeliness requirement coefficient.

4. The method according to claim 1, characterized in that, The step of analyzing the characteristics of the data to be transmitted based on the channel selection strategy and selecting the target transmission channel for the data to be transmitted from multiple candidate transmission channels includes: If the total size of the data to be transmitted corresponding to the transmission task is less than the first threshold, or the priority of the data to be transmitted is greater than the second threshold, then a direct connection channel is selected as the target transmission channel from multiple candidate transmission channels; otherwise, if the total size of the data is greater than the third threshold, then a fragmented storage channel is selected as the target transmission channel from multiple candidate transmission channels; otherwise, a message queue channel is selected as the target transmission channel from multiple candidate transmission channels.

5. The method according to claim 1, characterized in that, The transmission of the data to be transmitted to the target platform through the target transmission channel includes: Initiate a direct connection request to the target platform; Obtain the confirmation information and session key transmitted by the target platform; The data to be transmitted is transmitted to the target platform based on the session key.

6. The method according to claim 1, characterized in that, The transmission of the data to be transmitted to the target platform through the target transmission channel includes: The data to be transmitted is published to a message queue, which is used by the target platform to retrieve the data to be transmitted as needed after receiving a push notification sent by the message queue. Retrieve persistent confirmation information returned by the message queue; Obtain a first notification transmitted by the target platform, which is sent by the target platform after it has retrieved the data to be transmitted from the message queue.

7. The method according to claim 1, characterized in that, The transmission of the data to be transmitted to the target platform through the target transmission channel includes: The data to be transmitted is fragmented to obtain fragmented data; Each of the fragmented data and the corresponding Uniform Resource Locator (URL) is transmitted to the fragmented storage node; Transmit the Uniform Resource Locator to the target platform; Obtain a second notification transmitted by the target platform, which is transmitted after the target platform has obtained the data of each of the shards from the sharded storage nodes based on the Uniform Resource Locator.

8. The method according to claim 7, characterized in that, During the process of fragmenting the data to be transmitted, the fragment size is the maximum value between the minimum fragment size and the base fragment size. The base fragment size is determined based on the total data size, network quality factor, and number of available threads.

9. The method according to claim 8, characterized in that, The network quality factor is associated with one or more of the following: Current bandwidth; current latency; packet loss rate.

10. The method according to claim 9, characterized in that, The operation of determining the network quality factor includes: The ratio of the current bandwidth to the bandwidth limit is used as the first value; Determine the minimum value between the current delay and the set delay, and determine the difference between 1 and the minimum value as a second value; Determine the ratio of the packet loss rate to the upper limit of the packet loss rate, and determine the difference between the upper limit of the packet loss rate and the ratio as a third value; The product of the first value, the second value, and the third value is determined as the network quality factor.

11. The method according to claim 8, characterized in that, The number of available threads is associated with one or more of the following: CPU core count; CPU load.

12. The method according to claim 11, characterized in that, The operation of determining the number of available threads includes: Determine the first difference between the upper limit of the load and the CPU load; Determine the product of the first difference and the number of CPU cores; Determine the second difference between the product and the number of reserved cores; The maximum value between the second difference and the set number of threads is determined as the number of available threads.

13. The method according to claim 1, characterized in that, The transmission task includes control instructions, which contain one or more of the following fields: The control command includes the following information: identification information; type of control command; identification information of the target platform corresponding to the control command; set of parameters related to the control command; timestamp of the control command; and first signature information.

14. The method according to claim 1, characterized in that, The data reporting protocol used by the target agent node and the agent nodes in the target platform includes one or more of the following fields: The data reporting identifier information; the identifier information of the target intelligent agent node; the type of the corresponding platform; the set of status information of the corresponding platform; network status information; task queue status; information related to the last synchronization; The timestamp of the data report; the second signature information.

15. The method according to claim 1, characterized in that, The data transmission protocol between the target agent node and the agent node corresponding to the target platform includes one or more of the following fields: Data identification information; identification information of the transmission task; identification information of the source platform; identification information of the target platform; business data; data generation timestamp; encryption value; total data size; priority; transmission mode.

16. The method according to claim 1, characterized in that, Also includes: Get exception handling instructions; Perform one or more of the following operations corresponding to the exception handling instruction: Switch the transmission channel; reduce the transmission rate of the data to be transmitted; retransmit the data to be transmitted; schedule the transmission task to other intelligent agent nodes.

17. A data transmission method, characterized in that, Applied to a scheduling engine, the scheduling engine including an engine that communicates with target agent nodes in the source platform, the method includes: A transmission task is sent to the target intelligent agent node. The transmission task includes a channel selection strategy. The transmission task is used to trigger the target intelligent agent node to transmit the data to be transmitted to the target platform. Obtain the set indicators under multiple detection dimensions during the process of the target intelligent agent node executing the transmission task.

18. The method according to claim 17, characterized in that, The multiple detection dimensions are one or more of the following: Node status dimension; channel performance dimension; data transmission dimension.

19. The method according to claim 17, characterized in that, After obtaining the set indicators under multiple detection dimensions during the execution of the transmission task by the target intelligent agent node, the method further includes: The set indicators are analyzed by a dynamic decision-making model. In the event of an anomaly, a corresponding anomaly handling instruction is generated and transmitted. The anomalies include one or more of the following: Transmission channel error; data integrity error; node function error; task timeout error.

20. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-19.

Images