Data transmission method and apparatus, device, and storage medium

Abstract

This application provides a data transmission method, apparatus, device, and storage medium. The method includes: a first system acquiring the data transmission duration of a current batch of data transmitted by a second system to each of a plurality of first processing nodes; determining the current load of each first processing node based on its memory overflow index, data transmission duration, and multiple resource performance parameters; determining a target processing node from the plurality of first processing nodes based on its current load; and finally sending the identifier of the target processing node to the second system so that the next batch of data transmitted by the second system can be received through the target processing node. This solution fully considers the processing capabilities and states of different processing nodes in the first system, thereby improving load balancing.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a data transmission method, apparatus, device and storage medium. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field, and the traditional financial industry is gradually transforming into fintech, with data transmission technology being no exception. However, due to the explosive growth of financial business data, and considering factors such as security, real-time performance, and load balancing, higher demands are being placed on data transmission technology.

[0003] Currently, the transmission of financial business data typically involves connecting the financial business system to a third-party processing system (such as a log system), and then using a round-robin or weighted round-robin algorithm to transmit the data to the third-party processing system through relevant interfaces, so that the processing nodes in the third-party processing system can perform load balancing and data delivery.

[0004] However, in the existing technology, due to the different capabilities of each processing node in the third-party processing system, and the different amounts of data sent each time by the financial business system according to the polling algorithm or the weighted polling algorithm, there is a problem of poor load balancing. Summary of the Invention

[0005] This application provides a data transmission method, apparatus, device, and storage medium to solve the problem of unbalanced load in the prior art.

[0006] In a first aspect, embodiments of this application provide a data transmission method applied to a first system, the first system including multiple processing nodes, the method comprising:

[0007] The data transmission duration of the current batch of data transmitted by the second system to each of the plurality of first processing nodes is obtained, wherein the plurality of first processing nodes are determined from the plurality of processing nodes.

[0008] Based on the memory overflow index, data transmission duration, and multiple resource performance parameters of each first processing node, the current load of each first processing node is determined. The memory overflow index of each first processing node is used to indicate whether the memory of that first processing node has overflowed.

[0009] The target processing node is determined from the plurality of first processing nodes based on the current load of each first processing node.

[0010] The identifier of the target processing node is sent to the second system so that the next batch of data transmitted by the second system can be received through the target processing node.

[0011] In one possible design of the first aspect, determining the current load of each first processing node based on its memory overflow metric, data transfer duration, and multiple resource performance parameters includes:

[0012] Based on the memory overflow metric of each first processing node and multiple resource performance parameters, a weight value for each resource performance parameter is determined. The weight value of each resource performance parameter is used to represent the importance of that resource performance to the current load of the first processing node.

[0013] The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters.

[0014] Optionally, determining the weight value of each resource performance based on the memory overflow index of each first processing node and multiple resource performance parameters includes:

[0015] For each resource performance, multiple initial resource performance parameters are determined based on the resource performance parameter of each first processing node, and each initial resource performance parameter is within the parameter range of the resource performance parameter of the multiple first processing nodes.

[0016] Based on the memory overflow index of each first processing node and the resource performance parameter, determine the Gini index of each initial resource performance parameter.

[0017] The weight value of the resource performance is determined based on the minimum value of the Gini index.

[0018] Optionally, for each resource performance, based on the resource performance parameter of each first processing node, multiple initial resource performance parameters are determined, including:

[0019] For each resource performance parameter, the resource performance parameters of the multiple first processing nodes are sorted in descending order to obtain the sorting result.

[0020] Based on the sorting results, the average value of two adjacent resource performance parameters is determined as the initial resource performance parameter.

[0021] In another possible design of the first aspect, after determining the current load of each first processing node based on the memory overflow metric, data transfer duration, and multiple resource performance parameters of each first processing node, the method further includes:

[0022] The current load of each first processing node is determined as the current SLEWMA value of that first processing node.

[0023] Accordingly, determining the target processing node from the plurality of first processing nodes based on the current load of each first processing node includes:

[0024] Based on the current SLEWMA value of each first processing node, the target processing node that meets the preset conditions is determined from the plurality of first processing nodes. The preset conditions are that the current SLEWMA value of the first processing node is greater than a preset SLEWMA value, or a preset number of first processing nodes with the largest current SLEWMA value.

[0025] Optionally, after determining the target processing node that satisfies the preset conditions from the plurality of first processing nodes based on the current SLEWMA value of each first processing node, the method further includes:

[0026] Based on the current SLEWMA value of the target processing node, the identifier of the target processing node is placed into an initially empty max-heap to generate the first max-heap.

[0027] Accordingly, sending the identifier of the target processing node to the second system includes:

[0028] The identifiers of the target processing nodes in the first top heap are sent to the second system in a bottom-up order.

[0029] Optionally, after sending the identifier of the target processing node to the second system, the method further includes:

[0030] The current load of each first processing node is determined as the historical load, and multiple second processing nodes are re-determined from the plurality of processing nodes.

[0031] The current load of each second processing node is determined based on its memory overflow metric, multiple resource performance parameters, and data transmission duration.

[0032] Based on the current load of each second processing node and the historical load of the target processing node, a new target processing node is re-determined from the plurality of second processing nodes and the target processing node.

[0033] The identifier of the new target processing node is sent to the second system.

[0034] Optionally, after determining the current load of each second processing node based on its memory overflow metric, multiple resource performance parameters, and data transmission duration, the method further includes:

[0035] If any second processing node does not have the historical load, then the current load of the second processing node is determined as the current SLEWMA value of the second processing node.

[0036] If any second processing node has the historical load, then the current SLEWMA value of the second processing node is calculated based on the current load of the second processing node, the historical load, and a pre-set glitch variable, wherein the glitch variable is determined based on the historical load.

[0037] The historical SLEWMA value of the target processing node is determined based on the historical load of the target processing node and the glitch variable.

[0038] Accordingly, the step of re-determining a new target processing node from the plurality of second processing nodes and the target processing node based on the current load of each second processing node and the historical load of the target processing node includes:

[0039] Based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the new target processing node is re-determined from multiple second processing nodes and the target processing node.

[0040] Optionally, after determining the new target processing node from among multiple second processing nodes and the target processing node based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the method further includes:

[0041] Based on the current SLEWMA value and identifier of the new target processing node, the first max heap is updated to generate the second max heap.

[0042] Accordingly, sending the identifier of the new target processing node to the second system includes:

[0043] The identifiers of the new target processing nodes in the second large top heap are sent to the second system in a bottom-up order.

[0044] Optionally, determining the current load of each first processing node based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters includes:

[0045] The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, the preset weight value, and multiple resource performance parameters.

[0046] Optionally, the resource performance parameters of each first processing node include at least two of the following: the current traffic of the first processing node, CPU utilization, memory utilization, network interface card (NIC) input / output utilization, and disk NIC input / output utilization.

[0047] Optionally, before obtaining the data transmission duration of the current batch of data transmitted by the second system to each of the plurality of first processing nodes, the method further includes:

[0048] The system sends to the second system the identifier of each first processing node, the target data type of the current batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system.

[0049] Secondly, embodiments of this application provide a data transmission method applied to a second system, the method comprising:

[0050] Multiple first data transmission durations are sent to the first system, each first data transmission duration representing the duration for sending the current batch of data to the first processing node in the first system.

[0051] A first acquisition request is sent to the first system. The first acquisition request is used to acquire the identifier of the target processing node. The target processing node is a processing node among a plurality of first processing nodes whose current load meets preset conditions.

[0052] The system receives the identifier of the target processing node sent by the first system, so that the next batch of data to be transmitted is transmitted to the target processing node.

[0053] In one possible design of the second aspect, after receiving the identifier of the target processing node sent by the first system, the method further includes:

[0054] Multiple second data transmission durations are sent to the first system, each second data transmission duration representing the duration for sending new current batch data to the second processing node in the first system.

[0055] A second acquisition request is sent to the first system. The second acquisition request is used to acquire the identifier of a new target processing node. The new target processing node is one of the multiple second processing nodes and the processing node whose load meets the preset conditions among the target processing nodes.

[0056] The system receives the identifier of the new target processing node sent by the first system, so as to transmit the new batch of data to be transmitted to the new target processing node.

[0057] In another possible design of the second aspect, prior to the transmission of the plurality of first data transmission durations to the first system, the method further includes:

[0058] The system receives the identifiers of the multiple first processing nodes, the target data type of the current batch data, the identifier of the target port of the second system, and the identifier of the target process running in the second system, all sent by the first system.

[0059] The first data in the target process, which is in the target data format, is obtained from the target port, and the first data is transmitted to the plurality of first processing nodes.

[0060] Thirdly, embodiments of this application provide a data transmission apparatus applied to a first system, the first system including multiple processing nodes, the apparatus including:

[0061] The acquisition module is used to acquire the data transmission duration of the current batch of data transmitted by the second system to each of the plurality of first processing nodes, wherein the plurality of first processing nodes are determined from the plurality of processing nodes.

[0062] The processing module is used to determine the current load of each first processing node based on the memory overflow index, data transmission duration and multiple resource performance parameters of each first processing node. The memory overflow index of each first processing node is used to indicate whether the memory of the first processing node has overflowed.

[0063] The processing module is further configured to determine a target processing node from the plurality of first processing nodes based on the current load of each first processing node.

[0064] The sending module is used to send the identifier of the target processing node to the second system so that the target processing node can receive the next batch of data transmitted by the second system.

[0065] In one possible design of the third aspect, the processing module is specifically used for:

[0066] Based on the memory overflow metric of each first processing node and multiple resource performance parameters, a weight value for each resource performance parameter is determined. The weight value of each resource performance parameter is used to represent the importance of that resource performance to the current load of the first processing node.

[0067] The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters.

[0068] Optionally, the processing module is specifically used for:

[0069] For each resource performance, multiple initial resource performance parameters are determined based on the resource performance parameter of each first processing node, and each initial resource performance parameter is within the parameter range of the resource performance parameter of the multiple first processing nodes.

[0070] Based on the memory overflow index of each first processing node and the resource performance parameter, determine the Gini index of each initial resource performance parameter.

[0071] The weight value of the resource performance is determined based on the minimum value of the Gini index.

[0072] Optionally, the processing module is specifically used for:

[0073] For each resource performance parameter, the resource performance parameters of the multiple first processing nodes are sorted in descending order to obtain the sorting result.

[0074] Based on the sorting results, the average value of two adjacent resource performance parameters is determined as the initial resource performance parameter.

[0075] In another possible design of the third aspect, after determining the current load of each first processing node based on the memory overflow index, data transfer duration, and multiple resource performance parameters of each first processing node, the processing module is further configured to:

[0076] The current load of each first processing node is determined as the current SLEWMA value of that first processing node.

[0077] Correspondingly, the processing module is specifically used for:

[0078] Based on the current SLEWMA value of each first processing node, the target processing node that meets the preset conditions is determined from the plurality of first processing nodes. The preset conditions are that the current SLEWMA value of the first processing node is greater than a preset SLEWMA value, or a preset number of first processing nodes with the largest current SLEWMA value.

[0079] Optionally, after determining the target processing node that meets the preset conditions from the plurality of first processing nodes based on the current SLEWMA value of each first processing node, the processing module is further configured to put the identifier of the target processing node into an initially empty max-heap based on the current SLEWMA value of the target processing node, thereby generating a first max-heap.

[0080] Accordingly, the sending module is specifically used for:

[0081] The identifiers of the target processing nodes in the first top heap are sent to the second system in a bottom-up order.

[0082] Optionally, after sending the identifier of the target processing node to the second system, the processing module is further configured to determine the current load of each first processing node as the historical load, and to re-determine a plurality of second processing nodes from the plurality of processing nodes.

[0083] The processing module is also used to determine the current load of each second processing node based on the memory overflow index of each second processing node, multiple resource performance parameters, and data transmission duration.

[0084] The processing module is further configured to determine a new target processing node from the plurality of second processing nodes and the target processing node based on the current load of each second processing node and the historical load of the target processing node.

[0085] The sending module is also used to send the identifier of the new target processing node to the second system.

[0086] Optionally, after determining the current load of each second processing node based on its memory overflow metric, multiple resource performance parameters, and data transmission duration, the processing module is further configured to:

[0087] If any second processing node does not have the historical load, then the current load of the second processing node is determined as the current SLEWMA value of the second processing node.

[0088] If any second processing node has the historical load, then the current SLEWMA value of the second processing node is calculated based on the current load of the second processing node, the historical load, and a pre-set glitch variable, wherein the glitch variable is determined based on the historical load.

[0089] The historical SLEWMA value of the target processing node is determined based on the historical load of the target processing node and the glitch variable.

[0090] Correspondingly, the processing module is specifically used for:

[0091] Based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the new target processing node is re-determined from multiple second processing nodes and the target processing node.

[0092] Optionally, after determining the new target processing node from multiple second processing nodes and the target processing node based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the processing module is further configured to update the first max heap based on the current SLEWMA value and identifier of the new target processing node to generate a second max heap.

[0093] Accordingly, the sending module is specifically used for:

[0094] The identifiers of the new target processing nodes in the second large top heap are sent to the second system in a bottom-up order.

[0095] Optionally, the processing module is specifically used for:

[0096] The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, the preset weight value, and multiple resource performance parameters.

[0097] In another possible design in the third aspect, the multiple resource performance parameters of each first processing node include at least two of the following: the current traffic of the first processing node, CPU utilization, memory utilization, network interface card (NIC) input / output utilization, and disk NIC input / output utilization.

[0098] In another possible design of the third aspect, before obtaining the data transmission duration of the current batch data transmitted by the second system to each of the plurality of first processing nodes, the sending module is further configured to:

[0099] The system sends to the second system the identifier of each first processing node, the target data type of the current batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system.

[0100] Fourthly, embodiments of this application provide a data transmission device applied to a second system, the device comprising:

[0101] The sending module is used to send multiple first data transmission durations to the first system, each first data transmission duration representing the duration of sending the current batch of data to the first processing node in the first system.

[0102] The sending module is further configured to send a first acquisition request to the first system, the first acquisition request being used to acquire the identifier of a target processing node, the target processing node being a processing node among a plurality of first processing nodes whose current load meets preset conditions.

[0103] The receiving module is used to receive the identifier of the target processing node sent by the first system, so as to transmit the next batch of data to be transmitted to the target processing node.

[0104] In one possible design of the fourth aspect, after receiving the identifier of the target processing node sent by the first system, the sending module is further configured to send a plurality of second data transmission durations to the first system, each second data transmission duration representing the duration for sending new current batch data to the second processing node in the first system.

[0105] The sending module is further configured to send a second acquisition request to the first system. The second acquisition request is used to acquire the identifier of a new target processing node. The new target processing node is one of multiple second processing nodes and one of the target processing nodes whose load meets the preset conditions.

[0106] The receiving module is further configured to receive the identifier of the new target processing node sent by the first system, so as to transmit the new batch of data to be transmitted to the new target processing node.

[0107] In another possible design of the fourth aspect, before the duration of sending multiple first data transmissions to the first system, the receiving module is further configured to:

[0108] The system receives the identifiers of the multiple first processing nodes, the target data type of the current batch data, the identifier of the target port of the second system, and the identifier of the target process running in the second system, all sent by the first system.

[0109] The first data in the target process, which is in the target data format, is obtained from the target port, and the first data is transmitted to the plurality of first processing nodes.

[0110] Fifthly, embodiments of this application provide an electronic device, including: a processor, a transceiver, a memory, and computer program instructions stored in the memory and executable on the processor, wherein the processor executes the computer program instructions to implement the first aspect, the second aspect, and the methods provided by various possible designs.

[0111] In a sixth aspect, embodiments of this application may provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the first aspect, the second aspect, and various possible designs.

[0112] The data transmission method, apparatus, device, and storage medium provided in this application embodiment involve a first system acquiring the data transmission duration of the current batch of data transmitted by the second system to each of a plurality of first processing nodes. Based on the memory overflow index, data transmission duration, and multiple resource performance parameters of each first processing node, the first system determines the current load of each first processing node. Based on the current load of each first processing node, a target processing node is determined from the plurality of first processing nodes. Finally, the identifier of the target processing node is sent to the second system so that the next batch of data transmitted by the second system can be received through the target processing node. In this scheme, after the second system transmits data to the first system, the first system determines the target processing node for the next batch of data transmission based on the memory overflow index, multiple resource performance parameters, and data transmission duration of its own processing nodes. This fully considers the processing capabilities and states of different processing nodes, thereby improving load balancing. Attached Figure Description

[0113] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0114] Figure 1 This is a schematic diagram illustrating an application scenario of the data transmission method provided in an embodiment of this application.

[0115] Figure 2 A flowchart illustrating an embodiment of the data transmission method provided in this application;

[0116] Figure 3 A flowchart illustrating a second embodiment of the data transmission method provided in this application;

[0117] Figure 4 This is a schematic diagram of the structure of the first large top stack provided in an embodiment of this application;

[0118] Figure 5 A flowchart illustrating Embodiment 3 of the data transmission method provided in this application;

[0119] Figure 6 A schematic diagram illustrating the generation principle of the second largest top stack provided in an embodiment of this application;

[0120] Figure 7 A flowchart illustrating Embodiment 4 of the data transmission method provided in this application;

[0121] Figure 8 This is a schematic diagram of the structure of a data transmission device according to an embodiment of this application;

[0122] Figure 9This is a schematic diagram of the structure of a second embodiment of the data transmission device provided in this application.

[0123] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0124] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0125] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0126] First, the terms used in the embodiments of this application will be explained:

[0127] The Exponentially Weighted Moving-Average (EWMA) method assigns different weights to each observation, calculates a moving average based on these weights, and then uses this moving average as a basis to determine the predicted value. The weight of each observation decreases exponentially over time, with values ​​closer to the current time having larger weights. It does not require storing all past values, significantly reducing computational and storage costs.

[0128] Synthesis Load Exponentially Weighted Averages (SLEWMA): An exponentially weighted moving average of the synthesis load sample based on EWMA.

[0129] Load spikes: The overall load sample exhibits jitter.

[0130] The specific application scenario of this application is as follows: In existing technologies, third-party processing systems typically use round-robin or weighted round-robin algorithms to load balance financial business data. Round-robin or weighted round-robin refers to sending traffic data to the processing devices of the third-party processing system according to a specific value, where the value represents the frequency of data transmission. For example, assuming the third-party processing system includes two processing devices, device A and device B, the financial business system first sends data to device A for processing, then sends the data to device B for processing, and so on.

[0131] However, device A and device B have different processing capabilities. For example, if device A's processing capacity is 10 times that of device B, it's easy for device A to be idle while device B is heavily loaded. Furthermore, the amount of data sent by the financial business system varies each time; for instance, the financial business system might send 100MB of data to device A the first time and 1MB to device B the second time, leading to uneven traffic distribution. In summary, the existing technology suffers from poor load balancing.

[0132] Therefore, the data transmission method, apparatus, device, and storage medium provided in this application consider the impact of unit data size and the resource performance of each processing device in the third-party processing system on load balancing. Based on the memory overflow indicator used by each processing device to indicate whether there is a memory overflow, multiple resource performance parameters, and the data transmission time with the financial business system, the load of each processing device is determined so that the next batch of data sent by the financial business system is processed by the processing device with a smaller load, thereby effectively increasing load balancing.

[0133] For example, the data transmission method provided in this application embodiment can be applied to Figure 1 The diagram shows one application scenario. Figure 1 This diagram illustrates an application scenario of the data transmission method provided in this application, used to solve the aforementioned technical problems. For example... Figure 1 As shown, this application scenario may include: a data acquisition module 11, a data receiving module 12, and a comprehensive management module 13.

[0134] The data acquisition module 11 can be implemented through a traffic acquisition proxy. The data acquisition module 11 can be deployed on the data production node (the producer of the data to be transmitted) of the financial business system to collect bypass data traffic and forward the collected bypass data traffic to the data receiving module 12.

[0135] The data receiving module 12 is used to receive bypass data traffic from the data acquisition module 11, and to parse, process, and store it.

[0136] The comprehensive management module 13 is used to execute the algorithm for the data transmission method.

[0137] In this embodiment, the user can call the data type definer 132 and the data capture filter configurer 133 through the operation front end 131 of the integrated management module 13 to configure the relevant parameters of the data to be transmitted. These relevant parameters may include the data receiving module identifier (e.g., Figure 1 The data receiving module 12) contains data type, port identifier, and process identifier. The integrated management module 13, in response to user configuration operations, sends the aforementioned parameters to the data receiving module 12 and the data acquisition module 11.

[0138] Furthermore, regarding the data receiving module 12, it receives the aforementioned relevant parameters through the data type support interface 121 and configures the relational database management system (MySQL) parser 123 or the remote dictionary service (Redis) parser 124 according to these parameters. It should be understood that other types of parsers can also be configured here, and the type of parser can be determined according to the actual situation; this embodiment does not specifically limit this.

[0139] For the data acquisition module 11, it receives the aforementioned parameters through the capture filter and bypass address acquisition interface 111, acquires the current batch of data that conforms to the aforementioned parameters through the data acquisition engine 112, and sends the current batch of data to the data receiving module 12 through the data bypass interface 113. Simultaneously, the data acquisition module 11 obtains the data transmission duration of sending the aforementioned current batch of data to the data receiving module 12 through the time consumption statistics component 114, and sends the data transmission duration to the comprehensive management module 13 through the reporting interface 115.

[0140] Furthermore, the data receiving module 12 receives the aforementioned data, parses it using a MySQL parser 123 or a Redis parser 124, and stores the parsed data using a storage engine 125. Simultaneously, the data receiving module 12 collects multiple resource performance parameters through a performance metric acquisition component 126, a memory overflow metric monitor 127, and a traffic statistics component 128, and sends these parameters to the comprehensive management module 13 via a periodic reporting interface 129.

[0141] The integrated management module 13 receives multiple resource performance parameters sent by the data receiving module 12 and the data transmission duration sent by the data acquisition module 11 through the reporting and receiving interface 136, and calculates the data receiving module identifier (e.g., ...) for processing the next batch of data through the core computing engine 135. Figure 1 The data receiving module 12 sends the relevant parameters of the next batch of data to the data acquisition module 11 and the data receiving module corresponding to the above-mentioned data receiving module identifier.

[0142] The technical solution of this application will now be described in detail through specific embodiments.

[0143] It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0144] Figure 2 This is a flowchart illustrating an embodiment of the data transmission method provided in this application. Figure 2 As shown, this data transmission method is applied to a first system, which includes multiple processing nodes. The data transmission method may include the following steps:

[0145] S201. Obtain the data transmission duration of the current batch of data transmitted by the second system to each of the multiple first processing nodes.

[0146] In this embodiment of the application, the first system can be the aforementioned third-party processing system, and the second system can be the aforementioned financial business system. After the first system generates data, the data can be transmitted to the first system for storage.

[0147] It should be understood that the current batch data refers to the data transmitted by the second system to multiple first processing nodes. The second system can transmit the current batch data to each first processing node in real time, or it can transmit the current batch data to each first processing node at a certain frequency, which can be determined according to the actual situation. This application embodiment does not impose specific limitations on this.

[0148] The multiple first processing nodes are determined from multiple processing nodes. The first processing nodes can be some processing nodes or all processing nodes, which can be determined according to the actual situation. This application embodiment does not impose specific restrictions on this.

[0149] In a cloud computing environment, since the number of processing nodes in the first system is huge, some processing nodes can be pre-determined as the first processing nodes, which can effectively save the computing time and computing resources of subsequent processing and improve computing efficiency.

[0150] Optionally, prior to this step, the first system also needs to send the identifier of each first processing node to the second system so that the second system can transmit the current batch data to each first processing node.

[0151] S202. Determine the current load of each first processing node based on its memory overflow index, data transmission duration, and multiple resource performance parameters.

[0152] The memory overflow index for each first processing node is used to indicate whether the memory of that first processing node has overflowed.

[0153] It should be understood that the memory overflow metric can be the overload metric from the first processing node's own log. Specifically, the first processing node calculates the shared memory overflow amount in real time during operation and converts this amount into an overload metric, which is then written to its own log. If there is no memory overflow, the overload metric is 0. If a memory overflow occurs, the overload metric is 1. Once a memory overflow occurs, it indicates that data loss has occurred on the first processing node, and therefore, it cannot process or store the data transmitted from the second system.

[0154] Optionally, the resource performance parameters of each first processing node include at least two of the following: the current flow of the first processing node, the central processing unit (CPU) utilization (cpu_rate), the memory utilization (mem_rate), the network interface card input / output (IO) utilization (netio_rate), and the disk IO utilization (diskio_rate).

[0155] It should be understood that resource performance parameters may also include other content, which can be limited according to the actual situation. This application embodiment does not impose specific limitations on this.

[0156] This step determines the current load of the first processing node by analyzing resource performance across various dimensions, effectively improving the accuracy of load determination.

[0157] S203. Determine the target processing node from multiple first processing nodes based on the current load of each first processing node.

[0158] In practical applications, processing nodes with high loads have weak data processing capabilities, while those with low loads have strong data processing capabilities. Therefore, a target processing node can be determined based on the current load of each processing node, so that the next batch of data can be processed based on the determined target processing node.

[0159] The next batch of data refers to the data transmitted by the second system to the target processing node.

[0160] In one possible implementation, the target processing node can be determined by a preset number of first processing nodes with the lowest current load.

[0161] In another possible implementation, the first processing node whose current load is less than the preset load can be determined as the target processing node.

[0162] It should be understood that the number of target processing nodes can be one or more, depending on the actual situation. This application does not impose a specific limit on the number of target processing nodes.

[0163] S204. Send the identifier of the target processing node to the second system so that the target processing node can receive the next batch of data transmitted by the second system.

[0164] Optionally, when the second system needs to transmit the next batch of data to the first system, it can send an acquisition request to the first system. The first system then receives the acquisition request from the second system and sends the identifier of the target processing node to the second system according to the acquisition request.

[0165] Optionally, when there are multiple target processing nodes, the first system can send the identifiers of the target processing nodes to the second system sequentially, or it can send all the identifiers of the target processing nodes to the second system. When the first system sends the identifiers of the target processing nodes to the second system sequentially, the sending method can be random or sequential according to the current load of the target processing nodes from highest to lowest. This can be determined according to the actual situation, and this embodiment does not impose specific limitations on this.

[0166] The data transmission method provided in this application involves a first system acquiring the data transmission duration of the current batch of data transmitted by the second system to each of multiple first processing nodes. Based on the memory overflow index, data transmission duration, and multiple resource performance parameters of each first processing node, the first system determines the current load of each first processing node. Based on the current load of each first processing node, a target processing node is determined from the multiple first processing nodes. Finally, the identifier of the target processing node is sent to the second system so that the next batch of data transmitted by the second system can be received through the target processing node. In this scheme, after the second system transmits data to the first system, the first system determines the target processing node for the next batch of data transmission based on the memory overflow index, multiple resource performance parameters, and data transmission duration of its own processing nodes. This fully considers the processing capabilities and states of different processing nodes, thereby improving load balancing.

[0167] Optionally, in some embodiments, S202 can be implemented by the following steps:

[0168] Step 1: Determine the weight value of each resource performance based on the memory overflow index of each first processing node and multiple resource performance parameters.

[0169] The weight value of each resource performance is used to represent the importance of that resource performance to the current load of the first processing node.

[0170] It should be understood that the implementation principle and method of this step can be referred to Figure 3 The embodiments shown are not described in detail here.

[0171] Step 2: Determine the current load of each first processing node based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters.

[0172] The current load of each first processing node can be determined using the following formula:

[0173] Among them, load t Given the current load, 'a' is the weighted value of CPU utilization, 'b' is the weighted value of memory utilization, 'c' is the weighted value of disk I / O utilization, 'd' is the weighted value of network interface card I / O utilization, 'e' is the weighted value of current traffic, and 'cost_time' is the data transmission duration.

[0174] It should be understood that including (1-overload) in the denominator of the above formula does not mean that (1-overload) is actually being calculated as a denominator, but rather that it is being used to calculate the load. t When referring directly to the value of overload, when overload is 1, it represents load. t If the load is infinite, the current load of the first processing node can be set to a preset infinite load, or the first processing node can be directly removed so that it does not participate in subsequent calculation steps.

[0175] In the above embodiments, the importance of different resource performance to the current load of the first processing node is considered, and the current load of the first processing node is calculated according to the weight value of each resource performance, thereby improving the accuracy of the calculated current load of the first processing node and the accuracy of subsequent determination of the target processing node, and achieving true load balancing.

[0176] Optional, Figure 3 This is a flowchart illustrating a second embodiment of the data transmission method provided in this application. Figure 3 As shown, based on any of the above embodiments, determining the weight value of each resource performance based on the memory overflow index of each first processing node and multiple resource performance parameters can be achieved through the following steps:

[0177] S301. For each resource performance, determine multiple initial resource performance parameters based on the resource performance parameters of each first processing node.

[0178] Each initial resource performance parameter falls within the parameter range of the resource performance parameters of the multiple first processing nodes. For example, assuming the resource performance parameters of the first processing nodes are 1, 2, 3, 4, 5, 6, 7, 8, and 9, then each initial resource performance parameter falls within the range [1, 9].

[0179] In one possible implementation, for each resource performance parameter, the resource performance parameters of multiple first processing nodes are sorted in descending order to obtain the sorting result. Based on the sorting result, the average of two adjacent resource performance parameters is determined as the initial resource performance parameter.

[0180] In the above implementation, by determining the average value of two adjacent resource performance parameters in the sorting result as the initial resource performance parameter, the resource performance parameter of multiple first processing nodes can be divided into two parts according to each initial resource performance parameter, which improves the accuracy of the subsequent determination of the Gini index and thus improves the weight value of the calculated resource performance.

[0181] In another possible implementation, any value between two adjacent resource performance parameters in the sorting result can be used as the initial resource performance parameter. For example, assuming that the two adjacent resource performance parameters are 1 and 2, the initial resource performance parameter can be any value between 1 and 2, such as 1.2, 1.5, 1.8, etc.

[0182] In another possible implementation, the resource performance parameters can be directly determined as the initial resource performance parameters.

[0183] S302. Based on the memory overflow index of each first processing node and the resource performance parameter, determine the Gini index of each initial resource performance parameter.

[0184] For any initial resource performance parameter, the first processing nodes whose resource performance parameter is less than the initial resource performance parameter are defined as the first set, and the first processing nodes whose resource performance parameter is greater than or equal to the initial resource performance parameter are defined as the second set. Further, according to the following formula: Calculate the Gini index of the initial resource performance parameter. Here, M is the initial resource performance parameter, GINI(A, M) is the Gini index of the initial resource performance parameter M, A is the set of memory overflow indicators for each first processing node, A1 is the set of memory overflow indicators corresponding to the first set, and A2 is the set of memory overflow indicators corresponding to the second set. GINI(A1) can be calculated using the following formula: GINI(A1) = 2p1(1-p1), where p1 is the ratio of memory overflow indicators of 1 to those of 0 in A1. GINI(A2) can be calculated using the following formula: GINI(A2) = 2p2(1-p2), where p2 is the ratio of memory overflow indicators of 1 to those of 0 in A2.

[0185] S303. Determine the weight value of the resource performance based on the minimum value of the Gini index.

[0186] The criterion for selecting the Gini index is to achieve the highest purity for each first processing node, meaning that all first processing nodes falling within each observation partition (A1 and A2) belong to the same category. At this point, the Gini index is minimized, purity is highest, and uncertainty is lowest. For example, when the Gini index is minimized, the probability that all memory overflow indicators in A1 and A2 are consistent (both 1 or both 0) is highest.

[0187] The weight value of this resource performance can be calculated using the following formula: Weight = 100(1 - GINI(A, M)). Where, Weight is the weight value of this resource performance, GINI(A, M) is the minimum value of the Gini index calculated above, and 100 is the baseline weight value.

[0188] In the above embodiments, the impact of each resource performance on the current load of the first processing node varies, and may even dynamically affect the current load of the first processing node. Therefore, the weight value of the resource performance is determined in real time based on the minimum value of the Gini index, thereby reducing uncertainty, improving the accuracy of the calculated weight value, and enhancing generalization and overfitting resistance.

[0189] For example, to better illustrate the above embodiments, a specific example will be used to further illustrate them.

[0190] Assuming there are 10 first processing nodes, and the multiple resource performance parameters are current traffic, CPU utilization, memory utilization, network card IO utilization, and disk IO utilization, then the multiple resource performance parameters of each first processing node can be represented by Table 1.

[0191] Table 1

[0192]

[0193] 1. Resource performance is based on current traffic.

[0194] Extract the current traffic and corresponding overload data from Table 1, and rearrange them in descending order of current traffic. The rearranged data is then represented in Table 2.

[0195] Table 2

[0196] Identifier of the first processing node 1 2 3 4 5 6 7 8 9 10 Current traffic (Mb / s) 534 450 430 400 345 284 282 281 280 270 overload 1 1 1 0 1 1 0 0 0 0

[0197] The initial resource performance parameters and their Gini indexes are calculated using Table 2, and the current flow rate's Gini index is determined based on the calculation results.

[0198] When calculating initial resource performance parameters, a random selection can be made between two adjacent current traffic flows. This example uses the average value between two adjacent current traffic flows as the initial resource performance parameter for further explanation. First, the first initial resource performance parameter is calculated. The first processing node is divided into two parts: a first set {1} and a second set {1,2,3,4,5,6,7,8,9,10}. The overload set of the first set is A1, and the overload set of the second set is A2. A1 is {1}, and A2 is {1,1,0,1,1,0,0,0,0}.

[0199] For A1, p1 = 1. GINI(A1) = 0.

[0200] For A2,

[0201] so,

[0202] Further selection A1 is {1,1}, and A2 is {1,0,1,1,0,0,0,0}.

[0203] For A1, p1 = 1. GINI(A1) = 0.

[0204] For A2,

[0205] so,

[0206] Further selection A1 is {1,1,1}, and A2 is {0,0,0,0,1,1,0}.

[0207] For A1, p1 = 1. GINI(A1) = 0.

[0208] For A2,

[0209] so,

[0210] Based on the process and principles outlined above, select M = 492, 440, 415, 373, 315, 283, 281, 280, and 275 in sequence, and calculate the Gini index for each M. For example, the Gini index for each M can be found in Table 3.

[0211] Table 3

[0212] M 492 440 415 373 315 283 281 280 275 440 Gini index 0.44 0.38 0.29 0.42 0.32 0.17 0.29 0.38 0.44 0.44

[0213] As shown in Table 3, the minimum value of the Gini index is 0.17, and the corresponding M is 283. Therefore, the weight value of the current flow is e = 100(1-GINI(A,M)) = 100×(1-0.17) = 83.

[0214] 2. Resource performance refers to the network card's I / O utilization.

[0215] Extract the IO utilization rate and corresponding overload data of the network card from Table 1, and rearrange them in descending order of IO utilization rate. The rearranged data is represented in Table 4.

[0216] Table 4

[0217] Identifier of the first processing node 1 2 3 4 5 6 7 8 9 10 Network interface card (NIC) I / O utilization (%) 95 83 80 80 79 78 75 73 71 70 overload 1 1 1 1 1 0 0 0 0 0

[0218] The initial resource performance parameters and their Gini indexes are calculated using Table 4, and the current flow rate's Gini index is determined based on the calculation results.

[0219] First, calculate the first initial resource performance parameter. A1 is {1}, and A2 is {1,1,1,1,0,0,0,0,0}.

[0220] For A1, p1 = 1. GINI(A1) = 0.

[0221] For A2,

[0222] so,

[0223] Further selection A1 is {1,1}, and A2 is {1,1,1,0,0,0,0,0}.

[0224] For A1, p1 = 1. GINI(A1) = 0.

[0225] For A2,

[0226] so,

[0227] Further selection A1 is {1,1,1}, and A2 is {1,1,0,0,0,0,0,0}.

[0228] For A1, p1 = 1. GINI(A1) = 0.

[0229] For A2,

[0230] so,

[0231] Based on the process and principles outlined above, select M = 89, 82, 80, 79, 78, 76, 74, 72, and 70 in sequence, and calculate the Gini index for each M. For example, the Gini index for each M can be found in Table 5.

[0232] Table 5

[0233] M 89 82 80 79 78 76 74 72 70 89 Gini index 0.44 0.38 0.29 0.17 0 0.17 0.29 0.38 0.44 0.44

[0234] As shown in Table 5, the minimum value of the Gini index is 0, and the corresponding M is 78. Therefore, the weight value of the current flow is d = 100(1 - Gini(A, M) = 100 × (1 - 0) = 100.

[0235] 3. Resource performance refers to CPU utilization.

[0236] Extract the CPU utilization and corresponding overload data from Table 1, and rearrange them in descending order of CPU utilization. The rearranged data is then presented in Table 6.

[0237] Table 6

[0238] Identifier of the first processing node 1 2 3 4 5 6 7 8 9 10 CPU utilization (%) 72 60 54 50 45 43 40 35 30 28 overload 0 0 1 1 0 1 1 0 1 0

[0239] Based on the sorted CPU utilization and corresponding overload data in Table 6, determine each M and its corresponding Gini index. For example, the Gini index for each M can be found in Table 7.

[0240] Table 7

[0241] M 66 57 52 48 44 42 38 33 29 66 Gini index 0.44 0.38 0.48 0.5 0.48 0.5 0.48 0.5 0.44 0.44

[0242] As shown in Table 7, the minimum value of the Gini index is 0.38, and the corresponding M is 57. Therefore, the weight value of the current flow is a = 100(1-GINI(A,M)) = 100×(1-0.38) = 62.

[0243] 4. Resource performance refers to memory utilization.

[0244] Extract the memory usage rate and corresponding overload data from Table 1, and rearrange them in descending order of memory usage rate. The rearranged data is then presented in Table 8.

[0245] Table 8.

[0246] Identifier of the first processing node 1 2 3 4 5 6 7 8 9 10 Memory utilization (%) 90 67 64 60 50 48 48 47 45 40 overload 0 1 0 1 0 0 1 1 1 0

[0247] Based on the sorted memory usage and corresponding overload data in Table 8, determine each M and its corresponding Gini index. For example, the Gini index for each M can be found in Table 9.

[0248] Table 9

[0249] M 79 66 62 55 49 48 47 46 43 79 Gini index 0.44 0.5 0.48 0.5 0.48 0.42 0.48 0.5 0.44 0.44

[0250] As shown in Table 9, the minimum value of the Gini index is 0.42, and the corresponding M is 48. Therefore, the weight value of memory utilization is b = 100(1-GINI(A,M)) = 100×(1-0.42) = 58.

[0251] 5. When resource performance is measured by disk I / O utilization.

[0252] Extract the disk IO utilization rate and corresponding overload data from Table 1, and rearrange them in descending order of disk IO utilization rate. The rearranged data is then represented in Table 10.

[0253] Table 10.

[0254] Identifier of the first processing node 1 2 3 4 5 6 7 8 9 10 Disk I / O utilization (%) 82 75 70 69 65 64 58 56 46 40 overload 0 0 1 1 1 0 1 0 0 1

[0255] Based on the sorted disk IO utilization and corresponding overload data in Table 10, determine each M and its corresponding Gini index. For example, the Gini index of each M can be referred to in Table 11.

[0256] Table 11

[0257]

[0258]

[0259] As shown in Table 11, the minimum value of the Gini index is 0.38, and the corresponding M is 73. Therefore, the weight value of disk IO utilization is c = 100(1-GINI(A,M)) = 100×(1-0.38) = 62.

[0260] Optionally, in some embodiments, after S202, the data transmission method may further include the step of determining the current load of each first processing node as the current SLEWMA value.

[0261] If the first processing node has no historical load, then the current load of the first processing node is determined as the SLEWMA value.

[0262] Historical load refers to the load determined by the first processing node when processing the previous batch of data.

[0263] Accordingly, in this embodiment, determining the target processing node from multiple first processing nodes based on the current load of each first processing node can be achieved through the following steps:

[0264] Based on the SLEWMA value of each first processing node, target processing nodes that meet preset conditions are determined from multiple first processing nodes. The preset conditions are that the SLEWMA value of the first processing node is greater than a preset SLEWMA value, or a preset number of first processing nodes with the largest SLEWMA value.

[0265] In one possible implementation, the first processing node whose SLEWMA value is greater than a preset SLEWMA value can be determined as the target processing node. In this method, the preset condition is that the SLEWMA value of the first processing node is greater than the preset SLEWMA value.

[0266] In another possible implementation, a preset number of first processing nodes with the largest SLEWMA value can be determined as target processing nodes. For example, assuming the preset number is 7, this method determines the 7 first processing nodes with the largest SLEWMA value as target processing nodes. In this method, the preset condition is the preset number of first processing nodes with the largest SLEWMA value.

[0267] In the above embodiments, the target processing node can be determined based on the SLEWMA value and preset conditions. These preset conditions can be pre-configured according to specific needs, improving the flexibility of determining the target processing node. Furthermore, determining the target processing node based on the SLEWMA value eliminates the need to store all historical data of the first processing node, effectively reducing computational load and increasing the available data storage space of the first system.

[0268] Optionally, in some embodiments, after determining the target processing node that meets the preset conditions from a plurality of first processing nodes based on the SLEWMA value of each first processing node, the data transmission method may further include the following steps:

[0269] Based on the current SLEWMA value of the target processing node, the identifier of the target processing node is placed into an initially empty max-heap, generating the first max-heap.

[0270] In this first max-heap, the SLEWMA value of each node is greater than or equal to the SLEWMA values ​​of its left and right child nodes. For example, Figure 4 This is a schematic diagram of the structure of the first large top stack provided in an embodiment of this application. Figure 4 As shown, the SLEWMA value of target processing node 1 is greater than or equal to the SLEWMA values ​​of target processing node 2 and target processing node 3. Similarly, the SLEWMA value of target processing node 2 is greater than or equal to the SLEWMA values ​​of target processing node 4 and target processing node 5. The SLEWMA value of target processing node 3 is greater than or equal to the SLEWMA values ​​of target processing node 6 and target processing node 7.

[0271] Accordingly, in this embodiment, sending the identifier of the target processing node to the second system can be achieved through the following steps: sending the identifiers of the target processing nodes in the first top heap to the second system in a bottom-up order.

[0272] In the above embodiment, since the SLEWMA value of the target processing node in the first top heap decreases layer by layer, the identifier of the target processing node can be sent to the second system in a bottom-up order, so that the second system can transmit data to the processing node with the smallest SLEWMA value among the target processing nodes, thereby further increasing the load balancing.

[0273] Optionally, in some embodiments, the first system may also receive the current batch data transmitted by the second system through the first processing node, format the current batch data, and store the formatted data.

[0274] For example, the aforementioned current batch data can be a protocol packet, a response packet, or data in other formats; this application embodiment does not impose specific limitations on this.

[0275] Optionally, assuming the current batch of data is a MySQL protocol packet, the basic structure of the MySQL protocol packet can be represented by Table 11.

[0276] Table 11

[0277] MySQL package length (int) MySQL package number (int) MySQL package content string

[0278] One of the structures shown in Table 11 above can be called a MySQL L package.

[0279] Optionally, assuming the current batch of data is a MySQL response packet, the basic structure of the MySQL response packet can be represented by Table 12.

[0280] Table 12

[0281] MySQL package type illustrate [Result Set Header] Number of columns [Field] Column information (multiple MySQL packages) [EOF] End of column [Row Data] Row data (multiple MySQL packages) [EOF] End of data processing

[0282] Optionally, the current batch of data can be parsed using the MySQL protocol and its field content extracted, formatted, and then stored. For example, the stored keywords could be: session ID | source IP (identifier of the first system) | source PORT (port used for data transmission in the first system) | destination IP (identifier of the second system) | destination PORT (port used for data transmission in the second system) | database name | table name | key for each field | value for each field.

[0283] Optionally, the storage method can be to write the formatted data to a file in the first system, or to a database or big data platform. The database or big data platform can be deployed outside or inside the first system; this embodiment does not impose specific limitations on this.

[0284] Optional, Figure 5 This is a flowchart illustrating a third embodiment of the data transmission method provided in this application. Figure 5 As shown, based on any of the above embodiments, after sending the identifier of the target processing node to the second system, the data transmission method may further include the following steps:

[0285] S501. Determine the current load of each first processing node as the historical load, and re-determine multiple second processing nodes from multiple processing nodes.

[0286] After the second system transmits the current batch of data to the first system, a new target processing node needs to be determined to facilitate subsequent data transmission.

[0287] For example, multiple second processing nodes can be randomly determined from multiple processing nodes. The second processing nodes can be all processing nodes or some processing nodes. The second processing nodes can be the same as or different from the first processing nodes. The determination can be made according to the actual situation. This application embodiment does not impose specific restrictions on this.

[0288] S502. Determine the current load of each second processing node based on the memory overflow index, multiple resource performance parameters, and data transmission duration of each second processing node.

[0289] The implementation method and principle of this step can be referred to in sections S201 and S202, and will not be repeated here.

[0290] S503. Based on the current load of each second processing node and the historical load of the target processing node, a new target processing node is re-determined from multiple second processing nodes and the target processing node.

[0291] In one possible implementation, a processing node with a load greater than a preset limit can be identified as the target processing node.

[0292] In another possible implementation, the target processing nodes can be determined as a preset number of processing nodes with the highest load.

[0293] It should be understood that the processing nodes in each method in this step are both the second processing node and the target processing node.

[0294] S504. Send the identifier of the new target processing node to the second system.

[0295] The implementation method and principle of this step can be found in section S204, and will not be repeated here.

[0296] In the above embodiments, a new target processing node is determined based on the current load of the second processing node and the historical load of the previously determined target processing node. This allows the target processing node to be redefined as a new target processing node when the current load of the second processing node is greater than the historical load of the target processing node. This expands the selection range for determining a new target processing node and improves the accuracy of determining a new target processing node.

[0297] Optionally, in some embodiments, after determining the current load of each second processing node based on the memory overflow metric of each second processing node, multiple resource performance parameters, and data transmission duration, the data transmission method may further include the following steps:

[0298] Step 1: If any second processing node has no historical load, then the current load of the second processing node is determined as the current SLEWMA value of the second processing node.

[0299] Step 2: If any second processing node has historical load, calculate the current SLEWMA value of that second processing node based on its current load, historical load, and a pre-set spike variable. The spike variable is determined based on the historical load. The calculation of the current SLEWMA value of the second processing node can be achieved using the following formula: synload t+1 =β×synload t +(1-β)×load t+1 , among which, synload t+1 The current SLEWMA value of the second processing node, β is the glitch variable, and synload t For the historical SLEWMA value of this second processing node, load t+1 This represents the current load of the second processing node.

[0300] Optionally, β represents the weighted descent rate, which can be updated periodically according to the following formula: Where m is a pre-defined constant.

[0301] Step 3: Determine the historical SLEWMA value of the target processing node based on its historical load and spike variables. The determination method can be implemented using the formula described above, and will not be repeated here.

[0302] Accordingly, in this embodiment, the process of re-determining a new target processing node from multiple second processing nodes and the target processing node based on the current load of each second processing node and the historical load of the target processing node can be achieved through the following steps:

[0303] Based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, a new target processing node is re-determined from multiple second processing nodes and the target processing node.

[0304] It should be understood that the implementation method and principle of this step can refer to the above content on determining the target processing node that meets the preset conditions from multiple first processing nodes based on the SLEWMA value of each first processing node, which will not be repeated here.

[0305] Optionally, each second processing node can be polled in turn, and the node with the highest current SLEWMA value in the first max-heap can be replaced with the second processing node with the lowest current SLEWMA value. Then, the heap is rebuilt based on the replaced processing nodes to generate the second max-heap.

[0306] For example, Figure 6 This is a schematic diagram illustrating the generation principle of the second largest top heap provided in an embodiment of this application. (See attached diagram.) Figure 6As shown, the second processing node (such as...) Figure 6 The second processing node 1, second processing node 2, second processing node 3, second processing node 4, second processing node 5, second processing node 6 and second processing node 7 in the list are determined as the node to be polled, so as to determine whether it needs to be replaced with the target processing node in the first top heap based on the current SLEWMA value of each second processing node in the node to be polled list.

[0307] In the above embodiments, β is updated in real time according to the natural exponential decay function. The larger the historical load, the smaller β becomes, and the larger (1-β) becomes. t+1 Subject to load t+1 The greater the impact, the faster SLEWMA can monitor load spikes. Conversely, the smaller the historical load, the larger β is, and the smaller (1-β) is, indicating a lower synload. t+1 Subject to load t+1 The smaller the impact, the less sensitive it is to load spikes, and the more it reflects the overall average value. This achieves the goal of being able to quickly monitor load spikes when the load on the processing node is high, and being insensitive to load spikes when the load is low.

[0308] Optionally, in some embodiments, after a new target processing node is re-determined from a plurality of second processing nodes and the target processing node based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the data transmission method may further include the following steps:

[0309] Based on the current SLEWMA value and identifier of the new target processing node, the first max heap is updated to generate the second max heap.

[0310] It should be understood that the method and principle of rebuilding the heap based on the processing nodes after the replacement process can be referred to in the section on generating the first max heap, and will not be repeated here.

[0311] Accordingly, sending the identifier of the new target processing node to the second system can be achieved through the following steps:

[0312] The identifiers of the new target processing nodes in the second largest top heap are sent to the second system in a bottom-up order.

[0313] It should be understood that the implementation method and principle of this step can be referred to the content of sending the identifiers of the target processing nodes in the first top heap to the second system part in sequence, and will not be repeated here.

[0314] In the above embodiments, after the identifier of the target processing node is updated, the first top heap is updated according to the updated identifier of the new target processing node, so that the new identifier of the target processing node is sent to the second system according to the updated second top heap, thereby further ensuring load balancing.

[0315] Optionally, in some embodiments, determining the current load of each first processing node based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters can be achieved through the following steps:

[0316] The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, the preset weight value, and multiple resource performance parameters.

[0317] Optionally, the current load of the first processing node can be determined using the following formula:

[0318]

[0319] Where, weight 预设 The preset weight values.

[0320] In the above embodiments, staff can pre-configure preset weight values ​​based on actual application effects, thereby increasing the controllability of calculating the current load.

[0321] Optionally, in some embodiments, before obtaining the data transmission duration of the current batch of data transmitted by the second system to each of the plurality of first processing nodes, the data transmission method may further include the following steps:

[0322] Send to the second system the identifier of each first processing node, the target data type of the current batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system.

[0323] For example, the target data type can be a MySQL data type, the target port can be 3306, and the identifier of the target process can be MySQL.

[0324] In this embodiment, the first system can pre-send parameters for data collection to the second system, enabling the second system to collect data based on these parameters. Furthermore, these parameters can be pre-configured according to the user's actual needs, supporting pluggable configurations for various data types.

[0325] Optional, Figure 7 This is a flowchart illustrating Embodiment 4 of the data transmission method provided in this application. Figure 7As shown, this data transmission method is applied to a second system, and the data transmission method may include the following steps:

[0326] S701, Send multiple first data transmission durations to the first system.

[0327] In this embodiment, the second system can receive in advance the identifiers of multiple first processing nodes sent by the first system, and transmit the current batch data to the multiple first processing nodes. After transmitting the current batch data, the second system sends a first data transmission duration to the first system so that the first system can calculate the current load of each first processing node based on the first data transmission duration.

[0328] Each first data transmission duration is used to represent the duration for sending the current batch of data to the first processing node in the first system.

[0329] S702, Send the first acquisition request to the first system.

[0330] The first acquisition request is used to acquire the identifier of the target processing node, which is the processing node whose current load meets the preset conditions among multiple first processing nodes.

[0331] S703: Receive the identifier of the target processing node sent by the first system, so as to transmit the next batch of data to be transmitted to the target processing node.

[0332] In the above embodiments, after transmitting the current batch of data to the first system, the second system sends multiple first data transmission durations to the first system and sends a first acquisition request to the first system, so that the target processing node identifier sent by the first system can be received and the next batch of data to be transmitted can be transmitted to the target processing node, thereby improving the load balancing of the first system.

[0333] Optionally, in some embodiments, after receiving the identifier of the target processing node sent by the first system, the data transmission method may further include the following steps:

[0334] Step 1: Send multiple second data transmission durations to the first system. Each second data transmission duration represents the duration for sending new current batch data to each of the multiple second processing nodes in the first system.

[0335] Step 2: Send a second acquisition request to the first system. The second acquisition request is used to acquire the identifier of the new target processing node. The new target processing node is one of the multiple second processing nodes and the processing node whose load meets the preset conditions among the target processing nodes.

[0336] Step 3: Receive the identifier of the new target processing node sent by the first system, so that the new batch of data to be transmitted can be transmitted to the second target processing node.

[0337] The implementation method and principle of the above embodiments can be referred to Figure 6 The embodiments shown are not described in detail here.

[0338] In the above embodiment, the second system sends a second acquisition request to the first system to acquire the identifier of the new target processing node, ensuring that data is transmitted to the processing node with the lower current load of the first system each time, thereby further ensuring the load balance of the first system.

[0339] Optionally, in some embodiments, before sending multiple first data transmission durations to the first system, the data transmission method may further include the following steps:

[0340] It receives the identifiers of multiple first processing nodes sent by the first system, the target data type of the current batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system.

[0341] The first data in the target process is obtained from the target port and is in the target data format. The first data is then transmitted to multiple first processing nodes.

[0342] For example, assuming the target data type is MySQL, the target port is 3306, and the target process is identified as MySQL, the second system can listen on port 3306 of the network card and capture all data traffic of the MySQL process passing through port 3306.

[0343] In the above embodiments, data can be collected according to the parameters sent in advance by the second system, and the collected data can be transmitted to the first system for storage, which ensures data security, saves storage space of the second system, and ensures the operating speed of the second system.

[0344] Based on the data transmission methods in the above embodiments, the following is a specific example to illustrate the scheme.

[0345] In this example, the data transfer method may include the following steps:

[0346] Step 1: The first system sends the identifiers of multiple first processing nodes, the target data type of the first batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system to the second system.

[0347] For the second system, it receives the identifiers of multiple first processing nodes, the target data type of the first batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system from the first system.

[0348] Step 2: The second system collects the first batch of data based on the target data type, the target port identifier, and the target process identifier, and sends it to multiple first processing nodes.

[0349] Step 3: The second system obtains the data transmission duration for transmitting the first batch of data to each first processing node and sends the data transmission duration to the first system.

[0350] For the first system, it receives the first batch of data transmitted by the second system, as well as the duration of multiple data transmissions.

[0351] Step 4: The first system formats the first batch of data and writes it to a file.

[0352] Step 5: The first system obtains the memory overflow index and multiple resource performance parameters of each first processing node.

[0353] Step 6: The first system determines the current load of each first processing node based on the memory overflow index, data transmission duration, and multiple resource performance parameters of each first processing node.

[0354] Step 7: The first system calculates the current SLEWMA value of each first processing node based on the current load of each first processing node.

[0355] Step 8: The first system determines the target processing node from multiple first processing nodes based on the current SLEWMA value of each first processing node.

[0356] Step 9: The second system sends an acquisition request to the first system.

[0357] For the first system, it receives the acquisition request sent by the second system.

[0358] Step 10: The first system sends the identifier of the target processing node to the second system.

[0359] For the second system, it receives the identifier of the target processing node sent by the first system.

[0360] Step 11: The second system transmits the collected second batch of data to the target processing node.

[0361] For the first system, it receives the second batch of data transmitted from the second system, formats the second batch of data, and stores it.

[0362] Step 12: The first system re-determines the identifier of the second processing node and sends the identifier of the second processing node to the second system.

[0363] For the second system, it receives the identifier of the second processing node sent by the first system.

[0364] Step 12: The second system collects the third batch of data and transmits the third batch of data to the second processing node.

[0365] Step 13: The second system obtains the data transmission duration for transmitting the third batch of data to each second processing node and sends the data transmission duration to the first system.

[0366] For the first system, it receives the third batch of data transmitted by the second system, as well as multiple transmission durations.

[0367] Step 14: The first system formats the third batch of data and writes it to a file.

[0368] Step 15: The first system obtains the memory overflow index and multiple resource performance parameters of each second processing node.

[0369] Step 16: The first system determines the current load of each second processing node based on the memory overflow index, data transmission duration, and multiple resource performance parameters of each second processing node.

[0370] Step 17: The first system determines the current SLEWMA value of each second processing node based on the current load and historical load of each second processing node.

[0371] Step 18: The first system determines the new target processing node based on the current SLEWMA value of each second processing node and the historical SLEWMA value of each target processing node.

[0372] Step 19: The second system sends an acquisition request to the first system.

[0373] For the first system, it receives the acquisition request sent by the second system.

[0374] Step 20: The first system sends the identifier of the new target processing node to the second system.

[0375] For the second system, it receives the identifier of the new target processing node sent by the first system.

[0376] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0377] Figure 8 This is a schematic diagram of the structure of a data transmission device according to an embodiment of this application. Figure 8 As shown, the data transmission device is applied to a first system, which includes multiple processing nodes.

[0378] The acquisition module 801 is used to acquire the data transmission duration of the current batch of data transmitted by the second system to each of the multiple first processing nodes, wherein the multiple first processing nodes are determined from the multiple processing nodes.

[0379] The processing module 802 is used to determine the current load of each first processing node based on the memory overflow index, data transmission duration and multiple resource performance parameters of each first processing node. The memory overflow index of each first processing node is used to indicate whether the memory of the first processing node has overflowed.

[0380] The processing module 802 is also used to determine the target processing node from multiple first processing nodes based on the current load of each first processing node.

[0381] The sending module 803 is used to send the identifier of the target processing node to the second system so that the target processing node can receive the next batch of data transmitted by the second system.

[0382] In one possible design of this application embodiment, the processing module 802 is specifically used for:

[0383] Based on the memory overflow metric of each first processing node and multiple resource performance parameters, a weight value for each resource performance parameter is determined. The weight value of each resource performance parameter is used to represent the importance of that resource performance to the current load of the first processing node.

[0384] The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters.

[0385] Optionally, processing module 802 is specifically used for:

[0386] For each resource performance, multiple initial resource performance parameters are determined based on the resource performance parameter of each first processing node, and each initial resource performance parameter is within the parameter range of the resource performance parameter of the multiple first processing nodes.

[0387] Based on the memory overflow index of each first processing node and the resource performance parameter, determine the Gini index of each initial resource performance parameter.

[0388] The weight value of the resource performance is determined based on the minimum value of the Gini index.

[0389] Optionally, processing module 802 is specifically used for:

[0390] For each resource performance parameter, the resource performance parameters of multiple first processing nodes are sorted in descending order to obtain the sorting result.

[0391] Based on the sorting results, the average value of two adjacent resource performance parameters is determined as the initial resource performance parameter.

[0392] In another possible design of this application embodiment, after determining the current load of each first processing node based on the memory overflow index, data transmission duration, and multiple resource performance parameters of each first processing node, the processing module 802 is further configured to:

[0393] The current load of each first processing node is determined as the current SLEWMA value of that first processing node.

[0394] Accordingly, processing module 802 is specifically used for:

[0395] Based on the current SLEWMA value of each first processing node, target processing nodes that meet preset conditions are determined from multiple first processing nodes. The preset conditions are that the current SLEWMA value of the first processing node is greater than a preset SLEWMA value, or a preset number of first processing nodes with the largest current SLEWMA value.

[0396] Optionally, after determining the target processing node that meets the preset conditions from multiple first processing nodes based on the current SLEWMA value of each first processing node, the processing module 802 is further configured to put the identifier of the target processing node into an initially empty max heap based on the current SLEWMA value of the target processing node, thereby generating a first max heap.

[0397] Accordingly, the sending module 803 is specifically used for:

[0398] The identifiers of the target processing nodes in the first large top heap are sent to the second system in a bottom-up order.

[0399] Optionally, after sending the identifier of the target processing node to the second system, the processing module 802 is further configured to determine the current load of each first processing node as the historical load, and to re-determine multiple second processing nodes from the multiple processing nodes.

[0400] The processing module 802 is also used to determine the current load of each second processing node based on the memory overflow index of each second processing node, multiple resource performance parameters, and data transmission duration.

[0401] The processing module 802 is also used to re-determine a new target processing node from multiple second processing nodes and the target processing node based on the current load of each second processing node and the historical load of the target processing node.

[0402] The sending module 803 is also used to send the identifier of the new target processing node to the second system.

[0403] Optionally, after determining the current load of each second processing node based on its memory overflow metric, multiple resource performance parameters, and data transmission duration, the processing module 802 further configures itself to:

[0404] If any second processing node has no historical load, then the current load of that second processing node is determined as the current SLEWMA value of that second processing node.

[0405] If any second processing node has historical load, the current SLEWMA value of the second processing node is calculated based on its current load, historical load, and a pre-set glitch variable. The glitch variable is determined based on the historical load.

[0406] Determine the historical SLEWMA value of the target processing node based on its historical load and glitch variables.

[0407] Accordingly, processing module 802 is specifically used for:

[0408] Based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, a new target processing node is re-determined from multiple second processing nodes and the target processing node.

[0409] Optionally, after determining a new target processing node from multiple second processing nodes and the target processing node based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the processing module 802 is further configured to update the first max heap based on the current SLEWMA value and identifier of the new target processing node to generate a second max heap.

[0410] Accordingly, the sending module 803 is specifically used for:

[0411] The identifiers of the new target processing nodes in the second largest top heap are sent to the second system in a bottom-up order.

[0412] Optionally, processing module 802 is specifically used for:

[0413] The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, the preset weight value, and multiple resource performance parameters.

[0414] In another possible design of the embodiments of this application, the multiple resource performance parameters of each first processing node include at least two of the following: the current traffic of the first processing node, the CPU utilization, the memory utilization, the network interface card (NIC) input / output utilization, and the disk NIC input / output utilization.

[0415] In another possible design of this application embodiment, before obtaining the data transmission duration of the current batch data transmitted by the second system to each of the plurality of first processing nodes, the sending module 803 is further configured to:

[0416] The system sends to the second system the identifier of each first processing node, the target data type of the current batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system. In this embodiment, the acquisition module 801 is equivalent to... Figure 1 The data receiving module 12, processing module 802, and sending module 803 in the illustrated embodiment are equivalent to Figure 1 The integrated management module 13 in the illustrated embodiment.

[0417] The data transmission device provided in this application embodiment can be used to execute the data transmission method on the first system side in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0418] Figure 9 This is a schematic diagram of the structure of a second embodiment of the data transmission device provided in this application. Figure 9 As shown, the data transmission device is applied to the second system, and the data transmission device includes:

[0419] The sending module 901 is used to send multiple first data transmission durations to the first system, each first data transmission duration representing the duration of sending the current batch of data to the first processing node in the first system.

[0420] The sending module 901 is also used to send a first acquisition request to the first system. The first acquisition request is used to acquire the identifier of the target processing node. The target processing node is a processing node among multiple first processing nodes whose current load meets preset conditions.

[0421] The receiving module 902 is used to receive the identifier of the target processing node sent by the first system, so as to transmit the next batch of data to be transmitted to the target processing node.

[0422] In one possible design of this application embodiment, after receiving the identifier of the target processing node sent by the first system, the sending module 901 is further configured to send a plurality of second data transmission durations to the first system, each second data transmission duration representing the duration for sending new current batch data to the second processing node in the first system.

[0423] The sending module 901 is also used to send a second acquisition request to the first system. The second acquisition request is used to acquire the identifier of a new target processing node. The new target processing node is one of multiple second processing nodes and one of the target processing nodes whose load meets preset conditions.

[0424] The receiving module 902 is also used to receive the identifier of a new target processing node sent by the first system, so as to transmit the new batch of data to be transmitted to the new target processing node.

[0425] In another possible design of this application embodiment, before sending multiple first data transmission durations to the first system, the receiving module 902 is further configured to:

[0426] It receives the identifiers of multiple first processing nodes sent by the first system, the target data type of the current batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system.

[0427] The first data in the target process is obtained from the target port and is in the target data format. The first data is then transmitted to multiple first processing nodes.

[0428] The sending module 901 and receiving module 902 in the embodiments of this application are equivalent to Figure 1 The data acquisition module 11 in the illustrated embodiment.

[0429] The data transmission device provided in this application embodiment can be used to execute the data transmission method on the second system side in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0430] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls, or entirely in hardware. Alternatively, some modules can be implemented through processing element calls in software, while others are implemented in hardware. Moreover, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0431] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device may include: a processor 1001, a memory 1002, a transceiver 1003, and computer program instructions stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program instructions, it implements the data transmission method provided in any of the foregoing embodiments.

[0432] Optionally, the various components of the electronic device can be connected via a system bus.

[0433] The memory 1002 can be a separate memory unit or a memory unit integrated into the processor. The number of processors can be one or more.

[0434] Optionally, the electronic device may also include an interface for interacting with other devices.

[0435] The transceiver 1003 is used to communicate with other computers and forms a communication interface.

[0436] Optionally, in terms of hardware implementation, the above Figure 8 In the illustrated embodiment: the acquisition module 801 and the transmission module 803 correspond to the transceiver 1003 in this embodiment, and the processing module 802 corresponds to the processor 1001 in this embodiment; the above Figure 9 The transmitting module 901 and receiving module 902 in the illustrated embodiment correspond to the transceiver 1003 in this embodiment.

[0437] It should be understood that the processor 1001 can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0438] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0439] All or part of the steps in the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-described method embodiments. The aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0440] The electronic device provided in this application embodiment can be used to execute the data transmission method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0441] This application provides a system including the above-described electronic device, which can be used to execute the data transmission method provided in any of the above method embodiments. The implementation principle and technical effects are similar, and will not be repeated here.

[0442] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed on a computer, the computer performs the aforementioned data transmission method.

[0443] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0444] Optionally, a readable storage medium can be coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0445] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above-described data transmission method when executing the computer program.

[0446] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data transmission method, characterized in that, Applied to a first system, the first system comprising multiple processing nodes, the method includes: The data transmission duration of the current batch of data transmitted by the second system to each of the plurality of first processing nodes is obtained, wherein the plurality of first processing nodes are determined from the plurality of processing nodes; Based on the memory overflow index, data transmission duration, and multiple resource performance parameters of each first processing node, the current load of each first processing node is determined. The memory overflow index of each first processing node is used to indicate whether the memory of the first processing node is overflowing. The target processing node is determined from the plurality of first processing nodes based on the current load of each first processing node. The identifier of the target processing node is sent to the second system so that the next batch of data transmitted by the second system can be received through the target processing node; After determining the current load of each first processing node based on its memory overflow metric, data transfer duration, and multiple resource performance parameters, the method further includes: The current load of each first processing node is determined as the current comprehensive load sample index-weighted moving average (SLEWMA) value of that first processing node; Accordingly, determining the target processing node from the plurality of first processing nodes based on the current load of each first processing node includes: Based on the current SLEWMA value of each first processing node, the target processing node that meets the preset conditions is determined from the plurality of first processing nodes. The preset conditions are that the current SLEWMA value of the first processing node is greater than a preset SLEWMA value, or a preset number of first processing nodes with the largest current SLEWMA value. After determining the target processing node that meets the preset conditions from the plurality of first processing nodes based on the current SLEWMA value of each first processing node, the method further includes: Based on the current SLEWMA value of the target processing node, the identifier of the target processing node is placed into an initially empty max-heap to generate the first max-heap. Accordingly, sending the identifier of the target processing node to the second system includes: The identifiers of the target processing nodes in the first top heap are sent to the second system in a bottom-up order. After sending the identifier of the target processing node to the second system, the method further includes: The current load of each first processing node is determined as the historical load, and multiple second processing nodes are re-determined from the plurality of processing nodes. The current load of each second processing node is determined based on its memory overflow metric, multiple resource performance parameters, and data transmission duration. Based on the current load of each second processing node and the historical load of the target processing node, a new target processing node is re-determined from the plurality of second processing nodes and the target processing node. Send the identifier of the new target processing node to the second system; After determining the current load of each second processing node based on its memory overflow metric, multiple resource performance parameters, and data transmission duration, the method further includes: If any second processing node does not have the historical load, then the current load of the second processing node is determined as the current SLEWMA value of the second processing node. If any second processing node has the historical load, then the current SLEWMA value of the second processing node is calculated based on the current load of the second processing node, the historical load, and a pre-set glitch variable, wherein the glitch variable is determined based on the historical load. Based on the historical load of the target processing node and the glitch variable, determine the historical SLEWMA value of the target processing node; Accordingly, the step of re-determining a new target processing node from the plurality of second processing nodes and the target processing node based on the current load of each second processing node and the historical load of the target processing node includes: Based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the new target processing node is re-determined from multiple second processing nodes and the target processing node.

2. The method according to claim 1, characterized in that, The step of determining the current load of each first processing node based on its memory overflow index, data transmission duration, and multiple resource performance parameters includes: Based on the memory overflow metric of each first processing node and multiple resource performance parameters, a weight value for each resource performance is determined. The weight value of each resource performance is used to represent the importance of that resource performance to the current load of the first processing node. The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters.

3. The method according to claim 2, characterized in that, The step of determining the weight value of each resource performance based on the memory overflow index of each first processing node and multiple resource performance parameters includes: For each resource performance, multiple initial resource performance parameters are determined based on the resource performance parameter of each first processing node, and each initial resource performance parameter is within the parameter range of the resource performance parameter of the multiple first processing nodes. Based on the memory overflow index of each first processing node and the resource performance parameter, determine the Gini index of each initial resource performance parameter; The weight value of the resource performance is determined based on the minimum value of the Gini index.

4. The method according to claim 3, characterized in that, For each resource performance parameter, based on that resource performance parameter of each first processing node, multiple initial resource performance parameters are determined, including: For each resource performance parameter, the resource performance parameters of the multiple first processing nodes are sorted in descending order to obtain the sorting result. Based on the sorting results, the average value of two adjacent resource performance parameters is determined as the initial resource performance parameter.

5. The method according to claim 1, characterized in that, After determining the new target processing node from multiple second processing nodes and the target processing node based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, the method further includes: Based on the current SLEWMA value and identifier of the new target processing node, the first max heap is updated to generate the second max heap. Accordingly, sending the identifier of the new target processing node to the second system includes: The identifiers of the new target processing nodes in the second large top heap are sent to the second system in a bottom-up order.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining the current load of each first processing node based on the weight value of each resource performance, the data transmission duration of each first processing node, and multiple resource performance parameters includes: The current load of each first processing node is determined based on the weight value of each resource performance, the data transmission duration of each first processing node, the preset weight value, and multiple resource performance parameters.

7. The method according to any one of claims 1 to 4, characterized in that, Each first processing node has multiple resource performance parameters including at least two of the following: the current traffic of the first processing node, CPU utilization, memory utilization, network interface card (NIC) input / output utilization, and disk NIC input / output utilization.

8. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the data transmission duration of the current batch of data transmitted by the second system to each of the plurality of first processing nodes, the method further includes: The system sends to the second system the identifier of each first processing node, the target data type of the current batch of data, the identifier of the target port of the second system, and the identifier of the target process running in the second system.

9. A data transmission device, characterized in that, Applied to a first system, the first system including multiple processing nodes, the device includes: The acquisition module is used to acquire the data transmission duration of the current batch of data transmitted by the second system to each of the plurality of first processing nodes, wherein the plurality of first processing nodes are determined from the plurality of processing nodes; The processing module is used to determine the current load of each first processing node based on the memory overflow index, data transmission duration and multiple resource performance parameters of each first processing node. The memory overflow index of each first processing node is used to indicate whether the memory of the first processing node is overflowing. The processing module is further configured to determine a target processing node from the plurality of first processing nodes based on the current load of each first processing node. The sending module is used to send the identifier of the target processing node to the second system, so that the target processing node can receive the next batch of data transmitted by the second system. After determining the current load of each first processing node based on its memory overflow metric, data transfer duration, and multiple resource performance parameters, the processing module further performs the following: The current load of each first processing node is determined as the current SLEWMA value of that first processing node; The processing module is specifically used for: Based on the current SLEWMA value of each first processing node, a target processing node that meets the preset conditions is determined from multiple first processing nodes. The preset conditions are that the current SLEWMA value of the first processing node is greater than the preset SLEWMA value, or the preset number of first processing nodes with the largest current SLEWMA value. After determining the target processing node that meets the preset conditions from multiple first processing nodes based on the current SLEWMA value of each first processing node, the processing module is also used to put the identifier of the target processing node into an initially empty max heap based on the current SLEWMA value of the target processing node to generate the first max heap. The sending module is specifically used for: The identifiers of the target processing nodes in the first large top heap are sent to the second system in a bottom-up order. After sending the identifier of the target processing node to the second system, the processing module is further configured to determine the current load of each first processing node as the historical load, and to re-determine multiple second processing nodes from multiple processing nodes. The processing module is also used to determine the current load of each second processing node based on the memory overflow index of each second processing node, multiple resource performance parameters, and data transmission duration. The processing module is also used to re-determine a new target processing node from multiple second processing nodes and the target processing node based on the current load of each second processing node and the historical load of the target processing node. The sending module is also used to send the identifier of the new target processing node to the second system; After determining the current load of each second processing node based on its memory overflow metric, multiple resource performance parameters, and data transmission duration, the processing module is further configured to: If any second processing node has no historical load, then the current load of the second processing node is determined as the current SLEWMA value of the second processing node. If any second processing node has historical load, then the current SLEWMA value of the second processing node is calculated based on the current load, historical load and pre-set glitch variable of the second processing node. The glitch variable is determined based on the historical load. Determine the historical SLEWMA value of the target processing node based on its historical load and glitch variables; The processing module is specifically used for: Based on the historical SLEWMA value of the target processing node and the current SLEWMA value of each second processing node, a new target processing node is re-determined from multiple second processing nodes and the target processing node.

10. An electronic device, comprising: A processor, a transceiver, a memory, and computer program instructions stored in the memory and executable on the processor, characterized in that the processor executes the computer program instructions to implement the data transmission method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data transmission method as described in any one of claims 1 to 8.

Images