Method, device, storage medium and program product for transmitting performance data

Abstract

The application provides a performance data transmission method, device, storage medium and program product. The method comprises the following steps: obtaining performance data and network running parameters of a computing node; extracting corresponding data features based on the performance data; the data features comprise data structure features, data block capacity features and data information features; determining a compression algorithm of the performance data based on the data features and a preset compression strategy table; performing compression processing on the performance data based on the compression algorithm to obtain a compressed data file; determining a dynamic block threshold of the compressed data file based on the network running parameters; performing block processing on the compressed data file based on the dynamic block threshold to obtain a unit to be transmitted; the unit to be transmitted comprises a data block serial number; and transmitting the unit to be transmitted to a center service node based on the data block serial number. The application flexibly adjusts the transmission strategy according to the data features and the link state, and improves the timeliness and reliability of performance data transmission.

Description

Technical Field

[0001] This application relates to the field of computer network communication, and in particular to a method, device, storage medium and program product for transmitting performance data. Background Technology

[0002] With the explosive growth in demand for artificial intelligence, high-performance computing, and large-scale data processing, GPU (Graphics Processing Unit) clusters have become the core hardware architecture supporting complex computing tasks.

[0003] In GPU clusters, performance data aggregation across nodes typically employs shared-memory batch processing or message-based streaming. However, existing technologies are prone to issues such as high transmission latency, message backlog, low bandwidth utilization, and difficulty in ensuring data integrity under packet loss jitter when dealing with scenarios involving large volumes of performance data, high burstiness, and cross-node temporal correlations. Furthermore, resource contention intensifies when multiple nodes are concurrent.

[0004] Therefore, there is an urgent need for a performance data transmission solution that can adapt to the characteristics of GPU clusters and balance real-time performance with reliability, so as to support the efficient operation of large-scale distributed computing. Summary of the Invention

[0005] This application provides a method, device, storage medium, and program product for transmitting performance data, in order to improve the real-time performance and reliability of GPU cluster data transmission.

[0006] In a first aspect, embodiments of this application provide a method for transmitting performance data, including:

[0007] Obtain performance data and network operating parameters of compute nodes;

[0008] Based on performance data, corresponding data features are extracted; data features include data structure features, data block capacity features, and data information features.

[0009] Based on data characteristics and preset parameters, determine the compression algorithm for performance data;

[0010] Based on the compression algorithm, the performance data is compressed to obtain a compressed data file;

[0011] Based on network operating parameters, determine the dynamic block threshold for compressed data files;

[0012] Based on a dynamic block threshold, the compressed data file is divided into blocks to obtain units to be transmitted; each unit to be transmitted includes a data block number.

[0013] Based on the data block sequence number, the unit to be transmitted is transmitted to the central service node.

[0014] In this scheme, performance data and network operating parameters of computing nodes are acquired, and data structure features, data block capacity features, and data information features are extracted based on the performance data. Combined with a preset compression strategy table, an appropriate compression algorithm is determined, which effectively improves the targeting of performance data compression processing and reduces the amount of data in the unit to be transmitted. Based on the network operating parameters, a dynamic block threshold for compressed data files is determined, and the compressed data files are block-processed according to the dynamic block threshold. Combined with the data block sequence number, the unit to be transmitted is transmitted to the central service node, which enhances the orderliness and integrity of data transmission in scenarios with packet loss and jitter, thereby realizing the real-time and reliable cross-node transmission of performance data in the GPU cluster.

[0015] In some embodiments, transmitting the unit to be transmitted to the central service node based on the data block sequence number includes:

[0016] A connection health scoring model is constructed based on network operating parameters, including network bandwidth utilization, network latency, and packet loss rate.

[0017] Based on the connection health scoring model, calculate the connection health score of the computing node;

[0018] Based on the connection health score, the transmission strategy for the unit to be transmitted is determined;

[0019] Based on the transmission strategy, the unit to be transmitted is transmitted to the central service node.

[0020] In this scheme, network operation parameters, including network bandwidth utilization, network latency, and packet loss rate, are introduced into the connection health scoring model to quantitatively characterize the link status and jointly reflect the availability and stability of the current link. This allows the unit to be transmitted to be sent to the central service node using appropriate transmission strategies under different connection states, thereby improving the transmission flexibility of the unit to be transmitted.

[0021] In some embodiments, the transmission strategy of the unit to be transmitted is determined based on the connection health score, including:

[0022] The heartbeat interval of the computing node is determined based on the connection health score;

[0023] A heartbeat connection is established between the computing node and the central service node based on the heartbeat interval.

[0024] In this scheme, the heartbeat interval of computing nodes is determined by the connection health score, and a heartbeat connection between computing nodes and the central service node is established based on the heartbeat interval. This ensures that the heartbeat connection is consistent with the current link status, thereby enabling the transmission strategy to reflect network fluctuations in a timely manner, maintaining the effectiveness of communication between nodes, providing a stable connection foundation for subsequent transmission, and reducing the resource consumption caused by fixed-frequency heartbeats.

[0025] In some embodiments, the transmission strategy of the unit to be transmitted is determined based on the connection health score, including:

[0026] Based on the connection health score, determine whether there is a connection failure between the computing node and the central service node;

[0027] If a connection failure occurs, a reconnection operation is performed, and the breakpoint identifier is identified; the breakpoint identifier is the sequence number of the data block that has been successfully transmitted.

[0028] Based on the breakpoint identifier, the unit to be transmitted is incrementally transmitted to the central server.

[0029] In this scheme, link connection failures are dynamically determined by connection health scores, and only the units to be transmitted after the breakpoint are sent after reconnection. This reduces the number of duplicate transmission blocks, maintains the continuity of data block sequence numbers, and enables the central server to resume receiving the remaining data in order. This improves the continuity and consistency of the transmission process after the link connection failure.

[0030] In some embodiments, performing a reconnection operation includes:

[0031] Adjust the exponential backoff step size based on network operating parameters;

[0032] Based on the exponential backoff step size, the backoff reconnection strategy is executed until the computing node and the central service node re-establish a heartbeat connection, or the maximum number of reconnections is reached.

[0033] In this scheme, by dynamically adjusting the exponential backoff step size based on network operating parameters and executing the backoff reconnection strategy, the reconnection interval can be synchronized with the link status, reducing the number of invalid retries under weak network conditions and matching the connection recovery process with the current network carrying capacity. This improves the recovery stability of the heartbeat connection between the computing node and the central service node and reduces the occupation of cluster communication resources during the reconnection process.

[0034] In some embodiments, determining the dynamic block threshold for compressed data files based on network operating parameters includes:

[0035] The network bandwidth utilization rate is compared with the preset network threshold.

[0036] If the network bandwidth utilization exceeds the preset network threshold, then increase the dynamic block threshold for compressed data files.

[0037] If the network bandwidth utilization is not greater than the preset network threshold, then reduce the dynamic block threshold of the compressed data file.

[0038] In this scheme, the dynamic segmentation threshold is dynamically adjusted by using network bandwidth utilization and preset network thresholds, so that the segmentation granularity changes with the link status. This results in the data size of a single block increasing when the network load is high and decreasing when the network load is low, thereby matching the generation of the unit to be transmitted and the subsequent transmission process with the current link status.

[0039] In some embodiments, after transmitting the unit to be transmitted to the central service node based on the data block sequence number, the method further includes:

[0040] Based on the data block sequence number, the data of the unit to be transmitted is reassembled to obtain the corresponding data file;

[0041] Based on the characteristics of data information, generate the lifecycle of data files;

[0042] If the current cycle reaches the lifecycle of the data file, then clean up the data file.

[0043] In this scheme, the complete data file is restored by reassembling the unit to be transmitted based on the data block sequence number, and the life cycle is determined by combining the data information characteristics so that different data files can obtain matching storage durations. After the current cycle reaches the life cycle, cleanup is performed, which can make the storage space of the central service node reclaimed in a timely manner according to the arrival of data, so as to maintain the consistency between the reassembled files and the management data.

[0044] Secondly, embodiments of this application provide a performance data transmission apparatus, comprising:

[0045] The acquisition module is used to acquire performance data and network operating parameters of compute nodes;

[0046] The extraction module is used to extract corresponding data features based on performance data; data features include data structure features, data block capacity features, and data information features.

[0047] The first determining module is used to determine the compression algorithm for performance data based on data characteristics and a preset compression strategy table.

[0048] The first module is used to compress performance data based on a compression algorithm to obtain a compressed data file;

[0049] The second determining module is used to determine the dynamic block threshold of the compressed data file based on network operating parameters;

[0050] The second module is used to divide the compressed data file into blocks based on a dynamic block threshold to obtain the unit to be transmitted; the unit to be transmitted includes the data block number.

[0051] The transmission module is used to transmit the unit to be transmitted to the central service node based on the data block sequence number.

[0052] In some embodiments, the transmission module is specifically used for:

[0053] A connection health scoring model is constructed based on network operating parameters, including network bandwidth utilization, network latency, and packet loss rate.

[0054] Based on the connection health scoring model, calculate the connection health score of the computing node;

[0055] Based on the connection health score, the transmission strategy for the unit to be transmitted is determined;

[0056] Based on the transmission strategy, the unit to be transmitted is transmitted to the central service node.

[0057] In some embodiments, the transmission module is specifically used for:

[0058] The heartbeat interval of the computing node is determined based on the connection health score;

[0059] A heartbeat connection is established between the computing node and the central service node based on the heartbeat interval.

[0060] In some embodiments, the transmission module is specifically used for:

[0061] Based on the connection health score, determine whether there is a connection failure between the computing node and the central service node;

[0062] If a connection failure occurs, a reconnection operation is performed, and the breakpoint identifier is identified; the breakpoint identifier is the sequence number of the data block that has been successfully transmitted.

[0063] Based on the breakpoint identifier, the unit to be transmitted is incrementally transmitted to the central server.

[0064] In some embodiments, the transmission module is specifically used for:

[0065] Adjust the exponential backoff step size based on network operating parameters;

[0066] Based on the exponential backoff step size, the backoff reconnection strategy is executed until the computing node and the central service node re-establish a heartbeat connection, or the maximum number of reconnections is reached.

[0067] In some embodiments, the second determining module is specifically used for:

[0068] The network bandwidth utilization rate is compared with the preset network threshold.

[0069] If the network bandwidth utilization exceeds the preset network threshold, then increase the dynamic block threshold for compressed data files.

[0070] If the network bandwidth utilization is not greater than the preset network threshold, then reduce the dynamic block threshold of the compressed data file.

[0071] In some embodiments, after transmitting the unit to be transmitted to the central service node based on the data block sequence number, a generation module is further included, which is specifically used for:

[0072] Based on the data block sequence number, the data of the unit to be transmitted is reassembled to obtain the corresponding data file;

[0073] Based on the characteristics of data information, generate the lifecycle of data files;

[0074] If the current cycle reaches the lifecycle of the data file, then clean up the data file.

[0075] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0076] The memory stores instructions that the computer executes;

[0077] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0078] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0079] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0080] The performance data transmission method, device, storage medium, and program product provided in this application acquire the performance data and network operating parameters of computing nodes, and extract data structure features, data block capacity features, and data information features based on the performance data. Combined with a preset compression strategy table, an appropriate compression algorithm is determined, effectively improving the targeting of performance data compression processing and reducing the amount of data to be transmitted. Based on network operating parameters, a dynamic block threshold for compressed data files is determined, and the compressed data files are divided into blocks according to the dynamic block threshold. Combined with the data block sequence number, the units to be transmitted are transmitted to the central service node, enhancing the orderliness and integrity of data transmission in scenarios of packet loss and jitter, thereby achieving real-time and reliable cross-node transmission of performance data in GPU clusters. Attached Figure Description

[0081] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0082] Figure 1 This application provides a diagram of a performance data transmission system architecture.

[0083] Figure 2 Flowchart of the performance data transmission method provided in this application Figure 1 ;

[0084] Figure 3 Flowchart of the performance data transmission method provided in this application Figure 2 ;

[0085] Figure 4 A schematic diagram of the structure of the performance data transmission device provided in this application;

[0086] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0087] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0088] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0089] It should be noted that all data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0090] In a GPU cluster scenario consisting of multiple computing nodes, each computing node continuously generates kernel execution records, interface call trajectories, and hardware count information when performing model training, scientific computing, or heterogeneous acceleration tasks. These records need to be aggregated across nodes to a central service node for unified analysis.

[0091] In existing technologies, performance data is typically written to shared storage by each computing node and then read in batches by the central side, or sent to the central side as streaming messages via message brokers. However, in GPU clusters, performance data is often characterized by large volume, high burstiness, significant structural differences, and complex temporal relationships. Existing solutions are prone to problems such as file batch processing lag, streaming message backlog, insufficient utilization of link bandwidth, and excessively high retransmission costs under network jitter during actual operation. Especially when multiple nodes concurrently report performance data, if fixed compression and fixed block methods are still used, it is often difficult to balance transmission latency, data integrity, and resource overhead control, thus affecting the central side's timely analysis and judgment of the cluster's operating status.

[0092] In view of this, how to dynamically adjust the transmission method in a GPU cluster based on the characteristics of the performance data and the status of the link network to balance real-time performance, reliability and transmission efficiency has become an urgent technical problem to be solved.

[0093] To address the aforementioned issues, the technical concept of this application is as follows: After acquiring the performance data and network operating parameters of the computing nodes, corresponding data structure features, data block capacity features, and data information features are extracted based on the performance data. A compression algorithm is determined using a preset compression strategy table, and the performance data is compressed. Furthermore, based on the network operating parameters, a dynamic block threshold for the compressed data file is determined, thereby completing the block processing of the compressed data file and generating a transmission unit containing a data block sequence number. Based on the data block sequence number, the transmission unit is transmitted to the central service node. This enhances the orderliness and integrity of data transmission in scenarios involving packet loss and jitter, thereby achieving real-time and reliable cross-node transmission of performance data in the GPU cluster.

[0094] Optionally, Figure 1 This is a schematic diagram of a performance data transmission system architecture provided in this application. Figure 1 As shown, the performance data transmission system architecture includes multiple computing nodes 101, message transmission middleware 102, and central service node 103.

[0095] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the above architecture. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0096] In the specific implementation process, the computing node 101 is a node in the GPU cluster, used to collect, process and send the performance data in the corresponding computing node 101; the computing node 101 includes a data acquisition unit, a heartbeat unit, a compression unit and a block division unit; among them, the data acquisition unit is used to collect performance data, the heartbeat unit is used to establish a heartbeat connection with the central service node 103, the compression unit is used to compress the performance data, and the block division unit is used to divide the compressed performance data into blocks.

[0097] The message transmission middleware 102 is a lightweight message transmission middleware used for asynchronous transmission of performance data and concurrent communication.

[0098] The central service node 101 is used to receive, process, display and store the performance data of all computing nodes 103; wherein, the central service node 101 includes a heartbeat unit, a data reconstruction unit and a decompression unit; wherein, the heartbeat unit is used to establish a heartbeat connection with the computing node 101, the data reconstruction unit is used to reconstruct the data to be transmitted, and the decompression unit is used to decompress the compressed performance data.

[0099] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0100] Figure 2 Flowchart of the performance data transmission method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0101] S201. Obtain the performance data and network operating parameters of the computing nodes.

[0102] Performance data refers to the service data to be transmitted generated by computing nodes during the execution of model training, scientific computing, or heterogeneous acceleration tasks.

[0103] Network operating parameters are a set of parameters that characterize the state of the transmission link between the current computing node and the central service node. Network operating parameters include network bandwidth utilization, network latency, and packet loss rate.

[0104] In the specific implementation process, the client is deployed on each GPU computing node, and the performance data of the computing node is collected in event tracing mode by calling the GPU performance profiling tool.

[0105] The performance data includes kernel execution data, hardware performance event data, and interface operation trajectory data.

[0106] Kernel execution data consists of startup, execution, and completion events for each GPU kernel. Kernel execution data includes globally unique identifiers, grid or thread block dimensions, and timestamps.

[0107] Hardware performance event data refers to runtime low-level hardware events statistically analyzed through hardware performance counters. For example, low-level hardware events include clock cycle events, storage / cache access events, and memory bus events.

[0108] The interface execution trajectory data is the call sequence of the interface, including the start and end times of the interface call and the call parameters; for example, the interface execution trajectory data can be the start and end times of memory copy and synchronization operations and the corresponding call parameters.

[0109] In some embodiments, to ensure consistency in subsequent processing, when acquiring performance data, a batch identifier is first generated according to the acquisition batch, and the start time, end time, node number, and task number are recorded, so that the performance data within the same batch maintains a corresponding relationship in subsequent processing.

[0110] Furthermore, after acquiring the performance data, preprocessing operations are performed on the performance data to reduce the volume of the original performance data and decrease transmission bandwidth and compression performance consumption. The preprocessing operations include null value extraction, duplicate trajectory merging, invalid event filtering, and timestamp normalization. Based on the preprocessed performance data, files in corresponding formats are generated. For example, a plain text table format (with the .csv extension) file is generated based on kernel execution data; a lightweight text data exchange format file (with the .json extension) is generated based on hardware performance event data; and a database file (with the .db extension) is generated based on interface running trajectory data.

[0111] When acquiring network operating parameters, the sampling period of the network operating parameters is aligned with the batch of performance data collection, so that each batch of performance data corresponds to a set of time-sensitive network operating parameters; when the performance data suddenly increases in a short period of time, an instant network status update is triggered in addition to the original periodic sampling to update the current network operating parameters.

[0112] Furthermore, to prevent a single measurement from causing a shift in subsequent threshold judgments, smoothing processing can be performed on the network operating parameters of multiple consecutive sampling windows.

[0113] It should be understood that the above examples are for illustrative purposes only and are not intended to be limiting.

[0114] S202. Based on performance data, extract corresponding data features; data features include data structure features, data block capacity features, and data information features.

[0115] Data features are a set of features extracted from performance data to describe the data organization, volume distribution, and content attributes.

[0116] Among them, data structure features are used to characterize the field distribution, repetition, and degree of structure of performance data.

[0117] The data block capacity feature is used to characterize the overall size of performance data within the current batch.

[0118] The data information features include real-time performance data and historical performance data.

[0119] S203. Based on data characteristics and a preset compression strategy table, determine the compression algorithm for performance data.

[0120] The compression strategy table is a mapping table between compression algorithms and data features. Each strategy record in the compression strategy table may include structural feature conditions, capacity feature range, information feature threshold, corresponding compression algorithm identifier, and algorithm execution parameters.

[0121] For example, the compression algorithm includes one or a combination of character-based compression, entropy-coded compression, block compression, or time-series-oriented differential compression.

[0122] In the specific implementation process, the data structure characteristics, data block capacity characteristics, and data information characteristics obtained in step S202 are input to retrieve the compression strategy table.

[0123] For example, when the data structure characteristics indicate that the current batch consists of a large number of fixed-format records, the data information characteristics indicate that the proportion of repeated fields and repeated strings is high, and the data block capacity characteristics indicate that the overall size of the current batch exceeds the set compression starting capacity, a character-based compression or LZ (Lempel-Ziv) type compression algorithm can be matched from the compression strategy table; when the data content is mainly continuous count values ​​and time difference sequences, and the numerical fluctuation range is small, a combination of differential coding and entropy coding algorithms can be matched; when the performance data consists of a mixture of multiple types of records and has weak structural regularity, a general block compression algorithm can be matched.

[0124] By mapping the data characteristics of performance data to compression algorithms, a compression algorithm selection result is formed for the current batch of performance data. This allows the compression process to simultaneously consider data compressibility and node-end processing overhead, thereby reducing unnecessary data volume in subsequent transmission stages.

[0125] In some embodiments, the compression strategy table can also adopt a multi-level conditional filtering method. First, candidate compression algorithms are filtered according to data structure characteristics. Then, algorithms that do not provide sufficient compression benefits or have excessive computational costs under the current data scale are eliminated according to data block capacity characteristics. Finally, the final compression algorithm and corresponding algorithm execution parameters are determined according to data information characteristics.

[0126] In some embodiments, the compression function can be dynamically switched on and off based on the network bandwidth utilization rate; for example, if the network bandwidth utilization rate is higher than a preset network bandwidth utilization rate threshold, the compression function is enabled, and if the network bandwidth utilization rate is not higher than the preset network bandwidth utilization rate threshold, the compression function is disabled.

[0127] It should be understood that the above examples are for illustrative purposes only and are not intended to be limiting.

[0128] S204. Based on the compression algorithm, the performance data is compressed to obtain a compressed data file.

[0129] The compressed data file is the result file after performance data has been compressed using the corresponding compression algorithm.

[0130] In the specific implementation process, based on the compression algorithm and corresponding algorithm execution parameters determined in step S203, the current batch of performance data undergoes encoding, dictionary construction, differential conversion, block compression, or entropy encoding processing. If the performance data consists of multiple subtypes, it is first compressed according to record category, and then each compressed segment and segment description information are encapsulated into a unified compressed data file; if the compression strategy table returns a single general compression algorithm, the entire batch of data is directly compressed sequentially.

[0131] Before compression, file header metadata is generated, recording the computing node identifier, batch identifier, node number, original data size, compression algorithm identifier, compression parameter summary, original record count, and verification information. The file header and the compressed payload together form the compressed data file, so that the central service node can identify the compression method and complete decompression and verification after receiving it.

[0132] By converting performance data into compressed data files that are easy to transmit over the network, the number of bytes transmitted in subsequent transmissions is reduced, and by carrying algorithm identifiers and verification information, the decodability and integrity verification capabilities of the central service node are maintained.

[0133] It should be understood that the above examples are for illustrative purposes only and are not intended to be limiting.

[0134] S205. Determine the dynamic block threshold for compressed data files based on network operating parameters.

[0135] The dynamic chunking threshold is the size threshold used when processing compressed data files into chunks before sending them. The dynamic chunking threshold is adjusted based on changes in current network operating parameters.

[0136] In some embodiments, determining the dynamic segmentation threshold of the compressed data file based on network operating parameters includes: comparing the network bandwidth utilization rate with a preset network threshold; if the network bandwidth utilization rate is greater than the preset network threshold, increasing the dynamic segmentation threshold of the compressed data file; if the network bandwidth utilization rate is not greater than the preset network threshold, decreasing the dynamic segmentation threshold of the compressed data file.

[0137] Network bandwidth utilization is used to characterize the extent to which computing nodes are currently using network link bandwidth.

[0138] The preset network threshold is the bandwidth utilization threshold pre-written in the configuration file, which is set according to the cluster link capacity, node concurrent reporting scale, and historical transmission load.

[0139] In the specific implementation process, the network bandwidth utilization rate is compared with the preset network threshold. When it is determined that the current network bandwidth utilization rate is greater than the preset network threshold and the network latency is less than the preset network latency threshold, the dynamic segmentation threshold is automatically increased to reduce the number of segments and the overall protocol header overhead, thereby improving throughput. When it is determined that the current network bandwidth utilization rate is not greater than the preset network threshold and the network latency is not less than the preset network latency threshold, the dynamic segmentation threshold is automatically decreased to divide the compressed data file into smaller data blocks, thereby reducing the overhead and risk caused by retransmission of a single data block.

[0140] In some embodiments, the central service node can also add a concurrent scheduling module to monitor the network bandwidth, performance data capacity, and task priority of all online clients in real time, and dynamically allocate transmission time priority and network bandwidth quotas. Prioritize network bandwidth allocation and increase the dynamic block threshold for computing nodes executing critical GPU training tasks, while limiting the transmission rate of idle computing nodes. This avoids the impact of concurrent performance data surges from multiple computing nodes, which could lead to link congestion, false heartbeats, and data reassembly failures, thereby improving the overall transmission stability of the multi-cluster computing nodes.

[0141] S206. Based on the dynamic block threshold, the compressed data file is divided into blocks to obtain the unit to be transmitted; the unit to be transmitted includes the data block number.

[0142] The unit to be transmitted is the actual basic object to be sent after the compressed data file is divided according to the dynamic block threshold.

[0143] The data block number is a number that identifies the sequential position of each block in the entire compressed data file, and is used by the receiving end for identification, sorting, and data reassembly.

[0144] In the specific implementation process, the compressed data file is read, and sequential segmentation is performed according to the dynamic segmentation threshold determined in step S205. If the size of the compressed data file is less than or equal to the dynamic segmentation threshold, a single unit to be transmitted is generated from the compressed data file; if the size of the compressed data file is greater than the dynamic segmentation threshold, data payloads are sequentially extracted from the beginning of the compressed data file according to the length of the dynamic segmentation threshold until the end of the compressed data file. For the remaining data at the end of the compressed data file that is less than a complete dynamic segmentation threshold, a last unit to be transmitted is formed separately. The unit to be transmitted includes a block header and block data. The block header includes a globally unique transmission identifier, the total number of data blocks, the current data block sequence number, and the data block length; the block data is the block data obtained by segmenting the compressed data file.

[0145] In some embodiments, the data block sequence numbers are encoded using a continuously increasing sequence starting from zero or one, with each compressed data file corresponding to a unique total number of data blocks. After generating all units to be transmitted, a local transmission index table is established, recording the correspondence between each data block sequence number and its corresponding file offset, length, and transmission status. If the dynamic segmentation threshold is updated after the compressed data file is segmented, the remaining unsegmented portion of the compressed data file is further segmented using the updated dynamic segmentation threshold, while maintaining the continuity of data block sequence numbers in the index table. To support integrity verification at the receiving end, each unit to be transmitted can also carry a hash value or a cyclic redundancy check value in its block header to verify whether errors have occurred during the transmission of the segmented data.

[0146] By carrying data block sequence numbers in the units to be transmitted, the central service node can restore the order of compressed data files in the event of out-of-order arrival or partial retransmission; at the same time, the block granularity formed by the dynamic block threshold can match the current network state, so that the number of units to be transmitted and the capacity of units to be transmitted can be balanced between transmission efficiency and retransmission cost.

[0147] S207. Based on the data block sequence number, transmit the unit to be transmitted to the central service node.

[0148] The central service node is the aggregation receiver in the GPU cluster, used to receive data units to be transmitted from multiple computing nodes and to complete data reassembly, decompression and data storage.

[0149] In the specific implementation, a transmission channel is established between the computing node and the central service node. During the transmission phase, the units to be transmitted are transmitted in the order of data block sequence numbers. This allows the central service node to return acknowledgment information upon receipt, including the set of successfully received data block sequence numbers. The local transmission status is then updated based on the acknowledgment information.

[0150] Since each unit to be transmitted has a specific data block sequence number, the central service node can still classify and cache the data blocks according to their sequence numbers even when multiple nodes receive data concurrently or when the links are out of order. After receiving all the data blocks in the corresponding batch, the nodes can reassemble them in order to obtain the original compressed data file.

[0151] In some embodiments, the central service node maintains reception status information for each transmission batch, recording the sequence number of the arrived data blocks and the reception progress of the corresponding batch. After reassembly, the central service node performs corresponding decompression processing based on the compression algorithm identifier in the compressed data file header and verifies integrity based on file-level checksums. By driving the transmission and reception process with data block sequence numbers, the transmission process has a clear order recognition capability, enabling the stable aggregation of large-scale compressed performance data to the central service node when multiple nodes in the GPU cluster simultaneously report performance data.

[0152] Based on the aforementioned embodiments, after transmitting the unit to be transmitted to the central service node based on the data block sequence number, the method further includes: reassembling the data of the unit to be transmitted based on the data block sequence number to obtain the corresponding data file; generating the lifecycle of the data file based on the data information characteristics; and cleaning up the data file if the current period reaches the lifecycle of the data file.

[0153] Data reassembly is the process of piecing together multiple blocks in a predetermined order to restore a complete data file based on the sequence number of the data block carried by the unit to be transmitted.

[0154] Data files are used to carry the performance data that has been transmitted and serve as the object of subsequent lifecycle management and cleanup control.

[0155] The lifecycle is the retention period of a data file in the central service node, and the lifecycle can be generated based on the characteristics of the data information.

[0156] In the specific implementation process, after receiving each unit to be transmitted, the central service node reads the data block sequence number and caches the units to be transmitted with the same file identifier and consecutive data block sequence numbers in the corresponding reassembly queue. The units are then concatenated in ascending order of data block sequence number to form a data file corresponding to the original compressed data file.

[0157] If the unit to be transmitted corresponding to one of the sequence numbers has not yet arrived, the central service node remains in the state of waiting for completion and continues to wait for subsequent replacement units within the reception timeout threshold to avoid missing blocks in the reassembly result.

[0158] After the reorganization is completed, the central service node parses the data information characteristics in the data file. The data information characteristics also include data content type, data density, collection object identifier and information timeliness level, and maps the data information characteristics to a preset retention period as the life cycle of the data file.

[0159] When the current cycle reaches the lifecycle setting value, the central service node triggers a cleanup command to delete the corresponding data file, unassociate the index, or reclaim the storage space.

[0160] By reassembling the units to be transmitted based on data block sequence numbers, complete data files can be recovered. Furthermore, by combining data information characteristics to determine the retention period, data files with different information attributes can be assigned matching storage durations. Cleanup is performed after the current period reaches its lifespan, ensuring that the central service node's storage space is promptly reclaimed as data arrives, and maintaining consistency between subsequently reassembled files and managed data.

[0161] This application provides a method for transmitting performance data. The method involves acquiring performance data and network operating parameters of computing nodes; extracting corresponding data features based on the performance data, including data structure features, data block capacity features, and data information features; determining a compression algorithm for the performance data based on the data features and a preset compression strategy table; compressing the performance data using the compression algorithm to obtain a compressed data file; determining a dynamic block threshold for the compressed data file based on the network operating parameters; dividing the compressed data file into blocks based on the dynamic block threshold to obtain transmission units, each including a data block number; and transmitting the transmission units to a central service node based on the data block number. In this method, by characterizing the structure, capacity, and content attributes of the performance data, selecting a compression algorithm matching the current performance data using a compression strategy table, determining a dynamic block threshold based on network operating parameters, processing the performance data according to the compression algorithm and the dynamic block threshold, generating transmission units carrying data block numbers, and sending them to the central service node, the cross-node transmission of GPU cluster performance data no longer relies on fixed compression and block methods. In scenarios involving sudden increases in performance data, significant structural differences, and dynamic changes in network status, the compression method and block granularity of a single batch of performance data can be dynamically adjusted according to current conditions, thereby achieving coordination between real-time performance, integrity, and resource overhead during the transmission of performance data.

[0162] Figure 3 Flowchart of the performance data transmission method provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the step S207 above, "transmitting the unit to be transmitted to the central service node based on the data block sequence number," is described in detail. The method includes:

[0163] S301. Construct a connection health scoring model based on network operating parameters; network operating parameters include network bandwidth utilization, network latency, and packet loss rate.

[0164] The connection health rating model is an evaluation model used to characterize the connection status between computing nodes and central service nodes. The inputs are network bandwidth utilization, network latency, and packet loss rate, and the output is the connection health rating.

[0165] Network latency is used to characterize the round-trip time of a message.

[0166] Packet loss rate is used to characterize the reliability of link transmission.

[0167] Network bandwidth utilization, network latency, and packet loss rate collectively reflect the availability and stability of the current link.

[0168] S302. Calculate the connection health score of the computing node based on the connection health score model.

[0169] The connection health score is used to characterize the reliability of the connection link between the current computing node and the central service node. The connection health score can be set to a continuous value or a graded value so that subsequent policy matching can be performed based on the threshold range.

[0170] For example, the connection health scoring model is shown in the following formula:

[0171]

[0172] In the formula, To connect to health scores, For network bandwidth utilization, Due to network latency, For packet loss rate, , and These are the calculated weights corresponding to each network operating parameter.

[0173] It should be understood that the above examples are for illustrative purposes only and are not intended to be limiting.

[0174] S303. Based on the connection health score, determine the transmission strategy for the unit to be transmitted.

[0175] In some embodiments, the transmission strategy of the unit to be transmitted is determined based on the connection health score, including: determining the heartbeat interval of the computing node based on the connection health score; and establishing a heartbeat connection between the computing node and the central service node based on the heartbeat interval.

[0176] In the specific implementation process, after receiving the connection health score, the computing node can compare the connection health score with the preset score threshold to obtain the corresponding heartbeat interval parameter. If the connection health score is greater than the preset score threshold, the heartbeat interval of the computing node is extended; if the connection health score is not greater than the preset score threshold, the heartbeat interval of the computing node is shortened.

[0177] The compute node initiates a timed sending mechanism based on the determined heartbeat interval, sends heartbeat packets to the central service node at the heartbeat interval, and receives ACK (Acknowledgment) response packets returned by the central service node; the heartbeat packet includes the client IP (Internet Protocol) address deployed on the compute node.

[0178] By continuously feeding back the link status through heartbeat connections, the central service node can confirm the online status of the computing nodes. By combining the heartbeat interval with the connection health score, the heartbeat interval can adaptively change with the connection health score, thereby enabling the unit to be transmitted to maintain the corresponding transmission activity and connection guarantee under different network conditions. This provides a stable connection foundation for subsequent transmission, and allows the transmission strategy to reflect network fluctuations in a timely manner and maintain the effectiveness of communication between nodes, while reducing the resource consumption caused by fixed-frequency heartbeats.

[0179] In some embodiments, the transmission strategy for the unit to be transmitted is determined based on the connection health score, including: determining whether there is a connection failure between the computing node and the central service node based on the connection health score; if there is a connection failure, performing a reconnection operation and identifying the breakpoint identifier; the breakpoint identifier is the sequence number of the data block that has been successfully transmitted; and incrementally transmitting the unit to be transmitted to the central server based on the breakpoint identifier.

[0180] Connection failure is used to indicate situations where the current communication link is interrupted, times out, or experiences continuous abnormal responses, serving as a basis for triggering reconnection and resume transmission.

[0181] The reconnection operation is used to re-establish the communication connection between the computing node and the central service node after a connection failure is detected.

[0182] Breakpoint markers are used to record the sequence numbers of data blocks that have been successfully sent before reconnection, so that the transmission of unfinished parts can continue after the connection is restored.

[0183] In the specific implementation process, when the connection health score is lower than the preset health threshold, it is determined that there is a connection failure between the current computing node and the central service node. The subsequent transmission of units to be transmitted is stopped, and a reconnection request is initiated based on the transmission record. After receiving the connection restoration request, the central service node can return the sequence number of the most recently successfully received data block. The computing node identifies the sequence number of the most recently successfully received data block as the breakpoint identifier, and continues to send the untransmitted units to be transmitted after the breakpoint identifier as incremental transmission content to the central server.

[0184] The computing nodes dynamically determine link anomalies based on the connection health score, and after reconnection, only send the units to be transmitted after the breakpoint, reducing the number of duplicate transmissions, maintaining the continuity of data block sequence numbers, and enabling the central server to resume receiving the remaining data in order, thereby improving the continuity and consistency of the units to be transmitted during the recovery transmission process after the link anomaly.

[0185] In some embodiments, performing a reconnection operation includes: adjusting the exponential backoff step size based on network operating parameters; and executing a backoff reconnection strategy based on the exponential backoff step size until the compute node and the central service node re-establish a heartbeat connection, or the maximum number of reconnections is reached.

[0186] The exponential backoff step size is a time parameter used to control the increment of the reconnection interval. The exponential backoff step size can be adjusted based on a comprehensive assessment of network bandwidth utilization, network latency, and packet loss rate.

[0187] The maximum number of reconnections is used to limit the upper limit of reconnection retries.

[0188] For example, when network operating parameters indicate increased link jitter, higher network latency, or increased packet loss, the exponential backoff step size is increased to prolong the waiting time between two reconnection requests; when network operating parameters indicate that the link has stabilized, the exponential backoff step size is decreased to shorten the reconnection waiting time. The maximum number of reconnections can be 3.

[0189] Upon a connection failure, the compute node updates the exponential backoff step size based on the current network operating parameters and generates a backoff reconnection interval according to the exponential backoff step size. It then initiates a heartbeat reconnection request to the central service node according to the backoff reconnection interval. Each heartbeat reconnection request carries the compute node identifier, the timestamp of the last heartbeat, and information about the connection session to be restored. Upon receiving the heartbeat reconnection request, the central service node completes session verification and state restoration. When the compute node and the central service node re-establish a heartbeat connection, the connection status is set to normal, and subsequent reconnections are terminated. When the number of consecutive retries reaches the maximum reconnection limit, the node stops initiating requests, goes offline, and is removed from the central service node.

[0190] By dynamically adjusting the exponential backoff step size based on network operating parameters and implementing a backoff reconnection strategy, the reconnection interval can be synchronized with the link status, reducing the number of invalid retries under weak network conditions and matching the connection recovery process with the current network carrying capacity. This improves the recovery stability of the heartbeat connection between the computing node and the central service node and reduces the consumption of cluster communication resources during the reconnection process.

[0191] S304. Based on the transmission strategy, the unit to be transmitted is transmitted to the central service node.

[0192] After carrying the data block sequence number, the unit to be transmitted is sent to the central service node according to the determined transmission strategy, so that the central service node can identify, sort, reassemble and decompress the received unit to be transmitted based on the data block sequence number. The transmission strategy includes heartbeat connection strategy and breakpoint retransmission strategy.

[0193] In some embodiments, abnormal events can also be recorded, associated with corresponding network operating parameters, node load, and data types, and fault reports can be automatically generated. Abnormal events include heartbeat timeouts, reconnection failures, compression failures, chunk loss, and retransmission failures. Simultaneously, the central service node adds visualization capabilities, displaying the connection status, transmission rate, compression efficiency, and health score of each computing node in real time, and automatically triggering alarms when abnormal events occur. This enables full-process monitoring and maintenance of GPU performance data transmission, filling the gap in traditional transmission solutions that lack operational monitoring capabilities.

[0194] This application provides a method for transmitting performance data. By incorporating network bandwidth utilization, network latency, and packet loss rate into a connection health scoring model, it can quantitatively characterize the link status and dynamically select transmission strategies. This allows the unit to be transmitted to be sent to the central service node using appropriate transmission strategies under different connection states. The central service node can then continuously receive the units to be transmitted and identify, sort, reassemble, and decompress them. By dynamically adjusting the heartbeat interval, the overhead of sending invalid heartbeat packets is reduced, avoiding excessive bandwidth consumption in high-load scenarios. Simultaneously, in network fluctuation scenarios, the real-time performance of connection fault detection is improved, ensuring that link anomalies can be quickly detected and recovery mechanisms triggered, thereby optimizing overall link stability and resource utilization.

[0195] Figure 4 A schematic diagram of the structure of the performance data transmission device provided in this application is shown below. Figure 4 As shown, the performance data transmission device provided in this embodiment includes:

[0196] The acquisition module 401 is used to acquire the performance data and network operating parameters of the computing nodes.

[0197] The extraction module 402 is used to extract corresponding data features based on performance data; the data features include data structure features, data block capacity features, and data information features.

[0198] The first determining module 403 is used to determine the compression algorithm for performance data based on data characteristics and a preset compression strategy table.

[0199] The first module 404 is used to compress performance data based on a compression algorithm to obtain a compressed data file.

[0200] The second determining module 405 is used to determine the dynamic block threshold of the compressed data file based on network operating parameters.

[0201] The second module 406 is used to perform block processing on the compressed data file based on the dynamic block threshold to obtain the unit to be transmitted; the unit to be transmitted includes the data block sequence number.

[0202] The transmission module 407 is used to transmit the unit to be transmitted to the central service node based on the data block sequence number.

[0203] In one possible implementation, the transmission module 407 is specifically used to: construct a connection health score model based on network operating parameters; the network operating parameters include network bandwidth utilization, network latency, and packet loss rate; calculate the connection health score of the computing node based on the connection health score; determine the transmission strategy of the unit to be transmitted based on the connection health score; and transmit the unit to be transmitted to the central service node based on the transmission strategy.

[0204] In one possible implementation, the transmission module 407 is specifically used to: determine the heartbeat interval of the computing node based on the connection health score; and establish a heartbeat connection between the computing node and the central service node based on the heartbeat interval.

[0205] In one possible implementation, the transmission module 407 is specifically used to: determine whether there is a connection failure between the computing node and the central service node based on the connection health score; if there is a connection failure, perform a reconnection operation and identify the breakpoint identifier; the breakpoint identifier is the sequence number of the data block that has been successfully transmitted; based on the breakpoint identifier, incrementally transmit the unit to be transmitted to the central server.

[0206] In one possible implementation, the transmission module 407 is specifically used to: adjust the exponential backoff step size based on network operating parameters; and execute a backoff reconnection strategy based on the exponential backoff step size until the computing node and the central service node re-establish a heartbeat connection, or the maximum number of reconnections is reached.

[0207] In one possible implementation, the second determining module 405 is specifically used to: compare the network bandwidth utilization rate with a preset network threshold; if the network bandwidth utilization rate is greater than the preset network threshold, increase the dynamic block threshold of the compressed data file; if the network bandwidth utilization rate is not greater than the preset network threshold, decrease the dynamic block threshold of the compressed data file.

[0208] In one possible implementation, after transmitting the unit to be transmitted to the central service node based on the data block sequence number, a generation module is also included. The generation module is specifically used for: reassembling the data of the unit to be transmitted based on the data block sequence number to obtain the corresponding data file; generating the lifecycle of the data file based on the data information characteristics; and cleaning up the data file if the current cycle reaches the lifecycle of the data file.

[0209] The performance data transmission device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0210] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0211] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0212] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0213] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0214] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0215] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0216] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0217] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0218] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0219] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0220] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0221] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0222] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0223] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0224] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0225] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for transmitting performance data, characterized in that, include: Obtain performance data and network operating parameters of compute nodes; Based on the performance data, corresponding data features are extracted; the data features include data structure features, data block capacity features, and data information features. Based on the data characteristics and the preset compression strategy table, the compression algorithm for the performance data is determined; Based on the compression algorithm, the performance data is compressed to obtain a compressed data file; Based on the network operating parameters, determine the dynamic block threshold of the compressed data file; Based on the dynamic block threshold, the compressed data file is divided into blocks to obtain units to be transmitted; each unit to be transmitted includes a data block number. Based on the data block sequence number, the unit to be transmitted is transmitted to the central service node.

2. The method according to claim 1, characterized in that, The step of transmitting the unit to be transmitted to the central service node based on the data block sequence number includes: Based on the network operating parameters, a connection health scoring model is constructed; the network operating parameters include network bandwidth utilization, network latency, and packet loss rate. Based on the connection health scoring model, the connection health score of the computing node is calculated; Based on the connection health score, the transmission strategy of the unit to be transmitted is determined; Based on the transmission strategy, the unit to be transmitted is transmitted to the central service node.

3. The method according to claim 2, characterized in that, The step of determining the transmission strategy for the unit to be transmitted based on the connection health score includes: Based on the connection health score, the heartbeat interval of the computing node is determined; Based on the heartbeat interval, a heartbeat connection is established between the computing node and the central service node.

4. The method according to claim 3, characterized in that, The step of determining the transmission strategy for the unit to be transmitted based on the connection health score includes: Based on the connection health score, it is determined whether there is a connection failure between the computing node and the central service node; If the connection failure exists, a reconnection operation is performed, and the breakpoint identifier is identified; the breakpoint identifier is the sequence number of the data block that has been successfully transmitted. Based on the breakpoint identifier, the unit to be transmitted is incrementally transmitted to the central server.

5. The method according to claim 4, characterized in that, The reconnection operation includes: Based on the network operating parameters, adjust the exponential backoff step size; Based on the exponential backoff step size, a backoff reconnection strategy is executed until the computing node and the central service node re-establish a heartbeat connection, or the maximum number of reconnections is reached.

6. The method according to claim 5, characterized in that, Determining the dynamic block threshold of the compressed data file based on the network operating parameters includes: The network bandwidth utilization rate is compared with a preset network threshold. If the network bandwidth utilization rate is greater than the preset network threshold, then the dynamic block threshold of the compressed data file is increased. If the network bandwidth utilization is not greater than the preset network threshold, then the dynamic block threshold of the compressed data file is reduced.

7. The method according to any one of claims 1-6, characterized in that, After transmitting the unit to be transmitted to the central service node based on the data block sequence number, the process further includes: Based on the data block sequence number, the data of the unit to be transmitted is reassembled to obtain the corresponding data file; Based on the data information characteristics, the lifecycle of the data file is generated; If the current cycle reaches the lifecycle of the data file, then the data file is cleaned up.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images