Network data transmission method and device

Abstract

The invention discloses a network data transmission method and device, and relates to the technical field of data processing. According to the method, the hot objects and the cold objects are distinguished according to the comparison result of the cold degree value of the object to be transmitted and the preset cold degree threshold value, and the hot objects with the cold degree values smaller than or equal to the preset cold degree threshold value are directly written into the local memory through the first communication link, so that the operation speed of the business process depending on the hot objects is increased; the service failure caused by the delay problem is avoided; and for the cold object of which the cold degree value is greater than the preset cold degree threshold value, writing the cold object into the cache region, and then writing the data in the cache region into the remote memory in batches through the second communication link based on the preset batch transmission strategy, so that a large number of dispersed small data transmission requests are integrated into batch transmission requests, and the data transmission efficiency is improved. According to the invention, the number of network transmission requests is reduced, thereby reducing the occupation of network bandwidth resources, effectively reducing the occurrence of network congestion, and improving the overall data transmission efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a network data transmission method and device. BACKGROUND

[0002] With the rapid development of cloud computing, big data and distributed computing technology, remote memory technology is widely used in various data-intensive business scenarios. Remote memory technology pools and manages the memory resources of distributed nodes through network communication, so that the local node can access the remote memory resources of other nodes, thereby effectively improving the utilization of memory resources.

[0003] However, in the related art, the same transmission strategy is used for high access frequency objects and low access frequency objects in the remote memory in the remote memory data transmission method. Therefore, in the related art, the remote memory data transmission method has high delay when transmitting data of high access frequency objects, which may directly lead to substandard business performance or even business failure. When transmitting data of low access frequency objects, a large amount of network bandwidth resources is occupied, causing bandwidth waste and affecting overall data transmission efficiency. SUMMARY

[0004] The present application provides a network data transmission method and device to at least solve the problems that the data transmission method in the related art may directly lead to substandard business performance or even business failure, and a large amount of network bandwidth resources is occupied, causing bandwidth waste and affecting overall data transmission efficiency.

[0005] In a first aspect, the present application provides a network data transmission method, comprising:

[0006] obtaining a to-be-transmitted object;

[0007] comparing a coldness value of the to-be-transmitted object with a preset coldness threshold, and determining the to-be-transmitted object as a hot object in a case where the coldness value of the to-be-transmitted object is less than or equal to the preset coldness threshold, and determining the to-be-transmitted object as a cold object in a case where the coldness value of the to-be-transmitted object is greater than the preset coldness threshold;

[0008] in a case where the to-be-transmitted object is a hot object, writing the to-be-transmitted object to a local memory through a first communication link;

[0009] in a case where the to-be-transmitted object is a cold object, writing the to-be-transmitted object to a cache area, and batch writing data in the cache area to a remote memory through a second communication link based on a preset batch transmission strategy.

[0010] In a second aspect, the present application further provides a network data transmission device, comprising:

[0011] an obtaining module, configured to obtain a to-be-transmitted object;

[0012] determining that the to-be-transmitted object is a hot object when the coldness value of the to-be-transmitted object is less than or equal to the preset coldness threshold, and determining that the to-be-transmitted object is a cold object when the coldness value of the to-be-transmitted object is greater than the preset coldness threshold;

[0013] a first writing module configured to write the to-be-transmitted object into the local memory through the first communication link when the to-be-transmitted object is a hot object;

[0014] a second writing module configured to write the to-be-transmitted object into the cache area, and write data in the cache area into the remote memory in batches through the second communication link based on a preset batch transmission strategy when the to-be-transmitted object is a cold object.

[0015] In a third aspect, the present application further provides an electronic device, comprising:

[0016] a memory configured to store a computer program;

[0017] a processor configured to execute the computer program to implement the steps of the method according to the first aspect.

[0018] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to the first aspect.

[0019] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of the method according to the first aspect.

[0020] The network data transmission method and device provided by the present application distinguish hot objects and cold objects according to the comparison result of the coldness value of the to-be-transmitted object and the preset coldness threshold, directly write hot objects with a coldness value less than or equal to the preset coldness threshold into the local memory through the first communication link, avoid the high delay of the remote transmission link in the related art, improve the running speed of the business process relying on the hot object, and avoid business failure caused by delay problems; write cold objects with a coldness value greater than the preset coldness threshold into the cache area first, and then write data in the cache area into the remote memory in batches through the second communication link based on a preset batch transmission strategy, thereby integrating a large number of scattered small data transmission requests into batch transmission requests, reducing the number of network transmission requests, reducing the occupation of network bandwidth resources, avoiding bandwidth waste, reducing the processing pressure of network devices, effectively reducing the occurrence of network congestion, and further improving the overall data transmission efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0022] Figure 1 An application scenario diagram corresponding to a network data transmission method provided by an embodiment of the present application is shown in FIG. 1.

[0023] Figure 2 A flowchart of the network data transmission method provided by an embodiment of the present application is shown in FIG. 2.

[0024] Figure 3 A flowchart of the network data transmission method provided by another embodiment of the present application is shown in FIG. 3.

[0025] Figure 4 A structural diagram of a network data transmission device provided by an embodiment of the present application is shown in FIG. 4.

[0026] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.

[0028] In order to make those skilled in the art better understand the solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0029] With the rapid development of cloud computing, big data and distributed computing technology, remote memory technology is widely used in various data-intensive business scenarios due to its advantages of breaking through the limitation of local memory capacity and realizing resource elastic expansion. Remote memory technology pools and manages the memory resources of distributed nodes through network communication, so that the local node can access the remote memory resources of other nodes, thereby effectively improving the utilization rate of memory resources.

[0030] However, in the related art, the same transmission strategy is adopted for the "hot objects" with high access frequency and the "cold objects" with low access frequency in the remote memory. Specifically, in the related art, the hot objects are transmitted through the remote transmission link, and the transmission delay is usually maintained at the level of 40-80 microseconds (us), while the access delay of the local memory is only at the level of nanoseconds (ns), and the high delay characteristic will seriously slow down the business process relying on the hot objects, for example, in the delay-sensitive scenarios such as high-frequency trading, real-time data analysis, etc., which may directly lead to substandard business performance or even business failure. In addition, in the related art, the cold objects usually adopt the instant transmission mode, that is, one remote transmission is performed for each generated cold object, thereby causing a large number of small data transmission requests, which not only occupies a large amount of network bandwidth resources, causing bandwidth waste, but also increases the processing pressure of the network device, causing network congestion and other problems, affecting the overall data transmission efficiency.

[0031] Figure 1 An application scenario corresponding to a network data transmission method provided by an embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the application scenario includes a data transmission device 10, a local memory 11, a cache area 12, and a remote memory 13, and the local memory 11, the cache area 12, and the remote memory 13 are in communication connection with the data transmission device 10.

[0032] Optionally, after obtaining the to-be-transmitted object, the data transmission device 10 first compares the coldness value of the to-be-transmitted object with a preset coldness threshold, and determines the heat type of the to-be-transmitted object according to the comparison result. Specifically, in the case where the coldness value of the to-be-transmitted object is less than or equal to the preset coldness threshold, the heat type of the to-be-transmitted object is determined as a hot object; in the case where the coldness value of the to-be-transmitted object is greater than the preset coldness threshold, the heat type of the to-be-transmitted object is determined as a cold object.

[0033] Specifically, in the case where the to-be-transmitted object is a hot object, the data transmission device 10 writes the to-be-transmitted object to the local memory 11 through a first communication link, wherein the first communication link is a PICe link; in the case where the to-be-transmitted object is a cold object, the data transmission device 10 writes the to-be-transmitted object to the cache area 12, and writes the data in the cache area 12 to the remote memory 13 in batches through a second communication link based on a preset batch transmission strategy. The second communication link is a link based on the RDMA protocol.

[0034] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0035] Figure 2 This is a flowchart illustrating a network data transmission method according to an embodiment of this application, as shown below. Figure 2 As shown, the execution subject of this embodiment is a network data transmission device. This device can be implemented through a computer program, or through a medium storing the relevant computer program, such as a USB flash drive and / or optical disc; alternatively, it can be implemented through a physical device that integrates or installs the relevant computer program, such as a chip or electronic device. The electronic device may be a computer or a server, etc. The network data transmission method provided in this embodiment includes the following steps:

[0036] S201. Obtain the object to be transmitted.

[0037] The object to be transmitted refers to the data unit that needs to be transmitted between the local node and the memory resource pool (including local memory and remote memory) during the operation of the distributed system. The object to be transmitted can be a variable, a data block, a file fragment, etc., depending on the needs of the business scenario.

[0038] Optionally, the triggering conditions for obtaining the object to be transmitted include, but are not limited to, data writing requirements generated by the local node's business process, insufficient local memory requiring storage expansion, and data in remote memory being accessed by the local node.

[0039] Optionally, in practical applications, local nodes can obtain relevant information about the object to be transmitted from the business process through inter-process communication (IPC), system call interfaces, etc. This information includes not only the object's own data content but also metadata used for subsequent coldness value calculation, such as the object's creation time, last access time, number of accesses, and access interval. This metadata can be recorded and updated in real time through the memory management module, providing data support for subsequent hot / coldness differentiation.

[0040] S202. Compare the coldness value of the object to be transmitted with a preset coldness threshold. If the coldness value of the object to be transmitted is less than or equal to the preset coldness threshold, determine that the object to be transmitted is a hot object; if the coldness value of the object to be transmitted is greater than the preset coldness threshold, determine that the object to be transmitted is a cold object.

[0041] It should be noted that in this embodiment, the access frequency of the object to be transmitted is characterized by the coldness value. The lower the coldness value, the higher the access frequency of the object to be transmitted, that is, the "hotter" the object is; conversely, the higher the coldness value, the lower the access frequency of the object to be transmitted, that is, the "colder" the object is.

[0042] Optionally, the coldness value is a quantitative value calculated after comprehensively considering multiple access characteristics of the object to be transmitted. The calculation method of the coldness value can be adjusted according to the actual business scenario.

[0043] Optionally, there are multiple ways to calculate the "coldness" value of the object to be transmitted. For example, a preset time window (such as 1 minute or 5 minutes) can be used as the statistical period to count the number of times the object to be transmitted is accessed within that statistical period. The coldness value is negatively correlated with the number of accesses. For example, the coldness value can be set as K / (number of accesses + 1), where K is a preset proportional coefficient. This method of calculating the coldness value is suitable for scenarios with high sensitivity to access frequency.

[0044] Optionally, multiple dimensions such as the most recent access time interval (T) of the object to be transmitted, the number of accesses within the statistical period (N), and the data size (S) can be comprehensively considered, and a preset weight can be assigned to each dimension (e.g., the weight of T is 0.4, the weight of N is 0.5, and the weight of S is 0.1). The formula for calculating the coldness value L can be:

[0045] L=ω1×(T / Tmax)+ω2×(1 / (N+1))+ω3×(S / Smax)

[0046] Where ω1, ω2, and ω3 are the weights of each dimension and ω1+ω2+ω3=1, Tmax is the preset maximum time interval threshold, and Smax is the preset maximum data size threshold. This coldness value calculation formula can more comprehensively reflect the coldness and hotness characteristics of an object and is suitable for complex business scenarios.

[0047] It should be noted that the calculation period for the coldness value can be dynamically adjusted according to the business. For businesses with high real-time requirements, the period can be set to a shorter period (e.g., 10 seconds), while for non-real-time businesses, the period can be set to a longer period (e.g., 10 minutes).

[0048] The preset cooling threshold is a pre-set cooling value used to distinguish between hot and cold objects. Optionally, the preset cooling threshold can be determined based on historical business data statistics. For example, by analyzing the access characteristics of all data objects in the past month, the distribution range of cooling values ​​for hot and cold objects can be calculated, and the boundary point between the two ranges can be used as the preset cooling threshold.

[0049] Optionally, the system's operating status (such as local memory usage, network bandwidth utilization, and service latency metrics) can be monitored in real time, and the preset cold threshold can be dynamically adjusted based on the monitoring results. For example, when the local memory usage is below 30%, the preset cold threshold can be appropriately lowered to classify more objects as hot objects and store them in local memory to improve access speed; when the network bandwidth utilization is above 80%, the preset cold threshold can be appropriately increased to classify more objects as cold objects and perform batch transmission to reduce bandwidth consumption.

[0050] Optionally, after obtaining the coldness value of the object to be transmitted and determining the preset coldness threshold, the object type can be determined by numerical comparison. If the coldness value of the object to be transmitted is less than or equal to the preset coldness threshold, it means that the object is accessed frequently and is determined to be a hot object; if the coldness value of the object to be transmitted is greater than the preset coldness threshold, it means that the object is accessed frequently and is determined to be a cold object.

[0051] S203. If the object to be transmitted is a hot object, the object to be transmitted is written to the local memory through the first communication link.

[0052] Optionally, considering the high access frequency and latency sensitivity of hot objects, this embodiment adopts a local storage transmission strategy, that is, the hot object is directly written to the local memory through the first communication link, thereby solving the problem of high latency in remote transmission of hot objects.

[0053] The first communication link refers to the memory access link within the local node. Optionally, the first communication link can be a PCIe (Peripheral Component Interconnect Express Link) link, which can achieve nanosecond (ns) level low latency and perfectly match the low latency requirements of hot objects.

[0054] Optionally, during actual transmission, the local node's memory management module will allocate dedicated memory space for hot objects (which can be pre-allocated using a memory pool mechanism to reduce memory fragmentation) and achieve fast data writing through Direct Memory Access (DMA) technology to avoid increased latency caused by CPU intervention.

[0055] Optionally, to avoid hot objects occupying local memory for a long time and causing insufficient storage resources, the access status of hot objects in local memory can be monitored in real time. When the access frequency of a hot object decreases and its coldness value exceeds the preset coldness threshold, it can be marked as a "cold object to be migrated" and migrated to remote memory when local memory resources are scarce.

[0056] S204. If the object to be transmitted is a cold object, write the object to be transmitted to the buffer area, and write the data in the buffer area to the remote memory in batches through the second communication link based on the preset batch transmission strategy.

[0057] Optionally, considering the characteristics of cold objects being accessed infrequently and having low value per transmission, this embodiment adopts a strategy of caching first and then transmitting in batches, using a second communication link to achieve remote transmission of cold objects, thereby reducing bandwidth waste and lowering network pressure.

[0058] Optionally, the cache can be a storage space within the local node used for temporary storage of cold objects. The size of the cache can be configured according to system memory resources and business needs (e.g., the setting range can be 100KB-10GB).

[0059] Optionally, the cache can adopt a first-in-first-out or least recently used strategy. When the cache space is insufficient, the oldest or the coldest object that has not been accessed for the longest time will be automatically evicted. Before evicting, it must be ensured that the object has been successfully transferred to remote memory.

[0060] Optionally, the process of writing cold objects to the cache can be done asynchronously. That is, after the business process submits the cold object to the cache, it can continue to perform subsequent operations without waiting for the remote transmission to complete, thereby avoiding the blocking of the business process by the transmission of cold objects.

[0061] Optionally, deduplication can be performed on cold objects stored in the cache. If the same cold object is submitted multiple times, only the latest version is retained to further reduce invalid transmissions.

[0062] Optionally, the preset batch transfer strategy is a preset transfer method for cached data. The purpose of the preset batch transfer strategy is to integrate multiple cold objects into a batch data block for transfer, so as to reduce the number of transfer requests and network overhead.

[0063] Optionally, a buffer data size threshold (e.g., 64KB, to adapt to the maximum transfer unit) and a single data block size threshold (e.g., 64KB) can be set. When the total amount of cold object data stored in the buffer reaches the buffer data size threshold, or the size of a single cold object exceeds the single data block size threshold, a batch transfer is triggered.

[0064] Optionally, a fixed transmission period (such as 10 seconds or 30 seconds) can be set, triggering batch transmission at the end of each period regardless of whether the amount of data in the buffer has reached the buffer data volume threshold. This method is suitable for scenarios where cold objects are generated infrequently, avoiding the accumulation of cold objects in the buffer for extended periods.

[0065] Optionally, during batch transmission, cold objects in the buffer are packaged and the size of the batch data blocks is reduced by using data compression algorithms (such as GZIP), thereby further reducing bandwidth usage.

[0066] Optionally, the second communication link can adopt a communication protocol based on TCP / IP or RDMA (Remote Direct Memory Access). RDMA protocol can realize direct data transfer from memory to memory, bypassing the operating system kernel, which can effectively reduce the latency and CPU overhead of remote transmission and is suitable for batch transmission of large-scale cold objects.

[0067] This application provides a network data transmission method that distinguishes between hot and cold objects based on a comparison between the coldness value of the object to be transmitted and a preset coldness threshold. For hot objects with a coldness value less than or equal to the preset coldness threshold, they are directly written to local memory via a first communication link, avoiding the high latency of remote transmission links in related technologies. This improves the running speed of business processes that rely on hot objects and avoids business failures due to latency issues. For cold objects with a coldness value greater than the preset coldness threshold, the cold objects are first written to a buffer, and then the data in the buffer is written to remote memory in batches via a second communication link based on a preset batch transmission strategy. Thus, by integrating a large number of scattered small data transmission requests into batch transmission requests, the number of network transmission requests is reduced, thereby reducing the occupation of network bandwidth resources, avoiding bandwidth waste, alleviating the processing pressure on network devices, effectively reducing network congestion, and improving overall data transmission efficiency.

[0068] As an optional implementation, based on any of the above embodiments, data in the buffer is written to remote memory in batches via a second communication link according to a preset batch transmission strategy, including the following steps:

[0069] First, obtain the total amount of data in the cache.

[0070] It should be noted that in this embodiment, the total amount of data in the cache is the primary basis for triggering the batch transmission process, and the process of obtaining the total amount of data can be executed in real time by the cache management module. The cache management module will perform data volume statistics for each cold object stored in the cache and accumulate the data in real time to obtain the total amount of data in the cache. Optionally, to avoid the statistical process consuming too many system resources, an incremental statistical method is adopted, that is, the total amount of data is updated only when a new object is written or an old object is deleted, rather than periodically performing full statistics, to ensure the efficiency of the statistical process.

[0071] Secondly, when the total amount of data in the cache exceeds the preset batch threshold, the physical address of the remote memory is resolved through the direct memory access engine.

[0072] Optionally, the total amount of data obtained in the cache can be compared with a preset batch threshold in real time. The preset batch threshold is a value configured based on factors such as the bandwidth capacity of the second communication link and the write performance of the remote memory. For example, in a 10GbE network environment, the threshold can be set to 500MB, and in a 25GbE network environment, it can be increased to 1GB to make full use of the link bandwidth.

[0073] Optionally, when the total amount of data in the cache exceeds a preset batch threshold, an address resolution request is sent to the Direct Memory Access (DMA) engine.

[0074] The DMA engine, as a dedicated hardware unit responsible for memory address mapping and data transfer, possesses efficient address resolution capabilities. First, the DMA engine obtains the memory address space mapping table of the remote memory node from its memory controller via a distributed memory management protocol. Second, based on preset memory allocation rules (such as a contiguous physical address allocation mode), it resolves the contiguous physical address segments available for writing the current batch of data from the address mapping table. Finally, it verifies the resolved physical addresses to ensure their validity and accessibility, preventing transmission failures due to address errors.

[0075] The memory address space mapping table contains the mapping relationship between remote memory nodes and physical addresses. It should be noted that when the physical address of remote memory changes, only the physical address in the memory address space mapping table needs to be updated, and the DMA engine can still determine the latest physical address based on the updated memory address space mapping table.

[0076] It should be noted that the DMA engine can directly resolve physical addresses, reducing address resolution time from milliseconds to microseconds, significantly improving the efficiency of transmission preparation.

[0077] Finally, based on the physical address of the remote memory, the data in the cache is written to the remote memory in batches via the second communication link.

[0078] Optionally, after obtaining a valid remote memory physical address, the DMA engine takes over the data transfer control and writes the batch data in the buffer directly to the remote memory via a second communication link based on the remote memory's physical address. The entire transfer process uses "zero-copy" technology, meaning that data is transferred directly from the local buffer to the remote memory without passing through local and remote CPU registers, thus avoiding the overhead of copying data between multiple storage levels.

[0079] Optionally, the DMA engine first reassembles the batch data in the buffer according to the order of the remote memory physical addresses to ensure the continuity of data writing; then, it establishes a direct data channel with the remote memory node through the RDMA interface of the second communication link, and transmits the reassembled data to the corresponding physical address in the form of data blocks; during the transmission process, the DMA engine monitors the transmission progress and data integrity in real time, and verifies each data block through the Cyclic Redundancy Check (CRC) algorithm. If a data error is found, a retransmission mechanism is immediately triggered.

[0080] Optionally, after the data transfer in the cache is completed, the DMA engine will send a data transfer completion notification to the cache management module. The cache management module can mark the cold objects that have been successfully transferred in the cache as "cleanable" and delete them during the subsequent cache cleanup process to release cache space.

[0081] It should be noted that directly resolving physical addresses and performing zero-copy transfers through the DMA engine not only reduces CPU intervention and lowers CPU utilization, but also further shortens data transfer latency and improves the overall efficiency of batch transfers, making it suitable for batch transfer scenarios of ultra-large-scale cold objects.

[0082] This application provides a network data transmission method that first obtains the total amount of data in the cache. When the total amount of data in the cache exceeds a preset batch threshold, the method uses a direct memory access engine to parse the physical address of remote memory and then writes the data in batches. This allows for accurate control of the timing of batch transmission, avoiding transmission that is too early or too late. By using a direct memory access engine to parse the physical address, the method improves the accuracy and efficiency of writing data to remote memory, thereby enhancing the overall data transmission performance.

[0083] Figure 3 A flowchart illustrating a network data transmission method provided in another embodiment of this application is shown below. Figure 3 As shown, as an optional implementation, based on any of the above embodiments, the method further includes the following steps:

[0084] S301. In response to determining that the local memory usage exceeds a first preset ratio, the garbage collection function is triggered to scan each live object in the local memory.

[0085] Optionally, the local memory usage can be monitored in real time through the memory management module of the local node. The total memory capacity, used capacity, free capacity and other indicators can be obtained through the memory statistics interface provided by the system kernel (such as / proc / meminfo in Linux system), and the memory usage ratio can be calculated. The memory usage ratio = used capacity / total memory capacity × 100%.

[0086] The first preset percentage is a pre-defined percentage used to trigger garbage collection. Optionally, to balance memory utilization and business stability, the first preset percentage can be dynamically configured according to the business type. For example, for memory-sensitive businesses, the first preset percentage can be set to 70%; for compute-intensive businesses, the first preset percentage can be set to 85%. When the local memory usage exceeds the first preset percentage, garbage collection (GC) is triggered immediately. Compared to timed triggering, this can intervene more precisely when memory is tight, reducing unnecessary system overhead.

[0087] Optionally, after garbage collection is triggered, a generational garbage collection or mark-and-sweep algorithm will be used to perform a full scan of local memory to distinguish between live objects and garbage objects. Live objects are those that are still referenced by business processes or have been accessed within a preset time window; garbage objects are those that are not referenced by any process and have not been accessed for a long time.

[0088] Optionally, during the scanning process, the garbage collection function will work in conjunction with the metadata records of the memory management module to quickly determine the object status using information such as object reference counts and access timestamps. Furthermore, to reduce the impact of the scanning process on business processes, a concurrent scanning mode can be adopted, where garbage collection scanning and business process execution run in parallel, ensuring that business continuity is not significantly affected.

[0089] Optionally, after the scan is complete, the garbage collection function will immediately release the memory space occupied by garbage objects and generate a list of live objects.

[0090] S302. If it is determined that any surviving object has not been accessed, adjust the coldness value of any surviving object based on the preset value.

[0091] Optionally, after obtaining the list of surviving objects, the access records of each surviving object are checked one by one, with a focus on determining whether the object was accessed within a preset monitoring period (e.g., 5 minutes) before the waste collection scan. This monitoring period can be consistent with the coldness value calculation period to ensure the continuity of data statistics.

[0092] Optionally, when an application or program performs a read operation (such as reading an object field) or a write operation (such as modifying an object field) on an object in local memory, the read-write barrier automatically intercepts and records the object's coldness value. Each time the object is accessed, the object's coldness value can be set to 0 (indicating that the object has been active recently).

[0093] Optionally, if it is determined that any surviving object has not been accessed, it indicates that the access frequency of the surviving object has decreased. In this case, the coldness value of the surviving object is adjusted based on a preset value.

[0094] Optionally, if a surviving object remains unvisited for one monitoring cycle, the coldness value is increased by a preset base value (e.g., 1); if a surviving object remains unvisited for two consecutive monitoring cycles, the coldness value is increased by 1.5 times the base value (e.g., 1.5); if a surviving object remains unvisited for three or more consecutive cycles, the coldness value is increased by 2 times the base value (e.g., 2). Optionally, the specific value of the preset value can be calibrated according to the range of coldness values ​​to ensure that the adjusted coldness value still accurately reflects the object's thermal characteristics.

[0095] Optionally, for surviving objects whose adjusted coldness value exceeds the preset coldness threshold, they can be marked as "cold objects to be migrated" and the memory management module can be notified to migrate them to remote memory at an appropriate time (such as when local memory is tight again or when batch transmission is triggered); for objects whose adjusted coldness value is still lower than the preset coldness threshold, their hot object attributes are maintained and they continue to occupy local memory to ensure access efficiency.

[0096] This application provides a network data transmission method that triggers garbage collection to scan for live objects when local memory usage exceeds a first preset percentage. For unaccessed live objects, the cooling value is adjusted according to a preset value. This allows for timely detection of local memory storage pressure, and by adjusting the cooling value, the subsequent transmission strategy for live objects is changed, thus reasonably releasing local memory space and avoiding excessive memory resource consumption.

[0097] like Figure 3 As shown, as an optional implementation, based on any of the above embodiments, the method further includes the following steps:

[0098] S303. After the garbage collection scan is completed, obtain the size of each surviving object in local memory and its corresponding coldness value.

[0099] Optionally, after the garbage collection scan is completed, the size of each surviving object in local memory and its corresponding coldness value are further obtained. The size of the surviving object reflects the resource cost of memory usage, and the coldness value reflects the business value of the object.

[0100] S304. Based on the size of each surviving object and its corresponding coldness value, a dynamic threshold is determined through a preset algorithm, and the dynamic threshold is used to replace the preset coldness threshold.

[0101] Optionally, a weighted average algorithm can be used to determine the dynamic threshold. This algorithm uses the size of the surviving objects as the weight to calculate the weighted average of the coldness values ​​of all surviving objects, which is then used as the dynamic threshold P. The formula for calculating the dynamic threshold P is: P = Σ(surviving object size × L) / Σsize, where "size" represents the memory footprint of a single surviving object, "L" represents the coldness value of that object, "Σ(surviving object size × L)" is the sum of the products of "size - coldness value" for all surviving objects, and "Σsize" is the total memory footprint of all surviving objects. Optionally, if L ≤ P, the object is considered a hot object; if L > P, it is considered a cold object. When a surviving object is a hot object, a local storage strategy should be adopted, and it should be prioritized during transmission; when a surviving object is a cold object, a remote storage strategy can be adopted, and batch transmission can be performed.

[0102] Optionally, a dynamic threshold can be determined using a clustering algorithm. Optionally, a two-dimensional coordinate system is constructed with coldness value on the horizontal axis and object size on the vertical axis, mapping all surviving objects to data points in the coordinate system. The K-means clustering algorithm is used to divide the data points into two categories (hot object clusters and clusters to be evaluated). A memory constraint is introduced during the clustering process, meaning the total size of the hot object clusters does not exceed a preset proportion of the available local memory capacity (e.g., 60%). After clustering, the maximum coldness value in the hot object cluster is taken as the dynamic threshold. The clustering algorithm can adaptively match the current capacity of local memory, preventing the total size of the hot object clusters from exceeding the memory's carrying capacity.

[0103] Optionally, after the dynamic threshold is determined, the original preset coldness threshold is replaced with the dynamic threshold, and the new threshold is synchronized to modules such as the memory management module. Subsequent newly acquired objects to be transferred and surviving objects in local memory will all be judged using this dynamic threshold as the coldness / hotness criterion.

[0104] This application provides a network data transmission method that, after a garbage collection scan, acquires the size and coldness value of surviving objects in local memory, and uses a preset algorithm to determine a dynamic threshold to replace the preset coldness threshold. Thus, it can dynamically adjust the criteria for classifying hot and cold objects based on the characteristics of surviving objects, making the determination of hot and cold objects more relevant to the current situation, taking into account the actual operating status.

[0105] As an optional implementation, based on any of the above embodiments, the method further includes the following steps:

[0106] First, obtain the working status of the garbage collection function.

[0107] Optionally, a real-time communication link can be established between the system resource management module and the garbage collection module of the local node to continuously obtain the working status of the garbage collection function. The working status of the garbage collection function includes "running status" and "stopped status". The "running status" can include sub-stages such as initialization, marking, clearing, and sorting, while the "stopped status" can include two sub-states: idle and completed.

[0108] Optionally, the system resource management module can send a status query request to the garbage collection module every 10 milliseconds. When the working status of the garbage collection module changes (such as from stopped to running, or from running to stopped), it will actively send a status change notification to the system resource management module to ensure the real-time and accuracy of status acquisition.

[0109] Secondly, when the garbage collection function is detected to be running, the batch writing of data in the cache to remote memory via the second communication link is paused, while the writing of objects to be transferred to local memory via the first communication link is allowed.

[0110] Optionally, when the garbage collection function is detected to be running, the operation of writing data in the cache to remote memory in batches via the second communication link is immediately paused. Optionally, the currently executing batch transfer task is terminated first (if data transfer is in progress, it will wait for the current data block to be transferred before pausing to avoid integrity issues caused by data transfer interruption); secondly, the transfer progress at the time of pause is recorded, including the amount of data transferred, the amount of data not transferred, the current connection status, etc., so as to ensure seamless connection when the transfer is resumed later; finally, the batch transfer status is marked as "paused" and synchronized to the cache management module.

[0111] Optionally, to ensure that the access performance of hot objects is not affected by garbage collection and to maintain the normal operation of the hot object transmission channel, the part of the object to be transmitted that is determined to be a hot object is allowed to be written normally to the local memory through the first communication link.

[0112] Optionally, during a batch transfer pause, writing cold objects to the cache can still be performed normally. Optionally, the cache management module can expand the temporary cache partition (e.g., from 1GB to 2GB) to avoid the backlog of cold objects caused by the batch transfer pause. When the temporary cache partition is about to be full, the earliest stored cold object with the highest coldness value is evicted to ensure the availability of the cache.

[0113] Subsequently, when it is detected that the garbage collection function is stopped, the data in the cache is written to the remote memory in batches via the second communication link.

[0114] Optionally, when garbage collection is detected to be stopped, the batch transfer recovery process is initiated. First, the transmission status information recorded during the pause is verified. If a connection is found to be broken, the communication link with the remote memory node is re-established, and the connection handshake is quickly completed based on the RDMA protocol. Second, according to the transmission progress record, batch transfer is resumed from the pause point to avoid data duplication. Finally, during the transmission process, a gradual bandwidth utilization method can be adopted, that is, the transmission rate is gradually increased from 30% of the peak to 100%, to avoid network bandwidth fluctuations caused by sudden resumption of transmission, which may affect other services.

[0115] Optionally, if the total amount of cold object data in the cache exceeds 120% of the preset batch threshold after garbage collection stops, a multi-channel parallel transmission mechanism can be activated to establish 2-4 second communication links for batch transmission until the amount of data in the cache drops below the preset batch threshold, thereby improving the processing efficiency of backlogged data. After transmission is restored, the batch transmission status is reset to "normal," and the cache management module is notified to restore the original cache partition size.

[0116] This application provides a network data transmission method that, by acquiring the working status of the garbage collection function, pauses the batch writing of cached data to remote memory while the garbage collection function is running, allowing the objects to be transmitted to be written to local memory, and resumes batch writing when the garbage collection function is stopped. This coordinates garbage collection and data transmission, avoids resource contention caused by both occurring simultaneously, and ensures the stability of local memory writing and remote memory batch transmission.

[0117] As an optional implementation, based on any of the above embodiments, the method further includes the following steps:

[0118] First, in response to the remote memory usage exceeding a second preset ratio, the coldness value of each object in the remote memory is obtained.

[0119] Optionally, the local node's memory management module can establish real-time communication with the memory controller of the remote memory node through a remote memory monitoring protocol to continuously obtain the status data of the remote memory, including indicators such as the total capacity, used capacity, free capacity, and memory fragmentation rate of the remote memory. Among them, the key monitoring parameter is the remote memory occupancy ratio, which is calculated as: (used capacity / total capacity) × 100%.

[0120] The second preset ratio is a pre-set ratio value used to determine whether there is a resource shortage in remote memory. This second preset ratio can be configured in combination with the load capacity of remote memory and business needs. For remote memory clusters that carry multi-node cold object storage, the second preset ratio can be set to 85%; for remote memory nodes with dedicated local nodes, it can be increased to 90% to improve resource utilization.

[0121] Optionally, when the remote memory usage is detected to exceed the second preset ratio, the object migration evaluation process is triggered, and an object information retrieval request is sent to the remote memory node.

[0122] Secondly, the coldness value of each object in the remote memory is compared with the preset coldness threshold. If the coldness value of any object exceeds the preset coldness threshold, the data of the corresponding object is transferred from the remote memory to the local memory.

[0123] Optionally, after receiving a request, the memory management unit of the remote memory node traverses all storage objects in the remote memory and collects information for each object. The collected information may include a unique identifier, data size, coldness value, and recent access timestamp. The coldness value is the latest value that the local node can continuously update and synchronize based on real-time access records after the object is stored in the remote memory, ensuring that it reflects the object's current access characteristics.

[0124] Optionally, the remote memory node can sort object information in descending order of coldness value, generate a remote object coldness list, and feed it back to the local node.

[0125] Optionally, after receiving the list of remote object coldness values, the local node compares the coldness value of each object with the currently effective preset coldness threshold. If the coldness value of any object is less than or equal to the preset coldness threshold, it indicates that the access frequency of that object has increased, the "cold object" attribute is reversed, and migration can be performed. Optionally, if local memory is tight, data marked as "cold objects to be migrated" can be eliminated locally first according to "coldness value from high to low" to free up space before performing migration.

[0126] Optionally, if the coldness value of any object is greater than a preset coldness threshold, its remote storage state is maintained.

[0127] Optionally, a direct data channel between local and remote memory can be established via a second communication link, allowing the DMA engine to directly transfer object data from the physical address of the remote memory to the pre-allocated space in local memory. After the transfer is complete, the local node updates the object storage identifier and synchronizes it with the business process, so subsequent accesses will directly point to local memory; and notifies the remote memory node to delete the migrated objects, releasing the space in the remote memory.

[0128] This application provides a network data transmission method that, when remote memory usage exceeds a second preset percentage, acquires and compares the coldness values ​​of each object, and transfers the data of objects with coldness values ​​exceeding a threshold to local memory. Thus, by monitoring remote memory usage, data storage locations are rationally allocated based on the object's hotness / coldness attributes, and remote memory space is released in a timely manner.

[0129] As an optional implementation, based on any of the above embodiments, the method further includes the following steps:

[0130] First, after writing the objects to be transferred to local memory, verify the sequence number continuity of the objects to be transferred after the transfer is completed.

[0131] Optionally, during the acquisition of objects to be transmitted stage, the local node's transmission management module can assign a unique, consecutive sequence number to each object. This sequence number can be a combination of "business type code + timestamp + incrementing sequence," such as "TRD-20251204100000-001," where "TRD" represents a transaction-related business, the timestamp is accurate to milliseconds, and the incrementing sequence ensures the uniqueness of the object's sequence number under the same business and timestamp. The sequence number is then bound to the object data and transmitted along with the object through the first communication link.

[0132] Optionally, during the data transmission of the object to be transmitted, the local node's transmission management module can be used to record information such as the sequence number, transmission status (to be transmitted, in transmission, transmission completed), and target memory address of each object in real time, forming a full-process tracking of the transmission of the object to be transmitted, which is convenient for subsequent verification and problem localization.

[0133] Optionally, after the objects to be transmitted are written to local memory via the first communication link, a sequence number continuity verification process is triggered through the memory management module. This verification process primarily involves sequence number integrity verification. Specifically, all sequence numbers of objects transmitted in the current batch are extracted, sorted in ascending order, and then checked for missing, duplicate, or abrupt sequence numbers. For example, if the expected sequence numbers for the current batch are 001-010, and 005 is found to be missing after sorting, the integrity verification is deemed to have failed.

[0134] Optionally, if the sequence number continuity verification fails (due to missing, duplicate, or incorrect sequence numbers), the transmission management module can be notified to re-initiate the transmission of the corresponding object for the missing sequence number; for duplicate or incorrectly bound sequence numbers, the erroneous data in the local memory is deleted first, and then the correct object is re-transmitted.

[0135] Secondly, after the sequence number continuity verification of the objects to be transmitted is passed upon completion of the transmission, an object accessibility notification is generated and sent to the application completion queue. The target application retrieves the object accessibility notification by polling the application completion queue and accesses the object data in local memory based on the object accessibility notification.

[0136] Optionally, after the sequence number continuity verification of the object to be transmitted is passed after the transmission is completed, an object accessibility notification can be generated through the memory management module. The object accessibility notification may include information such as the object's unique sequence number, target memory address, data size, and access permission identifier. The object accessibility notification may adopt a structured data format (such as JSON) to ensure that the target application can parse it quickly.

[0137] Optionally, the generated object accessibility notification will be sent to the application completion queue, which can be allocated an independent queue partition for each target application to avoid interference between notifications from different applications.

[0138] Optionally, the target application obtains object accessibility notifications by polling the application completion queue. The polling period can be dynamically configured according to the real-time requirements of the business. For example, the polling period for high-frequency trading applications can be set to 10 microseconds, and the polling period for ordinary business applications can be set to 1 millisecond.

[0139] Optionally, to reduce the CPU resource consumption of polling, a polling mechanism that extends the cycle during idle periods is adopted. When no object access notification is obtained after three consecutive polls, the polling cycle is automatically extended to twice the original polling cycle; when an object access notification is obtained, the original polling cycle is immediately restored.

[0140] Optionally, after receiving the object access notification, the target application directly accesses the object data in local memory through the memory access interface based on the memory address and access permission identifier in the notification. During the access process, the application verifies whether its own access permissions match the permission identifier in the notification. If they do not match, access is denied and a permission exception alarm is triggered to ensure the security of data access.

[0141] This application provides a network data transmission method that, after writing the object to be transmitted into local memory, verifies the sequence number continuity. Upon successful verification, an object accessibility notification is generated and sent to the application completion queue. The target application can access the data by polling for the object accessibility notification. Thus, sequence number continuity verification ensures complete and accurate data transmission, preventing data loss or corruption; generating object accessibility notifications allows the target application to be promptly informed of the accessibility status and obtain data as needed, improving the timeliness and accuracy of data access and ensuring smooth business processing.

[0142] As an optional implementation, based on any of the above embodiments, the method further includes the following steps:

[0143] First, after writing the object to be transmitted to remote memory, a cyclic redundancy check is performed on the object after the transmission is completed, according to a preset delay time.

[0144] Optionally, after the objects to be transmitted (cold objects) are written to the remote memory in batches via the second communication link, the verification operation is not performed immediately, but the verification process is started after a preset delay.

[0145] The preset delay duration is a pre-set time threshold. The setting of the preset delay duration needs to take into account factors such as the write confirmation delay of remote memory and the stability compensation of network transmission. The setting range can be 100-300 microseconds to ensure that the remote memory has completed the persistent storage of data and avoid verification deviation caused by the completion of transmission but not the stable writing.

[0146] Optionally, a delay triggering mechanism can be implemented using a timer. When the batch transmission task is completed, the transmission completion timestamp is recorded. The timer compares the difference between the current time and the completion timestamp in real time. When the difference reaches the preset delay duration, the cyclic redundancy check (CRC) process is immediately triggered, and a verification request is sent to the remote memory node.

[0147] Cyclic redundancy check (CRC) involves generating a checksum using a specific algorithm and comparing the checksums of local and remote data to determine data integrity. Optionally, when writing cold objects to the cache using the local cache management module, a unique CRC-32 checksum is generated for each object. Before batch transmission, a mapping table of object identifiers and checksums for all objects in the batch is archived. When verification is triggered, the mapping table of object identifiers and checksums for the current batch is extracted from the archive and used as the verification benchmark.

[0148] Optionally, after receiving a verification request, the remote memory node locates the corresponding object data in the remote memory based on the object identifier list carried in the request, recalculates the checksum of each object using the same CRC-32 algorithm as the local node, and generates a response table of object identifiers and remote checksums. Optionally, the mapping table and the response table are compared one by one according to the object identifier. If the checksum of an object is completely consistent, the object is determined to have passed verification; if there is a difference, the verification is determined to have failed and a retransmission mechanism is triggered.

[0149] Secondly, if the cyclic redundancy check of the object to be transmitted passes after the transmission is completed, an object access notification is generated and sent to the application completion queue.

[0150] Optionally, an object accessibility notification is generated after all objects in the batch have passed cyclic redundancy check. The object accessibility notification uses a structured format and includes information such as batch identifier, object unique identifier, remote memory physical address, access permissions, and check pass time.

[0151] Optionally, the generated object access notification is sent to the application completion queue. This application completion queue can be partitioned and isolated from the application completion queue used by hot objects, allocating independent queue space for cold object access requests and avoiding mutual interference between notification information of hot and cold objects.

[0152] Optionally, after the target application obtains the object accessibility notification, it uses the address information and permission identifier in the object accessibility notification to call the object data in remote memory through the remote memory access interface.

[0153] This application provides a network data transmission method that performs cyclic redundancy check (CRC) after the object to be transmitted is written to remote memory, with a preset delay period. Only after the check passes is an accessibility notification generated. This delayed check approach allows for more comprehensive detection of the integrity of the transmitted data, preventing potential errors from going undetected during immediate checks. Furthermore, the CRC method is highly reliable, effectively ensuring the accuracy of data in remote memory. Access is only permitted after successful check, guaranteeing that subsequent business operations are based on correct data.

[0154] As an optional implementation, based on any of the above embodiments, the method further includes the following steps:

[0155] First, when the target application accesses data of any object in remote memory, it determines whether the coldness value of any accessed object is less than or equal to a preset coldness threshold.

[0156] Optionally, when the target application requests access to data of any object in remote memory through the remote memory access interface, the latest coldness value of the accessed object is obtained. This latest coldness value can be continuously updated by the local node based on the object's historical access records and synchronized to the real-time value in the remote memory metadata to ensure that the current popularity attribute of the object can be accurately reflected.

[0157] Optionally, the obtained coldness value is compared with the currently effective preset coldness threshold in real time. If the coldness value is greater than the preset coldness threshold, it means that the object is still in a low access frequency state, and its remote storage attribute is maintained, allowing the target application to access it normally through the second communication link. If the coldness value is less than or equal to the preset coldness threshold, it means that the access frequency of the object has reached the hot object standard, and its "cold object" attribute has been reversed.

[0158] Secondly, in response to any accessed object having a cold value less than or equal to a preset cold threshold, a cold-to-hot flag is added to any accessed object, and the preset cold threshold is updated based on the cold-to-hot flag.

[0159] Optionally, after determining that the coldness value of the accessed object is less than or equal to a preset coldness threshold, a cold-to-hot flag is added to the object. The cold-to-hot flag may include the object's unique identifier, the flag generation timestamp, and the access record that triggered the flag (access time, access application ID), etc.

[0160] Optionally, the cold-to-hot flag can be synchronized to the object state management table of the local node for subsequent memory scheduling; and synchronized to the object metadata of the remote memory node to prevent other nodes from triggering the same flag repeatedly.

[0161] Optionally, the method for calculating the updated preset coldness threshold can refer to the steps for calculating the dynamic threshold P in the above embodiments, which will not be repeated in this embodiment.

[0162] Optionally, after the preset cooling threshold is updated, the updated preset cooling threshold is synchronized to all relevant units such as the memory management module and remote memory nodes.

[0163] Optionally, for objects that have been marked as cold to hot, after the preset cold threshold is updated, the decision on whether to perform migration is made based on the local memory resource status. For example, if the local memory free capacity exceeds the total size of the marked objects, a batch migration is triggered; if the local memory free capacity is less than the total size of the marked objects, the marked objects are added to the priority migration queue and migration is performed first after the local memory space is released.

[0164] This application provides a network data transmission method that, when a target application accesses remote memory object data, determines the object's "coldness" value. If the value is less than or equal to a preset coldness threshold, a "cold to hot" flag is added, and the preset coldness threshold is updated. This allows for real-time capture of changes in object access, timely reflection of object attribute changes through the "cold to hot" flag, and timely updating of the preset coldness threshold, making subsequent hot and cold object classification more realistic.

[0165] Figure 4 This is a schematic diagram of the structure of a network data transmission device provided in an embodiment of this application, as shown below. Figure 4 As shown, the network data transmission device provided in this embodiment is located in an electronic device. The network data transmission device 40 provided in this embodiment includes: an acquisition module 41, a determination module 42, a first writing module 43, and a second writing module 44.

[0166] Specifically, the acquisition module 41 is used to acquire the object to be transmitted; the determination module 42 is used to compare the coldness value of the object to be transmitted with a preset coldness threshold, and determine the object to be transmitted as a hot object if the coldness value of the object to be transmitted is less than or equal to the preset coldness threshold; and determine the object to be transmitted as a cold object if the coldness value of the object to be transmitted is greater than the preset coldness threshold; the first writing module 43 is used to write the object to be transmitted to local memory through a first communication link if the object to be transmitted is a hot object; and the second writing module 44 is used to write the object to be transmitted to a buffer if the object to be transmitted is a cold object, and write the data in the buffer to remote memory in batches through a second communication link based on a preset batch transmission strategy.

[0167] Optionally, the first write module 43, when writing data in the cache to remote memory in batches via the second communication link based on a preset batch transfer strategy, is specifically used to: obtain the total amount of data in the cache; if the total amount of data in the cache exceeds a preset batch threshold, resolve the physical address of the remote memory through the direct memory access engine; and write the data in the cache to the remote memory in batches via the second communication link based on the physical address of the remote memory.

[0168] Optionally, the determination module 42 is further configured to: in response to determining that the local memory usage exceeds a first preset ratio, trigger the garbage collection function to scan each live object in the local memory; and if it is determined that any live object has not been accessed, adjust the coldness value of any live object based on a preset value.

[0169] Optionally, the determining module 42 is further configured to: after the garbage collection scan is completed, obtain the size of each surviving object in the local memory and the corresponding cold value; based on the size of each surviving object and the corresponding cold value, determine a dynamic threshold through a preset algorithm, and replace the preset cold threshold with the dynamic threshold.

[0170] Optionally, the second write module 44 is further configured to: obtain the working status of the garbage collection function; when the garbage collection function is detected to be running, suspend the batch writing of data in the cache to remote memory via the second communication link, and allow the object to be transferred to be written to local memory via the first communication link; when the garbage collection function is detected to be stopped, resume the batch writing of data in the cache to remote memory via the second communication link.

[0171] Optionally, the second writing module 44 is further configured to: in response to the remote memory usage exceeding a second preset ratio, obtain the coldness value of each object in the remote memory; compare the coldness value of each object in the remote memory with a preset coldness threshold, and if the coldness value of any object exceeds the preset coldness threshold, transfer the data of the corresponding object from the remote memory to the local memory.

[0172] Optionally, the network data transmission device provided in this embodiment further includes a verification module.

[0173] Optionally, the verification module is used to: after writing the object to be transferred to local memory, verify the sequence number continuity of the object to be transferred after the transfer is completed; after the sequence number continuity verification of the object to be transferred after the transfer is completed is passed, generate an object accessibility notification and send it to the application completion queue; the target application obtains the object accessibility notification by polling the application completion queue, and accesses the object data in local memory based on the object accessibility notification.

[0174] Optionally, the verification module is also used to: after writing the object to be transmitted to remote memory, perform cyclic redundancy check on the object to be transmitted after the transmission is completed according to a preset delay time; if the cyclic redundancy check of the object to be transmitted after the transmission is completed passes, generate an object accessibility notification and send it to the application completion queue.

[0175] Optionally, the network data transmission device provided in this embodiment further includes a tagging module.

[0176] Optionally, the marking module is used to: determine whether the coldness value of any object accessed is less than or equal to a preset coldness threshold when the target application accesses data of any object in remote memory; in response to the coldness value of any object accessed being less than or equal to the preset coldness threshold, add a cold-to-hot marker to any object accessed, and update the preset coldness threshold based on the cold-to-hot marker.

[0177] It should be noted that the technical effects of the network data transmission device in this embodiment have been explained in the above-described embodiment of the network data transmission method, and therefore will not be repeated in this embodiment.

[0178] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, as shown below. Figure 5 As shown, the electronic device 50 provided in the embodiments of this application includes: a memory 51 and a processor 52.

[0179] The memory 51 stores a computer program, and the processor 52 is configured to run the computer program to perform the steps in any of the above-described network data transmission method embodiments.

[0180] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described network data transmission method embodiments when it is run.

[0181] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0182] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described network data transmission method embodiments.

[0183] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described network data transmission method embodiments.

[0184] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0185] The solution provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A network data transmission method, characterized by, The method comprises: acquiring a to-be-transmitted object; comparing a coldness value of the to-be-transmitted object with a preset coldness threshold value, and determining the to-be-transmitted object as a hot object if the coldness value of the to-be-transmitted object is less than or equal to the preset coldness threshold value; determining the to-be-transmitted object as a cold object if the coldness value of the to-be-transmitted object is greater than the preset coldness threshold value; writing the to-be-transmitted object into a local memory through a first communication link if the to-be-transmitted object is a hot object; writing the to-be-transmitted object into a cache area, and writing data in the cache area into a remote memory through a second communication link in batches based on a preset batch transmission strategy if the to-be-transmitted object is a cold object.

2. The method of claim 1, wherein, The writing of the data in the cache area into the remote memory through the second communication link in batches based on the preset batch transmission strategy comprises: acquiring a total amount of data in the cache area; resolving a physical address of the remote memory through a direct memory access engine if the total amount of data in the cache area exceeds a preset batch threshold value; writing the data in the cache area into the remote memory through the second communication link based on the physical address of the remote memory.

3. The method of claim 1, wherein, The method further comprises: triggering a garbage collection function to scan each live object in the local memory in response to determining that an occupancy of the local memory exceeds a first preset proportion; adjusting a coldness value of any live object based on a preset value if it is determined that the live object is not accessed.

4. The method of claim 3, wherein, The method further comprises: acquiring a size and a corresponding coldness value of each live object in the local memory after the garbage collection scan is completed; determining a dynamic threshold value through a preset algorithm based on the size and the corresponding coldness value of each live object, and replacing the preset coldness threshold value with the dynamic threshold value.

5. The method of claim 3, wherein, The method further comprises: acquiring a working state of the garbage collection function; suspending the writing of the data in the cache area into the remote memory through the second communication link and allowing the writing of the to-be-transmitted object into the local memory through the first communication link if it is detected that the garbage collection function is in a running state; resuming the writing of the data in the cache area into the remote memory through the second communication link if it is detected that the garbage collection function is in a stopped state.

6. The method of claim 1, wherein, The method further comprises: acquiring a coldness value of each object in the remote memory in response to an occupancy of the remote memory exceeding a second preset proportion; comparing the coldness value of each object in the remote memory with a preset coldness threshold value, and transferring data of a corresponding object from the remote memory to the local memory if the coldness value of any object exceeds the preset coldness threshold value.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: verifying serial number continuity of the to-be-transmitted object after the to-be-transmitted object is written into the local memory. After the sequence number continuity verification of the to-be-transmitted object after transmission is passed, an object accessible notification is generated and sent to an application completion queue; the target application obtains the object accessible notification by polling the application completion queue, and accesses the object data in the local memory based on the object accessible notification.

8. The method according to any one of claims 1 to 6, characterized in that, Further comprising: After writing the to-be-transmitted object into the remote memory, cyclic redundancy check is performed on the to-be-transmitted object after transmission according to a preset delay duration; In the case that the cyclic redundancy check of the to-be-transmitted object after transmission is passed, an object accessible notification is generated and sent to an application completion queue.

9. The method according to any one of claims 1 to 6, characterized in that, Further comprising: When the target application accesses the data of any object in the remote memory, it is judged whether the coldness value of the accessed any object is less than or equal to the preset coldness threshold value; In response to the coldness value of the accessed any object being less than or equal to the preset coldness threshold value, a cold-to-hot mark is added to the accessed any object, and the preset coldness threshold value is updated based on the cold-to-hot mark.

10. An electronic device, comprising: Comprise: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the method according to any one of claims 1-9.

Images