A method for transmitting service protocol data that requires multi-packet transmission

Abstract

This invention discloses a method for transmitting service protocol data that requires multi-packet transmission. By setting up two allocators, when the receiver receives protocol data, it first allocates cache nodes from memory. If the memory nodes reach the configured maximum memory usage, it then allocates cache nodes from a cache file. The allocated cache nodes copy and cache the protocol data, and submit it to the upper layer for processing after the service protocol is fully received; otherwise, it continues to wait for data reception. This transmission method uses a chain of hybrid-type protocol transmission unit cache nodes to provide read and write capabilities, thereby alleviating the resource bottleneck caused by insufficient user memory space in processes.

Description

Technical Field

[0001] This invention pertains to computer data transmission methods, and particularly relates to a method for transmitting business protocol data that requires multi-packet transmission. Background Technology

[0002] In non-standalone systems, different computers typically need to exchange information using TCP / IP technology. To differentiate data exchanges for various services, it's necessary to define the format of the exchanged data; in other words, appropriate network protocols need to be specified. A network protocol is a set of rules, standards, or conventions established for data exchange in a computer network. This is a broad concept; for example, the OSI seven-layer model and the TCP / IP four-layer model each have their own protocol support, encompassing not only data formats but also various data exchange process specifications.

[0003] In network data exchange, a fundamental issue that must be addressed is the size of a single data block exchanged. The service protocols requiring multi-packet transmission studied in this invention, commonly referred to as "large protocols," typically refer to service function protocols defined by the service layer whose total size exceeds the size of a single protocol frame, necessitating packet definition and transmission. Depending on different service needs, service layer protocols may define protocols far exceeding the size of a single protocol frame. From the perspective of service function processing, these protocols need to be treated as a single protocol during processing, but during transmission, they manifest as multiple service protocol frames. The protocols mentioned in this invention primarily refer to service layer protocols above the underlying network protocols (such as TCP / IP). Aside from the necessary procedures required for data exchange during service processes, this section mainly discusses the format of protocol data, particularly its data size. Correspondingly, network transmission has the concept of a Maximum Transmission Unit (MTU), used to inform the other party of the maximum size of the data service unit that can be accepted, indicating the effective payload size that the sender can accept.

[0004] MTU (Maximum Transmission Unit) is the maximum length of a packet or frame, usually measured in bytes. Ethernet and 802.3 have MTUs of 1500 bytes and 1492 bytes, respectively. If the length of a data frame to be transmitted by the IP layer is larger than the link layer's MTU, it will be rejected by a router because it cannot handle excessively large packets. In this case, the IP layer needs to perform fragmentation, dividing the datagram into smaller fragments, each smaller than the MTU. If it's too small, the actual amount of data transmitted will be too small because the protocol must add a header to the packet (or frame), making it inefficient. Most operating systems provide a default value that is generally suitable for the user.

[0005] Generally, if there are no special requirements, when many business systems define protocols, they usually choose to design the maximum size of a frame of business data protocol transmission unit between 1000 and 1400 bytes. This is usually sufficient for most protocols, but sometimes the need for large protocol exchange and transmission cannot be completely avoided. Therefore, the design of large protocol packet transmission and cache merging has become a new problem.

[0006] Take a 1MB business protocol as an example. Obviously, for a protocol transmission unit designed to be 1KB in size, the 1MB byte stream needs to be cut into at least 1000 protocol transmission units at the sending end, and in each split packet data, supplementary description information such as protocol sequence number, total number of packets, and packet sequence number needs to be attached to enable caching and splicing at the receiving end, so as to finally achieve the function of transmitting a larger business protocol based on a smaller protocol transmission unit.

[0007] However, there is a relatively fatal hidden danger in this solution.

[0008] Consider the following situation. When a server provides connection services to N terminal devices, each terminal has a certain amount of business to send a business protocol to the server (assuming the data size of the protocol is 1MB). If N is large enough and the large protocol interaction is frequent enough, then there may be enough terminal devices (assuming x, x < N) sending business protocols to the server at almost the same pace.

[0009] A possible scenario is that the server has received business protocols sent by M terminals, but at a certain moment, none of these M business protocols have been received completely. The problem caused at this time is that all the data of the M incomplete protocols that have been received have to stay in the memory cache, waiting for the last packet or several packets of protocol transmission unit data. It can be estimated that the memory cache size required for these incomplete business protocols in the memory is close to x MB in size.

[0010] Assume that N is large enough (for example, 10000), making it possible for x to take the value of 3000. Then the required memory cache size is close to 3GB in size.

[0011] If only considering 64-bit operating systems equipped with more than 8G of memory, a single process running requires 3GB of memory size, which is not impossible to achieve, but it is not reasonable (usually, the memory occupied by 64-bit / 32-bit processes running in a 64-bit operating system is less than 1G, except for the Java virtual machine).

[0012] In addition, if the program needs to be compatible with a 32-bit operating system platform, the user memory space of the 32-bit process cannot exceed 2GB, which will directly determine that the 3GB memory cache requirement cannot be met, and there is a considerable chance that the process will run out of memory, or even crash. Summary of the Invention

[0013] The purpose of this invention is to provide a method for transmitting business protocol data that requires multiple packet transmissions. This method uses a chain of mixed-type protocol transmission unit cache nodes to provide read and write capabilities, thereby alleviating the resource bottleneck caused by insufficient user memory space in the process.

[0014] To achieve the above objectives, the solution of the present invention is:

[0015] A method for transmitting service protocol data requiring multi-packet transmission is provided, which enables the receiver to cache and read data received from protocol transmission units; the receiver is equipped with a first allocator and a second allocator for allocating cache nodes, the first allocator being used to allocate cache nodes from memory, and the second allocator being used to allocate cache nodes from cache files; the method includes the following steps:

[0016] Step A: When the receiver receives the protocol data, the first allocator first allocates a cache node to copy the cached protocol data. If the first allocator fails to allocate the cache node, the second allocator allocates the cache node. If the second allocator also fails to allocate the cache node, it returns an allocation failure signal. If the first or second allocator successfully allocates the cache node, proceed to step B.

[0017] Step B: After the cache node copies and caches the received protocol data, it checks whether the protocol is a sub-packet protocol of the business protocol. If not, it submits it to the upper layer - the business logic processing layer for immediate protocol processing and returns it after processing is completed; if yes, it proceeds to step C.

[0018] Step C: Based on the identifier ID of the sending device at the other end and the serial number of the service protocol, obtain the cache structure of the service protocol, add the latest received sub-packet protocol, and check whether the service protocol has been received completely. If it is complete, submit the cache structure of the service protocol to the upper layer - the business logic processing layer for immediate protocol processing. If it is still not received completely, continue to wait for the reception of new protocol sub-packets.

[0019] In step A above, the first allocator fails to allocate memory when the memory nodes that the first allocator can allocate reach the maximum memory usage configured.

[0020] In step C above, once the business protocol has been fully received and submitted to the upper-level business logic processing layer for immediate protocol processing, each cache node is returned to the corresponding first or second allocator according to its node type, and is reserved as an idle node for subsequent reuse.

[0021] By adopting the above scheme, this invention utilizes disk files as data storage nodes for protocol transmission units. At the same time, it unifies the data read / write interfaces of file-cached protocol transmission unit cache nodes and memory-cached protocol transmission unit cache nodes into a completely consistent form. This allows the business protocol to be provided without being a continuous memory byte stream buffer, but rather by providing read / write access through a chain of hybrid protocol transmission unit cache nodes. Ultimately, this achieves the goal of alleviating the resource bottleneck caused by insufficient user memory space in processes. Attached Figure Description

[0022] Figure 1 This is a flowchart of the present invention;

[0023] Figure 2 It is a multi-cached node linked list storage scheme for business protocols. Detailed Implementation

[0024] The technical solution and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Typical protocol processing generally involves storing protocol data in a contiguous, flat memory region for access. This approach is generally suitable for small protocols (within a few KB) because accessing a specific portion of a contiguous byte array is very straightforward. 、 convenient.

[0026] However, the above method has significant drawbacks for larger protocols. In practice, both small and large protocols may exist. Implementing all protocols by allocating contiguous flat memory regions will lead to memory fragmentation due to inconsistent memory allocation and deallocation sizes. If the protocol is designed to be too large, there is a high probability of memory allocation failure when the process is under memory pressure (for example, in a 32-bit process with complex logic, frequent memory allocation and deallocation, and hundreds of megabytes of memory already allocated after a long period of operation, allocating a 200MB contiguous flat memory block is highly likely to fail, but allocating 200 contiguous flat memory blocks of 1MB each is more likely to succeed). In worse memory fragmentation situations, even allocating a single 1MB contiguous flat memory block is difficult. However, as long as the memory usage of the 32-bit process has not reached the limit of the 32-bit addressable user memory space and there is still some margin, allocating a 1MB contiguous flat memory block may fail, but allocating a sufficient number of contiguous flat memory blocks of only 1KB each is usually feasible, given the same total memory allocation).

[0027] Furthermore, considering that when there are a large number of terminals involved in communication, there may be a large amount of incompletely received business protocol data that needs to be cached. This makes it highly impractical to have each business protocol cache data in a contiguous, flat memory area. Therefore, this invention uses multiple segmented and block-based cache nodes concatenated to form a business protocol cache.

[0028] This invention designs two types of protocol transmission unit data buffer node allocators, in conjunction with Figure 1 As shown, allocator A allocates memory from memory and sets a configurable maximum memory footprint (e.g., 256MB), while allocator B allocates memory from the cache file.

[0029] When the application receives data from a protocol transmission unit, it first allocates a memory cache node from allocator A to copy the temporary data. When the memory node allocated by allocator A reaches its configured maximum memory usage, the allocation fails, and it then tries to allocate a memory cache node from allocator B to copy the temporary data. If allocation from allocator B also fails, the process is completely unsuccessful. In this case, the receiver will retain the currently read data and will not discard it. It will also stop submitting asynchronous read data requests on the TCP link to prevent the read cache from overwriting the data that has not yet been copied and transferred to the protocol cache after receiving new TCP data from the other end. After the protocol data in the historical cache has been processed and some protocol cache nodes have been released, the protocol data in the read cache is sent to the protocol cache, and an asynchronous read data request is submitted on the TCP link to continue receiving new protocol data from the other end.

[0030] After a memory cache node is allocated to copy and cache the received protocol data, it checks whether the protocol is a sub-packet of a business protocol. If not, it can be directly submitted to the upper layer—the business logic processing layer—for immediate protocol processing, and returned after processing. If the currently received protocol is a sub-packet of a business protocol, the cache structure of that business protocol can be obtained based on the identifier ID of the sending device at the other end and the serial number of the business protocol. The latest received sub-packet is added, and it is checked whether the business protocol has been received completely. If it has not been received completely, no further action is taken, and it continues to wait for new protocol sub-packets to arrive from the network. If the corresponding business protocol has been received completely, the cache structure (data node linked list) of that business protocol is submitted to the upper layer—the business logic processing layer—for immediate protocol processing. After processing, each cached data node in the linked list should be returned to the corresponding allocator according to its node type (memory / file) as an idle node for later reuse.

[0031] The technical solution and beneficial effects of the present invention will be described in detail below through a specific embodiment.

[0032] First, make the following settings:

[0033] 1) The maximum limit for the protocol transmission unit size of the service layer protocol is BLOCK_SIZE = 1024 bytes (excluding other underlying protocol headers, referring only to the size of the pure service data).

[0034] 2) The upper limit of free allocation for the memory cache node allocator used for business layer protocol caching is 256MB, so the maximum number of memory cache nodes that can be allocated is 256MB / 1KB = 256*1024.

[0035] 3) A maximum of 32 cache files can be used, and the maximum size of each file used for caching is limited to 2GB.

[0036] Define the maximum number of file blocks that can be allocated for each cache file as MAX_BLOCK_IDX = (2*1024*1024*1024) / BLOCK_SIZE;

[0037] Design of a multi-cached node linked list concatenated storage scheme for business protocols, such as Figure 2 As shown.

[0038] Data nodes are of a hybrid type, which can be either memory-based cache nodes or file-block-based cache nodes. They define some common data structures, including: node type (memory-based / file-block-based), cached data length, and data pointers. Memory-based cache nodes directly define memory blocks for caching data within the node, while file-block-based cache nodes primarily store the file block label `block_idx` (the actual packet protocol data is stored in the corresponding cache file). When it is necessary to read the packet protocol data from a file-block-based cache node, a sufficiently large temporary memory block must first be provided for it, and then the packet protocol data is loaded from the file into this temporary memory area to provide the same packet protocol data reading capability as a memory-block-based cache node.

[0039] Example

[0040] 1. Business Protocol Caching Process

[0041] Step 1. A packetization protocol for a business protocol (packet size not greater than BLOCK_SIZE) is received from the network;

[0042] Step 2. Determine if the number of memory cache nodes allocated (but not yet released) by the memory cache node allocator exceeds 256*1024. If it does, proceed to Step 4; otherwise, continue.

[0043] Step 3. Write the received packet protocol data into the memory cache node allocated in the above steps, then proceed to Step 10;

[0044] Step 4. Set the initial cache file number idx to 0;

[0045] Step5. Increment 1 in idx and examine the idx-th cache file file_idx;

[0046] Step 6. Check if the free block node linked list of the cache file file_idx is empty. If it is not empty, take the next file block cache node from the head of the list (the core information is the label of the file block cache node, block_idx. Based on this label, block_idx, the starting address of the file block cache node in the cache file can be calculated by formula), and go to Step 10. Otherwise, continue.

[0047] Step 7. Determine whether the currently allocated and used maximum file block number cur_max_block_idx (initially 0) is greater than MAX_BLOCK_IDX. If it is greater, go to Step 6; otherwise, continue.

[0048] Step 8. Set the file block cache node with the label cur_max_block_idx as the currently allocable node, and then increment cur_max_block_idx by one;

[0049] Step 9. Write the received packet protocol data into the file block corresponding to the file block cache node (with the corresponding label block_idx) allocated in the above steps. The starting address is calculated as BLOCK_SIZE*block_idx.

[0050] Step 10. Based on the identifier ID of the source device of the sub-packet protocol and the serial number of the service protocol, find the cache data structure of the incomplete service protocol (which can be understood as an array of sub-packet protocols), and add the cache node to the correct position in the cache structure according to the sub-packet sequence number;

[0051] Step 11. Determine whether the number of cached nodes in the cached data structure of the business protocol is equal to the total number of sub-packets of the business protocol. If they are not equal, proceed to Step 1 and continue waiting for the arrival of other sub-packet protocols.

[0052] Step 12. When the number of cached nodes in the cached data structure of the business protocol is equal to the total number of packets of the business protocol, the cached nodes of all packet protocol data are sequentially linked into a data cache node linked list according to the corresponding packet sequence number, and finally the head pointer of the list is submitted to the upper business layer for protocol processing.

[0053] Considering the analysis and processing of the protocol, the core requirement is to extract the necessary N bytes sequentially, which can be designed as follows:

[0054] 2. Business Protocol Reading Process

[0055] Step 1. Receive a complete business agreement;

[0056] Step 2. Initialize the pointer of the currently accessed cache node to the pointer of the head node of the cache node linked list of the business protocol;

[0057] Step 3. Initialize the current read position rnode_pos of the currently accessed cache node to the effective data length of the currently accessed cache node;

[0058] Step 4. Initialize the number of bytes read to 0;

[0059] Step 5. Determine if the number of bytes already read (got_n) is less than the initial number of bytes N to be read. If it is not less, proceed to Step 14; otherwise, continue.

[0060] Step 6. Check if the pointer of the currently accessed cache node is null. If it is null, end; otherwise, continue.

[0061] Step 7. Determine if the length of valid data in the currently accessed cache node is greater than the current read position rnode_pos of the currently accessed cache node. If it is greater, go to Step 10; otherwise, continue.

[0062] Step 8. Assign the pointer of the currently accessed cache node to its successor node pointer, and check if it is null. If it is null, end; otherwise, continue.

[0063] Step 9. Set the current read position of the currently accessed cache node to 0, and go to Step 5;

[0064] Step 10. Calculate the difference between the initial number of bytes N to be read and the number of bytes already read, got_n, to obtain the remaining number of bytes to be read, need_n;

[0065] Step 11. Calculate the difference between the effective data length in the currently accessed cache node and the current read position rnode_pos of the currently accessed cache node to obtain the remaining usable bytes usable_len in the currently accessed cache node;

[0066] Step 11. Take the minimum value of need_n (the remaining number of bytes to be read) and usable_len (the remaining number of available bytes in the current access cache node) as copy_len;

[0067] Step 12. Copy the data buffer of the currently accessed cache node from the current read position rnode_pos to the end of the read buffer, starting from copy_len bytes.

[0068] Step 13. Increase the number of bytes read (got_n) by copy_len bytes, and also increase the current read position (rnode_pos) of the currently accessed cache node by copy_len bytes, then proceed to Step 5;

[0069] Step 14. Determine if the number of bytes read, got_n, is not less than the initial number of bytes to be read, N. If it is not less, the process ends successfully; otherwise, it fails.

[0070] 3. Cache node release process

[0071] The cache node release process is relatively simple, mainly involving two aspects:

[0072] 1) Determine the type of cache node (memory block type / file block type) and select the correct cache node allocator to release it;

[0073] 2) When released to the cache node allocator, the resource is not directly returned to the system, but is instead attached to a free list.

[0074] In summary, the improvements of this invention are reflected in the following aspects:

[0075] (1) When the protocol memory cache reaches the pre-specified size, the received protocol transmission units are cached in the file;

[0076] (2) Divide the 2G file into continuous blocks according to the preset size of the protocol transmission unit, and use a memory block of more than ten bytes to map a file block of 1KB or larger. This will enable the use of a small memory footprint to map and access a file cache space of about 100 times (for example, 100MB of memory data can map and access a file cache space of about 10G) to expand the cache limit of the packet splitting protocol.

[0077] (3) Use a linked list of data cache nodes containing both memory cache type and file cache type to identify a business protocol, instead of a continuous memory byte stream buffer.

[0078] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for transmitting service protocol data that requires multi-packet transmission, used to enable the receiver to buffer and read data received from the protocol transmission unit; characterized in that: The receiver is equipped with a first allocator and a second allocator for allocating cache nodes. The first allocator is used to allocate cache nodes from memory, and the second allocator is used to allocate cache nodes from cache files. The method includes the following steps: Step A: When the receiver receives the protocol data, the first allocator first allocates a cache node to copy the cached protocol data. If the first allocator fails to allocate, the second allocator allocates a cache node. If the second allocator also fails to allocate, an allocation failure signal is returned. If the first or second allocator successfully allocates a cache node, proceed to step B; Step B: After the cache node copies and caches the received protocol data, it checks whether the protocol is a sub-packet protocol of the business protocol. If not, it submits it to the upper layer - the business logic processing layer for immediate protocol processing and returns it after processing is completed. If so, proceed to step C; Step C: Based on the identifier ID of the sending device at the other end and the serial number of the service protocol, obtain the cache structure of the service protocol, add the latest received sub-packet protocol, and check whether the service protocol has been received completely. If it is complete, submit the cache structure of the service protocol to the upper layer - the business logic processing layer for immediate protocol processing. If it is still not received completely, continue to wait for the reception of new protocol sub-packets.

2. The data transmission method for service protocols requiring multi-packet transmission as described in claim 1, characterized in that: In step A, the first allocator fails to allocate memory when the memory nodes that the first allocator can allocate reach the maximum memory usage configured.

3. The data transmission method for service protocols requiring multi-packet transmission as described in claim 1, characterized in that: In step C, once the business protocol has been fully received and submitted to the upper-level business logic processing layer for immediate protocol processing, each cache node is returned to the corresponding first or second allocator according to its node type, and is reserved as an idle node for subsequent reuse.

Images