High-concurrency network file processing system, method, computer device and storage medium

Abstract

Embodiments of the present application provide a high-concurrency network file processing system, method, computer device and storage medium. The system comprises a DPDK data plane module configured to receive network data packets from a network card transceiver queue and store the network data packets in an MBUF memory pool; a coroutine scheduling module configured to preprocess the network data packets based on an event loop and generate corresponding files; a task dispatching module configured to add a file writing task to a task queue and assign the task to a target file cache shard according to system load conditions; a file cache module configured to receive and cache data segments of the writing task and reorder the data segments according to offset information; and an asynchronous I / O module configured to submit the reordered data to a storage device interface in an asynchronous manner. The introduction of the DPDK enables user mode direct access to the network card transceiver queue, avoids the overhead of system calls and kernel protocol stacks, and significantly improves network I / O performance.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and more specifically, to a high-concurrency network file processing system, method, computer device, and storage medium. Background Technology

[0002] With the rapid development of internet and cloud computing technologies, the concurrent access volume of network file processing systems is constantly increasing. The traditional kernel network stack-based input / output I / O model has significant performance bottlenecks in high-concurrency scenarios. In large-scale distributed architectures, systems typically need to handle a large number of remote procedure calls (RPCs) and file stream transfer tasks. However, the traditional epoll / poll model (poll is an I / O multiplexing technique that checks the readiness status of file descriptors by polling, while epoll is an enhanced version of poll) requires frequent system calls and context switching, severely impacting response performance.

[0003] To address this, an asynchronous network model based on coroutines emerged, enabling lightweight scheduling of high-concurrency tasks through user-space coroutines. However, the coroutine model still relies on the kernel network stack, resulting in limited network I / O performance.

[0004] Therefore, the related technologies suffer from input / output I / O performance bottlenecks, insufficient coroutine resource scheduling efficiency, and system blocking risks in high-concurrency network file processing scenarios. Summary of the Invention

[0005] This application provides a high-concurrency network file processing system, method, computer device, and storage medium.

[0006] A first aspect of this application provides a high-concurrency network file processing system, the system comprising: The DPDK data plane module is used to receive network data packets from the network card's transceiver queue, store the network data packets in the MBUF memory pool, and send the network data packets to the coroutine scheduling module. Here, DPDK is the data plane development kit, and MBUF is the core structure in DPDK used to encapsulate network data packets. The coroutine scheduling module is used to receive the network data packets and preprocess the network data packets based on the event loop to generate corresponding files; The task dispatch module is used to add the file writing task to the task queue and distribute the task to the target file cache shard according to the system load through the load balancer. The file caching module is used to receive and cache data fragments of the file's write task using the target file cache fragments in the fragmented cache area, and to reorder the data fragments according to the offset information using the file reorganizer; The asynchronous I / O module is used to submit the reordered data to the storage device interface asynchronously and write it to the underlying storage device. The monitoring and scheduling module is used to obtain the system resource status through the performance monitor, and the dynamic expansion controller adjusts the coroutine scheduling strategy or load balancing strategy according to the status.

[0007] In an optional embodiment of this application, in the DPDK data plane module, the network interface card (NIC) transceiver queue is used to receive and send packets at high speed from the NIC, and the MBUF memory pool is used to manage the data packet buffer and to allocate and release data packet memory without locking.

[0008] In an optional embodiment of this application, the preprocessing of network packets based on the event loop in the coroutine scheduling module is achieved through the following steps: The coroutine manager creates or wakes up the processing coroutines corresponding to network packets and executes the coroutine logic in the event loop to perform protocol parsing, file ID parsing, and offset calculation on the network packets. The coroutine manager is used to manage the life cycle of coroutines and realize cooperative scheduling, while the event loop is used to drive the execution of coroutines and realize I / O or event-driven logic.

[0009] In an optional embodiment of this application, in the task dispatch module, the task queue is used to store write tasks of files to be processed, and the load balancer is used to select the optimal execution path based on CPU, memory, and I / O pressure.

[0010] In an optional embodiment of this application, in the file caching module, the fragmented cache area is used to provide independent cache space for different data fragments of the file, and the file reassembler is used to reassemble the data fragments into a complete file data stream according to the file offset.

[0011] In an optional embodiment of this application, in the asynchronous I / O module, the submission of reordered data to the storage device interface in an asynchronous manner is implemented through an asynchronous write engine, wherein the asynchronous write engine is used to implement batch asynchronous writes using an asynchronous I / O model and an asynchronous I / O interface, and the storage device interface is used to interact with the storage device.

[0012] In an optional embodiment of this application, in the monitoring and scheduling module, adjusting the coroutine scheduling strategy or load balancing strategy according to the status includes: Adjust the task queue weight, coroutine concurrency, or cache shard count according to the state. The performance monitor is used to monitor CPU, queue length, number of coroutines, and I / O pressure in real time. The dynamic expansion controller is used to dynamically adjust the amount of system resources or scheduling strategy based on the monitoring indicators.

[0013] A second aspect of this application provides a high-concurrency network file processing method, including: The DPDK data plane module receives network data packets from the network card's transceiver queue, stores the network data packets in the MBUF memory pool, and sends the network data packets to the coroutine scheduling module. Here, DPDK is the data plane development kit, and MBUF is the core structure in DPDK used to encapsulate network data packets. The network data packets are received using a coroutine scheduling module, and the network data packets are preprocessed based on an event loop to generate corresponding files; The task of writing the file is added to the task queue using the task dispatch module, and the task is distributed to the target file cache shard by the load balancer according to the system load. The file caching module uses the target file cache fragments of the fragmented cache area to receive and cache the data fragments of the file write task, and the file reorderer uses the offset information to reorder the data fragments; The asynchronous I / O module is used to submit the reordered data to the storage device interface asynchronously and write it to the underlying storage device. The monitoring and scheduling module obtains the system resource status through the performance monitor, and the dynamic expansion controller adjusts the coroutine scheduling strategy or load balancing strategy according to the status.

[0014] A third aspect of this application provides a computer device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a high-concurrency network file processing method.

[0015] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a high-concurrency network file processing method.

[0016] Compared with the prior art, the technical solutions provided in this application have at least some or all of the following advantages: The high-concurrency network file processing system described in this application embodiment includes: a DPDK data plane module, used to receive network data packets from the network interface card (NIC) transceiver queue, store the network data packets in the MBUF memory pool, and send the network data packets to a coroutine scheduling module, wherein DPDK is a data plane development kit, and MBUF is the core structure in DPDK used to encapsulate network data packets; a coroutine scheduling module, used to receive the network data packets and preprocess them based on an event loop to generate corresponding files; and a task dispatch module, used to add the file writing tasks to a task queue and distribute the tasks according to the system load through a load balancer. The system includes a target file cache fragment module, a file caching module, and a file reordering module. The file caching module receives and caches data fragments from the target file cache fragments in the fragmented cache area, and reorders these data fragments based on offset information. An asynchronous I / O module submits the reordered data asynchronously to the storage device interface and writes it to the underlying storage device. A monitoring and scheduling module obtains the system resource status through a performance monitor, and the dynamic extension controller adjusts the coroutine scheduling strategy or load balancing strategy based on the status. By introducing DPDK, it enables direct user-space access to the network interface card's transmit / receive queue, avoiding the overhead of system calls and the kernel protocol stack, and significantly improving network I / O performance. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the structure of a high-concurrency network file processing system provided in one embodiment of this application; Figure 2 A flowchart of a coroutine-based task scheduling process is provided as an embodiment of this application; Figure 3 This is a schematic diagram of a DPDK-based data transmission and reception path provided in one embodiment of this application; Figure 4 A flowchart of file fragmentation and cache management provided for one embodiment of this application; Figure 5 A flowchart illustrating a high-concurrency network file processing method provided in one embodiment of this application; Figure 6 This is a schematic diagram of a computer device structure provided in one embodiment of this application. Detailed Implementation

[0018] In the process of developing this application, the inventors discovered that current high-concurrency network file processing methods are poor in terms of input / output I / O performance, coroutine resource scheduling efficiency, and system blocking risk.

[0019] To address the aforementioned issues, this application provides a high-concurrency network file processing system, method, computer device, and storage medium to improve input / output I / O performance and coroutine resource scheduling efficiency.

[0020] The solutions in this application embodiment can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0021] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0022] Please see Figure 1 The high-concurrency network file processing system provided in this application includes: The DPDK data plane module 10 is used to receive network data packets from the network card transceiver queue, store the network data packets in the MBUF memory pool, and send the network data packets to the coroutine scheduling module. Here, DPDK is a data plane development kit, and MBUF is the core structure in DPDK used to encapsulate network data packets. The coroutine scheduling module 20 is used to receive the network data packets and preprocess the network data packets based on the event loop to generate corresponding files; The task dispatch module 30 is used to add the file writing task to the task queue and distribute the task to the target file cache shard according to the system load through the load balancer. The file caching module 40 is used to receive and cache data fragments of the file's write task using the target file cache fragments in the fragmented cache area, and to reorder the data fragments according to the offset information using the file reorganizer. The asynchronous I / O module 50 is used to submit the reordered data to the storage device interface asynchronously and write it to the underlying storage device. The monitoring and scheduling module 60 is used to obtain the system resource status through the performance monitor, and the dynamic expansion controller adjusts the coroutine scheduling strategy or load balancing strategy according to the status.

[0023] In one optional embodiment of this application, the DPDK data plane module is responsible for high-speed reception and transmission of network data packets; the coroutine scheduling module manages file processing coroutines in user space, achieving non-blocking switching. Specifically, a lightweight coroutine scheduler is created in user space, with each coroutine corresponding to an independent file processing task; the coroutine scheduler adopts an event-driven mechanism to achieve non-blocking cooperative scheduling. The coroutine scheduler's task scheduling flowchart based on coroutines is as follows: Figure 2 As shown; the task dispatch module is responsible for allocating data blocks to corresponding coroutines according to file identifiers; the file caching module provides fragment caching and reassembly; the asynchronous I / O module performs asynchronous disk writing; the monitoring and load balancing module dynamically adjusts the coroutine number and CPU core binding strategy, and the coroutine scheduler dynamically creates / destroys coroutines according to CPU utilization and task queue length, realizing multi-core parallelism and NUMA-aware optimization of inconsistent memory access.

[0024] The high-concurrency network file processing system provided in this application combines DPDK with coroutines. In high-concurrency file transmission and streaming parsing scenarios, it coordinates the polling model of DPDK and the scheduling model of coroutines. It can efficiently introduce DPDK into the coroutine architecture, achieve seamless integration of the network layer and the task scheduling layer, and reduce context switching and copying processes by integrating DPDK with coroutines.

[0025] The high-concurrency network file processing system provided in this application embodiment introduces DPDK to enable direct access to the network card's send / receive queue in user space, avoiding the overhead of system calls and kernel protocol stack, significantly improving network I / O performance, and realizing high-speed network access in user space.

[0026] The high-concurrency network file processing system provided in this application adopts a coroutine pool combined with the DPDK polling mechanism. In the coroutine scheduling module, the coroutine scheduler is started to create and manage the coroutine pool. In the task assignment module, the DPDK polling mechanism is combined with the coroutine pool to realize the parallel execution of multiple tasks within a single thread. This enables high-concurrency task processing, avoids thread blocking problems, improves CPU utilization, and realizes coroutine-based high-concurrency scheduling. The coroutine pool and the DPDK polling mechanism work together to ensure high-concurrency network file processing within a single thread.

[0027] The high-concurrency network file processing system provided in this application initializes the MBUF memory pool in the DPDK data plane module and performs zero-copy file processing in the file cache module. Combined with the DPDK MBUF memory pool, it realizes zero-copy file transmission during the reception, parsing, and forwarding of file data, significantly reducing memory bandwidth consumption.

[0028] The high-concurrency network file processing system provided in this application embodiment, in the monitoring and scheduling module, monitors the DPDK queue and the execution status of coroutines, dynamically adjusts the coroutine pool size and CPU core binding relationship, and uses comprehensive strategy scheduling to maintain high performance and load balance, thereby achieving efficient load distribution.

[0029] The high-concurrency network file processing system provided in this application embodiment is applicable to multiple scenarios, and is particularly suitable for high-concurrency file processing scenarios such as electronic record management, cloud storage agent, smart gateway, and edge agent node.

[0030] In an optional embodiment of this application, in the DPDK data plane module, the network interface card (NIC) transceiver queue is used for high-speed packet reception and transmission from the NIC, the MBUF memory pool is used to manage the data packet buffer, and for lock-free allocation and release of data packet memory. Data packets are passed to the coroutine scheduling module via pre-registered callbacks. A schematic diagram of the DPDK-based data transmission path is shown below. Figure 3 As shown.

[0031] In an optional embodiment of this application, the preprocessing of network packets based on the event loop in the coroutine scheduling module is achieved through the following steps: The coroutine manager creates or wakes up the processing coroutines corresponding to network packets and executes the coroutine logic in the event loop to perform protocol parsing, file ID parsing, and offset calculation on the network packets. The coroutine manager is used to manage the life cycle of coroutines and realize cooperative scheduling, while the event loop is used to drive the execution of coroutines and realize I / O or event-driven logic.

[0032] In an optional embodiment of this application, in the task dispatch module, the task queue is used to store the file writing tasks to be processed, and the load balancer is used to select the optimal execution path based on CPU, memory, and I / O pressure. The coroutine submits the parsed file writing tasks to the task queue, and the load balancer allocates the tasks to the corresponding file cache shards according to the system load, thereby improving the processing parallelism in multi-file / large file scenarios.

[0033] In an optional embodiment of this application, in the file caching module, the fragmented cache area is used to provide independent cache space for different data fragments of a file, and the file reassembler is used to reassemble the data fragments into a complete file data stream according to the file offset. The file write tasks in the task queue are assigned to the fragmented cache area for temporary storage, and the file reassembler reorders the data fragments according to the file offset and sequence number to provide continuous data blocks for the asynchronous write module. The file fragmentation and cache management flowchart is shown below. Figure 4 As shown.

[0034] In an optional embodiment of this application, the file reorderer reorders data segments based on file offset and sequence number, including: After the data fragment arrives, the data fragment is written to the fragmented buffer (inserted into an ordered structure according to the offset) and the bitmap is updated. After all the data fragments in all the fragmented buffers have arrived, the data fragments in all the fragmented buffers are traversed in the order of the offset and concatenated into a continuous fragmented buffer. The asynchronous write task of delivering data fragments according to the continuous fragmented buffer is then performed.

[0035] In an optional embodiment of this application, the basic file information is as follows: File ID: file_123; File size: 10 KB; Shard size: 2 KB; Total number of shards: 5 (numbered 0-4). Shard arrival scenario: Shard 2 → Shard 0 → Shard 3 → Shard 1 → Shard 4. Shard cache context (by file): Using offset as the sorting key, a red-black tree is the core of the reordering. The steps for reordering the data fragments of the 5 shards are as follows: Step 1: Receive fragmented data segments → Write to fragmented buffer Taking the data segment of fragment 2 as an example: block_no = 2 offset = 2 2048 = 4096 implement: block = alloc_block_node(); block->block_no = 2; block->offset = 4096; block->len = 2048; block->data = malloc(2048); memcpy(block->data, payload, 2048); rb_insert(&ctx->block_tree, block->offset, block); set_bit(ctx->recv_bitmap, 2); ctx->recv_blocks++; At this point, the logical order of the cache (by offset) is: [4KB] Step 2: Continue receiving out-of-order fragments and inserting them. Receive fragment 0: offset = 0 KB After automatic sorting by red-black tree: [0KB][4KB] Receive fragment 3: offset = 6 KB Sorting results: [ 0KB ][ 4KB ][ 6KB ] Receive fragment 1: offset = 2 KB Sorting results: [ 0KB ][ 2KB ][ 4KB ][ 6KB ] Receive fragment 4: offset = 8 KB Final sort: [ 0KB ][ 2KB ][ 4KB ][ 6KB ][ 8KB ] Determine whether recombination can be triggered bitmap judgment: if (bitmap_all_set(ctx->recv_bitmap, total_blocks)) { / / ready } The following is the specific execution process of file reorganization. Step 3: Sequentially traverse the cache area (core reordering point) uint32_t expect_offset = 0; rb_for_each(node, ctx->block_tree) { if (node->offset != expect_offset) { / / Fragmentation not contiguous, exception return ERROR; } memcpy(rebuild_buf + expect_offset, node->data, node->len); expect_offset += node->len; } This does not rely on the "receive order," but entirely on the offset sorting. The following provides contiguous data blocks for the asynchronous write module. Step 4: Submit the write task async_write_task task; task.file_id = ctx->file_id; task.buf = rebuild_buf; task.len = total_file_size; push_to_write_queue(&task); In an optional embodiment of this application, in the asynchronous I / O module, the asynchronous submission of reordered data to the storage device interface is implemented through an asynchronous write engine. The asynchronous write engine is used to implement batch asynchronous writes using the asynchronous I / O model io_uring and the asynchronous I / O interface AIO. The storage device interface is used to interact with the storage device. The reorganized data blocks are submitted to the storage device interface asynchronously by the asynchronous write engine, so that the write process does not block the business thread and the file data is persisted to the underlying storage device.

[0036] In an optional embodiment of this application, in the monitoring and scheduling module, adjusting the coroutine scheduling strategy or load balancing strategy according to the status includes: The system adjusts the task queue weight, coroutine concurrency, or cache shard count based on the stated status. The performance monitor monitors CPU, queue length, number of coroutines, and I / O pressure in real time. The dynamic scaling controller dynamically adjusts the system resource quantity or scheduling strategy based on the monitoring indicators. The performance monitor collects system operation indicators in real time and feeds them back to the scheduling module. The dynamic scaling controller adjusts the task queue weight, coroutine concurrency, or cache shard count based on system pressure to achieve automatic load balancing and elastic scaling.

[0037] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0038] Please see Figure 5One embodiment of this application provides a high-concurrency network file processing method, including: S510 uses the DPDK data plane module to receive network data packets from the network card transceiver queue, stores the network data packets in the MBUF memory pool, and sends the network data packets to the coroutine scheduling module. Here, DPDK is the data plane development kit, and MBUF is the core structure in DPDK used to encapsulate network data packets. S520, the coroutine scheduling module receives the network data packets and preprocesses the network data packets based on the event loop to generate corresponding files; S530, the task of writing the file is added to the task queue by the task dispatch module, and the task is distributed to the target file cache fragment by the load balancer according to the system load. S540, the file caching module uses the target file cache fragment of the fragmented cache area to receive and cache the data fragments of the file write task, and the file reorderer uses the offset information to reorder the data fragments; The S550 uses an asynchronous I / O module to submit the reordered data to the storage device interface asynchronously and write it to the underlying storage device. S560 uses the monitoring and scheduling module to obtain the system resource status through the performance monitor, and the dynamic expansion controller adjusts the coroutine scheduling strategy or load balancing strategy according to the status.

[0039] In an optional embodiment of this application, in step S510, the method further includes: Initialize the DPDK environment and memory pool: The DPDK environment is initialized through rte_eal_init (rte_eal_init is the core initialization function in DPDK, responsible for the initialization of the environment abstraction layer), and an MBUF memory pool is created for high-speed caching and reuse of network packets.

[0040] In an optional embodiment of this application, in step S520, the network data packets are received using a coroutine scheduling module, and the network data packets are preprocessed based on an event loop to generate corresponding files, including: Create a lightweight coroutine scheduler in user space; Each coroutine corresponds to an independent file processing task; The scheduler uses an event-driven mechanism to achieve non-blocking cooperative scheduling.

[0041] In an optional embodiment of this application, step S530, adding the file writing task to the task queue, includes: Batch receive network file data via the DPDK network card packet receiving interface; Data is dispatched to the corresponding coroutine task queue based on the data packet header information (file ID, block ID).

[0042] In an optional embodiment of this application, in step S540, the file caching module uses the target file cache fragment of the fragmented cache area to receive and cache the data fragment of the file write task, including: The coroutine retrieves file data blocks from the task queue; Perform fragment verification, reassembly, and cache writing; The zero-copy mechanism is used to directly map data to the user-space cache.

[0043] In an optional embodiment of this application, in step S550, the reordered data is submitted asynchronously to the storage device interface using an asynchronous I / O module and written to the underlying storage device, including: Data is written to the storage medium using asynchronous I / O via coroutines; A confirmation packet was sent back to the client synchronously, confirming that the block was written successfully.

[0044] In an optional embodiment of this application, in step S560, the monitoring and scheduling module obtains the system resource status through a performance monitor, and the dynamic expansion controller adjusts the coroutine scheduling strategy or load balancing strategy according to the status, including: Dynamically create / destroy coroutines based on CPU utilization and task queue length; Achieve multi-core parallelism and NUMA-aware optimization.

[0045] The high-concurrency network file processing method of this application improves the processing efficiency of network files by reducing data copying and context switching.

[0046] In an optional embodiment of this application, the high-concurrency network file processing method of this application is applied to the hierarchical management of a molecular dynamics computing cluster, wherein the network interface card (NIC) transceiver queue is used to propagate change events. For each node in the three-tier architecture of the molecular dynamics computing cluster, when the current node is started, it obtains configuration information from the parent node of the current node, establishes the version number, change log and event subscription relationship in the configuration information, wherein the three-tier architecture includes management nodes, service nodes and computing nodes; When the configuration information of the current node changes, a change event and its corresponding version number are generated, a change log is recorded based on the change event, and the change event is propagated through the event bus; Periodically check whether the configuration version number of the current node is consistent with that of its parent node; If the configuration version number of the current node is inconsistent with that of its parent node, a synchronization request is sent to the parent node, wherein the synchronization request carries the configuration version number of the current node; Based on the configuration version number of the current node, incremental data is extracted from the parent node, enabling the current node to apply incremental configuration changes to its configuration information. The incremental data is determined based on the configuration version numbers of the current node and its parent node.

[0047] In an optional embodiment of this application, for each node in the three-tier architecture of the molecular dynamics computing cluster, when the current node is started, obtaining configuration information from the parent node of the current node and establishing the version number, change log, and event subscription relationship in the configuration information includes: In the three-tier architecture of the molecular dynamics computing cluster, the management node layer, service node layer, and computing node layer each manage their own configuration domain independently. The service node configuration inherits the configuration template of the management node and overrides the first preset configuration item, while the computing node configuration inherits the configuration template of the service node and overrides the second preset configuration item.

[0048] In an optional embodiment of this application, the step of generating a change event and its corresponding version number when the configuration information of the current node changes, recording a change log based on the change event, and propagating the change event through the event bus includes: When the configuration information of the management node changes, a first change event is generated and pushed to all service nodes through the event bus. The service nodes receive the first change event, update their local configuration, and pass the first change event to the compute nodes for local configuration updates. After modifying their local configuration, each node replies with a confirmation message. The first change event includes a version number, change content, and timestamp. The change content is recorded in the change log. When the configuration information of the compute node changes, a second change event is generated and pushed to all service nodes through the event bus. The service nodes determine whether to synchronize with the service nodes, compute nodes and management nodes based on the changes in the second change event.

[0049] In an optional embodiment of this application, the process of determining whether to synchronize with the service node, compute node, and management node based on the change content of the second change event at the service node includes: If the change in the second change event is to expand the memory of the compute node, the change content of the second change event will be synchronized to the management node; If the change in the second change event is a change in the location of the temporary directory of the compute node, there is no need to synchronize the change in the second change event to the management node.

[0050] In an optional embodiment of this application, incremental data is extracted from the parent node based on the configuration version number of the current node, including: Extract all changes from the current node onwards from the change log of the parent node, and treat all changes as incremental data.

[0051] In an optional embodiment of this application, the method further includes: When the configuration items of the first compute node are modified, the version number and timestamp of the first compute node are V1 and T1, respectively. After the modified configuration items are propagated to the second compute node, the second compute node modifies the configuration items based on the version number V1. The version number and timestamp of the second compute node are V2 and T2, respectively. V2>V1 and T2>T1. The version numbers V1 and V2 are in an increasing relationship, and it is determined that there is no conflict in the configuration of the compute node. When the configuration items of the first computing node are modified, the version number and timestamp of the first computing node are V1 and T1 respectively. After the modified configuration items are propagated to the second computing node, the second computing node modifies the configuration items based on the version number V1. The version number and timestamp of the second computing node are V1 and T2 respectively. It is determined that there is a first conflict in the configuration of the computing node. When the global version number is V4, the third compute node receives an event with version number V3. It determines that V4 > V3, and therefore determines that there is a second conflict in the compute node's configuration. There is no need to modify the configuration items of the third compute node, and it obtains the latest configuration information from the superior node.

[0052] In an optional embodiment of this application, the method further includes: In the event of a first conflict in the configuration of compute nodes, the first conflict can be resolved using one of the following strategies: timestamp priority strategy, node priority strategy, merging strategy, or manual resolution.

[0053] For specific limitations on the above methods, please refer to the limitations on high-concurrency network file processing systems mentioned above, which will not be repeated here.

[0054] In one embodiment, a computer device is provided, the internal structure of which can be as follows: Figure 6As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a high-concurrency network file processing method as described above. It includes: memory and a processor; the memory stores the computer program; and the processor executes the computer program to implement any step of the high-concurrency network file processing method described above.

[0055] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can perform any of the steps in the high-concurrency network file processing method described above.

[0056] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0061] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A high-concurrency network file processing system, characterized in that, The system includes: The DPDK data plane module is used to receive network data packets from the network card's transceiver queue, store the network data packets in the MBUF memory pool, and send the network data packets to the coroutine scheduling module. Here, DPDK is the data plane development kit, and MBUF is the core structure in DPDK used to encapsulate network data packets. The coroutine scheduling module is used to receive the network data packets and preprocess the network data packets based on the event loop to generate corresponding files; The task dispatch module is used to add the file writing task to the task queue and distribute the task to the target file cache shard according to the system load through the load balancer. The file caching module is used to receive and cache data fragments of the file's write task using the target file cache fragments in the fragmented cache area, and to reorder the data fragments according to the offset information using the file reorganizer; The asynchronous I / O module is used to submit the reordered data to the storage device interface asynchronously and write it to the underlying storage device. The monitoring and scheduling module is used to obtain the system resource status through the performance monitor, and the dynamic expansion controller adjusts the coroutine scheduling strategy or load balancing strategy according to the status.

2. The method according to claim 1, characterized in that, In the DPDK data plane module, the network interface card (NIC) transceiver queue is used for high-speed packet reception and transmission from the NIC, and the MBUF memory pool is used to manage the data packet buffer and for lock-free allocation and release of data packet memory.

3. The method according to claim 1, characterized in that, In the coroutine scheduling module, the preprocessing of network packets based on the event loop is achieved through the following steps: The coroutine manager creates or wakes up the processing coroutines corresponding to network packets and executes the coroutine logic in the event loop to perform protocol parsing, file ID parsing, and offset calculation on the network packets. The coroutine manager is used to manage the life cycle of coroutines and realize cooperative scheduling, while the event loop is used to drive the execution of coroutines and realize I / O or event-driven logic.

4. The method according to claim 1, characterized in that, In the task dispatch module, the task queue is used to store file writing tasks to be processed, and the load balancer is used to select the optimal execution path based on CPU, memory, and I / O pressure.

5. The method according to claim 1, characterized in that, In the file caching module, the segmented cache area is used to provide independent cache space for different data segments of the file, and the file reassembler is used to reassemble the data segments into a complete file data stream according to the file offset.

6. The method according to claim 1, characterized in that, In the asynchronous I / O module, the asynchronous submission of reordered data to the storage device interface is achieved through an asynchronous write engine. The asynchronous write engine is used to implement batch asynchronous writes using an asynchronous I / O model and an asynchronous I / O interface, and the storage device interface is used to interact with the storage device.

7. The method according to claim 1, characterized in that, In the monitoring and scheduling module, adjusting the coroutine scheduling strategy or load balancing strategy according to the status includes: Adjust the task queue weight, coroutine concurrency, or cache shard count according to the state. The performance monitor is used to monitor CPU, queue length, number of coroutines, and I / O pressure in real time. The dynamic expansion controller is used to dynamically adjust the amount of system resources or scheduling strategy based on the monitoring indicators.

8. A high-concurrency network file processing method, characterized in that, include: The DPDK data plane module receives network data packets from the network card's transceiver queue, stores the network data packets in the MBUF memory pool, and sends the network data packets to the coroutine scheduling module. Here, DPDK is the data plane development kit, and MBUF is the core structure in DPDK used to encapsulate network data packets. The network data packets are received using a coroutine scheduling module, and the network data packets are preprocessed based on an event loop to generate corresponding files; The task of writing the file is added to the task queue using the task dispatch module, and the task is distributed to the target file cache shard by the load balancer according to the system load. The file caching module uses the target file cache fragments of the fragmented cache area to receive and cache the data fragments of the file write task, and the file reorderer uses the offset information to reorder the data fragments; The asynchronous I / O module is used to submit the reordered data to the storage device interface asynchronously and write it to the underlying storage device. The monitoring and scheduling module obtains the system resource status through the performance monitor, and the dynamic expansion controller adjusts the coroutine scheduling strategy or load balancing strategy according to the status.

9. A computer device, comprising: A memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the high-concurrency network file processing method of claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the high-concurrency network file processing method of claim 8.

Images