Distributed RDMA model parallel processing method and system based on hardware multithreading

Abstract

The invention discloses a distributed RDMA model parallel processing method and system based on hardware multithreading, and relates to the technical field of computer network communication and data processing, and the method comprises the steps: receiving a plurality of messages to be processed, and analyzing the messages to extract key information; according to the key information, loading the received message into a corresponding thread; according to the thread corresponding to each message, performing multi-stage task processing on the message through a preset processing rule to obtain a processing result; and sending the processing result to a corresponding destination node according to a preset sending strategy. According to the method, efficient parallel processing of tasks can be achieved, hardware multi-thread resources are fully utilized, the processing efficiency and throughput of a system are improved, and the requirements of high-performance calculation and large-scale data processing are effectively met. Meanwhile, dynamic load balancing is achieved, a task allocation strategy is adjusted in real time, the resource utilization rate is increased, and system performance reduction caused by single-thread overload is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of computer network communication and data processing, and in particular to a distributed RDMA model parallel processing method and system based on hardware multithreading. Background Art

[0002] With the rapid development of the internet and the explosive growth of data, traditional network data processing methods are increasingly facing challenges such as low processing efficiency, high latency, and uneven resource utilization when faced with the demands of large-scale distributed systems and high-performance computing. As a highly efficient network communication technology, Remote Direct Memory Access (RDMA) enables direct data transfer between memories without passing through the operating system kernel, significantly reducing latency and improving bandwidth utilization.

[0003] However, existing RDMA systems are mostly based on single-core or single-thread designs, making it difficult to fully utilize the parallel computing capabilities of modern multi-core processors. In multi-core environments, competition for RDMA resources among multiple threads leads to a significant increase in context switching overhead. This is especially true in distributed scenarios, where cross-node thread synchronization and load balancing lead to low parallel efficiency. Summary of the Invention

[0004] In view of the problem of low parallel efficiency caused by cross-node thread synchronization and load balancing in distributed scenarios in the prior art, the present invention provides a distributed RDMA model parallel processing method and system based on hardware multithreading. The specific technical solution is as follows:

[0005] A distributed RDMA model parallel processing method based on hardware multithreading, comprising:

[0006] Receive multiple messages to be processed and parse them to extract key information;

[0007] Loading the received message into the corresponding thread according to the key information;

[0008] According to the thread corresponding to each message, the message is processed in multiple stages according to the preset processing rules to obtain the processing results;

[0009] The processing results are sent to the corresponding destination node according to the preset sending strategy.

[0010] Preferably, the key information includes the type, destination address and priority of the message.

[0011] Preferably, the step of loading the received message into the corresponding thread according to the key information includes:

[0012] According to the type, destination address and priority of the message in the key information and the load of the current thread, the optimal thread is matched from the processor's resource pool to process the corresponding message, and the processed message is loaded into the corresponding thread.

[0013] Preferably, the multi-stage task processing is performed on the message according to the thread corresponding to each message according to the preset processing rules, and the processing results obtained include:

[0014] According to the thread corresponding to each message, the message is processed through the preset processing rules, including data verification, address translation, protocol analysis, flow control and priority scheduling stages;

[0015] The task processing results of each stage are used as the input of the next stage until the task processing of all stages is completed and the final processing results are obtained.

[0016] Preferably, the distributed RDMA model parallel processing method based on hardware multithreading also includes:

[0017] When performing multi-stage task processing on messages, the thread allocation and task scheduling strategies are dynamically adjusted by sharing the memory area between threads and based on the thread execution status and feedback information.

[0018] Preferably, dynamically adjusting thread allocation and task scheduling strategies based on thread execution status and feedback information includes:

[0019] Monitor the load of each thread and the progress of task execution in real time. When it detects that the load rate of a thread reaches a preset threshold or the task backlog reaches a preset number, it transfers some tasks to the thread with the lowest load rate or the lowest task backlog.

[0020] According to the changes in network traffic and the priority of the message, the priority and weight of task scheduling are adjusted, and the highest priority message is processed and transmitted first.

[0021] Preferably, the preset sending strategy includes routing selection based on destination address, path optimization based on traffic engineering, or priority scheduling based on quality of service.

[0022] A distributed RDMA model parallel processing system based on hardware multithreading is applied to the aforementioned distributed RDMA model parallel processing method based on hardware multithreading, comprising:

[0023] A parsing unit, configured to receive multiple messages to be processed and parse the messages to extract key information;

[0024] A scheduling unit, configured to load the received message into a corresponding thread according to the key information;

[0025] The processing unit is used to perform multi-stage task processing on the message according to the thread corresponding to each message and the preset processing rules to obtain the processing result;

[0026] The sending unit is used to send the processing result to the corresponding destination node according to a preset sending strategy.

[0027] An electronic device, comprising:

[0028] processor;

[0029] a memory for storing processor-executable instructions;

[0030] The processor implements the aforementioned hardware multi-threaded distributed RDMA model parallel processing method by running the executable instructions.

[0031] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the steps of the aforementioned hardware multi-threaded distributed RDMA model parallel processing method.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention provides a distributed RDMA model parallel processing method based on hardware multi-threading. The method receives multiple messages to be processed and parses the messages to extract key information. According to the key information, the received messages are loaded into corresponding threads, and each thread focuses on processing a specific type of message to avoid resource competition. According to the thread corresponding to each message, the message is subjected to multi-stage task processing according to a preset processing rule to obtain a processing result. The processing result is sent to the corresponding destination node according to a preset sending strategy. The present invention can realize efficient parallel processing of tasks, improve the processing efficiency and throughput of the system by making full use of hardware multi-threading resources, and effectively meet the needs of high-performance computing and large-scale data processing. At the same time, dynamic load balancing can adjust the task allocation strategy in real time according to the execution status and feedback information of the thread, ensure the load balance of each thread, improve resource utilization, and avoid system performance degradation caused by single-thread overload. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0035] Figure 1 This is a flow chart of the distributed RDMA model parallel processing method based on hardware multithreading of the present invention.

[0036] Figure 2 This is a schematic diagram of the distributed RDMA model parallel processing system based on hardware multithreading of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0039] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0040] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0041] Please refer to the following examples Figure 1 and Figure 2 .

[0042] The present invention provides a distributed RDMA model parallel processing method based on hardware multithreading, including:

[0043] Step S1: receiving multiple messages to be processed and parsing the messages to extract key information;

[0044] RDMA packets are received using high-speed network interfaces, such as RoCE networks. RoCE network cards enable remote direct memory access (RDMA) in Ethernet environments, providing low-latency, high-bandwidth data transmission. Hardware queues (such as RxQueuePair) are used to acquire data in batches, improving efficiency and reducing overhead. During the receiving process, the network card continuously monitors the network interface. When an RDMA packet arrives, it is stored in the RxQueuePair for further processing.

[0045] By extracting RDMA protocol fields (such as Opcode, Flags, LID / GID, QPN, SQN / RQN) from the received RDMA message, the Opcode identifies the type of RDMA operation (such as Send or Write); Flags contains some operation flags that indicate special attributes of the operation; LID (Local Identifier) ​​and GID (Global Identifier) ​​identify nodes in the network; QPN (QueuePairNumber) is the queue pair number used to identify a specific queue pair; SQN (SendQueueNumber) and RQN (ReceiveQueueNumber) represent the send queue and receive queue numbers, respectively. The destination node identifier, data buffer address, length, checksum, and priority are extracted based on the RDMA operation type (Send / Write / Read / Atomic).

[0046] Destination node identifier (IP+Port / QPN). By combining this information, the destination node can be uniquely identified, so that the message can be sent to the correct node. Operation type (such as Write with Immediate) specifies the specific operation type of the message. Data buffer pointer and priority tag, etc. The data buffer pointer is used to point to the location where data is stored, and the priority tag can be used to prioritize messages during processing.

[0047] Step S2: Load the received message into the corresponding thread according to the key information;

[0048] The scheduler selects threads based on the parsed message information. Message information includes the operation type, destination address, priority, and more. Different operation types may require different processing resources and logic. For example, Write and Read operations may be handled differently.

[0049] The destination address can be used to determine the target node of the message so that the thread related to the node can be selected for processing. The priority tag can be used to determine the order in which the messages are processed, and high-priority messages should be processed first.

[0050] When selecting a thread, consider the current thread load. Thread load can be measured using various metrics, such as CPU utilization and memory usage. If a thread is already highly loaded, avoid assigning new messages to it to prevent thread overload.

[0051] A weighted round-robin strategy is used to achieve load balancing and optimal resource utilization. In weighted round-robin, each thread is assigned a weight, which is determined based on factors such as the thread's processing power and current load. Threads with high processing power and low load have higher weights and a higher probability of being selected. The scheduler selects threads to process packets based on their weights, achieving load balancing.

[0052] The scheduler selects the optimal thread from the processor's resource pool to process the parsed message based on the parsed message information. The thread selection takes into account the message type, destination address, priority, and current thread load, and adopts a weighted round-robin strategy to achieve load balancing and optimal resource utilization.

[0053] Step S3: According to the thread corresponding to each message, the message is processed in multiple stages according to the preset processing rules to obtain the processing result;

[0054] The processor performs multi-stage task processing according to the preset processing order and rules based on the thread corresponding to each message. This multi-stage task processing includes operations such as data verification, address translation, protocol parsing, and flow control. The processing logic and rules at each stage are customized and optimized according to the specific needs of the distributed storage system. Different distributed storage systems may have different performance requirements and application scenarios, and the processing logic needs to be adjusted according to actual conditions. For example, for systems with high latency requirements, the data verification and protocol parsing algorithms can be optimized to reduce processing time; for systems with high throughput requirements, the flow control and address translation logic can be optimized to improve data transmission efficiency.

[0055] Step S4: Send the processing result to the corresponding destination node according to the preset sending strategy.

[0056] The preset sending strategy is determined by the destination address of the message and system requirements. The destination address can be used to determine the next-hop device or destination node of the message. The sending strategy can include selecting the optimal network path and adjusting the sending rate based on network load. For example, for certain latency-sensitive messages, a low-latency network path can be selected for transmission; for messages with large data volumes, flow control can be implemented based on network load to avoid network congestion.

[0057] After processing, packets are sent to the corresponding next-hop device or destination node via the network card based on their destination address and the preset delivery policy. The network card encapsulates the packets into appropriate network data packets and selects the appropriate network interface for delivery based on the delivery policy. During delivery, the network card may also perform additional processing, such as adding network and transport layer header information. Furthermore, it collects statistics and counts on successfully and unsuccessfully delivered packets, generating corresponding performance metrics and log information.

[0058] Specifically, the preset sending strategy includes routing selection based on destination address, path optimization based on traffic engineering, or priority scheduling based on quality of service.

[0059] The routing process based on the destination address is as follows:

[0060] When a router receives a data packet, it first checks the destination IP address. The router then searches its routing table for a route that matches the destination IP address. Routing table entries typically include information such as the destination network address, subnet mask, next-hop address, and outbound interface. If multiple matching routes exist, the router selects the optimal path based on the longest match, routing protocol priority, and route cost. Based on the selected route, the packet is forwarded to the corresponding outbound interface and continues on to the destination address using the next-hop address.

[0061] The implementation process of path optimization based on traffic engineering is as follows:

[0062] Each node in the network collects traffic engineering information, such as link status and bandwidth, and distributes this information to other nodes through extended IGP protocols (such as OSPF-TE), building a traffic engineering database (TEDB). The Constrained Shortest Path First (CSPF) algorithm combines the TEDB data with specified constraints (such as bandwidth and delay) to calculate a path that meets the requirements. Constrained Routing Label Switched Path (CR-LSP) tunnels are established using the RSVP-TE signaling protocol. RSVP-TE signaling carries constraints such as tunnel bandwidth and explicit paths, enabling dynamic path establishment. Qualified traffic is introduced into the established CR-LSP tunnel for MPLS forwarding. Traffic can be introduced into the tunnel using static routing or policy-based routing.

[0063] The implementation process of priority scheduling based on quality of service is as follows:

[0064] Traffic entering the network is classified and assigned to different classes or queues based on predefined criteria such as IP address, port number, and DSCP value. Data packets are then tagged with a different type of service (DTS) tag. Priority queues are set for different types of traffic based on business needs and QoS policies. For example, voice and video traffic with high real-time requirements can be placed in a high-priority queue, while file download traffic that is less sensitive to latency can be placed in a low-priority queue. In the output queue, the order in which data packets are sent is determined by a corresponding scheduling algorithm (such as priority scheduling) based on the priority queue settings. Traffic shaping techniques (such as the token bucket algorithm) can also be used to control the rate and pattern of data packet transmission to avoid network congestion. QoS policies are applied through policy enforcement points on network devices (such as routers and switches), and network status and traffic conditions are continuously monitored to adjust policy configurations based on actual needs.

[0065] In specific implementation, a corresponding sending strategy can be selected according to actual needs.

[0066] Specifically, in a preferred embodiment of the present application, loading the received message into the corresponding thread according to the key information includes:

[0067] According to the type, destination address and priority of the message in the key information and the load of the current thread, the optimal thread is matched from the processor's resource pool to process the corresponding message, and the message is loaded into the corresponding thread.

[0068] In practice, load balancing requires real-time collection of the load of each thread in the processor resource pool. This load can be measured using a variety of metrics, including CPU utilization, memory usage, and task queue length. Load information is obtained through system calls or monitoring tools and updated in real time to a load information table.

[0069] In this embodiment, a weighted round-robin strategy is used to match the optimal thread from the processor resource pool based on key information and thread load. The specific steps are as follows:

[0070] 1) Filter available threads: Filter out currently available threads from the resource pool.

[0071] 2) Calculate the matching score: For each available thread, a matching score is calculated based on the message type, destination address, priority, and thread load. The overall matching score is calculated based on message type matching, destination address matching, and priority matching. The specific calculation is as follows:

[0072] S total (i) = S type (i)ω1+S addr (i)ω2+Spri (i)ω3

[0073] Where S total (i) indicates the total matching score of message type i; S type (i) S addr (i) and S pri (i) represents the message type matching score, destination address matching score and priority adjustment score respectively; ω1, ω2 and ω3 represent the message type matching weight, destination address matching weight and priority adjustment weight respectively.

[0074] 3) Select the optimal thread: Select the thread with the highest matching score as the optimal thread to process the message.

[0075] When the optimal thread is determined, the message is loaded into the task queue of the thread.

[0076] In this embodiment, by comprehensively considering the message type, destination address, priority, and thread load, a weighted round-robin strategy is adopted for thread matching. This evenly distributes messages to each thread, avoiding situations where some threads are overloaded while others are idle, thereby achieving load balancing, improving processor resource utilization, and reducing system response time and processing delays. Furthermore, the appropriate thread is selected for processing based on the message type and destination address, improving processing efficiency. For example, for certain message types that require specific algorithms or data structures, assigning them to threads that excel at processing these types of messages can reduce processing time and resource consumption. High-priority messages are preferentially assigned to threads with lighter loads, ensuring that high-priority messages are processed promptly.

[0077] Specifically, in a preferred embodiment of the present application, the multi-stage task processing is performed on the message according to the thread corresponding to each message according to the preset processing rules, and the processing results obtained include:

[0078] According to the thread corresponding to each message, the message is processed through the preset processing rules, including data verification, address translation, protocol analysis, flow control and priority scheduling stages;

[0079] In this embodiment, the following steps are used for processing, including:

[0080] The first step, data verification, uses algorithms such as CRC (Cyclic Redundancy Check) to calculate the checksum of the message data and compare it with the checksum carried in the message. The check also checks whether the message format complies with the RDMA protocol specification. If the check passes, the message is marked as valid and passed to the next stage. If the check fails, the message is processed according to pre-set rules, discarded, an error log is recorded, and the sender is notified to retransmit.

[0081] The second step, address translation, is to receive packets that have passed verification in the previous step. In a distributed system, packets may carry logical addresses, which need to be translated into physical addresses by querying an address mapping table. For packets that communicate across networks, network address translation (NAT) is performed. The translated address information is then updated in the packet and passed to the protocol parsing stage.

[0082] Step 3: Protocol Parsing: Receive the message after address conversion. Determine the specific RDMA operation type (Send, Write, Read, or Atomic) based on the Opcode field in the message. Parse other protocol fields in the message, such as the Flags field, LID / GID, QPN, SQN / RQN, etc., to obtain detailed operation information and parameters. Create a processing context based on the parsing results and store the key information of the message in the context. Pass the parsed context and message to the flow control stage.

[0083] Step 4, flow control phase: Receive the context and message delivered during the protocol parsing phase. Check the local system's resource usage, such as CPU utilization, memory usage, and network bandwidth. If local resources are limited, it may be necessary to limit the message processing speed. By monitoring network status, such as packet loss rate and latency, determine whether the network is congested. If the network is congested, appropriate flow control measures need to be taken. A control strategy is implemented, and based on the evaluation results, the message sending rate is limited. A sliding window protocol is used to dynamically adjust the sending window size based on the receiver's feedback to control the amount of data sent. After flow control processing, the message is passed to the priority scheduling phase.

[0084] Step 5, Priority Scheduling: Receive the message and context processed in the flow control phase. The message's priority is assessed based on the priority tag carried in the message and information such as the operation type obtained previously. Messages are placed into different queues based on their priority, with high-priority messages being processed first. System resources, such as CPU time slices and network bandwidth, are allocated to each message based on its priority. For example, high-priority messages receive preferential CPU processing. After priority scheduling, the thread performs final processing on the message, completing the entire message processing process and obtaining the processing result.

[0085] The task processing results of each stage are used as the input of the next stage until the task processing of all stages is completed and the final processing results are obtained.

[0086] In this embodiment, through multi-stage task processing, each stage can process different messages in parallel, improving the overall processing capacity and throughput of the system. At the same time, each stage focuses on a specific task, making the processing logic clearer and more efficient.

[0087] Specifically, in a preferred embodiment of the present application, the distributed RDMA model parallel processing method based on hardware multithreading further includes:

[0088] When performing multi-stage task processing on messages, the thread allocation and task scheduling strategies are dynamically adjusted by sharing the memory area between threads and based on the thread execution status and feedback information.

[0089] Specifically, dynamically adjusting thread allocation and task scheduling strategies based on thread execution status and feedback information includes:

[0090] Monitor the load of each thread and the progress of task execution in real time. When it detects that the load rate of a thread reaches a preset threshold or the task backlog reaches a preset number, it dynamically transfers some tasks to the thread with the lowest load rate or the lowest task backlog.

[0091] According to the changes in network traffic and the priority of the message, the priority and weight of task scheduling are adjusted, and the highest priority message is processed and transmitted first.

[0092] In this embodiment, a shared memory area is planned in the system and is accessible to all threads involved in message processing. The shared memory is partitioned, for example, into a data storage area, a status information area, and a task queue area. The data storage area stores key information about messages to be processed, the status information area records the execution status of each thread, and the task queue area stores pending tasks.

[0093] At the same time, establish synchronization mechanisms to ensure secure access to shared memory by multiple threads. This can be achieved through the use of locking mechanisms. For example, when a thread accesses a data storage area in shared memory, it must first acquire a mutex lock and then release the lock after the access is complete. Alternatively, use semaphores to control the number of concurrent accesses to shared resources, preventing data inconsistencies caused by multiple threads modifying shared memory simultaneously.

[0094] For each thread, its load is monitored in real time. This load is assessed by monitoring metrics such as the thread's CPU utilization and memory usage. As each thread processes a task, it must report its progress to the shared memory status area. Each task is assigned a unique identifier, and the thread updates its execution status at different stages of the task, such as "Received," "Processing," or "Completed." Other threads or monitoring programs can access the shared memory status area to understand the progress of each task.

[0095] By presetting thresholds for thread load rates and task backlogs, for example, the thread load rate threshold can be set to 80% and the task backlog threshold to 10. Furthermore, these thresholds can be dynamically adjusted based on the system's hardware resources and business needs. The monitoring program regularly checks the load status and task backlog of each thread. When it detects that a thread's load rate reaches a preset threshold or the task backlog reaches a preset number, the task transfer mechanism is triggered. Tasks are selected from the task queue of the overloaded thread and their information is transferred to the task queue with a lower load.

[0096] In this embodiment, network traffic changes are monitored in real time using a network monitoring tool. Network traffic rates are calculated by reading network interface statistics, such as byte counts and packet counts. During the message reception and parsing phase, each message is assigned a priority tag. For example, service messages with high real-time requirements are assigned a higher priority, while general log messages are assigned a lower priority.

[0097] Dynamically adjust task scheduling priorities and weights based on changes in network traffic and message priorities. When network traffic is high, prioritize high-priority messages to ensure the normal operation of critical services. Priority queues can also be used to manage tasks, placing high-priority messages at the front of the queue for priority processing and transmission. Furthermore, queue sorting rules can be adjusted based on network traffic to adapt to different network environments.

[0098] In this example, by real-time monitoring of thread loads and task execution progress, when a thread's load reaches a preset threshold or the task backlog reaches a preset number, some tasks can be promptly transferred to the thread with the lowest load or the smallest task backlog. This effectively avoids performance bottlenecks caused by overloading a single thread and fully utilizes the system's overall processing capacity. Dynamic adjustment of thread allocation ensures a more balanced distribution of tasks across multiple threads, allowing each thread to process tasks in parallel, thereby accelerating message processing. During multi-stage task processing, the processing time for each stage is more evenly distributed, reducing overall process blockages caused by slow processing by a single thread and improving system processing efficiency. Furthermore, adjusting task scheduling priorities and weights based on network traffic fluctuations allows the system to better adapt to varying network environments. During peak network traffic periods, high-priority messages are prioritized for processing and transmission, ensuring the normal operation of critical services. During low network traffic periods, strict priority requirements can be appropriately relaxed to improve overall processing efficiency. This dynamic adjustment mechanism allows system resources to be flexibly allocated based on actual needs, optimizing resource utilization efficiency.

[0099] The present application also provides a hardware multi-threaded distributed RDMA model parallel processing system, which is applied to the aforementioned hardware multi-threaded distributed RDMA model parallel processing method, including:

[0100] A parsing unit, configured to receive multiple messages to be processed and parse the messages to extract key information;

[0101] A scheduling unit, configured to load the received message into a corresponding thread according to the key information;

[0102] The processing unit is used to perform multi-stage task processing on the message according to the thread corresponding to each message and the preset processing rules to obtain the processing result;

[0103] The sending unit is used to send the processing result to the corresponding destination node according to a preset sending strategy.

[0104] The functional explanation of each unit in this embodiment is the same as that of the distributed RDMA model parallel processing method based on hardware multi-threading, and the technical effect is the same, so it will not be repeated here.

[0105] An embodiment of the present application further provides an electronic device, including:

[0106] processor;

[0107] a memory for storing processor-executable instructions;

[0108] The processor implements the aforementioned hardware multi-threaded distributed RDMA model parallel processing method by running the executable instructions.

[0109] The technical effect of this embodiment is the same as the technical effect of the distributed RDMA model parallel processing method based on hardware multithreading in the embodiment, and will not be repeated here.

[0110] The processor in this embodiment may be a central processing unit (CPU), a controller, a microcontroller, or other data processing chips.

[0111] An embodiment of the present application further provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the aforementioned hardware multi-threaded distributed RDMA model parallel processing method.

[0112] The technical effect of this embodiment is the same as the technical effect of the distributed RDMA model parallel processing method based on hardware multithreading in the embodiment, and will not be repeated here.

[0113] The present invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0114] Those skilled in the art will appreciate that the units of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0115] In the embodiments provided by the present invention, it should be understood that the division of units is merely a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0116] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nly Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A distributed RDMA model parallel processing method based on hardware multithreading, characterized in that: include: Receive multiple messages to be processed and parse them to extract key information; Loading the received message into the corresponding thread according to the key information; According to the thread corresponding to each message, the message is processed in multiple stages according to the preset processing rules to obtain the processing results; The processing results are sent to the corresponding destination node according to the preset sending strategy.

2. The distributed RDMA model parallel processing method based on hardware multithreading according to claim 1 is characterized in that: The key information includes the type, destination address and priority of the message.

3. The distributed RDMA model parallel processing method based on hardware multithreading according to claim 2, characterized in that: The step of loading the received message into the corresponding thread according to the key information includes: According to the type, destination address and priority of the message in the key information and the load of the current thread, the optimal thread is matched from the processor's resource pool to process the corresponding message, and the processed message is loaded into the corresponding thread.

4. The distributed RDMA model parallel processing method based on hardware multithreading according to claim 1, characterized in that: The multi-stage task processing is performed on the message according to the thread corresponding to each message according to the preset processing rules, and the processing results include: According to the thread corresponding to each message, the message is processed through the preset processing rules, including data verification, address translation, protocol analysis, flow control and priority scheduling stages; The task processing results of each stage are used as the input of the next stage until the task processing of all stages is completed and the final processing results are obtained.

5. The distributed RDMA model parallel processing method based on hardware multithreading according to claim 4 is characterized in that: Also includes: When performing multi-stage task processing on messages, the thread allocation and task scheduling strategies are dynamically adjusted by sharing the memory area between threads and based on the thread execution status and feedback information.

6. The distributed RDMA model parallel processing method based on hardware multithreading according to claim 5, characterized in that: The dynamically adjusting thread allocation and task scheduling strategy based on the thread execution status and feedback information includes: Monitor the load of each thread and the progress of task execution in real time. When it detects that the load rate of a thread reaches a preset threshold or the task backlog reaches a preset number, it transfers some tasks to the thread with the lowest load rate or the lowest task backlog. According to the changes in network traffic and the priority of the message, the priority and weight of task scheduling are adjusted, and the highest priority message is processed and transmitted first.

7. The distributed RDMA model parallel processing method based on hardware multithreading according to any one of claims 1 to 6, characterized in that: The preset sending strategy includes routing selection based on destination address, path optimization based on traffic engineering, or priority scheduling based on quality of service.

8. A distributed RDMA model parallel processing system based on hardware multithreading, characterized in that: The distributed RDMA model parallel processing method based on hardware multithreading applied to any one of claims 1 to 7 comprises: A parsing unit, configured to receive multiple messages to be processed and parse the messages to extract key information; A scheduling unit, configured to load the received message into a corresponding thread according to the key information; The processing unit is used to perform multi-stage task processing on the message according to the thread corresponding to each message and the preset processing rules to obtain the processing result; The sending unit is used to send the processing result to the corresponding destination node according to a preset sending strategy.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the distributed RDMA model parallel processing method based on hardware multithreading as described in any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the steps of the distributed RDMA model parallel processing method based on hardware multi-threading as described in any one of claims 1 to 7 are implemented.

Images