Remote direct memory access network damage simulation method and device and computer equipment
By setting up a remote direct memory access protocol stack on the server side and using damage configuration information to process messages, the problem of poor effectiveness of traditional network damage simulation methods is solved, and efficient and low-cost network failure simulation is achieved, suitable for testing and real environments .
Patent Information
- Application Number
- CN202411388757.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-12-10
AI Technical Summary
The traditional network damage simulation method is not effective in simulating network failures, and the structure simulation on the switch side is complex and costly, which affects other services and makes it difficult to achieve fine-grained failure behavior simulation.
By setting up a remote direct memory access protocol stack on the server side, using the damage configuration information to process messages, directly simulate network failure scenarios in the user-state space, realizing finer-grained network failure behavior simulation, reducing costs and improving simulation results.
It realizes efficient and low-cost network failure simulation on the server side, avoids the impact on other services, can be lightly deployed in test and real environments, improving user satisfaction and simulation results.
Smart Images
Figure CN119109753A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of remote direct memory access, and in particular to a remote direct memory access network damage simulation method, device, and computer equipment. Background Art
[0002] As the size of the cluster increases, the number of servers in the cluster continues to increase, so reliable communication between servers and other devices is particularly important; especially in business scenarios such as distributed computing and distributed storage, a low-latency, high-bandwidth communication environment is very important, and the RDMA (Remote Direct Memory Access) communication solution has become the first choice. The RDMA communication solution has played a very important role in a variety of high-performance scenarios. However, with the complexity and diversification of business forms, RDMA network failures have also increased, so it is crucial to detect network failures in advance and configure fault resolution or prevention strategies in advance. Relevant technicians can generally construct business traffic on the switch side to simulate remote direct memory access network damage. However, the constructed business traffic is often different from the actual business traffic, so the traditional network damage simulation method has the problem of poor network damage simulation effect. Summary of the invention
[0003] In view of this, the present disclosure provides a remote direct memory access network damage simulation method, device, and computer equipment to effectively solve the problem that traditional network damage simulation methods have poor network damage simulation effects.
[0004] In a first aspect, the present disclosure provides a remote direct memory access network impairment simulation method, the method comprising:
[0005] Establishing a remote direct memory access connection between a first host and a second host, the first host being used to send a first message to the second host and to receive a second message sent by the second host;
[0006] Obtain damage configuration information, which is used to characterize network failure scenarios;
[0007] According to the damage configuration information, a third message passing through a remote direct memory access protocol stack is processed. The remote direct memory access protocol stack is set in a user space of the first host and is used to connect a remote direct memory access driver and a network interface card. The third message includes a first message and a second message.
[0008] Compared with the traditional network damage simulation method, the present disclosure takes the damage configuration information used to characterize the network failure scenario as the basis, and realizes the technical purpose of network damage simulation by processing the third message of the remote direct memory access protocol stack passing through the user space of the first host; it can be seen that the present disclosure directly simulates the network failure scenario on the server side, solves the problems of the complexity of simulating network failure on the switch side and the impact on other services in the related technology, and significantly reduces the cost of implementing the network damage simulation solution. The network damage simulation solution provided by the present disclosure can not only be lightweight deployed in the test environment, but also can be lightweight deployed in the real environment, and can realize the construction of finer-grained network failure behaviors such as packet loss, guide and verify related services to bypass network failures, higher user satisfaction and lower implementation costs.
[0009] In a second aspect, the present disclosure provides a remote direct memory access network damage simulation device, the device comprising:
[0010] A communication establishment module, used to establish a remote direct memory access connection between a first host and a second host, the first host is used to send a first message to the second host and is used to receive a second message sent by the second host;
[0011] A configuration acquisition module is used to acquire damage configuration information, which is used to characterize network failure scenarios;
[0012] The message processing module is used to process a third message passing through a remote direct memory access protocol stack according to damage configuration information. The remote direct memory access protocol stack is set in the user state space of the first host and is used to connect a remote direct memory access driver and a network interface card. The third message includes a first message and a second message.
[0013] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the remote direct memory access network damage simulation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0014] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the remote direct memory access network damage simulation method of the first aspect or any corresponding embodiment thereof.
[0015] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the remote direct memory access network damage simulation method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0017] Figure 1 It is a flowchart of a remote direct memory access network damage simulation method according to an embodiment of the present disclosure.
[0018] Figure 2 It is a schematic diagram of the damage simulation test networking principle according to an embodiment of the present disclosure.
[0019] Figure 3 It is a flowchart of another remote direct memory access network damage simulation method according to an embodiment of the present disclosure.
[0020] Figure 4 It is a schematic diagram of the implementation logic of the hook program according to an embodiment of the present disclosure.
[0021] Figure 5 It is a flowchart of another remote direct memory access network damage simulation method according to an embodiment of the present disclosure.
[0022] Figure 6 It is a schematic diagram of the setting location of the hook program according to an embodiment of the present disclosure.
[0023] Figure 7 It is a schematic diagram of the implementation principle of a lightweight, user-mode, high-performance RDMA protocol stack according to an embodiment of the present disclosure.
[0024] Figure 8 It is a schematic diagram of an RDMA network damage scenario involved in an embodiment of the present disclosure.
[0025] Fig. 9 It is a flowchart of an RDMA network damage simulation method taking random packet loss as an example according to an embodiment of the present disclosure.
[0026] Fig.10 It is a flowchart of an RDMA network damage simulation method taking network anomaly as an example according to an embodiment of the present disclosure.
[0027] Fig.11 It is a flowchart of an RDMA network damage simulation method taking a network card failure as an example according to an embodiment of the present disclosure.
[0028] Fig.12It is a schematic diagram of the RDMA communication process between the first host and the second host according to an embodiment of the present disclosure.
[0029] Fig.13 It is a structural block diagram of a remote direct memory access network damage simulation device according to an embodiment of the present disclosure.
[0030] Fig.14 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0032] RDMA (Remote Direct Memory Access) technology has the advantages of high throughput and low latency. In business scenarios with high requirements for communication bandwidth or latency, such as storage and recommendation search, RDMA technology has become the preferred technical solution for various high-performance scenarios and can be deployed on a large scale in data centers.
[0033] However, with the gradual increase in cluster size, the increase in the number of servers, and the diversification and complexity of business forms, RDMA network failures are also on the rise. For example, a series of fault-related tasks such as alarm, isolation, positioning, and recovery require technical personnel to invest tremendous energy, which poses higher challenges and requirements for network operation and maintenance and business linkage. In the face of the above situation, it is very valuable and meaningful to be able to detect network failures in advance and successfully bypass them to avoid losses to the business.
[0034] In the related technology, network failure behavior can be simulated by constructing a network failure on a switch. However, simulating the construction of business traffic will cause traffic noise and distortion, and setting up a test environment is time-consuming, labor-intensive, and costly. The solution for constructing a network failure on a switch is very complex and can easily affect other businesses.
[0035] According to an embodiment of the present disclosure, an embodiment of a remote direct memory access network damage simulation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0036] In this embodiment, a remote direct memory access network damage simulation method is provided, which can be used for a first host. Figure 1 is a flowchart of a remote direct memory access network damage simulation method according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:
[0037] Step S101, establishing a remote direct memory access connection between a first host and a second host, wherein the first host is used to send a first message to the second host and to receive a second message sent by the second host.
[0038] Combination Figure 2As shown, the first host is represented as Host0, the second host is represented as Host1, and the remote direct memory access connection is RDMA connection. The damage simulation test networking architecture shown in this embodiment is explained by taking the RDMA connection between one host and another host as an example. Specifically, in this embodiment, the first application (App1) in Host0 is used as a server (server), and the second application (App2) in Host1 is used as a client (client). The relevant business program uses Byte-S-RDMA in the server and hardware RDMA in the client. The relevant business program establishes an RDMA connection between the server and the client through an out-of-band (for example, TCP, Transmission Control Protocol) connection or an in-band (RDMA) connection to run real business traffic or test business traffic; More specifically, Host0 is configured with a remote direct memory access driver (Byte-S-RDMA provider), a remote direct memory access protocol stack (Byte-S-RDMA) and a network interface card (NIC), and Host1 is configured with a Mellanox driver (MLX provider) and a Mellanox network interface card (MLX NIC). In this embodiment, a first communication port (port1) and a second communication port (port2) are configured on the switch, and Host1 and Host0 communicate with each other through the MLX provider, MLX protocol stack, and MLX NIC in turn. NIC, port2, switch, port1, NIC, Byte-S-RDMA, Byte-S-RDMA provider or a reverse link of the link are communicated and connected; wherein the first message and / or the second message can be a message in real business traffic, so that the present embodiment can avoid the distortion or noise problem caused by the method of simulating business traffic in the related art.
[0039] like Figure 2 As shown, the remote direct memory access driver (Byte-S-RDMA provider) and the remote direct memory access protocol stack (Byte-S-RDMA) of this embodiment constitute a damage instrument, which is used to implement damage simulation testing. It implements the network damage simulation function at the software level to avoid excessive dependence on hardware. Therefore, this embodiment provides a lightweight, stateful software version of the RDMA network damage simulation technical solution. The remote direct memory access protocol stack of this embodiment can directly communicate with the RDMA hardware, so that more fine-grained protocol stack exceptions can be constructed to simulate various scenarios.
[0040] Combination Fig.12As shown, in some optional implementations, the combination of App1 and the damage instrument is regarded as the server, and the combination of App2 and the network card under test is regarded as the client, and the connection process between the client and the server is as follows: App2 sends a request to create a QP (Queue Pair) to the network card under test, and sends a request to App1 to request connection; App1 can respond to the request for connection by creating a QP and setting the damage instrument to enter the RTR (Ready To Receive) state, and feedback the connection response to App2; after receiving the connection response, App2 sets the QP and makes the network card under test enter the RTS (Ready To Send) state, and connects to App1 and enters the state of sending an RTU (Remote Terminal Unit) message to App1; then App1 sets the QP and makes the damage instrument enter the RTS state. Next, App2 sends a data sending request to the network card under test, thereby sending data message 1, data message 2, data message 3, ..., data message n-2, data message n-1, data message n to the damage instrument through the network card under test. Data messages 1 to n in this example are the data messages in the aforementioned second message. The damage instrument then feeds back an ack (confirmation) message to the network card under test, that is, the confirmation message in the aforementioned first message, and generates a recv complete (reception complete) message to notify App1 that data reception is complete; when the network card under test receives the ack message, it generates a send complete message to notify App2 that data reception is complete. The remote direct memory access network damage simulation method in one or more embodiments of the present disclosure can execute at the damage instrument processing of the data messages sent from the network card under test and the confirmation messages sent to the network card under test. The process of the damage instrument sending a message to the network card under test is similar to the above process and will not be repeated here. The remote direct memory access network damage simulation method involved in one or more embodiments of the present disclosure can also execute the data message sent to the network card under test and the confirmation message fed back by the network card under test at the damage instrument for processing.
[0041] Step S102: obtaining damage configuration information, where the damage configuration information is used to characterize a network failure scenario.
[0042] Specifically, the damage configuration information can be sent to Byte-S-RDMA by configuration, and Byte-S-RDMA calls it when performing network damage simulation; in an optional implementation, the damage configuration information can also be pre-configured in the above-mentioned Byte-S-RDMA.
[0043] In some optional implementations, the network failure scenarios of this embodiment include end-side failure scenarios and network-side failure scenarios. The end-side failure scenarios include server failure scenarios, and the network-side failure scenarios include switch failure scenarios.
[0044] The server includes a first host and / or a second host, and the switch is used to connect the first host and the second host.
[0045] like Figure 8 As shown, the damage meter provided in this embodiment can be used to implement the end-side network damage simulation function and the network-side (switch packet loss) network damage simulation function. The application scope of the RDMA network damage simulation function is very wide.
[0046] Specifically, the end-side failure scenarios may include one or more of random packet loss, fixed packet loss, packet sequence number (PSN) error or PSN disorder, bit (BIT) flip, and ICRC (Invariant Cyclic Redundancy Check) error. The network-side failure scenarios include one or more of black hole (silent) packet loss, congestion packet loss, abnormal packet loss, link packet loss, latency, and jitter. Abnormal packet loss may include packet loss caused by CRC (Cyclic Redundancy Check) error or forwarding logic error. Forwarding logic errors include ACL (Access Control List) entry errors and / or routing errors.
[0047] Step S103, according to the damage configuration information, the third message passing through the remote direct memory access protocol stack is processed, the remote direct memory access protocol stack is set in the user state space of the first host and is used to connect the remote direct memory access driver and the network interface card, and the third message includes the first message and the second message.
[0048] Combination Figure 7As shown, the remote direct memory access protocol stack in this embodiment is set in the user state space. The entire RDMA protocol stack architecture in this embodiment includes the following parts: remote direct memory access protocol stack (Byte-S-RDMA), kernel driver (Byte-S-RDMA kernel driver) and remote direct memory access driver (Byte-S-RDMAprovider), Byte-S-RDMA can be a user state RDMA protocol stack implemented based on DPDK (Data Plane Development Kit, data plane development kit), Byte-S-RDMA kernel driver is mainly responsible for resource application and management, generating transport layer interface (Uverbs) equipment, and establishing Netlink (network link) control channel with Byte-S-RDMA, Byte-S-RDMA provider is mainly responsible for verbs api (transport layer interface application programming interface) conversion, control path interaction with kernel driver and data path interaction with Byte-S-RDMA. Figure 7 The solid arrows in the figure represent the control channel (control), and the dotted arrows represent the data channel (data). The RDMA protocol stack of this embodiment can support RDMA intercommunication with the underlying hardware.
[0049] In combination with the foregoing embodiments, the remote direct memory access driver set in the user space (Userspace) in this embodiment constructs a data communication channel between the application (App) and the remote direct memory access protocol stack, a control channel is also set between the application and the remote direct memory access driver, and the remote direct memory access protocol stack set in the user space constructs a data communication channel between the remote direct memory access driver and the network interface card (NIC) on the physical network card. For the kernel state space (Kernel), the kernel state space of this embodiment includes a kernel driver, and the kernel driver is configured with a queue pair management module (QP mgmt, Queue Pairs management) for managing queue pairs, a completion queue management module (CQ mgmt, Complete Queue management) for managing completion queues, a memory registration management module (MR mgmt, Memory Registration, management) for managing memory registration processes, a ring buffer management module (mmap mgmt, memory mapping management) for managing ring buffers, a transport layer interface (Uverbs) device and a network link (Netlink) interface, the network link (Netlink) interface is used to connect to the remote direct memory access protocol stack to build a control channel between the remote direct memory access protocol stack and the kernel driver, and the transport layer interface device is used to connect to the remote direct memory access driver to build a data write control channel between the remote direct memory access protocol stack and the kernel driver. For the detailed workflow related to the control channel, it can be combined with the processing method of the control information in the control channel adopted in the relevant RDMA communication process on the basis of the disclosed embodiment, and it will not be repeated here.
[0050] In this embodiment, the process of processing the third message simulates the process of the third message being affected when the corresponding network is damaged.
[0051] In some optional implementations, the processed third message involved in this embodiment is used to simulate at least one of packet loss behavior, hardware logic abnormal behavior, and delay jitter behavior.
[0052] For example, it simulates delay jitter behaviors caused by network anomalies, simulates hardware logic abnormal behaviors caused by network card hardware failure, simulates packet loss behaviors caused by protocol stack packet loss, etc. Of course, the network damage that can be simulated by this embodiment is not limited to this.
[0053] This embodiment realizes the function of simulating RDMA network damage such as network anomalies, network card hardware failures, and protocol stack packet loss, and has a wide range of applications. This embodiment solves the complexity of simulating network failures on the switch side and the impact on other services by directly simulating various network failure scenarios on the server side, and reduces the cost of implementing network damage simulation solutions. This embodiment specifically provides a simple and fast solution for constructing network failures, and can use service traffic or test traffic to directly observe the behavior of services under the current network failure. It can be seen that the simulation method provided by this embodiment can not only be lightweight deployed in a test environment, but also lightweight deployed in a real environment, and can realize the construction of finer-grained network failure behaviors such as packet loss, so as to better guide and verify related services to bypass network failures (such as network failover, network failover is a mechanism that automatically switches to a backup system or backup link when a main system or main link fails to maintain service continuity and availability), and the remote direct memory access network damage simulation method provided by the present disclosure is a stateful lightweight support RDMA network damage simulation method.
[0054] In this embodiment, a remote direct memory access network damage simulation method is provided, which can be used for a first host. Figure 3 is a flowchart of a remote direct memory access network damage simulation method according to an embodiment of the present disclosure. Figure 3 As shown, the process includes the following steps:
[0055] Step S301, establish a remote direct memory access connection between a first host and a second host, the first host is used to send a first message to the second host and to receive a second message sent by the second host. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0056] Step S302: Obtain damage configuration information, which is used to characterize network failure scenarios. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0057] Step S303, according to the damage configuration information, the third message passing through the remote direct memory access protocol stack is processed, the remote direct memory access protocol stack is set in the user state space of the first host and is used to connect the remote direct memory access driver and the network interface card, and the third message includes the first message and the second message.
[0058] Specifically, the above step S303 includes:
[0059] Step S3031, filter out a target message from the third message according to a message filtering condition, the message filtering condition includes at least one preset message parameter, and the target message includes a data message and a confirmation message having a message parameter matching the preset message parameter and passing through the remote direct memory access protocol stack.
[0060] Specifically, for the first message and the second message transmitted between the first host and the second host, it is determined whether the current message is to be used as a to-be-processed message for network damage simulation according to preset message parameters in the message filtering condition.
[0061] In some optional implementations, at least one preset message parameter includes message transmission direction, destination address, source address, source port, queue pair number and data packet sequence number; the above step S3031 includes but is not limited to step a1 and step a2.
[0062] Step a1: determining a message filtering condition based on preset message parameters pre-configured in the remote direct memory access protocol stack and / or preset message parameters in the damage configuration information.
[0063] Specifically, the preset message parameters may be message parameters sent by the host to the remote direct memory access protocol stack, and / or message parameters that have been pre-configured in the remote direct memory access protocol stack.
[0064] Step a2: determine the third message that matches the message transmission direction, target address, source address, source port, queue pair number and data packet sequence number as the target message.
[0065] like Figure 4 As shown, the message transmission direction is expressed as direction, the destination address may specifically be the destination IP (Internet Protocol Address), the source address may specifically be the source IP (Internet Protocol Address), the queue pair number (QPN) is used as an identifier of each queue pair (Queue Pair) in the message, and the packet sequence number (PSN) is used to identify each data packet and to be used by the receiving end to identify whether it is consistent with the expected data packet sequence number. Of course, the preset message parameters that can be used in this embodiment are not limited to the above examples. For example, the preset message parameters may also include a bit flip parameter and a latency parameter. Then, a third message that matches the message transmission direction, destination address, source address, source port, queue pair number, data packet sequence number, bit flip parameter and latency parameter in the preset message parameters is determined as the target message.
[0066] Specifically, this embodiment can preferentially determine whether the preset message parameters are configured in the damage configuration information, and determine whether the message parameters in the third message match the preset message parameters if the preset message parameters are configured in the damage configuration information. Figure 4 As shown, this embodiment can determine in parallel whether the message transmission direction, target address, source address, source port, queue pair number and data packet sequence number in the third message are respectively configured and matched one by one with the message transmission direction, target address, source address, source port, queue pair number and data packet sequence number in the preset message parameters, and perform simulation actions when the above parameters are configured and matched, simulating at least one of packet loss-type behaviors, hardware logic abnormal behaviors, and delay jitter-type behaviors.
[0067] Step S3032: Process the target message according to the damage configuration information.
[0068] Specifically, the damage configuration information includes packet loss processing strategy information and message modification strategy information; the above step S3032 includes: performing packet loss processing on the target message according to the packet loss processing strategy information and / or modifying the content in the target message according to the message modification strategy information.
[0069] This embodiment performs packet loss processing on the target message, which can be applicable to various packet loss-related scenarios such as random packet loss, fixed packet loss, congestion packet loss, abnormal packet loss, link packet loss, etc., and modifies the content of the target message, which can be applicable to various message content error scenarios such as PSN error / disorder, bit flipping, etc.
[0070] Therefore, this embodiment only processes the messages that meet the requirements according to the message filtering conditions. The advantage of this method is that the RDMA network damage simulation method only acts on the messages that meet the requirements, and has no effect on non-target messages, that is, the services related to the non-target messages will not be affected by the RDMA network damage simulation method. Therefore, the RDMA network damage simulation method provided by the present disclosure can be lightweight deployed in the production environment, and the RDMA network damage simulation method can be directly executed under the service traffic. In addition, based on the preset message parameters including the message transmission direction, target address, source address, source port, queue pair number and data packet sequence number, this embodiment can further accurately filter out the target message from the third message, has a strong filtering effect on the message, realizes accurate processing of the message, and greatly improves the authenticity and reliability of the RDMA network damage simulation effect.
[0071] In this embodiment, a remote direct memory access network damage simulation method is provided, which can be used for a first host. Figure 5 is a flowchart of a remote direct memory access network damage simulation method according to an embodiment of the present disclosure. Figure 5As shown, the process includes the following steps:
[0072] Step S501: establish a remote direct memory access connection between a first host and a second host, wherein the first host is used to send a first message to the second host and to receive a second message sent by the second host. Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.
[0073] Step S502: Acquire damage configuration information, where the damage configuration information is used to characterize a network failure scenario.
[0074] Specifically, the damage configuration information includes current configuration information determined according to the test requirements; the above step S502 includes: respectively sending the current configuration information to the first hook program, the second hook program, the third hook program and the fourth hook program through the configuration sending component.
[0075] Combination Figure 2 As shown, this embodiment can use an interactive command window to send damage configuration information to the remote direct memory access protocol stack (Byte-S-RDMA). The interactive command window can be, for example, bfctl. Then, the user can use bfctl to send the damage configuration information to the corresponding hook program (impairment-hook) according to specific test requirements, wherein the hook program includes but is not limited to the first hook program, the second hook program, the third hook program, and the fourth hook program of the embodiment described later. By sending configuration information to each hook program respectively, this embodiment can determine the network message to be filtered according to the current network damage function to be simulated, so that the RDMA network damage simulation method of this scheme has stronger flexibility.
[0076] Step S503, according to the damage configuration information, the third message passing through the remote direct memory access protocol stack is processed, the remote direct memory access protocol stack is set in the user state space of the first host and is used to connect the remote direct memory access driver and the network interface card, and the third message includes the first message and the second message.
[0077] Specifically, the remote direct memory access protocol stack includes a request sending component, a response receiving component, a response sending component and a request receiving component. The above step S503 includes:
[0078] Step S5031, filtering out data packets sent from the first host to the second host from the third packet according to the packet filtering condition by the first hook program set in the request sending component;
[0079] Step S5032, filtering out the data message sent by the second host to the first host from the third message according to the message filtering condition by the second hook program set in the response receiving component;
[0080] Step S5033, filtering out the confirmation message sent by the first host to the second host from the third message according to the message filtering condition by the third hook program set in the response sending component;
[0081] Step S5034: filter out the confirmation message sent by the second host to the first host from the third message according to the message filtering condition through the fourth hook program set in the request receiving component.
[0082] The state machine of the remote direct memory access protocol stack of this embodiment includes a request component (requester) and a responder component (responder). The request component is responsible for sending data (data) messages and receiving confirmation (ack) messages, and the responder component is responsible for receiving data (data) messages and sending confirmation (ack) messages.
[0083] Specifically, the request component (requester) includes a request sending component (requester-tx) and a request receiving component (requester-rx), the request sending component is used to send data messages, and the request receiving component is used to receive confirmation messages, the response component (responder) includes a response receiving component (responder-rx) and a response sending component (responder-tx), the response receiving component is used to receive data messages, and the response sending component is used to send confirmation messages.
[0084] like Figure 6 As shown, the hook program specifically includes the above-mentioned first hook program (impairment-hook-01), second hook program (impairment-hook-02), third hook program (impairment-hook-03) and fourth hook program (impairment-hook-04). Specifically, the first hook program is set in the request sending component (requester-tx), the second hook program is set in the response receiving component (responder-rx), the third hook program is set in the response sending component (responder-tx), and the fourth hook program is set in the request receiving component (requester-rx). It can be seen that this embodiment places the impairment-hook hook point in the four places of requester-tx / requester-rx / responder-tx / responder-rx. In response to the interface trigger prompt instruction (such as Doorbell, doorbell instruction), the corresponding hook program can match and filter all RDMA messages and perform damage simulation actions.
[0085] This embodiment sets the first hook program, the second hook program, the third hook program, and the fourth hook program in a one-to-one correspondence at the request sending component, the response receiving component, the response sending component, and the request receiving component, respectively. This method can achieve comprehensive capture of sent or received data packets and sent or received confirmation packets, and can further construct finer-grained network damage behaviors such as packet loss, so that the RDMA network damage simulation method of this embodiment can be applicable to the simulation of more network damage scenarios.
[0086] like Fig. 9 As shown in the figure, taking random packet loss in network anomalies as an example, specifically taking single random packet loss as an example, assuming that the packet loss rate is 1% and the message transmission direction is the ingress message direction (ingress), in this case, the expected business App will perceive the decrease in throughput bandwidth. For example, bfctl sends the following example of damage configuration information: bfctl--type random-drop--rate1 / 100--direction ingress; the message received by the damage meter may include data messages and confirmation messages (or response messages).
[0087] For data messages, they are filtered through impairment-hook-02 configured in responder-rx. If random packet loss information (random-drop) is configured in the damage configuration information, a random number is generated, such as a 32-bit random number rand1(). If the modulus (%) calculation result of the random number rand1() and the first preset value is 0, packet loss processing is performed on the current data packet in the data message. The first preset value can be 100, for example. If random packet loss information (random-drop) is not configured in the damage configuration information or the modulus calculation result is not 0, the subsequent processing of the protocol stack is performed.
[0088] For the confirmation message, it is filtered through the impairment-hook-04 configured in the requester-rx. If the random packet loss information (random-drop) is configured in the damage configuration information, a random number is generated, such as a 32-bit random number rand2(). If the modulus (%) calculation result of the random number rand2() and the second preset value is 0, the current data packet in the confirmation message is subjected to packet loss processing. The second preset value can be 100, for example. If the random packet loss information (random-drop) is not configured in the damage configuration information or the modulus calculation result is not 0, the subsequent processing of the protocol stack is performed.
[0089] like Fig.10As shown, take the switch queue backlog in the network anomaly as an example, specifically take the switch queue backlog causing an additional 1000μs (microsecond) delay (latency), and the message transmission direction is the ingress message direction (ingress) as an example. In this case, the expected business App will perceive more timeouts. For example, bfctl sends the following example of damage configuration information: bfctl--type latency--time 1000--direction ingress.
[0090] The data packets received by the damage instrument are filtered through the impairment-hook-02 configured in responder-rx. If the relevant delay information is configured in the damage configuration information, the packet can be placed in the pending queue and the timer (for example, a 1000μs timer) can be started. After the timer is triggered, the protocol stack performs subsequent processing. If the experimental information is not configured in the damage configuration information, the protocol stack performs subsequent processing directly.
[0091] The confirmation message (or response message) sent by the damage instrument is filtered through impairment-hook-03 configured in responder-tx. If the relevant delay (latency) information is configured in the damage configuration information, the message can be placed in the pending queue, and the timer (for example, a 1000μs timer) can be started. After the timer is triggered, the protocol stack performs subsequent processing. If the experimental information is not configured in the damage configuration information, the protocol stack performs subsequent processing directly.
[0092] like Fig.11 As shown in the figure, take the network card failure in the end-side failure scenario as an example, specifically take the bit flip failure as an example, for example, the data packet (packet-type tx-data, the data packet type is the data sent) sent by the damage instrument simulation has a bit flip (bitflap), and the flip probability is one in a thousand. In this case, the expected business app will perceive a decrease in data throughput. For example, bfctl can send the following example of damage configuration information: bfctl--type bitflap--packet-type tx-data--rate 1 / 1000.
[0093] The data message sent by the damage meter is filtered through the impairment-hook-01 configured in the requester-tx. If the relevant bit flip (bitflap) information is configured in the damage configuration information, a random number is generated, such as a 32-bit random number rand(). If the modulus (%) calculation result of the random number rand() and the third preset value is 0, the payload information of the currently sent data message is modified. The third preset value can be 1000, for example. If the relevant bit flip (bitflap) information is not configured in the damage configuration information or the modulus calculation result is not 0, the data message is forwarded normally.
[0094] In summary, the present disclosure designs a stateful, pure user-mode high-performance RDMA protocol stack. The traditional RDMA protocol stack needs to rely on the hardware network card and must be unloaded to the network card hardware to achieve high performance. The present disclosure implements running the entire RDMA network protocol stack on the pure user-mode software side, which has less dependence on hardware; the present disclosure designs a lightweight, stateful software design solution that can simulate network anomalies, network card hardware failures, and protocol stack packet loss software solutions; the present disclosure can be directly and quickly deployed in a test environment or a production environment in a lightweight manner, and on the end side, by simulating various network failure behaviors such as network failures, network card hardware failures, and protocol stack abnormal failures, the failover effect of the business traffic in the failure scenario can be directly observed, which greatly improves the test efficiency and saves labor costs.
[0095] In this embodiment, a remote direct memory access network damage simulation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0096] This embodiment provides a remote direct memory access network damage simulation device, such as Fig.13 As shown, including:
[0097] The communication establishment module 1301 is used to establish a remote direct memory access connection between a first host and a second host, and the first host is used to send a first message to the second host and to receive a second message sent by the second host;
[0098] The configuration acquisition module 1302 is used to acquire damage configuration information, where the damage configuration information is used to characterize a network failure scenario.
[0099] The message processing module 1303 is used to process the third message passing through the remote direct memory access protocol stack according to the damage configuration information. The remote direct memory access protocol stack is set in the user state space of the first host and is used to connect the remote direct memory access driver and the network interface card. The third message includes the first message and the second message.
[0100] In some optional implementations, the message processing module 1303 includes:
[0101] The screening unit is used to screen out a target message from the third message according to a message filtering condition, wherein the message filtering condition includes at least one preset message parameter, and the target message includes a data message and a confirmation message having a message parameter matching the preset message parameter and passing through a remote direct memory access protocol stack.
[0102] The processing unit is used to process the target message according to the damage configuration information.
[0103] In some optional implementations, at least one preset message parameter includes a message transmission direction, a destination address, a source address, a source port, a queue pair number, and a data packet sequence number; and the screening unit includes:
[0104] A condition determination subunit, configured to determine a message filtering condition based on a preset message parameter preconfigured in the remote direct memory access protocol stack and / or a preset message parameter in the impairment configuration information;
[0105] The message determination subunit is used to determine a third message that matches the message transmission direction, target address, source address, source port, queue pair number and data packet sequence number as a target message.
[0106] In some optional implementations, the remote direct memory access protocol stack includes a request sending component, a response receiving component, a response sending component and a request receiving component; the screening unit includes:
[0107] A first screening subunit is used to screen out data packets sent by the first host to the second host from the third packet according to the packet filtering condition through a first hook program set in the request sending component;
[0108] A second screening subunit is used to screen out data packets sent by the second host to the first host from the third message according to the message filtering condition through a second hook program set in the response receiving component;
[0109] A third screening subunit is used to screen out the confirmation message sent by the first host to the second host from the third message according to the message filtering condition through a third hook program set in the response sending component;
[0110] The fourth screening subunit is used to screen out the confirmation message sent by the second host to the first host from the third message according to the message filtering condition through a fourth hook program set in the request receiving component.
[0111] In some optional implementations, the damage configuration information includes current configuration information determined according to the test requirements; the configuration acquisition module 1302 is specifically used to send the current configuration information to the first hook program, the second hook program, the third hook program and the fourth hook program respectively through the configuration sending component.
[0112] In some optional implementations, the damage configuration information includes packet loss processing strategy information and message modification strategy information; the processing unit is specifically used to perform packet loss processing on the target message according to the packet loss processing strategy information and / or to modify the content in the target message according to the message modification strategy information.
[0113] In some optional implementations, the network failure scenario includes an end-side failure scenario and a network-side failure scenario, the end-side failure scenario includes a server failure scenario, and the network-side failure scenario includes a switch failure scenario; wherein the server includes a first host and / or a second host, and the switch is used to connect the first host and the second host.
[0114] In some optional implementations, the processed third message is used to simulate at least one of packet loss behavior, hardware logic abnormal behavior, and delay jitter behavior.
[0115] The further functional description of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0116] The remote direct memory access network damage simulation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0117] The present disclosure also provides a computer device having the above Fig.13 The remote direct memory access network impairment simulation device is shown.
[0118] See also Fig.14 , Fig.14 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Fig.14As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.14 A processor 10 is taken as an example.
[0119] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0120] The memory 20 stores instructions to be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0121] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0122] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0123] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0124] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0125] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.
[0126] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A remote direct memory access network impairment simulation method, It is characterized in that The method comprises: Establishing a remote direct memory access connection between a first host and a second host, wherein the first host is used to send a first message to the second host and to receive a second message sent by the second host; Acquire damage configuration information, where the damage configuration information is used to characterize a network failure scenario; According to the damage configuration information, a third message passing through a remote direct memory access protocol stack is processed, wherein the remote direct memory access protocol stack is set in a user space of the first host and is used to connect a remote direct memory access driver and a network interface card, and the third message includes the first message and the second message.
2. The method according to claim 1, It is characterized in that The processing of the third message passing through the remote direct memory access protocol stack according to the damage configuration information includes: Filtering a target message from the third message according to a message filtering condition, wherein the message filtering condition includes at least one preset message parameter, and the target message includes a data message and a confirmation message having a message parameter matching the preset message parameter and passing through the remote direct memory access protocol stack; The target message is processed according to the damage configuration information.
3. The method according to claim 2, It is characterized in that The at least one preset message parameter includes a message transmission direction, a destination address, a source address, a source port, a queue pair number, and a data packet sequence number; The step of filtering out the target message from the third message according to the message filtering condition includes: Determining the message filtering condition based on the preset message parameters preconfigured in the remote direct memory access protocol stack and / or the preset message parameters in the damage configuration information; The third message that matches the message transmission direction, the target address, the source address, the source port, the queue pair number and the data packet sequence number is determined as the target message.
4. The method according to claim 2, It is characterized in that The remote direct memory access protocol stack includes a request sending component, a response receiving component, a response sending component and a request receiving component; the step of filtering out a target message from the third message according to the message filtering condition includes: filtering, by a first hook program set in the request sending component, a data message sent by the first host to the second host from the third message according to the message filtering condition; filtering, by a second hook program set in the response receiving component, the data message sent by the second host to the first host from the third message according to the message filtering condition; filtering, by a third hook program set in the response sending component, a confirmation message sent by the first host to the second host from the third message according to the message filtering condition; The confirmation message sent by the second host to the first host is filtered out from the third message according to the message filtering condition through the fourth hook program set in the request receiving component.
5. The method according to claim 4, It is characterized in that The damage configuration information includes current configuration information determined according to the test requirements; the obtaining of the damage configuration information includes: The current configuration information is respectively issued to the first hook program, the second hook program, the third hook program and the fourth hook program through the configuration issuing component.
6. The method according to any one of claims 2 to 5, It is characterized in that The damage configuration information includes packet loss processing strategy information and message modification strategy information; the processing of the target message according to the damage configuration information includes: The target message is subjected to packet loss processing according to the packet loss processing strategy information and / or the content in the target message is modified according to the message modification strategy information.
7. The method according to any one of claims 1 to 5, It is characterized in that The network failure scenario includes a terminal failure scenario and a network failure scenario. The terminal failure scenario includes a server failure scenario, and the network failure scenario includes a switch failure scenario. The server includes the first host and / or the second host, and the switch is used to connect the first host and the second host.
8. The method according to any one of claims 1 to 5, It is characterized in that The third message after processing is used to simulate at least one of packet loss behavior, hardware logic abnormal behavior, and delay jitter behavior.
9. A remote direct memory access network damage simulation device, It is characterized in that The device comprises: A communication establishment module, used to establish a remote direct memory access connection between a first host and a second host, wherein the first host is used to send a first message to the second host and to receive a second message sent by the second host; A configuration acquisition module, used to acquire damage configuration information, where the damage configuration information is used to characterize a network failure scenario; A message processing module is used to process a third message passing through a remote direct memory access protocol stack according to the damage configuration information, wherein the remote direct memory access protocol stack is set in the user state space of the first host and is used to connect a remote direct memory access driver and a network interface card, and the third message includes the first message and the second message.
10. A computer device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the remote direct memory access network damage simulation method according to any one of claims 1 to 8 by executing the computer instructions.
11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the remote direct memory access network damage simulation method according to any one of claims 1 to 8.
12. A computer program product, It is characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the remote direct memory access network damage simulation method according to any one of claims 1 to 8.