A load testing method, medium and device for RDMA device

Abstract

The present application relates to the technical field of remote direct memory access device testing, and particularly relates to a load testing method for RDMA device, medium and equipment. The method establishes mapping of logical queue pair index and hardware queue pair, and sets global state flag and index state flag; a flow thread issues work queue elements through logical queue pair indexes in an active state; when a queue pair update condition is met, a control thread selects part of logical queue pair indexes and freezes them, destroys corresponding hardware queue pairs, creates new hardware queue pairs and reestablishes mapping, and unselected logical queue pair indexes continue to be used for flow; after receiving resource preparation completion, a target logical queue pair index is restored to an active state. The present application can test application, release and reuse of queue pairs and associated resources under the condition that the device continuously bears load, and avoids continuously issuing work queue elements to frozen or destroyed hardware queue pairs.

Description

Technical Field

[0001] This invention relates to the field of remote direct memory access device testing technology, and in particular to a load testing method, medium, and device for RDMA devices. Background Technology

[0002] Remote Direct Memory Access (RDMA) is a data transfer technology that allows one computing device to directly access the memory of another computing device. RDMA reduces data copying and processor involvement in the data transfer process, and features high bandwidth and low latency, making it widely used in distributed storage, high-performance computing, and data center networks.

[0003] During RDMA communication, the communicating parties typically manage data transmission through queue pairs (QPs). Each QP includes a transmit work queue and a receive work queue, and corresponds to a QP context describing the connection state, transmit state, and receive state. The software enables the RDMA device to perform remote direct memory access write operations, remote direct memory access read operations, or transmit operations by submitting work queue elements (WQEs) to the corresponding work queues.

[0004] To test the performance and operational stability of RDMA devices, testing tools such as PerfTest are typically used to continuously send WQEs to the RDMA device. In existing testing processes, the testing tools generally create a preset number of QPs at the start of the test, perform load balancing through these QPs, and destroy all QPs at the end of the test. Therefore, during a single test, each QP and its associated QP context, memory, and other resources typically remain unchanged.

[0005] However, in real-world business scenarios, RDMA devices may maintain a large number of QPs simultaneously. Furthermore, as business tasks start and end, or communication relationships change, some QPs may be destroyed and recreated while other QPs continue transmitting data. Therefore, in addition to handling data transmission loads, RDMA devices also need to handle the allocation, release, reclamation, and reallocation of QP contexts and associated resources.

[0006] Because existing testing tools typically maintain the QP's lifecycle unchanged during testing, it's difficult to test the dynamic changes of QPs and their associated resources under continuous data transmission loads on the device. If a QP is destroyed directly during the existing streaming process, the streaming thread might still issue new WQEs to the destroyed or rebuilt QPs; furthermore, if data transmission resumes after a new QP is created but the receive buffer and other receive resources are not yet ready, arriving data may not be received correctly. Therefore, achieving dynamic destruction and reconstruction of QPs during continuous streaming requires addressing the coordination issue between the streaming process and QP lifecycle changes.

[0007] Therefore, how to dynamically destroy and rebuild some QPs during continuous load transfer in remote direct memory access devices, and coordinate WQE issuance with QP state changes, in order to test the ability of remote direct memory access devices to apply for, release, and reuse QPs and their associated resources under load, is an urgent technical problem to be solved. Summary of the Invention

[0008] To address one of the aforementioned technical problems, the present invention adopts the following technical solution:

[0009] According to one aspect of the present invention, a load testing method for RDMA devices is provided, comprising the following steps:

[0010] Establish a set of logical queue pairs indexes, create hardware queue pairs for each logical queue pair index and establish a mapping relationship, set a global status flag to record the processing stage of the load test, and set an index status flag for each logical queue pair index; the global status flag includes running status and update status, and the index status flag includes active status, frozen status and reactivation preparation status.

[0011] The streaming thread reads the index status flags of each logical queue pair index and sends work queue elements to the remote direct memory access device through the hardware queue pairs mapped by the active logical queue pair indexes.

[0012] In response to the fulfillment of preset queue pair update conditions when the global state flag is in the running state, the control thread switches the global state flag to the update state, determines a portion of logical queue pair indices from the logical queue pair index set as target logical queue pair indices, and switches the index state flag of each target logical queue pair index from the active state to the frozen state to stop issuing new work queue elements through each target logical queue pair index; among them, other logical queue pair indices that are not determined as target logical queue pair indices but are in the active state continue to be used to issue work queue elements;

[0013] By controlling the thread to destroy the hardware queue pairs mapped to each target logical queue pair index, creating new hardware queue pairs for each target logical queue pair index, remapping the new hardware queue pairs to the corresponding target logical queue pair index, and switching the index status flag of each target logical queue pair index from frozen state to reactivation ready state.

[0014] Prepare receiving resources for the new hardware queue pairs mapped to the indexes of each target logical queue pair; in response to the completion of receiving resource preparation, switch the corresponding index status flag from the reactivation preparation state to the active state, so that the streaming thread can resume issuing work queue elements through the new hardware queue pairs; in response to the indexes of each target logical queue pair being restored to the active state, switch the global status flag to the running state through the control thread.

[0015] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above-described load testing method for RDMA devices.

[0016] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described load testing method for RDMA devices.

[0017] This invention has at least one of the following beneficial effects:

[0018] This invention sets global status flags to indicate the running phase and queue pair update phase, and assigns active, frozen, and reactivation preparation states to each logical queue pair index. When the queue pair update conditions are met, the control thread switches the target logical queue pair index to a frozen state, thereby stopping the streaming thread from issuing new work queue elements through the target logical queue pair index. Simultaneously, other logical queue pairs indexes that are not identified as the target logical queue pair index but are active continue to be used to issue work queue elements. This avoids continuing to issue tasks to hardware queue pairs during their destruction or reconstruction, and allows the RDMA device to continue handling data transmission loads even when some hardware queue pairs are dynamically changing. This allows the test process to simulate the working state of some communication queues being continuously destroyed and rebuilt, while other communication queues continue to perform high-load data transmission during actual business operations, making the test process closer to the actual use case of RDMA devices and improving coverage of dynamic connection scenarios.

[0019] Meanwhile, this invention establishes a mapping relationship between logical queue pair indices and hardware queue pairs. After destroying the hardware queue pairs originally mapped to the target logical queue pair index, it creates new hardware queue pairs for each target logical queue pair index and remaps the new hardware queue pairs to their corresponding target logical queue pair indices. Since the logical queue pair index remains unchanged before and after the hardware queue pair update, while its mapped hardware queue pairs can change dynamically, there is no need to rebuild the logical queue pair index set due to the destruction and reconstruction of hardware queue pairs, reducing the impact of dynamic changes in hardware queue pairs on the test program management structure. At the same time, by repeatedly destroying and creating hardware queue pairs, the application, release, reclamation, and reuse of queue pair contexts and associated resources can be repeatedly triggered under continuous load scenarios to test the stability of RDMA devices in dynamically managing related resources.

[0020] Furthermore, this invention, after remapping the new hardware queue pair to the target logical queue pair index, first switches the target logical queue pair index to a reactivation ready state and prepares receiving resources for the new hardware queue pair. Once the receiving resources are ready, the target logical queue pair index is switched to an active state and the distribution of work queue elements resumes. Since the new hardware queue pair only re-participates in the load test after it has the conditions to receive data, it avoids the inability to receive data normally due to resuming data transmission before the receive buffer is ready, thus improving the reliability of the dynamic recovery process and the load test process for the hardware queue pair. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a load testing method for RDMA devices provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of a queue state synchronization mechanism based on state flags, provided as an embodiment of the present invention.

[0024] Figure 3 This is a flowchart illustrating a randomized mixed load generation, data verification, and fault replay method provided in an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention primarily describes the load testing process for RDMA devices. Embodiment 1 mainly explains how to dynamically destroy and rebuild certain hardware queue pairs while the RDMA device continuously bears the data transmission load, and how to coordinate the streaming process with the state changes of the hardware queue pairs. Embodiment 2, based on Embodiment 1, further explains how to generate random mixed loads, verify transmitted data, and replay load test sequences corresponding to faults. These two embodiments together constitute a specific implementation of the load testing method for RDMA devices.

[0027] Example 1

[0028] As one possible embodiment of the present invention, such as Figure 1 As shown, a load testing method for RDMA devices is provided, including the following steps.

[0029] Step S1: Establish a set of logical queue pairs indexes, create hardware queue pairs for each logical queue pair index and establish a mapping relationship, set a global status flag to record the processing stage of the load test, and set an index status flag for each logical queue pair index.

[0030] In this embodiment, the logical queue pair index is a logical identifier set by the test program at the software level. The logical queue pair index persists during the load test, while the hardware queue pair mapped to it can be destroyed and replaced with a newly created hardware queue pair. Both the streaming thread and the control thread use the logical queue pair index as the software management object, and then determine the corresponding hardware queue pair based on the mapping relationship.

[0031] At the start of load testing, the number of logical queue pairs requiring indexes can be determined based on the test scale, and a set of logical queue pair indexes can be created accordingly. The test program creates corresponding hardware queue pairs for each logical queue pair index and records the mapping relationship between the logical queue pair indexes and the hardware queue pairs. Subsequently, both the streaming thread and the control thread can determine which hardware queue pairs need to be used or operated through the logical queue pair indexes and their management records.

[0032] By separating the logical queue pair indexes from the hardware queue pairs, even if a hardware queue pair is destroyed and rebuilt during load testing, the test program does not need to delete or rebuild the logical queue pair index set accordingly. Instead, it only needs to update the hardware queue pairs mapped by the corresponding logical queue pair indexes. This allows for dynamic updates of hardware queue pairs and their associated resources while maintaining a relatively stable test program management structure.

[0033] The global status flags include running status and update status, while the index status flags include active status, frozen status, and reactivation ready status.

[0034] Furthermore, the logical queue pair index set includes multiple logical queue pair indexes that remain in existence during load testing, and each logical queue pair index has a corresponding management record.

[0035] The management records include at least the logical queue pair index identifier, the hardware queue pair identifier, and the index status flag.

[0036] The information in the management records and their functions are shown in Table 1.

[0037] Table 1

[0038]

[0039] The activity status is used to indicate that the corresponding logical queue pair index mapping has a hardware queue pair that is allowed to issue work queue elements.

[0040] The frozen state is used to indicate the cessation of issuing new work queue elements to the index through the corresponding logical queue, and to allow the control thread to destroy the hardware queue pair mapped to the logical queue to the index.

[0041] The reactivation of the ready state indicates that the corresponding logical queue pair index has been mapped to a new hardware queue pair and is preparing to receive resources for the new hardware queue pair.

[0042] The correspondence between global status flags and index status flags can be seen in Table 2.

[0043] Table 2

[0044]

[0045] The global status flag is used to indicate the overall processing stage of a load test, while the index status flag is used to indicate whether a single logical queue can participate in WQE deployment. The two have different scopes of application.

[0046] In this embodiment, the streaming thread refers to the execution thread that continuously generates, organizes, and sends WQEs to the RDMA device during load testing. The streaming thread reads the index status flags of each logical queue pair index and determines whether to send a new WQE through the corresponding logical queue pair index based on the reading results. The control thread is used to determine the logical queue pair indexes that need to be dynamically updated and to control the switching of the corresponding status flags as well as the destruction and reconstruction of hardware queue pairs.

[0047] With the above settings, a relatively stable logical queue can be used to manage dynamically changing hardware queue pairs, and two-level status flags can be used to coordinate the control thread and the streaming thread.

[0048] like Figure 2 As shown, when the global status flag is in the running state, each logical queue pair index participates in the work queue element distribution according to its index status; after the preset queue pair update conditions are met, the global status flag switches to the update state, and the target logical queue pair index sequentially goes through the frozen state, the reactivation preparation state, and the active state. After all the target logical queue pair indexes are restored to the active state, the global status flag is switched to the running state.

[0049] Step S2: The streaming thread reads the index status flags of each logical queue pair index and sends work queue elements to the remote direct memory access device through the hardware queue pairs mapped by the active logical queue pair indexes.

[0050] Step S3: In response to the preset queue pair update condition being met when the global state flag is in the running state, the global state flag is switched to the update state through the control thread, and a portion of the logical queue pair indices are determined from the logical queue pair index set as target logical queue pair indices. The index state flag of each target logical queue pair index is switched from the active state to the frozen state to stop issuing new work queue elements through each target logical queue pair index.

[0051] The target logical queue pair index refers to the logical queue pair index determined by the control thread from the active logical queue pair indexes during the current round of queue pair updates. This index determines the logical queue pair whose mapped hardware queue pairs need to be frozen, destroyed, or rebuilt. Setting the target logical queue pair index distinguishes the logical queue pair indexes that require hardware queue pair updates in the current round from other logical queue pair indexes that continue to bear load transmission.

[0052] Among them, other logical queue pairs that are not identified as target logical queue pairs but are in an active state continue to be used to issue work queue elements.

[0053] Further, step S3, the step of determining a portion of the logical queue pair indices from the set of logical queue pair indices as the target logical queue pair indices, includes:

[0054] Step S31: By controlling the thread, determine the target logical queue pair index from the currently active logical queue pair indexes according to a preset number or preset ratio.

[0055] Steps S2, S3, and S31 together define the coordination process between normal streaming and freezing of the target logical queue for the index. During normal load testing, the streaming thread reads the index status flag of each logical queue for the index and only issues WQE to the index through the logical queue that is in an active state.

[0056] The preset queue pair update condition can be the expiration of a preset time, the completion of a preset load test round, or the receipt of a queue pair update command. In response to meeting this condition, the control thread switches the global state flag to the update state and determines the target logical queue pair index according to a preset number or preset proportion. The above triggering conditions are merely examples and do not constitute a limitation on the preset queue pair update conditions.

[0057] In one possible implementation, the control thread can randomly select unique target logical queue pairs from the currently active logical queue pairs, provided that a preset number or proportion is met. Alternatively, the target logical queue pairs can be selected according to a preset index order or other selection rules.

[0058] After the control thread switches the target logical queue pair index to a frozen state, the streaming thread no longer issues new WQEs through that target logical queue pair index. However, other logical queue pairs that are still active continue to be used to issue WQEs. Therefore, switching the global state flag to the update state does not mean that all logical queue pairs have stopped streaming.

[0059] In real-world business scenarios such as distributed storage, high-performance computing, or data center networks, some communication tasks may end or be re-established, requiring the corresponding hardware queue pairs to be destroyed and recreated; meanwhile, other communication tasks still need to continue data transmission. By freezing only the target logical queue pair index and allowing other active indexes to continue issuing WQE, a dynamic communication scenario corresponding to the above business states can be formed during testing, avoiding the complete halt of load transmission due to updating some hardware queue pairs.

[0060] Step S4: By controlling the thread, destroy the hardware queue pairs mapped to each target logical queue pair index, create new hardware queue pairs for each target logical queue pair index, remap the new hardware queue pairs to the corresponding target logical queue pair index, and switch the index status flag of each target logical queue pair index from frozen state to reactivation ready state.

[0061] Further, step S4 includes:

[0062] Step S41: When destroying the hardware queue pair mapped to the target logical queue pair index, retain the target logical queue pair index and its management record, and remove the mapping relationship between the target logical queue pair index and the destroyed hardware queue pair.

[0063] Step S42: After creating a new hardware queue pair, write the hardware queue pair identifier of the new hardware queue pair into the management record to update the mapping relationship corresponding to the target logical queue pair index.

[0064] Steps S4, S41, and S42 together define the process of keeping the logical queue pair index unchanged while dynamically replacing the hardware queue pair. The control thread first determines the hardware queue pair that needs to be destroyed based on the hardware queue pair identifier in the management record and removes the original mapping relationship; after creating a new hardware queue pair, the new hardware queue pair identifier is written to the same management record.

[0065] For example, the original hardware queue pair identifier mapped to logical queue pair index 2 could be 7. After the control thread destroys hardware queue pair 7 and removes the original mapping, logical queue pair index 2 and its management record continue to exist. Assuming the newly created hardware queue pair identifier is 20, then hardware queue pair identifier 20 is written to the management record corresponding to logical queue pair index 2. After the update, logical queue pair index 2 is mapped to hardware queue pair 20. The above identifiers are merely examples to illustrate the mapping change.

[0066] After completing the new mapping relationship, the control thread first switches the corresponding index status flag to the reactivation preparation state, instead of directly switching it to the active state, so as to further prepare the receiving resources before resuming WQE issuance.

[0067] Step S5: Prepare receiving resources for the new hardware queue pairs mapped to the indices of each target logical queue pair. In response to the completion of receiving resource preparation, switch the corresponding index status flag from the reactivation preparation state to the active state, so that the streaming thread can resume issuing work queue elements through the new hardware queue pairs. In response to all target logical queue pair indices being restored to the active state, switch the global status flag to the running state through the control thread.

[0068] Further, step S5, the step of preparing receiving resources for the new hardware queue pairs mapped to each target logical queue pair index, includes:

[0069] Step S51: Configure a receive buffer for the new hardware queue pair to receive data.

[0070] Following step S51, the following is also included:

[0071] In step S52, in response to determining that the receive buffer can accept the data transmitted through the new hardware queue pair, the index status flag of the corresponding target logical queue pair is switched from the reactivation ready state to the active state, so that the streaming thread can send work queue elements through the new hardware queue pair according to the switched index status flag.

[0072] After all target logical queue pairs have been restored to the active state and the global state flags have been restored to the running state, the following steps are also included:

[0073] Step S6: In response to the preset queue pair update condition being met again, the target logical queue pair index is redefined to periodically destroy and rebuild some hardware queue pairs.

[0074] Steps S5, S51, and S52 collectively define the process from the completion of mapping to the resumption of streaming for the new hardware queue pair. The receiving resources in this embodiment primarily include, and at least explicitly as defined in the claims, the receive buffer, i.e., the memory space at the receiving end used to receive data transmitted through the new hardware queue pair.

[0075] For a transmit operation, receive resources may include a receive buffer allocated by the receiver for the new hardware queue pair. In implementations that require submitting receive tasks to a receive work queue, receive resources may also include a receive WQE corresponding to the receive buffer, so that data transmitted through the new hardware queue pair can be written into the appropriate receive buffer.

[0076] For RDMA write operations, the receive resource may include a target memory region used by the receiver to receive the written data. This target memory region is used to receive the data written by the sender through a new hardware queue.

[0077] For other data transmission operations, a memory area or buffer capable of receiving data can be configured for the new hardware queue according to the data flow direction of the corresponding operation, so that the data generated by the corresponding operation has a definite receiving location.

[0078] The above-described receiving resource formats are examples and do not constitute a limitation on the specific structure of receiving resources. "Receiving resources ready" in this solution mainly refers to the receiving end having a receiving buffer or target memory area capable of handling data transmitted through the new hardware queue. The size, number, and organization of the receiving buffer can be determined based on message length, operation type, and load test scale.

[0079] Step S51 is used to configure the receive buffer, and step S52 is used to restore the active state after confirming that the receive buffer can receive data. These two are different processing stages that are connected one after the other. The new hardware queue only participates in load balancing after the receive resources are ready, which can avoid prematurely resuming WQE transmission when the receiving end is not yet ready to receive data.

[0080] When all target logical queue pairs are restored to the active state in this round, the control thread restores the global state flag to the running state. In response to the preset queue pair update conditions being met again, the process of determining, freezing, destroying, rebuilding, and restoring the target logical queue pair indexes is re-executed.

[0081] Different update rounds can determine the same or different target logical queue pair indices. For example, in the first round, logical queue pair index 2 and logical queue pair index 4 can be selected, while in the next round, logical queue pair index 3 and logical queue pair index 5 can be selected. This example does not constitute a limitation on the order or number of selections.

[0082] In real-world applications, RDMA devices may maintain a large number of hardware queue pairs simultaneously, continuously requesting, releasing, and reallocating queue pair resources as different business tasks start and end. By periodically destroying and rebuilding some hardware queue pairs while allowing others to continue handling data transmission loads, the requesting, releasing, reclamation, and reuse of queue pair contexts and associated resources can be repeatedly triggered under continuous load scenarios, thereby testing the stability of RDMA devices in handling dynamic queue pair resource changes.

[0083] The above embodiment 1, through the mapping relationship between logical queue pairs and hardware queue pairs, and the coordination between global status flags and index status flags, enables some hardware queue pairs to be destroyed, rebuilt, and restored during the continuous data transmission of other hardware queue pairs, thus solving the coordination problem between the continuous streaming process and the life cycle changes of hardware queue pairs.

[0084] However, existing RDMA testing tools typically require pre-specifying an opcode and using fixed message lengths, WQE chain lengths, and barrier flag configurations when generating test loads. In this case, the WQEs issued in a single test are relatively homogeneous, making it difficult to cover mixed traffic scenarios in actual business where the same communication connection continuously carries different operation types, message lengths, and task combinations. Furthermore, existing testing tools usually determine data transmission anomalies primarily based on WQE completion information, making it difficult to detect situations where the completion information is error-free but the transmitted data content has changed. For faults occurring in random load testing, the inability to regenerate the random load sequence at the time of the fault may increase reproduction and location time. Therefore, the following Example 2, based on Example 1, further refines the WQE generation and issuance process in step S2. By generating random mixed WQE chains, verifying transmitted data, and recording and replaying random seeds, the load test can cover a wider variety of business loads and improve data error detection and random fault reproduction capabilities.

[0085] Example 2

[0086] As another possible embodiment of the present invention, such as Figure 3 As shown, step S2 in Embodiment 1, which is the step of issuing work queue elements to a remote direct memory access device through the hardware queue pair mapped by the logical queue pair index that is in an active state, includes the following steps.

[0087] Step S21: Obtain the hybrid load configuration information set according to the target business scenario.

[0088] The mixed load configuration information includes the set of allowed opcodes, message length range, work queue element chain length range, and fence flag configuration information.

[0089] The opcode set includes at least two of the following: remote direct memory access write operation, remote direct memory access read operation, and send operation.

[0090] In the mixed load configuration information, the opcode set is used to limit the types of WQE operations that are allowed to be generated. An opcode (OP code) represents the data transmission operation type corresponding to the WQE. In some implementations, at least two operations can be determined from RDMA write operations, RDMA read operations, and transmit operations, based on the target service scenario, to form the allowed opcode set.

[0091] The message length range is used to limit the amount of data to be transmitted for a single WQE. By setting the message length range, different WQEs generated subsequently can have different data transmission lengths, and the randomly generated message length is limited to the range that may be used in the target business scenario.

[0092] The WQE chain length range is used to limit the number of WQEs included in a single WQE chain issued to the hardware queue. For example, when the determined WQE chain length is 10, the corresponding WQE chain can include 10 WQEs; when the next determined WQE chain length is 100, the corresponding WQE chain can include 100 WQEs. The above values ​​are only used to illustrate that the WQE chain length can vary and do not constitute a limitation on the range of WQE chain lengths.

[0093] Fence flags are used to indicate the execution constraints between related WQEs. Fence flag configuration information is used to limit the fence flag configurations allowed when generating WQEs, so that subsequently generated WQEs can form different task combinations according to the corresponding fence configurations.

[0094] The target business scenario represents the actual business scenario that needs to be simulated or covered through load testing. Different business scenarios may use different RDMA operation types and have different message length distributions and task combinations. Therefore, the hybrid load configuration information is used to limit the range of optional random parameters, rather than allowing various load parameters to be generated arbitrarily without constraints.

[0095] For example, if a specific business scenario primarily uses RDMA write and send operations, and message lengths are concentrated within a certain range, the opcode set can be set to include both RDMA write and send operations, and the message length range can be set to the range corresponding to that business scenario. The above example does not constitute a limitation on opcode types or message length ranges.

[0096] By configuring the opcode set, message length range, WQE chain length range, and fence flag configuration information according to the target business scenario, random parameter combinations that are obviously unrelated to the target business scenario can be reduced, making the subsequently generated random mixed load more targeted in covering the possible task combinations of the target business.

[0097] Step S22: Determine the opcode, message length, work queue element chain length, and fence flag within the range defined by the mixed load configuration information, based on the random seed corresponding to the index of each logical queue pair.

[0098] Step S23: Generate a hybrid work queue element chain based on the determined opcode, message length, work queue element chain length, and fence flag, and distribute the hybrid work queue element chain through the corresponding hardware queue.

[0099] In this context, the same hardware queue may have different opcodes and / or message lengths for different work queue elements.

[0100] A random seed is an initial value used to initialize a pseudo-random number generator function. The pseudo-random number generator function generates a sequence of random numbers according to defined computation rules; it can repeatedly generate the same random number sequence using the same random seed, the same pseudo-random number generator function, and the same function call order. Therefore, a random seed can be used both to generate randomized loads and as a basis for regenerating the same load sequence.

[0101] In this embodiment, each logical queue pair index can correspond to a random seed. The test program calls the pseudo-random number generation function according to the corresponding random seed, and determines the corresponding value from the opcode set, message length range, WQE chain length range, and fence flag configuration information based on the generated random number.

[0102] For example, random numbers used to determine each parameter can be generated sequentially according to a preset generation order of opcode, message length, WQE chain length, and fence flag. For the opcode, an opcode can be determined from the opcode set based on the random number and the number of opcodes included in the opcode set; for the message length and WQE chain length, the corresponding random numbers can be mapped to their respective value ranges.

[0103] For example, assuming the random seed for logical queue index 1 is 10, after initializing the pseudo-random number generation function, the test program calls the function for the first time to obtain a random number R1, and determines the opcode from the opcode set based on R1; the second call obtains a random number R2, and determines the message length based on R2; the third call obtains a random number R3, and determines the WQE chain length based on R3; the fourth call obtains a random number R4, and determines the fence flag based on R4. When using a random seed of 10 again and calling the same pseudo-random number generation function in the same order, R1, R2, R3, and R4 can be obtained again, and the same load parameters can be re-determined. The random seed value and random number symbol mentioned above are only used to illustrate the process of generating random parameters.

[0104] The test program generates a corresponding number of WQEs based on the determined WQE chain length, and configures the opcode, message length, and barrier flag for each WQE to form a hybrid WQE chain. The streaming thread reads the index status flag of the corresponding logical queue pair, and when the index status flag is active, it sends the hybrid WQE chain through its currently mapped hardware queue pair.

[0105] In the queue pair update process of Implementation Example 1, after the target logical queue pair index is switched to a frozen state or reactivated to a ready state, the streaming thread stops issuing new hybrid WQE chains through that index; the active index that is not selected continues to issue hybrid WQE chains; after the target index is restored to an active state, it resumes issuing through the newly mapped hardware queue pair.

[0106] Different WQEs issued by the same hardware queue can have different opcodes and / or message lengths. For example, a hybrid WQE chain can include RDMA write operations, send operations, and RDMA read operations in sequence, with different WQEs corresponding to different message lengths. This example does not constitute a limitation on the order of WQEs or the values ​​of their parameters.

[0107] In real-world business scenarios, the same communication connection may continuously perform write, read, and send operations, transmitting data of varying lengths. By randomly generating opcodes, message lengths, WQE chain lengths, and barrier flags within the defined scope of the target business scenario, the same hardware queue can be used to carry different types and lengths of WQE combinations, thereby improving the test load's coverage of actual mixed business traffic. The effect of this technique is reflected in the increased coverage of the test scenario, rather than directly equating randomization with more accurate test results.

[0108] Furthermore, this embodiment also includes the following steps.

[0109] Step S24: In response to receiving a data verification instruction for indicating that the transmitted data in the load test should be verified, mutually isolated memory regions are configured for different opcodes, and the source memory location and target memory location of the data to be transmitted are determined according to the opcode to be executed and the hardware queue pair to perform the corresponding operation.

[0110] Step S25: Synchronize the random seed used to generate verification data between the two communication ends of the load test, so that the two communication ends generate the same data sequence according to the same random seed.

[0111] Step S26: According to the determined source memory location and target memory location, perform data transmission through the corresponding hardware queue, and compare the actual data located at the corresponding target memory location after data transmission with the expected data generated according to the random seed to determine whether the data transmission of the remote direct memory access device is correct.

[0112] Steps S24 to S26 collectively define the data verification process. For RDMA write operations, data in the local memory can be written to the target memory area corresponding to the peer, and a relatively independent memory area is configured for the write operation to avoid overwriting data corresponding to other operations. For RDMA read operations, data in the peer's memory can be read into the target memory area corresponding to the local memory, and verification is performed in a separate read operation memory area. For transmit operations, local data can be sent to the peer's receive buffer, and the receive area used for the transmit operation is isolated from the memory area used by the unilateral RDMA operation.

[0113] The data transmission relationships described above are only used to illustrate the data flow of different opcodes and the reasons for setting up isolated memory regions, and do not constitute a limitation on specific memory addresses, memory capacities, or memory layout methods. When multiple hardware queue pairs transmit concurrently, the test program can further determine the source memory location and target memory location in the memory region of the corresponding opcode based on the hardware queue pair performing the corresponding operation, so that the data of different opcodes and different hardware queue pairs are placed in identifiable locations.

[0114] In step S25, both communication terminals can initialize the same pseudo-random number generation function using the same random seed and call the function in the same data generation order, thereby generating the same data sequence respectively. One communication terminal uses the generated data sequence as the data to be transmitted, while the other communication terminal uses the independently generated identical data sequence as the expected data.

[0115] For example, suppose both communication endpoints synchronize using a random seed of 20. The first communication endpoint initializes a pseudo-random number generation function with a random seed of 20 and continuously calls the function to generate data D1, D2, and D3, then writes D1, D2, and D3 sequentially into the source memory region where the data to be transmitted resides. The second communication endpoint also initializes the same pseudo-random number generation function with a random seed of 20 and calls the function in the same order, thus generating the same data D1, D2, and D3 sequentially, which is then used as the expected data. After the first communication endpoint transmits data through the corresponding hardware queue, the second communication endpoint can compare the actual data in the target memory location with D1, D2, and D3. The random seed values ​​and data symbols described above are only used to illustrate the process of the two communication endpoints generating the same data sequence.

[0116] In this way, the two communication ends do not need to synchronize the complete expected data sequence before data transmission. They only need to synchronize the random seed used to initialize the pseudo-random number generation function to obtain the corresponding data to be transmitted and expected data, thereby reducing the amount of data transmitted between the two communication ends to establish the expected data.

[0117] After the data transfer is complete, the test program retrieves the actual data from the corresponding target memory location and compares it with the expected data according to the corresponding data length and order. If the actual data matches the expected data, it can be determined that the data transfer meets expectations; if they do not match, it can be determined that there are data content errors in the data transfer.

[0118] In high-load testing scenarios for RDMA devices, numerous data transfer tasks may return normal WQE completion information. However, normal completion information does not directly prove that the data content in the target memory is consistent with expectations. By further comparing the actual data with the expected data, data content errors can be detected even when the completion information does not report any anomalies, thereby improving the ability to detect the correctness of data transfer on RDMA devices.

[0119] Furthermore, this embodiment also includes the following steps.

[0120] Step S27: Record the random seed used by each logical queue for indexing in each round of load testing.

[0121] Step S28: In response to a load test failure, determine the load test round corresponding to the failure and obtain the random seed corresponding to the determined load test round.

[0122] Step S29: Based on the obtained random seed, regenerate the corresponding opcode, message length, work queue element chain length, and fence flag according to the same random parameter generation order as the determined load test round, so as to replay the load test sequence corresponding to the fault.

[0123] Steps S27 to S29 together define the recording of the random load test sequence and the fault replay process. The random seed can be used in three interrelated stages: first, in the hybrid WQE generation process, to determine the opcode, message length, WQE chain length, and barrier flag within the range defined by the hybrid load configuration information; second, in the data verification process, to ensure that the two communication ends generate the same data sequence; and third, in the fault replay process, to regenerate the load parameter sequence at the time of the fault by combining the same random parameter generation order.

[0124] The test program can record the random seed used by each logical queue pair index in the current load test round, and establish the association between the load test round, the logical queue pair index, and the random seed.

[0125] When a load test fails, the load test round is determined based on the test progress at the time of the failure, and the random seed used by each logical queue pair index in that round is obtained. Subsequently, the obtained random seed is reconfigured to the corresponding logical queue pair index, and the opcode, message length, WQE chain length, and barrier flag are regenerated according to the same random parameter generation order as the original load test round.

[0126] If the random seed and the order of random parameter generation remain consistent, the same random parameter sequence as the original load test round can be regenerated, and the corresponding hybrid WQE chain can be regenerated and issued accordingly.

[0127] In random load tests lasting several hours or longer, some low-probability failures may only occur under specific combinations of random parameters. Re-executing the entire test process is not only time-consuming but also may not regenerate the same load sequence. By recording the random seed for the round corresponding to the failure and replaying the corresponding load test sequence in the same random parameter generation order, the scope of tests that need to be repeated can be narrowed, the reproducibility of low-probability random failures can be improved, and the time required for fault localization can be reduced.

[0128] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0129] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0130] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0131] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as “circuit,” “module,” or “system.”

[0132] An electronic device according to this embodiment of the invention. The electronic device is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the invention.

[0133] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0134] The storage device stores program code that can be executed by the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of the present invention.

[0135] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0136] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0137] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0138] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0139] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0140] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0143] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0144] Furthermore, the accompanying drawings are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes shown in the above drawings do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0145] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0146] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A load testing method for RDMA devices, characterized in that, Includes the following steps: Establish a set of logical queue pairs indexes, create hardware queue pairs for each logical queue pair index and establish a mapping relationship, set a global status flag to record the processing stage of the load test, and set an index status flag for each logical queue pair index; wherein, the global status flag includes running status and update status, and the index status flag includes active status, frozen status and reactivation preparation status. The streaming thread reads the index status flags of each logical queue pair index and sends work queue elements to the remote direct memory access device through the hardware queue pairs mapped by the logical queue pair indexes that are in the active state. In response to the global state flag being in the running state and satisfying a preset queue pair update condition, the control thread switches the global state flag to the update state, determines a portion of logical queue pair indices from the logical queue pair index set as target logical queue pair indices, and switches the index state flag of each target logical queue pair index from the active state to the frozen state to stop issuing new work queue elements through each target logical queue pair index; wherein, other logical queue pair indices that are not determined as target logical queue pair indices but are in the active state continue to be used to issue work queue elements; The control thread destroys the hardware queue pairs mapped to each target logical queue pair index, creates new hardware queue pairs for each target logical queue pair index, remaps the new hardware queue pairs to the corresponding target logical queue pair index, and switches the index status flag of each target logical queue pair index from the frozen state to the reactivation preparation state. Prepare receiving resources for the new hardware queue pairs mapped to the indexes of each target logical queue pair; in response to the completion of the receiving resource preparation, switch the corresponding index status flag from the reactivation preparation state to the active state, so that the streaming thread resumes issuing work queue elements through the new hardware queue pairs; in response to the indexes of each target logical queue pair being restored to the active state, switch the global status flag to the running state through the control thread.

2. The method according to claim 1, characterized in that, The logical queue pair index set includes multiple logical queue pair indexes that remain present during the load test, and each logical queue pair index has a corresponding management record; the management record includes at least a logical queue pair index identifier, a hardware queue pair identifier, and an index status flag; When destroying the hardware queue pair mapped to the target logical queue pair index, the target logical queue pair index and its management record are retained, and the mapping relationship between the target logical queue pair index and the destroyed hardware queue pair is released. After creating the new hardware queue pair, the hardware queue pair identifier of the new hardware queue pair is written into the management record to update the mapping relationship corresponding to the target logical queue pair index.

3. The method according to claim 1, characterized in that, The activity status is used to indicate that the corresponding logical queue pair index mapping has a hardware queue pair that allows the issuance of work queue elements; The frozen state is used to indicate the cessation of issuing new work queue elements to the index through the corresponding logical queue, and to allow the control thread to destroy the hardware queue pair mapped to the logical queue to the index; The reactivation readiness state indicates that the corresponding logical queue pair index has been mapped to the new hardware queue pair, and is preparing to receive resources for the new hardware queue pair.

4. The method according to claim 1, characterized in that, The step of determining a portion of the logical queue pair indices from the set of logical queue pair indices as target logical queue pair indices includes: The control thread determines the target logical queue pair index from the currently active logical queue pair indexes according to a preset number or a preset ratio. After all target logical queue pairs have been restored to the active state and the global state flag has been restored to the running state, in response to the preset queue pair update condition being met again, the target logical queue pair index is redefined so as to periodically destroy and rebuild some hardware queue pairs.

5. The method according to claim 1, characterized in that, The steps for preparing new hardware queue pairs mapped to the indexes of each target logical queue pair to receive resources include: Configure a receive buffer for receiving data for the new hardware queue pair; Following the step of preparing resources to receive new hardware queue pairs mapped to each target logical queue pair index, the method further includes: In response to determining that the receive buffer can accept data transmitted through the new hardware queue pair, the index status flag of the corresponding target logical queue pair is switched from the reactivation ready state to the active state, so that the streaming thread issues work queue elements through the new hardware queue pair according to the switched index status flag.

6. The method according to claim 1, characterized in that, The step of issuing work queue elements to the remote direct memory access device through the hardware queue pair mapped by the logical queue pair index in the active state includes: Obtain hybrid load configuration information set according to the target business scenario; the hybrid load configuration information includes the set of allowed opcodes, message length range, work queue element chain length range, and fence flag configuration information; the set of opcodes includes at least two of remote direct memory access write operations, remote direct memory access read operations, and send operations. Based on the random seed corresponding to the index of each logical queue pair, the opcode, message length, work queue element chain length, and fence flag are determined within the range defined by the hybrid load configuration information. A hybrid work queue element chain is generated based on the determined opcode, message length, work queue element chain length, and fence flag, and the hybrid work queue element chain is issued to the corresponding hardware queue; wherein, the same hardware queue has different opcodes and / or message lengths for different corresponding work queue elements.

7. The method according to claim 6, characterized in that, The method further includes: In response to receiving a data verification instruction indicating that the transmitted data in the load test should be verified, mutually isolated memory regions are configured for different opcodes, and the source memory location and target memory location of the data to be transmitted are determined according to the opcode to be executed and the hardware queue pair to perform the corresponding operation. A random seed for generating verification data is synchronized between the two communication ends of the load test, so that the two communication ends generate the same data sequence according to the same random seed. According to the determined source memory location and target memory location, data transmission is performed through the corresponding hardware queue, and the actual data located at the corresponding target memory location after data transmission is compared with the expected data generated according to the random seed to determine whether the data transmission of the remote direct memory access device is correct.

8. The method according to claim 6, characterized in that, The method further includes: Record the random seed used by each logical queue for the index in each round of load testing; In response to a failure in the load test, the load test round corresponding to the failure is determined, and a random seed corresponding to the determined load test round is obtained. Based on the obtained random seed, the corresponding opcode, message length, work queue element chain length, and fence flag are regenerated in the same random parameter generation order as the determined load test round, so as to replay the load test sequence corresponding to the fault.

9. A non-transitory computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the load testing method for RDMA devices as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the load testing method for RDMA devices as described in any one of claims 1 to 8.

Images