Distributed IO access routing method based on middleware and related equipment

Abstract

The invention provides a middleware-based distributed IO access routing method and related equipment, and the method comprises the steps: deploying a middleware layer between a client and server nodes, periodically collecting the load state information of each server node, and dynamically adjusting a scheduling parameter according to the load state information; in response to an IO access request sent by a client, querying a preset path mapping rule according to a target path in the request to obtain a candidate server node set; and performing load evaluation on the candidate server nodes according to the load state information and the scheduling parameters to obtain a target server node, and routing the IO access request to the target server node. According to the method, response type regulation and control on system load changes are achieved by adjusting the scheduling parameters in a self-adaptive mode, load inclination is effectively relieved while path mapping locality is kept, and system tail delay is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a middleware-based distributed I / O access routing method and related equipment. Background Technology

[0002] Distributed storage systems have become a core infrastructure supporting massive data storage and processing. In distributed storage systems, efficient routing of I / O access requests is a key factor in ensuring system performance. Traditional distributed I / O access routing methods typically employ static load balancing strategies, such as round-robin, random selection, or hash-based fixed mapping, to distribute client access requests to various server nodes.

[0003] While existing dynamic routing methods can make routing decisions based on the real-time load of server nodes, they typically use fixed routing policy parameters and cannot adaptively adjust the routing strength according to changes in the overall system load. This results in uneven load distribution and excessively high latency at the tail of the system in scenarios with sudden load surges or load imbalances. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that existing distributed I / O access routing methods use fixed routing strategy parameters and cannot adaptively adjust the routing strength according to changes in the overall system load, resulting in uneven load distribution and excessively high latency at the tail of the system. This invention provides a middleware-based distributed I / O access routing method applied to a distributed storage system, the distributed storage system including a client and multiple server nodes, characterized in that the distributed I / O access routing method includes: The middleware layer deployed between the client and the server node periodically collects the load status information of each server node, and adjusts the scheduling parameters periodically based on the load status information. In response to the IO access request sent by the client, a set of candidate server nodes is obtained by querying a preset path mapping rule according to the target path in the IO access request. Based on the load status information and the scheduling parameters, the load of each candidate server node in the candidate server node set is evaluated to obtain the target server node, and the IO access request is routed to the target server node.

[0005] This invention also provides a middleware-based distributed I / O access routing device applied to a distributed storage system, the distributed storage system including a client and multiple server nodes, characterized in that the distributed I / O access routing device includes: The load monitoring module is used to periodically collect the load status information of each server node through the middleware layer deployed between the client and the server node, and periodically adjust the scheduling parameters according to the load status information. The candidate query module is used to respond to the IO access request sent by the client, query the preset path mapping rules according to the target path in the IO access request, and obtain a set of candidate server nodes. The routing decision module is used to perform load assessment on each candidate server node in the candidate server node set according to the load status information and the scheduling parameters, obtain the target server node, and route the IO access request to the target server node.

[0006] The present invention also provides a middleware-based distributed I / O access routing device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; the at least one processor invokes the instructions in the memory to cause the middleware-based distributed I / O access routing device to execute the steps of the middleware-based distributed I / O access routing method described above.

[0007] The present invention also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the middleware-based distributed I / O access routing method described above.

[0008] The aforementioned middleware-based distributed I / O access routing method and related devices deploy a middleware layer between the client and server nodes to periodically collect load status information of each server node and dynamically adjust scheduling parameters based on this information. In response to I / O access requests from clients, the method queries preset path mapping rules based on the target path in the request to obtain a set of candidate server nodes. Then, it performs load assessment on the candidate server nodes based on the load status information and scheduling parameters to determine the target server node and routes the I / O access request to it. This invention achieves responsive control to system load changes by adaptively adjusting scheduling parameters, effectively mitigating load skew and reducing system tail latency while maintaining path mapping locality.

[0009] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0010] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the first embodiment of the middleware-based distributed I / O access routing method in this invention. Figure 2 This is a schematic diagram of a second embodiment of the middleware-based distributed I / O access routing method in this invention. Figure 3 This is a schematic diagram of an embodiment of the middleware-based distributed I / O access routing device of the present invention; Figure 4 This is a schematic diagram of an embodiment of a middleware-based distributed I / O access routing device according to the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0014] To facilitate understanding of this embodiment, a middleware-based distributed I / O access routing method disclosed in this embodiment will first be described in detail. This distributed I / O access routing method is applied to a distributed storage system, which includes a client and multiple server nodes. For example... Figure 1 As shown, this method includes the following steps: 101. The middleware layer deployed between the client and the server node periodically collects the load status information of each server node, and adjusts the scheduling parameters periodically based on the load status information. In this embodiment, the periodic collection of load status information of each server node through the middleware layer deployed between the client and the server node includes: collecting real-time load data of each server node at preset time intervals through the middleware layer deployed between the client and the server node, wherein the real-time load data includes the current queue length, median processing latency, and tail processing latency; performing an exponentially weighted moving average calculation on the real-time load data to obtain a smoothed load index for each server node, and storing and maintaining the smoothed load index as load status information.

[0015] Specifically, the middleware layer is deployed using a proxy process. In a distributed storage system, independent physical machines or virtual machines are selected as middleware nodes, and these nodes are located between the client and server nodes on the network. After the middleware process starts, it opens a listening port locally. Clients modify their configuration files to redirect I / O access requests originally sent to the server node to the middleware's listening port.

[0016] Two types of long-lived TCP connections are established between the middleware layer and the server nodes. The data channel is used to forward I / O access requests and receive response data, while the management channel is dedicated to collecting load information. Upon startup, the middleware layer proactively initiates a management connection request to each server node. After accepting the connection, the server node allocates an independent management connection and maintains it. The management channel and data channel operate separately and do not interfere with each other.

[0017] In the implementation of load data collection, the middleware process creates a background monitoring thread using the pthread library. The thread's main loop sets a timer and calls the `sleep` function to sleep for a preset time interval. When the timer expires, the thread is awakened, reads the identifiers of all server nodes and the socket descriptors of the management channel from the node list in memory, and iterates through each node sequentially.

[0018] For each node, a thread constructs a query message. The query message uses a custom protocol format; the message header includes a type field indicating that it is a load balancing query request, and the message body is empty. The thread sends the query message by calling the `send` function on the management channel's socket. The server node's management module listens for the message, parses the message header, and identifies the load balancing query request.

[0019] The statistics module running inside the server node communicates with the I / O processing module via shared memory, reading the length of the current request queue from the shared memory. The statistics module also maintains a circular buffer, recording the processing time of the most recently completed requests. Whenever a request is completed, its processing time is written to the buffer, and the position pointer is moved forward in a circular fashion.

[0020] When a server node receives a query request, it reads the queue length from shared memory. For latency metrics, the statistics module iterates through the circular buffer, copies valid duration data to a temporary array, and sorts it using a quicksort algorithm. After sorting, the value at the middle position of the array is used as the median latency, and the value at the end of the array is used as the tail latency. These three values ​​are encapsulated in a response message with the message header type set to load response, and sent back to the middleware layer via the management channel socket.

[0021] After sending a query message, the monitoring thread in the middleware layer calls the `recv` function on the socket to wait for a response. Upon receiving the response message, it parses the message body to extract three real-time load data points. The thread reads the previous smoothed load metric for that node from the node list, recording it as the previous value, and the currently collected real-time data as the current value. The thread performs an exponentially weighted moving average calculation, multiplying the previous value by a smoothing coefficient close to 1 and the current value by a smaller coefficient, adding the two together to obtain a new smoothed load metric. This calculation makes the load metric change smoother, and the influence of historical data gradually diminishes. The calculation result replaces the smoothed metric value of that node in the node list.

[0022] The middleware layer maintains a hash table in memory as a server status table. The keys are node identifiers, and the values ​​are structures containing smoothing metrics such as queue length, median latency, tail latency, and update timestamps. After completing the smoothing calculation, the thread writes the result to the hash table and updates the timestamp to the current system time.

[0023] In terms of scheduling parameter adjustment, after collecting data from all nodes, the thread reads all smoothing load metrics from the status table and calculates the mean and standard deviation of the queue length. The standard deviation reflects the dispersion of load among nodes; a large value indicates an unbalanced load. The thread also identifies the maximum tail latency for all nodes.

[0024] The thread compares the standard deviation with a preset imbalance target value, and any deviation exceeding this target is counted as imbalance deviation. Similarly, the tail delay deviation is calculated. The two deviation values ​​are multiplied by weighting coefficients and then summed to obtain the system stress value. The middleware layer maintains a scheduling parameter structure containing two fields: the number of samples and the load difference threshold. The thread compares the stress value with the upper and lower thresholds. If the stress value consistently exceeds the upper limit, the number of samples is increased and the threshold is decreased; if the stress value consistently falls below the lower limit, the number of samples is decreased and the threshold is increased. To prevent frequent oscillations, the thread maintains a counter, and adjustments are only performed when the stress value exceeds the limit for multiple consecutive periods. The adjusted parameters are shared with the routing module via shared memory, and the routing module reads the latest parameters for decision-making.

[0025] Furthermore, the step of periodically adjusting the scheduling parameters based on the load status information includes: calculating the load distribution of each server node based on the load status information to obtain a load imbalance index; calculating a system tail latency index based on the tail processing latency in the load status information; calculating a system pressure value based on the load imbalance index and the system tail latency index; when the system pressure value exceeds a preset upper limit threshold, increasing the number of candidate node samples in the scheduling parameters and decreasing the load difference threshold; when the system pressure value is lower than a preset lower limit threshold, decreasing the number of candidate node samples and increasing the load difference threshold.

[0026] Specifically, after the monitoring thread completes load collection for all nodes, it calls the malloc function to allocate a contiguous block of memory on the heap as a temporary array, the size of which is equal to the total number of server nodes. The thread traverses the server status table in the form of a hash table, accessing the status structure of each node sequentially through the hash table's iterator, reading the smoothing index value of the queue length from the structure, and writing these values ​​sequentially into the temporary array.

[0027] The thread first calculates the sum of all queue lengths in the array, accumulating the value of each element through an incrementing variable as it iterates through the array. After the iteration is complete, the value of the incrementing variable is divided by the array length to obtain the average queue length. Then, the thread iterates through the array again, subtracting the average from each queue length to obtain the deviation, calling the `pow` function to calculate the square of the deviation, and accumulating the squared value into another variable. After completing the iteration, the sum of squares is divided by the array length, and the `sqrt` function is called to take the square root of the result. The resulting value is the standard deviation of the queue lengths, which the thread temporarily stores as a load imbalance indicator in a local variable.

[0028] To calculate the system's tail latency metric, the thread again iterates through the server state table using an iterator, this time reading the smoothing metric value of tail processing latency from the state structure of each node. The thread maintains a variable recording the maximum value, initially set to zero. During the iteration, for each tail latency value read, an if condition is used to check if the value is greater than the current maximum value; if it is, the maximum value variable is updated. After the iteration is complete, the maximum value variable stores the largest tail latency among all nodes, and the thread uses this value as the system's tail latency metric.

[0029] The thread reads the preset target imbalance threshold and target tail delay threshold from the configuration file or configuration table. It subtracts the previously calculated standard deviation from the target imbalance threshold; if the difference is negative, it sets it to zero, retaining the positive value as the imbalance deviation. Similarly, it subtracts the target tail delay threshold from the maximum tail delay; negative values ​​are set to zero, and positive values ​​are retained as the tail delay deviation. The thread reads two weighting coefficients, multiplies them by the two deviation values ​​respectively, and then adds the two products to obtain the system stress value.

[0030] The thread reads the preset upper and lower threshold values ​​for stress from shared memory. It uses an `if` statement to check if the current stress value exceeds the upper threshold. If the condition is true, the thread accesses a static counter and increments it by one. The thread then checks if the counter has reached the threshold for consecutive triggers; if so, it performs parameter adjustments. During adjustment, the thread locks the scheduling parameter structure in shared memory using the `pthread_mutex_lock` function. It reads the current number of candidate node samples from the structure and increments it by a step value, but checks if the incremented value exceeds the preset maximum sample size limit; if so, it sets it to the maximum value. Similarly, it reads the load difference threshold, decrements it by a step value, and checks if it falls below the minimum threshold limit; if so, it sets it to the minimum value. The thread writes the adjusted parameter values ​​back to the structure, calls the `pthread_mutex_unlock` function to release the lock, and then resets the upper counter to zero.

[0031] If the pressure value has not exceeded the upper limit, the thread checks if the upward counter is non-zero. If so, it clears it to zero, as the pressure has decreased and is no longer continuously exceeding the limit. Next, the thread checks if the pressure value is less than the lower threshold. If so, it increments the downward counter. When the downward counter reaches the threshold, the thread also locks the scheduling parameter structure, decreases the sampling quantity by one step and checks the lower limit, increases the difference threshold by one step and checks the upper limit, writes back to the structure, releases the lock, and clears the downward counter. If the pressure value is between the upper and lower limits, both counters remain zero, and the parameters are not adjusted.

[0032] After adjustment, the thread calls the `free` function to release the previously allocated temporary array memory, and then continues the next round of timed sleep waiting. When the routing decision module receives an IO access request, it uses `pthread_mutex_lock` to access the shared memory scheduling parameter structure, reads the current sample count and difference threshold, and uses these parameters to select candidate nodes and evaluate the load after releasing the lock.

[0033] 102. In response to the IO access request sent by the client, query the preset path mapping rules according to the target path in the IO access request to obtain a set of candidate server nodes; In this embodiment, the step of responding to the IO access request sent by the client and querying a preset path mapping rule according to the target path in the IO access request to obtain a candidate server node set includes: responding to the IO access request sent by the client, determining an initial mapping server node according to the target path in the IO access request and a consistent hashing algorithm; selecting consecutive server nodes starting from the initial mapping server node according to the hash ring structure of the consistent hashing algorithm to obtain a preliminary candidate node range; and filtering out server nodes from the preliminary candidate node range that match the namespace mapping relationship of the target path of the IO access request to form the candidate server node set.

[0034] Specifically, the main thread of the middleware layer listens for client connections on the listening port using the epoll mechanism. When a client sends an I / O access request, the `epoll_wait` function returns a ready socket descriptor. The main thread calls the `accept` function to accept the client connection and then calls the `recv` function to read the request data from the socket. The request data is encapsulated according to a predefined protocol format. The main thread parses the request header to extract the message type field, confirming that it is an I / O access request. Next, it parses the request body to extract the target path string. The target path is a string similar to a file system path, consisting of multi-level directory names separated by forward slashes.

[0035] The middleware layer has already constructed a consistent hashing ring during the initialization phase. During construction, the middleware layer reads the list of server nodes and calculates the hash values ​​of several virtual nodes for each server node. Specifically, it concatenates the server node's identifier with different suffix strings and then calls the SHA256 hash function to calculate the hash value of the concatenated string. Each server node generates multiple virtual nodes, and the hash values ​​of these virtual nodes are distributed throughout the hash space. The middleware layer stores the hash values ​​of all virtual nodes into an array and calls the qsort function to sort the array by hash value. The sorted array forms a logical circular structure, with the element after the last element of the array being the first element. Simultaneously, it maintains a mapping table from virtual nodes to actual server nodes, recording which actual server each virtual node belongs to.

[0036] After obtaining the target path string, the main thread calls the SHA256 function to calculate the hash value of the path, obtaining a hash value. Then, the main thread calls a binary search algorithm to search for the first virtual node in the sorted virtual node hash array that is greater than or equal to the path hash value. The binary search narrows the search range by repeatedly comparing the middle element with the target value, eventually locating the index position of the target virtual node in the array. If the path hash value is greater than the hash values ​​of all virtual nodes in the array, the index of the first element of the array is returned, utilizing the characteristics of a circular structure to achieve wrapping.

[0037] The main thread looks up the actual server node identifier corresponding to the virtual node in the mapping table from virtual nodes to server nodes. This server node is the initial mapping server node. The consistent hashing algorithm ensures that the same path always maps to the same server node, and that adding or removing server nodes only affects the mapping relationship of a small number of paths.

[0038] After determining the initial mapping node, the main thread needs to select a contiguous range of nodes on the hash ring as candidates. The main thread records the index position of the virtual node corresponding to the initial mapping node in the array, and then traverses the array backward from this position. During the traversal, the actual server node corresponding to each virtual node is added to a temporary set. Because a server node has multiple virtual nodes on the ring, the temporary set uses a hash set data structure to avoid adding the same server node repeatedly. The main thread continues to traverse until the number of actual server nodes in the temporary set reaches the preset candidate range size, or until the entire array has been traversed and the thread returns to the starting position. The resulting temporary set contains several actual server nodes near the initial mapping node on the hash ring, forming the initial candidate node range.

[0039] However, not all nodes within the initial scope are suitable for handling the current request. Distributed storage systems typically use namespace partitioning, with different path prefixes managed by different server node groups. The middleware layer maintains a namespace mapping table, recording the correspondence between path prefixes and server node groups. The main thread parses the target path string, extracting the directory names at each level of the path. Starting from the top-level directory, the main thread searches the namespace mapping table for the server node group identifier corresponding to that directory name. If a match is found, the corresponding node group is recorded; otherwise, the search continues to the next lower directory level until a matching mapping rule is found or all directory levels have been traversed.

[0040] After obtaining the namespace node group identifier corresponding to the path, the main thread iterates through each server node in the initial candidate node range. For each node, the main thread queries its own node group information and determines whether the node group identifier is the same as the namespace node group identifier of the path. If they are the same, the node is retained and added to the final candidate server node set; otherwise, the node is discarded. This filtering process ensures that all server nodes in the candidate set have the permissions and data access capabilities to process the target path.

[0041] After the filtering process, the main thread obtains a set of candidate server nodes that are both close to the initial mapped node on the hash ring and satisfy the namespace constraints. This set is passed to the subsequent load assessment module via pointers or references. The set is stored using a dynamic array or linked list structure for easy traversal and access by subsequent modules.

[0042] 103. Based on the load status information and the scheduling parameters, perform load assessment on each candidate server node in the candidate server node set to obtain the target server node, and route the IO access request to the target server node.

[0043] In this embodiment, the middleware layer's process of obtaining the target server node from the candidate server node set is relatively straightforward. After receiving the candidate set, the main thread iterates through each server node in the set. For each candidate node, the main thread accesses the server status table and uses the candidate node's identifier as the key to look up the corresponding status structure in the hash table. From the structure, it reads the node's smooth load metrics, including queue length, median latency, and tail latency.

[0044] The main thread performs a simple comparison of the read load metrics. A common approach is to directly compare queue lengths, maintaining a variable to record the minimum queue length found during the current iteration, along with the corresponding server node identifier. After reading the queue length of each candidate node, an if statement is used to check if the value is less than the currently recorded minimum. If it is less, the minimum value variable and the node identifier variable are updated. After iterating through all candidate nodes, the node corresponding to the minimum value variable is the node with the lowest load, and the main thread uses it as the target server node.

[0045] Another approach is to calculate a comprehensive load score. For each candidate node, the main thread multiplies the queue length, median latency, and tail latency by preset weighting coefficients, then sums these three weighted values ​​to obtain the node's comprehensive load score. The main thread also maintains variables for the minimum score and its corresponding node identifier, iterates through the candidate set to update these variables, and finally selects the node with the lowest comprehensive score as the target server node.

[0046] After identifying the target node, the main thread needs to forward I / O access requests to that node. The main thread searches for the data channel socket corresponding to the target node in the internally maintained server connection pool. The connection pool uses a hash table structure, with the server node identifier as the key and the socket descriptor and connection status information as the values. The main thread searches the connection pool based on the target node identifier. If an established connection is found and its status is normal, the socket is used directly. If the connection does not exist or has been broken, the main thread calls the `socket` function to create a new socket, and then calls the `connect` function to initiate a connection request to the target server node's listening address. After the connection is successfully established, the new socket descriptor is added to the connection pool.

[0047] The main thread sends the raw data of the I / O access request to the target node's socket using the `send` function. When sending, a protocol header may need to be added before the request data to identify the request type and data length, etc. After sending, the main thread calls the `recv` function on the socket to wait for the target server's processing result. Upon receiving the response data, the main thread parses the response content, extracts the processing status and returned data, and then forwards the response back to the client through the original client's socket, completing the entire request routing and forwarding process.

[0048] In this embodiment, a middleware layer is deployed between the client and server nodes to periodically collect load status information of each server node and dynamically adjust scheduling parameters based on this information. In response to an I / O access request from the client, a preset path mapping rule is queried based on the target path in the request to obtain a set of candidate server nodes. Then, the candidate server nodes are evaluated for load based on the load status information and scheduling parameters to determine the target server node, and the I / O access request is routed to the target server node. This invention achieves responsive control to system load changes by adaptively adjusting scheduling parameters, effectively mitigating load skew and reducing system tail latency while maintaining path mapping locality.

[0049] Please see Figure 2 Another embodiment of the middleware-based distributed I / O access routing method in this application includes: 201. The middleware layer deployed between the client and the server node periodically collects the load status information of each server node, and adjusts the scheduling parameters periodically according to the load status information. 202. In response to the IO access request sent by the client, query the preset path mapping rules according to the target path in the IO access request to obtain a set of candidate server nodes; In this embodiment, steps 201-202 are similar to steps 101-102 in the first embodiment, and will not be described again here.

[0050] 203. Determine the initial mapping server node in the candidate server node set based on the target path of the IO access request; In this embodiment, although step 202 has determined the initial mapping server node using the consistent hashing algorithm, this node may not be in the candidate server node set after namespace filtering. Therefore, the main thread needs to re-determine which node is the initial mapping node within the candidate set.

[0051] The main thread reads the initial mapped server node identifier, calculated using a consistent hashing algorithm, from a saved temporary variable. Then, the main thread iterates through the candidate server node set, which stores server node identifiers using an array or linked list structure. During iteration, the main thread compares the identifier of each node in the set with the saved initial mapped node identifier, calling the strcmp function to determine if the two identifiers are identical.

[0052] If a matching node is found in the candidate set, it means that the initial mapping node calculated by the hash algorithm satisfies both the distance requirement on the hash ring and the access permission constraints of the namespace. The main thread directly marks this node as the initial mapping node in the candidate set and records its index position or pointer address in the candidate set.

[0053] If no matching node is found after traversing the entire candidate set, it means that the initial mapping node calculated by the hash algorithm was excluded by the namespace filtering process. In this case, the main thread needs to select a node from the candidate set as a replacement for the initial mapping node. One approach is to select the first node in the candidate set, because the candidate set is constructed according to the order on the hash ring, and the first node is closest to the original initial mapping node on the ring. The main thread directly marks the first element of the candidate set as the initial mapping node.

[0054] Another approach is to recalculate the distance between each node in the candidate set and the target path hash value on the hash ring. The main thread iterates through the candidate set, looking up the hash value of the first virtual node for each node in the virtual node mapping table, and calculating the difference between the virtual node hash value and the path hash value. Considering the circular nature of the hash ring, if the virtual node hash value is less than the path hash value, the difference needs to be increased by the maximum value of the hash space to achieve wraparound. The main thread records the node with the smallest distance to the path hash value among all candidate nodes and uses it as the initial mapping node.

[0055] After determining the initial mapping node, the main thread stores the node's identifier and its position in the candidate set into local variables, which are then passed to subsequent sampling and evaluation steps. The initial mapping node serves as a benchmark for load comparison; subsequent steps compare the load of sampled nodes with it to determine whether a route target switch is necessary.

[0056] 204. Select a sampling node from the candidate server node set according to the scheduling parameters; In this embodiment, selecting sampling nodes from the candidate server node set according to the scheduling parameters includes: obtaining the number of candidate nodes to be sampled in the scheduling parameters, and determining the distribution range of the candidate server node set on the hash ring according to the path mapping rules; selecting a number of server nodes corresponding to the number of candidate nodes to be sampled from the distribution range as the sampling nodes.

[0057] Specifically, the main thread locks the scheduling parameter structure in shared memory using the `pthread_mutex_lock` function, and reads the current candidate node sampling count `d` from the structure. The value of the sampling count `d` is dynamically set by the aforementioned adaptive adjustment mechanism; `d` is larger when the system load is severely unbalanced, and smaller when the load is balanced. After reading, `pthread_mutex_unlock` is called to release the lock.

[0058] The main thread checks the actual size of the candidate server node set. The candidate set is obtained through consistent hashing and namespace filtering, and the nodes are distributed continuously on the hash ring. The main thread reads the virtual node hash values ​​corresponding to the first and last nodes on the hash ring from the candidate set. The range between these two hash values ​​represents the distribution range of the candidate set on the hash ring. This range includes all server nodes that are close to the target path in the hash space and satisfy the namespace constraints.

[0059] If the actual size of the candidate set is less than or equal to the number of samples d, the main thread directly uses the entire candidate set as the sampling node set without further filtering. The main thread calls the malloc function to allocate the sampling node array and copies the identifiers of all nodes in the candidate set to the sampling array one by one.

[0060] If the candidate set size is greater than the sampling number d, the main thread needs to select d nodes from it. The main thread starts from the determined position of the initial mapping server node in the candidate set. This initial mapping node is the master node calculated by the consistent hashing algorithm based on the target path. The main thread locates the index position of the initial mapping node in the candidate set, and then traverses the candidate set backward from that position.

[0061] During the traversal, the main thread sequentially selects d minus one nodes following the initial mapping node. Because the candidate set is constructed according to the hash ring order, the nodes selected consecutively starting from the initial mapping node are also consecutive on the hash ring. These d minus one nodes, plus the initial mapping node itself, constitute a total of d sampling nodes. The main thread then sequentially writes the identifiers of these d nodes into the sampling node array.

[0062] If, after traversing from the initial mapped node position to the end of the candidate set, not enough nodes (d) have been selected, the main thread utilizes the continuity of the candidate set on the hash ring to wrap the index back to the beginning of the candidate set and continue selection. Specifically, the main thread maintains the current index variable, incrementing the index by one after each node selection. If the index value equals the size of the candidate set, the index is set to zero, and selection begins again from the beginning of the set.

[0063] After sampling is complete, the main thread obtains a sample array containing d server node identifiers. These nodes are all located near the initially mapped nodes on the hash ring and satisfy namespace access constraints. The main thread passes the pointer to the sample array and its size to the subsequent load comparison and evaluation step.

[0064] 205. Based on the load status information and the scheduling parameters, perform a load comparison and evaluation between the sampling node and the initial mapping server node to obtain the target server node, and route the IO access request to the target server node.

[0065] In this embodiment, the step of comparing and evaluating the load of the sampling node and the initial mapping server node based on the load status information and the scheduling parameters to obtain the target server node includes: acquiring the load status information of the sampling node and the initial mapping server node; comparing the load status information of the sampling node with the load status information of the initial mapping server node, filtering out candidate nodes whose load is lower than that of the initial mapping server node and whose difference meets the load difference threshold in the scheduling parameters, and obtaining a set of nodes that meet the conditions; when the set of nodes that meet the conditions is not empty, selecting the node with the lowest load from the set of nodes that meet the conditions as the target server node; otherwise, determining the initial mapping server node as the target server node.

[0066] Specifically, the main thread first retrieves the load status information of the initially mapped server node from the server status table. Using the identifier of the initially mapped node as the key, the main thread searches the hash table-structured status table. Once the corresponding status structure is found, it reads the queue length smoothing index and the median processing delay smoothing index, which are recorded as the initial node queue length and the initial node delay, respectively.

[0067] The main thread iterates through the array of sampling nodes, looking up the load status information of each node from the status table. The main thread reads the queue length smoothing metric and the latency smoothing metric of the sampling node. Then, the main thread accesses the shared memory scheduling parameter structure using `pthread_mutex_lock`, reads the current load difference thresholds, and records them as the queue length threshold and latency threshold, respectively. After reading, the lock is released.

[0068] The main thread compares the queue length of the sampling node with the queue length of the initial node. If the queue length of the sampling node is greater than or equal to the queue length of the initial node, it means that the load of the sampling node is not lower than that of the initial node, and the main thread skips this node and continues to the next sampling node. If the queue length of the sampling node is less than the queue length of the initial node, the main thread calculates the difference between the two and determines whether the difference is greater than or equal to a queue length threshold. The difference must reach the threshold to indicate that the load advantage of the sampling node is significant enough to warrant a route switch.

[0069] Similarly, the main thread compares the median processing latency of the sampled node with that of the initial node. The latency of the sampled node must be less than that of the initial node, and the difference must reach a latency threshold. Only when both the queue length and latency conditions are met simultaneously, does the main thread consider the sampled node to meet the switching conditions and add its identifier to a temporary array of nodes that meet the conditions.

[0070] After the main thread has traversed all sampling nodes, it checks whether the array of nodes that meet the conditions is empty. If the array is empty, it means that the load of all sampling nodes does not meet the switching conditions; either the load is not low enough, or although it is slightly low, the difference has not reached the threshold. In this case, the main thread uses the initial mapped server node as the final target server node, keeping the original path mapping relationship unchanged.

[0071] If the array of nodes that meet the criteria is not empty, the main thread needs to select the node with the lowest load. The main thread iterates through the array of nodes that meet the criteria, reads the queue length index of each node, and maintains a variable that records the minimum queue length and a variable that records the corresponding node identifier. During the traversal, these two variables are continuously updated. After the traversal is completed, the node with the minimum queue length is obtained, and the main thread uses it as the target server node.

[0072] After identifying the target node, the main thread searches for the corresponding data channel socket in the server's connection pool. The connection pool uses a hash table to store the mapping between node identifiers and socket descriptors. If a valid connection is found, the main thread uses that socket directly; otherwise, it calls the `socket` and `connect` functions to establish a new connection and adds it to the connection pool. The main thread calls the `send` function to send the I / O access request data to the target node's socket, and then calls the `recv` function to wait for a response. Upon receiving the response, the main thread forwards the response data back to the client, completing the request processing.

[0073] In this embodiment, a middleware layer is deployed between the client and server nodes to periodically collect load status information of each server node and dynamically adjust scheduling parameters based on this information. In response to an I / O access request from the client, a preset path mapping rule is queried based on the target path in the request to obtain a set of candidate server nodes. Then, the candidate server nodes are evaluated for load based on the load status information and scheduling parameters to determine the target server node, and the I / O access request is routed to the target server node. This invention achieves responsive control to system load changes by adaptively adjusting scheduling parameters, effectively mitigating load skew and reducing system tail latency while maintaining path mapping locality.

[0074] The middleware-based distributed I / O access routing method in the embodiments of the present invention has been described above. The middleware-based distributed I / O access routing device in the embodiments of the present invention is described below, applied to a distributed storage system. The distributed storage system includes a client and multiple server nodes. For details on this middleware-based distributed I / O access routing device, please refer to [link to relevant documentation]. Figure 3 One embodiment of the middleware-based distributed I / O access routing device of the present invention includes: The load monitoring module 301 is used to periodically collect the load status information of each server node through the middleware layer deployed between the client and the server node, and periodically adjust the scheduling parameters according to the load status information. The candidate query module 302 is used to respond to the IO access request sent by the client, query the preset path mapping rules according to the target path in the IO access request, and obtain a set of candidate server nodes. The routing decision module 303 is used to perform load evaluation on each candidate server node in the candidate server node set according to the load status information and the scheduling parameters, obtain the target server node, and route the IO access request to the target server node.

[0075] In this embodiment of the invention, the middleware-based distributed I / O access routing device runs the aforementioned middleware-based distributed I / O access routing method. The middleware-based distributed I / O access routing device deploys a middleware layer between the client and server nodes, periodically collects the load status information of each server node, and dynamically adjusts scheduling parameters based on the load status information. In response to an I / O access request sent by the client, it queries a preset path mapping rule based on the target path in the request to obtain a set of candidate server nodes. Then, it performs load evaluation on the candidate server nodes based on the load status information and scheduling parameters to obtain the target server node, and routes the I / O access request to the target server node. This invention achieves responsive control to changes in system load by adaptively adjusting scheduling parameters, effectively alleviating load skew while maintaining path mapping locality and reducing system tail latency.

[0076] above Figure 3 The middleware-based distributed I / O access routing device in this embodiment of the invention will be described in detail from the perspective of unitized functional entities. The middleware-based distributed I / O access routing device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0077] Figure 4 This is a schematic diagram of a middleware-based distributed I / O access routing device 400 provided in an embodiment of the present invention. The middleware-based distributed I / O access routing device 400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 433 or data 432. The memory 420 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more units (not shown in the diagram), each unit may include a series of instruction operations on the middleware-based distributed I / O access routing device 400. Furthermore, the processor 410 may be configured to communicate with the storage media 430 and execute the series of instruction operations on the storage media 430 on the middleware-based distributed I / O access routing device 400 to implement the steps of the aforementioned middleware-based distributed I / O access routing method.

[0078] The middleware-based distributed I / O access routing device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The illustrated middleware-based distributed I / O access routing device structure does not constitute a limitation on the middleware-based distributed I / O access routing device provided by the present invention. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0079] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the middleware-based distributed I / O access routing method.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A middleware-based distributed I / O access routing method, applied to a distributed storage system, the distributed storage system comprising a client and multiple server nodes, characterized in that, The distributed I / O access routing method includes: The middleware layer deployed between the client and the server node periodically collects the load status information of each server node, and adjusts the scheduling parameters periodically based on the load status information. In response to the IO access request sent by the client, a set of candidate server nodes is obtained by querying a preset path mapping rule according to the target path in the IO access request. Based on the load status information and the scheduling parameters, the load of each candidate server node in the candidate server node set is evaluated to obtain the target server node, and the IO access request is routed to the target server node.

2. The distributed I / O access routing method according to claim 1, characterized in that, The periodic collection of load status information of each server node through the middleware layer deployed between the client and the server node includes: The middleware layer deployed between the client and the server node collects real-time load data of each server node at preset time intervals, wherein the real-time load data includes the current queue length, median processing latency, and tail processing latency. An exponentially weighted moving average is calculated on the real-time load data to obtain a smoothed load index for each server node, and the smoothed load index is stored and maintained as load status information.

3. The distributed I / O access routing method according to claim 1, characterized in that, The periodic adjustment of scheduling parameters based on the load status information includes: The load distribution of each server node is calculated based on the load status information to obtain the load imbalance index. Calculate the system tail delay index based on the tail processing delay in the load status information; Calculate the system pressure value based on the load imbalance index and the system tail delay index; When the system pressure value exceeds the preset upper limit threshold, the number of candidate node samples in the scheduling parameters is increased and the load difference threshold is decreased. When the system pressure value is lower than the preset lower threshold, the number of candidate node samples is reduced and the load difference threshold is increased.

4. The distributed I / O access routing method according to claim 1, characterized in that, In response to the IO access request sent by the client, the candidate server node set is obtained by querying a preset path mapping rule according to the target path in the IO access request, including: In response to the IO access request sent by the client, the initial mapping server node is determined based on the target path and consistent hashing algorithm in the IO access request; Based on the hash ring structure of the consistent hashing algorithm, consecutive server nodes are selected starting from the initial mapping server node to obtain a preliminary candidate node range; The candidate server node set is formed by selecting server nodes from the initial candidate node range that match the namespace mapping relationship of the target path of the IO access request.

5. The distributed I / O access routing method according to claim 1, characterized in that, The step of performing load evaluation on each candidate server node in the candidate server node set based on the load status information and the scheduling parameters to obtain the target server node includes: The initial mapping server node in the candidate server node set is determined based on the target path of the IO access request. Sampling nodes are selected from the candidate server node set according to the scheduling parameters; Based on the load status information and the scheduling parameters, the load of the sampling node and the initial mapping server node are compared and evaluated to obtain the target server node.

6. The distributed I / O access routing method according to claim 5, characterized in that, The step of selecting a sampling node from the candidate server node set according to the scheduling parameters includes: Obtain the number of candidate nodes sampled in the scheduling parameters, and determine the distribution range of the candidate server node set on the hash ring according to the path mapping rules; A number of server nodes corresponding to the number of candidate nodes sampled are selected from the distribution range as the sampling nodes.

7. The distributed I / O access routing method according to claim 5, characterized in that, The step of comparing and evaluating the load of the sampling node and the initial mapping server node based on the load status information and the scheduling parameters to obtain the target server node includes: Obtain the load status information of the sampling node and the initial mapping server node; The load status information of the sampling node is compared with the load status information of the initial mapping server node, and candidate nodes whose load is lower than that of the initial mapping server node and whose difference meets the load difference threshold in the scheduling parameters are selected to obtain a set of nodes that meet the conditions. When the set of nodes that meet the conditions is not empty, the node with the lowest load is selected from the set of nodes that meet the conditions as the target server node; otherwise, the initial mapping server node is determined as the target server node.

8. A middleware-based distributed I / O access routing device, applied to a distributed storage system, the distributed storage system comprising a client and multiple server nodes, characterized in that, The distributed I / O access routing device includes: The load monitoring module is used to periodically collect the load status information of each server node through the middleware layer deployed between the client and the server node, and periodically adjust the scheduling parameters according to the load status information. The candidate query module is used to respond to the IO access request sent by the client, query the preset path mapping rules according to the target path in the IO access request, and obtain a set of candidate server nodes. The routing decision module is used to perform load assessment on each candidate server node in the candidate server node set according to the load status information and the scheduling parameters, obtain the target server node, and route the IO access request to the target server node.

9. A middleware-based distributed I / O access routing device, characterized in that, The middleware-based distributed I / O access routing device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the middleware-based distributed I / O access routing device to perform the steps of the middleware-based distributed I / O access routing method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instruction is executed by the processor, it implements the steps of the middleware-based distributed I / O access routing method as described in any one of claims 1-7.

Images