A distributed network disk scheduling processing method, system, medium and product

Abstract

A distributed network disk scheduling processing method, system, medium and product, relating to the technical field of distributed storage. Implementing the method, the scheduling processing system maps the unique identifier of the data block to be transmitted to a virtual hash ring through a consistent hash algorithm, accurately circumscribes a predetermined number of candidate nodes, both retaining the fast distribution characteristics of hash mapping and avoiding the limitations of single node binding. The scheduling processing system obtains the real-time state of the disk queue depth, available bandwidth, etc. of the candidate nodes through link probe packets, and calculates the real-time load resistance value in combination with the network round-trip time, while estimating the data transmission time consumption and queuing waiting time consumption, and finally screening out the target execution node with the shortest expected task completion time. This closed-loop design of "hash positioning + real-time state evaluation + comprehensive time consumption sorting" realizes the dynamic matching of data access requests and storage node performance, effectively avoiding the resource waste situation of some nodes being overloaded and congested while some nodes being idle.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a scheduling and processing method, system, medium and product for a distributed cloud storage. Background Technology

[0002] With the rapid development of internet technology and the continuous growth of data scale, distributed cloud storage has been widely used. By distributing data across multiple storage nodes, distributed cloud storage can effectively improve system storage capacity and data access efficiency. In distributed cloud storage, how to rationally schedule data access requests to achieve efficient data transmission and load balancing has become an important technical challenge.

[0003] Currently, distributed cloud storage systems use a fixed hash mapping method for data scheduling. Specifically, the distributed cloud storage system calculates a hash value based on the identifier information of the data block and directly maps the hash value to a pre-divided hash space. Each storage node is assigned a fixed hash value range. When a data access request is received, the distributed cloud storage system assigns the data access request to the storage node corresponding to the hash value range for processing. This method enables rapid scheduling and distribution of data access requests.

[0004] However, in practical applications, the hardware performance, network status, and workload of storage nodes often vary significantly. The fixed hash mapping scheduling method does not consider the real-time status of storage nodes, which may result in some lower-performing or heavily loaded storage nodes continuously receiving data access requests, while other storage nodes remain relatively idle. When there are a large number of data access requests, this load imbalance can cause severe congestion on some storage nodes, affecting data transmission efficiency. Summary of the Invention

[0005] This application provides a scheduling and processing method, system, medium, and product for distributed cloud storage, which can improve overall data transmission efficiency, reduce average task completion latency, and ensure the load balance of the distributed cloud storage system.

[0006] Firstly, this application provides a scheduling method for a distributed cloud storage system, applied to a scheduling system. The method includes: receiving a data access request sent by a client, the data access request including a unique identifier and the size of the data block to be transmitted; calculating the unique identifier based on a consistent hashing algorithm to obtain a target hash value, mapping the target hash value onto a virtual hash ring, the virtual hash ring representing the logical positions of multiple storage nodes in the hash space; traversing the virtual hash ring clockwise from the corresponding position of the target hash value on the virtual hash ring, selecting a preset number of storage nodes traversed sequentially as candidate nodes; sending link probe packets marked with a sending timestamp to the candidate nodes, and receiving status information from the candidate nodes including disk queue depth and available bandwidth. The system generates a status response packet and records the timestamp of the received status response packet. Based on the difference between the sending and receiving timestamps, it calculates the network round-trip latency of the candidate node. Based on the disk queue depth and network round-trip latency, it calculates the real-time load resistance value of the candidate node and maps the real-time load resistance value to the corresponding queuing waiting time. Combining the data volume and the available bandwidth of the candidate node, it estimates the data transmission time and adds the data transmission time to the queuing waiting time to obtain the expected task completion time of the candidate node. It compares the expected task completion times of the candidate nodes, selects the candidate node with the smallest expected task completion time as the target execution node, generates a scheduling instruction including the target network address corresponding to the target execution node, and sends it to the client to instruct the client to establish a data transmission connection with the target execution node.

[0007] By adopting the above technical solution, firstly, the scheduling and processing system maps the unique identifier of the data block to be transmitted to a virtual hash ring using a consistent hashing algorithm, accurately identifying a preset number of candidate nodes. This retains the fast distribution characteristics of hash mapping while avoiding the limitations of single-node binding. Subsequently, the scheduling and processing system obtains the real-time status of candidate nodes, such as disk queue depth and available bandwidth, through link probe packets. It then calculates the real-time load resistance value based on network round-trip latency, while simultaneously estimating data transmission time and queuing time. Finally, it selects the target execution node with the shortest expected task completion time. This closed-loop design of "hash positioning + real-time status evaluation + comprehensive time-based sorting" achieves dynamic matching between data access requests and storage node performance, effectively avoiding resource waste caused by some nodes being overloaded and congested while others are idle. In large-scale data access scenarios, it can significantly improve overall data transmission efficiency, reduce average task completion latency, ensure the load balance of the distributed cloud storage system, enhance the stability and reliability of the distributed cloud storage system, and provide users with a more efficient and smooth data access experience.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the real-time load resistance value of a candidate node is calculated based on the disk queue depth and network round-trip time (RTD). Specifically, this includes: determining the ratio of the RTD to a preset baseline network delay as the transmission impedance exponent; determining the ratio of the disk queue depth to the maximum concurrent I / O threshold of the candidate node as the first I / O saturation; exponentially weighting the first I / O saturation to obtain a second I / O saturation; and linearly summing the transmission impedance exponent and the second I / O saturation to obtain the real-time load resistance value of the candidate node.

[0009] By adopting the above technical solution, the scheduling system determines the transmission impedance index as the ratio of network round-trip latency to the baseline network latency, accurately reflecting the load pressure at the network transmission layer. Simultaneously, the scheduling system determines the first I / O saturation by the ratio of disk queue depth to the maximum concurrent I / O threshold, and then applies an exponentially weighted amplification to obtain the second I / O saturation, fully highlighting the sharp increase in disk I / O load pressure. By linearly summing the transmission impedance index and the second I / O saturation, a comprehensive quantification of network transmission load and disk I / O load is achieved, avoiding load misjudgments caused by single-dimensional evaluation. This multi-dimensional and refined load quantification method can accurately capture the differences in the true load status of different candidate nodes. For example, for nodes with high network latency but low disk load, and nodes with high disk load but good network conditions, accurate load resistance values ​​can be obtained through this calculation logic, thereby ensuring the rationality of subsequent target execution node selection, further optimizing the system load balancing effect, and improving the stability and efficiency of data transmission.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, mapping the real-time load resistance value to the corresponding queuing time specifically includes: obtaining the storage medium type of the candidate node and determining the unit resistance time coefficient corresponding to the storage medium type, wherein the unit resistance time coefficient is used to represent the average time required to process data corresponding to the storage medium type under unit load resistance; multiplying the real-time load resistance value by the unit resistance time coefficient to obtain the basic queuing time; obtaining the average I / O response jitter value of the candidate node within a preset historical time window, and correcting the basic queuing time based on the average I / O response jitter value to obtain the queuing time.

[0011] By adopting the above technical solution, the scheduling and processing system constructs a queuing waiting time mapping mechanism of "basic time calculation + dynamic jitter correction," effectively improving the accuracy of time estimation and providing key support for the optimal selection of target execution nodes. First, the scheduling and processing system determines the unit resistance time coefficient based on the storage medium type of the candidate nodes, fully considering the performance differences of different storage media (such as mechanical hard drives and solid-state drives) when processing data, making the calculation of basic queuing time more closely match actual hardware characteristics. Based on this, the scheduling and processing system introduces the average I / O response jitter value within a preset historical time window to correct the basic queuing time, compensating for the deficiency of static coefficient calculation in dealing with node performance fluctuations. For example, for storage media such as solid-state drives with fast I / O response speeds but potentially occasional jitter, jitter value correction can avoid time estimation deviations caused by instantaneous performance fluctuations; while for mechanical hard drives, their jitter value is relatively stable, and high reliability can still be maintained after correction. This mapping method, which takes into account both hardware characteristics and dynamic fluctuations, makes the estimated queuing time closer to the actual operation of candidate nodes. This ensures that the calculation of the expected task completion time of candidate nodes is more accurate. It can not only screen out truly efficient target execution nodes and reduce scheduling errors caused by time estimation deviations, but also further optimize the timeliness and stability of data transmission and improve the overall service quality of the distributed cloud storage system.

[0012] In some embodiments of the first aspect, the data transmission time is estimated by combining the data size and the available bandwidth of the candidate nodes. Specifically, this includes: determining whether the data size is less than a preset TCP slow start threshold; if so, calculating a bandwidth utilization coefficient based on the data size, wherein the bandwidth utilization coefficient is positively correlated with the data size; using the bandwidth utilization coefficient to correct the available bandwidth to obtain the effective transmission bandwidth; and calculating the ratio of the data size to the effective transmission bandwidth to obtain the data transmission time.

[0013] By adopting the above technical solution, a refined optimization scheme is provided for data transmission time estimation, effectively solving the problem that traditional estimation methods do not consider the impact of the TCP slow start mechanism on small data volume transmission, and significantly improving the accuracy of time estimation. In distributed cloud storage scenarios, when the data volume is less than the preset TCP slow start threshold, the TCP connection cannot immediately make full use of the available bandwidth. If the data transmission time is directly calculated based on the available bandwidth, the estimated result will deviate significantly from the actual result. By judging the relationship between the data volume and the preset TCP slow start threshold, a bandwidth utilization coefficient is introduced for small data volume scenarios. This bandwidth utilization coefficient is positively correlated with the data volume, and can accurately reflect the actual bandwidth utilization under small data volume. The available bandwidth is corrected using this bandwidth utilization coefficient to obtain the effective transmission bandwidth, and then the data transmission time is calculated based on this, making the estimation result more consistent with the transmission characteristics of the TCP protocol. This method not only makes the calculation of the expected task completion time of candidate nodes more accurate and filters out truly efficient target execution nodes, but also avoids scheduling decision errors caused by prediction deviations, reduces waiting time during data transmission, improves the user experience in scenarios with small data volume transmission, and optimizes the resource scheduling efficiency of the entire distributed cloud storage system.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, before sending link probe packets marked with sending timestamps to candidate nodes respectively, the method further includes: querying whether there are historical state records of candidate nodes in the local cache; if there are historical state records of candidate nodes in the local cache and the update time of the historical state records is within a preset valid time window, then directly reading the disk queue depth and available bandwidth in the historical state records; if there are no historical state records of candidate nodes in the local cache and / or the update time of the historical state records is outside the preset valid time window, then performing the step of sending link probe packets marked with sending timestamps to candidate nodes respectively.

[0015] By adopting the above technical solution and introducing a local cache query mechanism, the communication overhead and response latency of the scheduling system are significantly reduced, and the overall performance is optimized, while ensuring the validity of node status data. In traditional solutions, a link probe packet needs to be sent to the node to obtain status information after each candidate node selection. Frequent communication interactions consume a large amount of network resources and increase the time spent on scheduling decisions. This method prioritizes querying the historical status records of candidate nodes in the local cache. If the historical status record exists and is within a preset valid time window, the disk queue depth and available bandwidth are read directly without initiating additional link probe requests. The probe step is only executed when the historical status record of the candidate node does not exist in the local cache and / or the update time of the historical status record is outside the preset valid time window. This avoids the resource waste caused by repeated probes and ensures the real-time nature of status data through the limitation of the preset valid time window, preventing scheduling deviations caused by the use of expired data. In high-concurrency access scenarios, this significantly reduces network communication pressure, speeds up scheduling decisions, shortens the client's data access waiting time, reduces the processing burden on candidate nodes, achieves efficient collaborative operation between the scheduling system and storage nodes, and improves the overall service response speed of the distributed network disk system.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, after the step of traversing the virtual hash ring clockwise from the corresponding position of the target hash value on the virtual hash ring and selecting a preset number of storage nodes as candidate nodes, the method further includes: determining whether the unique identifier matches the target data identifier in the hot data list, the hot data list including multiple data identifiers, the target data identifier being any one of the data identifiers; if so, querying the target node corresponding to the target data identifier; determining whether there is an intersection node between the target node and the candidate nodes; if so, reducing the disk queue depth of the intersection node according to a preset cache hit weight to reduce the real-time load resistance value of the intersection node.

[0017] By adopting the above technical solution, a dedicated optimization mechanism was designed for hot data access scenarios, effectively alleviating the problem of excessive load on storage nodes corresponding to hot data and ensuring the stability and access efficiency of the distributed cloud storage system. In distributed cloud storage, hot data is often frequently accessed by a large number of clients. If scheduling is done solely according to conventional load assessment logic, nodes storing hot data are prone to congestion due to continuous requests. By judging whether a unique identifier matches the hot data list, hot data access requests are accurately identified. When there is an intersection between a candidate node and the target node corresponding to the hot data, the disk queue depth of the intersection node is reduced by pre-setting a cache hit weight, thereby reducing its real-time load resistance. This method gives the intersection node a greater advantage in the subsequent expected task completion time ranking, enabling it to be selected as the target execution node first, fully utilizing its advantage of caching hot data and reducing data read overhead.

[0018] In some embodiments, in conjunction with the first aspect, the method further includes: obtaining the total transmission count of link probe packets sent to the candidate node within a preset time period, and the total reception count of successfully receiving the corresponding status response packets; calculating the difference between the total transmission count and the total reception count, and determining the ratio of the difference to the total transmission count as the data packet loss rate; calculating the bandwidth loss coefficient based on the data packet loss rate, wherein the higher the data packet loss rate, the larger the bandwidth loss coefficient; reducing the available bandwidth of the candidate node using the bandwidth loss coefficient to obtain the actual effective bandwidth of the candidate node; and dividing the data size by the actual effective bandwidth to calculate the data transmission time.

[0019] By adopting the above technical solution, the available bandwidth is dynamically corrected by introducing the packet loss rate, which solves the problem of ignoring the impact of data loss on transmission efficiency in traditional bandwidth assessment, and further improves the accuracy and reliability of data transmission time estimation. In real network environments, packet loss is a common phenomenon. A high packet loss rate will lead to data retransmission, thereby reducing the actual effective bandwidth. If the data transmission time is still calculated based on the theoretical available bandwidth, serious deviations will occur. By statistically counting the sending and receiving of link probe packets within a preset time period, the packet loss rate is calculated, and the bandwidth loss coefficient is determined based on the packet loss rate. The higher the packet loss rate, the larger the bandwidth loss coefficient, thus achieving accurate quantification of bandwidth loss. The available bandwidth of candidate nodes is reduced using the bandwidth loss coefficient to obtain the actual effective bandwidth, and then the data transmission time is calculated based on this, making the estimation results more consistent with the real network transmission situation. This method can accurately identify nodes with poor network quality and high data loss rates, avoid scheduling a large number of data access requests to such nodes, reduce retransmission time and failure risk during data transmission, improve the data transmission stability and efficiency of the entire distributed network disk system, and provide users with a smoother access experience.

[0020] In a second aspect, embodiments of this application provide a scheduling processing system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the scheduling processing system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a scheduling processing system, cause the scheduling processing system to execute the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a scheduling processing system, cause the scheduling processing system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the scheduling and processing system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By adopting the above technical solution, firstly, the scheduling and processing system maps the unique identifier of the data block to be transmitted to a virtual hash ring using a consistent hash algorithm, accurately identifying a preset number of candidate nodes. This retains the fast distribution characteristics of hash mapping while avoiding the limitations of single-node binding. Subsequently, the scheduling and processing system obtains the real-time status of candidate nodes, such as disk queue depth and available bandwidth, through link probe packets. It then calculates the real-time load resistance value based on network round-trip latency, while simultaneously estimating data transmission time and queuing time. Finally, it selects the target execution node with the shortest expected task completion time. This closed-loop design of "hash positioning + real-time status evaluation + comprehensive time-based sorting" achieves dynamic matching between data access requests and storage node performance, effectively avoiding resource waste caused by some nodes being overloaded and congested while others are idle. In large-scale data access scenarios, it can significantly improve overall data transmission efficiency, reduce average task completion latency, ensure the load balance of the distributed cloud storage system, enhance the stability and reliability of the distributed cloud storage system, and provide users with a more efficient and smooth data access experience.

[0026] 2. By adopting the above technical solution, the scheduling system determines the ratio of network round-trip latency to baseline network latency as the transmission impedance index, accurately reflecting the load pressure at the network transmission layer. Simultaneously, the scheduling system determines the first I / O saturation by the ratio of disk queue depth to the maximum concurrent I / O threshold, and then applies an exponentially weighted amplification to obtain the second I / O saturation, fully highlighting the sharp increase in disk I / O load pressure. By linearly summing the transmission impedance index and the second I / O saturation, a comprehensive quantification of network transmission load and disk I / O load is achieved, avoiding load misjudgments caused by single-dimensional evaluation. This multi-dimensional and refined load quantification method can accurately capture the differences in the true load status of different candidate nodes. For example, for nodes with high network latency but low disk load, and nodes with high disk load but good network conditions, accurate load resistance values ​​can be obtained through this calculation logic, thereby ensuring the rationality of subsequent target execution node selection, further optimizing the system load balancing effect, and improving the stability and efficiency of data transmission.

[0027] 3. By adopting the above technical solution, a refined optimization scheme is provided for data transmission time estimation, effectively solving the problem that traditional estimation methods do not consider the impact of the TCP slow start mechanism on small data volume transmission, and significantly improving the accuracy of time estimation. In distributed cloud storage scenarios, when the data volume is less than the preset TCP slow start threshold, the TCP connection cannot immediately make full use of the available bandwidth. If the data transmission time is directly calculated based on the available bandwidth, the estimated result will deviate significantly from the actual result. By judging the relationship between the data volume and the preset TCP slow start threshold, a bandwidth utilization coefficient is introduced for small data volume scenarios. This bandwidth utilization coefficient is positively correlated with the data volume, and can accurately reflect the actual bandwidth utilization under small data volume. The available bandwidth is corrected using this bandwidth utilization coefficient to obtain the effective transmission bandwidth, and then the data transmission time is calculated based on this, making the estimation result more consistent with the transmission characteristics of the TCP protocol. This method not only makes the calculation of the expected task completion time of candidate nodes more accurate and filters out truly efficient target execution nodes, but also avoids scheduling decision errors caused by prediction deviations, reduces waiting time during data transmission, improves the user experience in scenarios with small data volume transmission, and optimizes the resource scheduling efficiency of the entire distributed cloud storage system. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the scheduling and processing method of the distributed network disk in an embodiment of this application;

[0029] Figure 2 This is another flowchart illustrating the scheduling and processing method of the distributed network disk in this application embodiment;

[0030] Figure 3This is a schematic diagram of the physical device structure of the scheduling and processing system in the embodiments of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] The following describes the process of the method provided in this implementation. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating the scheduling method of the distributed network disk in this embodiment of the application.

[0034] S101. Receive a data access request sent by the client. The data access request includes a unique identifier of the data block to be transmitted and the size of the data.

[0035] In this context, "client" refers to the terminal device that initiates data access operations to the distributed cloud storage, which can be a personal computer, mobile device, etc. "Data access request" refers to the network request sent by the client to read or write data. "Data block to be transmitted" represents the specific data content that needs to be transmitted. "Unique identifier" is used to uniquely identify the data block to be transmitted, and can be a hash value, UUID, etc. "Data size" refers to the number of bytes in the data block to be transmitted.

[0036] Specifically, when a client needs to access data in the distributed cloud drive, the scheduling and processing system receives the data access request sent by the client. The scheduling and processing system parses the content of the data access request, extracts the unique identifier of the data block to be transmitted, which is used to locate the data storage location later; at the same time, it obtains the size of the data block to be transmitted to evaluate the transmission time.

[0037] S102. Calculate the unique identifier based on the consistent hashing algorithm to obtain the target hash value, and map the target hash value onto the virtual hash ring. The virtual hash ring is used to represent the logical position of multiple storage nodes in the hash space.

[0038] In this context, consistent hashing is a special hashing algorithm used to evenly distribute data across storage nodes in a distributed cloud drive. The target hash value is the hash result obtained by calculating a unique identifier using the consistent hashing algorithm. The virtual hash ring represents the ring structure formed by connecting the head and tail of the hash space. Storage nodes are the server nodes in the distributed cloud drive that actually store the data. Logical locations represent the mapping points of storage nodes on the virtual hash ring.

[0039] Specifically, first, upon receiving a data access request, the scheduling system uses a consistent hashing algorithm to hash the unique identifier of the data block to be transmitted, obtaining a target hash value within a fixed range. Then, the scheduling system maps the target hash value to a connected virtual hash ring. On this virtual hash ring, each storage node occupies several evenly distributed locations. This mapping method allows for the rapid determination of the likely storage area for the data block to be transmitted.

[0040] S103. Starting from the position of the target hash value on the virtual hash ring, traverse the virtual hash ring in a clockwise direction and select a preset number of storage nodes as candidate nodes.

[0041] Here, "corresponding position" represents the specific point on the virtual hash ring where the target hash value is mapped. "Clockwise direction" refers to a fixed traversal direction on the virtual hash ring. "Preset quantity" represents the pre-configured number of candidate nodes. Candidate nodes are the set of storage nodes that may be used to process data access requests.

[0042] Specifically, after determining the corresponding position of the target hash value on the virtual hash ring, the scheduling system uses that position as the starting point and begins traversing the virtual hash ring in a clockwise direction. During the traversal, the scheduling system sequentially selects the storage nodes encountered until the number of selected nodes reaches a preset number. This selection method ensures the determinism of node selection while providing multiple alternatives, thus offering operational flexibility for subsequent load balancing. The preset number of candidate nodes is typically determined based on the system scale and reliability requirements, and can be set to, for example, 3 to 5.

[0043] S104. Send link probe packets marked with sending timestamps to candidate nodes respectively, receive status response packets from candidate nodes including disk queue depth and available bandwidth, and record the receiving timestamp of the status response packets.

[0044] The link probe packet represents a data packet used to test the network connectivity status. The sending timestamp is the system timestamp when the link probe packet was sent. The disk queue depth represents the number of I / O requests currently being processed by the storage node. Available bandwidth refers to the network transmission rate currently available to the storage node. The status response packet represents a data packet containing performance metrics returned by the candidate node. The receiving timestamp is the system timestamp when the status response packet was received.

[0045] Specifically, after identifying candidate nodes, the scheduling system needs to obtain their real-time status information. The system constructs a link probe packet containing a 32-bit or 64-bit timestamp and sends it to each candidate node via TCP or UDP. Upon receiving the link probe packet, the candidate node checks its own status and encapsulates its current disk queue depth (e.g., the number of I / O requests waiting) and available bandwidth (e.g., idle bandwidth obtained through traffic statistics) into a status response packet and returns it. Upon receiving the status response packet, the scheduling system immediately records a millisecond-accurate timestamp for subsequent calculations of network round-trip latency.

[0046] On each storage node of the distributed cloud drive, a status monitoring agent module is pre-deployed. The status monitoring agent module runs as a background daemon process and is used to collect the underlying hardware status of the storage node in real time, including disk queue depth and available bandwidth.

[0047] When the status monitoring agent module receives a link probe packet from the scheduling processing system, it immediately encapsulates the latest cached disk queue depth and available bandwidth into a status response packet and returns it to the scheduling processing system. Preferably, the link probe packet uses the UDP protocol to minimize the network resource consumption during the probe process.

[0048] It should be noted that after sending link probe packets to candidate nodes, the scheduling system starts a short timeout timer (e.g., 200ms). If a candidate node does not respond with a status response packet within the timeout period, the scheduling system marks it as "unavailable" by default or assigns it an extremely large load resistance value, thus excluding it from the preferred selection range. If all preset number of candidate nodes (e.g., 3) time out or become unreachable, to ensure service availability, the scheduling system will trigger a degradation mode: either randomly selecting one candidate node or falling back to the original consistent hash mapping node that has not undergone real-time status correction, ensuring that client requests are always responded to by a node and preventing the scheduling system itself from becoming a single point of failure.

[0049] S105. Calculate the network round-trip time of the candidate node based on the difference between the sending timestamp and the receiving timestamp. Calculate the real-time load resistance value of the candidate node based on the disk queue depth and the network round-trip time, and map the real-time load resistance value to the corresponding queuing waiting time.

[0050] Network round-trip time (RTT) represents the time it takes for a data packet to travel from send to receive. Real-time load resistance is a quantitative indicator that comprehensively reflects the current load status of a storage node. Queuing time represents the estimated time a data access request needs to wait on the storage node. The difference between the send and receive timestamps represents the time consumed by network transmission.

[0051] Specifically, after receiving the status response packets from candidate nodes, the scheduling system first calculates the network round-trip time (RTT) in milliseconds by measuring the difference between the sending and receiving timestamps of each candidate node. The scheduling system can then map the RTT and disk queue depth to a dimensionless value according to preset rules. The system then substitutes the disk queue depth and RTT of each candidate node into a preset calculation formula to calculate a real-time load resistance value reflecting the current load status of the candidate node. This preset calculation formula typically uses a weighted summation method, assigning different weight coefficients to disk queue depth and network RTT; for example, the weight of disk queue depth can be set to 0.7, and the weight of network RTT to 0.3. The scheduling system converts the calculated real-time load resistance value into a specific queuing time using a pre-established mapping curve or lookup table. This mapping relationship typically uses a piecewise function; for example, when the real-time load resistance value is in the range of 0-50, the queuing time increases linearly; when the real-time load resistance value exceeds 50, the queuing time increases exponentially. This allows for a more accurate prediction of the actual waiting time for data access requests under different load conditions.

[0052] Optionally, in general, the real-time load resistance value of a candidate node can be calculated based on disk queue depth and network round-trip latency in the following ways, without limitation: the ratio of network round-trip latency to a preset baseline network delay is determined as the transmission impedance exponent; the ratio of disk queue depth to the maximum concurrent I / O threshold of the candidate node is determined as the first I / O saturation; the first I / O saturation is exponentially weighted to obtain the second I / O saturation; the transmission impedance exponent and the second I / O saturation are linearly summed to obtain the real-time load resistance value of the candidate node.

[0053] Among them, the baseline network latency refers to a pre-defined ideal network round-trip time benchmark value, measured in milliseconds. The transmission impedance index is a proportionality coefficient of the network round-trip time relative to the baseline network latency, used to quantify the degree of network transmission obstruction. The maximum concurrent I / O threshold represents the maximum number of I / O requests a candidate node can handle simultaneously while ensuring normal service quality. The first I / O saturation refers to the proportion of disk queue depth occupied relative to the maximum concurrent I / O threshold. Exponential weighted amplification is a mathematical process that non-linearly amplifies the first I / O saturation using an exponential function. The second I / O saturation represents the I / O load level indicator after exponential weighted amplification. Linear summation is a calculation method that combines multiple indicators in an additive manner.

[0054] Specifically, after obtaining the network round-trip time (RTT) and disk queue depth of candidate nodes, the scheduling system first divides the actual measured RTT by a pre-configured baseline network delay (e.g., 20 milliseconds) to obtain the transmission impedance exponent. Simultaneously, the scheduling system divides the current disk queue depth by a pre-configured maximum concurrent I / O threshold (e.g., 100) to obtain the first I / O saturation. Then, the scheduling system performs exponential weighted amplification on the first I / O saturation, specifically using the exponential function y=x^α (where α is an exponential coefficient greater than 1, e.g., α=2), substituting the first I / O saturation into the calculation to obtain the second I / O saturation. This exponential weighted amplification allows the resistance value of the I / O load to increase faster when it approaches saturation, thus better preventing system overload. Finally, the scheduling system linearly sums the transmission impedance exponent and the second I / O saturation (either by simple addition or by setting different weighting coefficients such as 0.4 and 0.6) to obtain the final real-time load resistance value. This real-time load resistance value comprehensively reflects the performance limitations of candidate nodes in both network transmission and I / O processing dimensions. The larger the value, the greater the load pressure on the candidate node.

[0055] When calculating real-time load resistance, those skilled in the art should understand that network round-trip time (in milliseconds) and disk queue depth (in units) are physical quantities with different dimensions. Therefore, this embodiment eliminates the influence of dimensions through normalization:

[0056] Transmission impedance index: It is the ratio of the network round-trip time to the reference network delay. It is a dimensionless coefficient that represents the relative multiple of the network congestion level.

[0057] I / O saturation: This is the ratio of disk queue depth to the maximum concurrent I / O threshold. It is also a dimensionless coefficient that represents the percentage of I / O load.

[0058] Since both have been converted into dimensionless relative indices, linear summation or weighted calculation can be performed.

[0059] Furthermore, this application is not limited to the linear summation method mentioned in the embodiments. In other feasible alternative embodiments, the real-time load resistance value can also be calculated in one of the following ways:

[0060] Nonlinear product model: Multiply the transmission impedance exponent by the second I / O saturation to highlight the bottleneck effect;

[0061] Vector magnitude calculation: Using the transmission impedance index and the second I / O saturation as two-dimensional vector coordinates, calculate its magnitude as the resistance value;

[0062] Machine learning model: It takes network round-trip latency and disk queue depth as input features, inputs them into a pre-trained neural network or logistic regression model, and outputs an instantaneous load resistance score.

[0063] Any fusion computing logic based on network state and I / O state should be included within the scope of the technical concept of this application.

[0064] Optionally, in general, mapping the real-time load resistance value to the corresponding queuing time can be achieved in the following ways, without limitation: obtain the storage medium type of the candidate node and determine the unit resistance time coefficient corresponding to the storage medium type. The unit resistance time coefficient is used to represent the average time required to process data corresponding to the storage medium type under unit load resistance; multiply the real-time load resistance value by the unit resistance time coefficient to obtain the basic queuing time; obtain the average I / O response jitter value of the candidate node within a preset historical time window, and correct the basic queuing time based on the average I / O response jitter value to obtain the queuing time.

[0065] The storage medium type indicates the type of physical storage device used by the candidate node, such as hard disk drive (HDD), solid-state drive (SSD), or RAM. The unit resistance time factor represents the average time consumed by different storage media to process data under unit load resistance conditions. The base queuing time represents the waiting time initially calculated based on real-time load resistance and storage medium characteristics, in milliseconds. The preset historical time window refers to a pre-defined historical observation period for statistically analyzing I / O performance fluctuations, such as the last 5 or 10 minutes. The average I / O response jitter value refers to the relative fluctuation of I / O response time within the preset historical time window. The queuing wait time represents a more accurate estimated request wait time after correction, in milliseconds.

[0066] Specifically, after obtaining the real-time load resistance value, the scheduling system first queries the storage media type information of the candidate nodes to obtain their storage media types. For different storage media types, the scheduling system retrieves the corresponding unit resistance time coefficient from a pre-configured parameter table. For example, it might be configured as 10 milliseconds / unit load for mechanical hard drives, 2 milliseconds / unit load for solid-state drives, and 0.1 milliseconds / unit load for memory. Then, the scheduling system multiplies the real-time load resistance value by the obtained unit resistance time coefficient to calculate the basic queuing time. Next, the scheduling system queries the I / O response time data recorded by the candidate nodes within a preset historical time window (e.g., the last 5 minutes) and calculates its coefficient of variation (standard deviation divided by the mean) as the average I / O response jitter value. Finally, the scheduling system uses the average I / O response jitter value to calculate a correction coefficient: correction coefficient = 1 + average I / O response jitter value. The scheduling system multiplies the basic queuing time by the correction coefficient to obtain the final queuing wait time (in milliseconds).

[0067] Regarding the method for obtaining the unit resistance time consumption coefficient, this embodiment provides a specific preset and calibration scheme. The unit resistance time consumption coefficient can be a physical benchmark value obtained through offline stress testing or online machine learning. The scheduling and processing system pre-runs benchmark testing tools (such as FIO) for different types of storage media (such as NVMe SSD, SATA SSD, SAS HDD, etc.) to test their average data processing latency under unit load resistance (i.e., standardized I / O stress).

[0068] For example, after testing, the average processing time of a certain SSD model under a unit load is 2ms, so its unit resistance time factor is set to 2ms / unit load; while the unit resistance time factor of a certain HDD model is set to 12ms / unit load.

[0069] Furthermore, the unit resistance time consumption coefficient supports dynamic updates: the scheduling processing system can collect the actual job completion records of storage nodes in the past 24 hours, and use a linear regression algorithm to fit the current true unit resistance time consumption coefficient to adapt to performance changes caused by hardware aging or firmware updates.

[0070] S106. Combining the data volume and the available bandwidth of the candidate nodes, estimate the data transmission time, add the data transmission time to the queuing time, and obtain the expected task completion time of the candidate nodes.

[0071] Data transmission time represents the actual time required for data transmission. Available bandwidth refers to the network transmission capacity currently available to the storage node. Expected task completion time represents the total estimated time from the start of processing to the completion of transmission.

[0072] Specifically, first, the scheduling system converts the data size to the same unit as the available bandwidth (e.g., all converted to bits). Then, considering network protocol overhead (e.g., TCP / IP header overhead is approximately 5-10%) and actual bandwidth utilization (typically 60%-80% of the theoretical bandwidth), the scheduling system adjusts the available bandwidth. Next, the scheduling system divides the adjusted data size by the adjusted available bandwidth to obtain the theoretical data transmission time. Taking into account factors such as retransmissions and flow control in network transmission, the scheduling system adds a 10%-20% redundancy time to the theoretical data transmission time, resulting in the final data transmission time. Finally, the scheduling system adds the calculated data transmission time (in milliseconds) to the previously obtained queuing time (in milliseconds) to obtain the expected task completion time in milliseconds.

[0073] Optionally, under normal circumstances, the estimated data transmission time can be achieved by combining the data volume and the available bandwidth of the candidate nodes in the following ways, which are not limited here: determine whether the data volume is less than the preset TCP slow start threshold; if so, calculate the bandwidth utilization coefficient based on the data volume, which is positively correlated with the data volume; use the bandwidth utilization coefficient to correct the available bandwidth to obtain the effective transmission bandwidth; calculate the ratio of the data volume to the effective transmission bandwidth to obtain the data transmission time.

[0074] The preset TCP slow start threshold represents the upper limit of the data volume for which the TCP protocol uses a slow start mechanism during the initial stage of transmission, measured in bytes. The bandwidth utilization coefficient refers to the effective utilization ratio of theoretical bandwidth during actual transmission, a dimensionless coefficient ranging from 0 to 1. Effective transmission bandwidth represents the actual available data transmission rate after bandwidth utilization correction, measured in the same units as available bandwidth (bps or multiples thereof). Positive correlation refers to the relationship where the bandwidth utilization coefficient increases accordingly with the increase in data volume.

[0075] Specifically, the scheduling system compares the received data size with a preset TCP slow start threshold (typically 64KB or 128KB). When the data size is less than the preset TCP slow start threshold, the scheduling system considers the data transmission process to be significantly affected by the TCP slow start mechanism. In this case, the scheduling system calculates a bandwidth utilization coefficient based on the data size, using the formula: Bandwidth utilization coefficient = MIN(0.2 + 0.8 × (data size / preset TCP slow start threshold), 1.0). Then, the scheduling system multiplies the available bandwidth of the candidate node by this bandwidth utilization coefficient to obtain the effective transmission bandwidth. Finally, the scheduling system divides the data size by the effective transmission bandwidth (note the unit conversion to ensure the final result is in milliseconds) to obtain the data transmission time considering the impact of TCP slow start.

[0076] Optionally, under normal circumstances, the data transmission time can be obtained in the following ways, without limitation: obtain the total number of link probe packets sent to the candidate node within a preset time period, and the total number of received packets that successfully receive the corresponding status response packets; calculate the difference between the total number of sent packets and the total number of received packets, and determine the packet loss rate as the ratio of the difference to the total number of sent packets; calculate the bandwidth loss coefficient based on the packet loss rate, the higher the packet loss rate, the larger the bandwidth loss coefficient; reduce the available bandwidth of the candidate node using the bandwidth loss coefficient to obtain the actual effective bandwidth of the candidate node; divide the data size by the actual effective bandwidth to calculate the data transmission time.

[0077] Regarding the calculation of the bandwidth loss factor, it should be noted that the packet loss rate is based on statistical values ​​obtained from historical statistical periods, rather than being calculated in real time during this single scheduling process.

[0078] Specifically, the scheduling system runs an asynchronous statistics thread in the background, which maintains a sliding time window (e.g., the past 5 minutes). This thread continuously counts the total number of probe packets sent to each storage node and the total number of response packets received within the sliding time window. When the main scheduling process needs to calculate the bandwidth loss of a candidate node, it directly reads the latest packet loss rate data maintained by this asynchronous statistics thread. This design avoids the need for lengthy packet capture waits within a single user request to obtain the packet loss rate, ensuring millisecond-level response speeds for scheduling decisions while also incorporating the impact of network quality on effective bandwidth.

[0079] S107. Compare the expected task completion times of candidate nodes, select the candidate node with the shortest expected task completion time as the target execution node, generate a scheduling instruction including the target network address corresponding to the target execution node, and send it to the client to instruct the client to establish a data transmission connection with the target execution node.

[0080] Here, the target execution node refers to the storage node ultimately selected to execute the data transmission task. The target network address refers to the IP address and port number of the target execution node. The scheduling instruction represents the control instructions that guide the client to establish a connection. The data transmission connection refers to the data transmission channel established between the client and the target execution node.

[0081] Specifically, the scheduling system sorts and compares the expected task completion times of all candidate nodes, selecting the candidate node with the shortest expected task completion time as the target execution node (which currently has the best processing capacity and network status) and is expected to complete the data transmission task the fastest. Subsequently, the scheduling system generates a scheduling instruction containing the target execution node's network address information (e.g., IP:PORT format) and sends it to the client through an initial connection. Upon receiving the scheduling instruction, the client establishes a new TCP connection with the target execution node and begins the actual data transmission process.

[0082] By adopting the above technical solution, firstly, the scheduling and processing system maps the unique identifier of the data block to be transmitted to a virtual hash ring using a consistent hashing algorithm, accurately identifying a preset number of candidate nodes. This retains the fast distribution characteristics of hash mapping while avoiding the limitations of single-node binding. Subsequently, the scheduling and processing system obtains the real-time status of candidate nodes, such as disk queue depth and available bandwidth, through link probe packets. It then calculates the real-time load resistance value based on network round-trip latency, while simultaneously estimating data transmission time and queuing time. Finally, it selects the target execution node with the shortest expected task completion time. This closed-loop design of "hash positioning + real-time status evaluation + comprehensive time-based sorting" achieves dynamic matching between data access requests and storage node performance, effectively avoiding resource waste caused by some nodes being overloaded and congested while others are idle. In large-scale data access scenarios, it can significantly improve overall data transmission efficiency, reduce average task completion latency, ensure the load balance of the distributed cloud storage system, enhance the stability and reliability of the distributed cloud storage system, and provide users with a more efficient and smooth data access experience.

[0083] The following provides a more detailed description of the process of the method provided in this implementation. Please refer to [link / reference]. Figure 2 This is another flowchart illustrating the scheduling and processing method of the distributed network disk in this application embodiment.

[0084] S201. Receive a data access request sent by the client. The data access request includes a unique identifier of the data block to be transmitted and the size of the data.

[0085] For details, please refer to step S101, which will not be repeated here.

[0086] S202. Calculate the unique identifier based on the consistent hashing algorithm to obtain the target hash value, and map the target hash value onto the virtual hash ring. The virtual hash ring is used to represent the logical position of multiple storage nodes in the hash space.

[0087] For details, please refer to step S102, which will not be repeated here.

[0088] S203. Starting from the position of the target hash value on the virtual hash ring, traverse the virtual hash ring in a clockwise direction and select a preset number of storage nodes as candidate nodes.

[0089] For details, please refer to step S103, which will not be repeated here.

[0090] S204. Determine whether the unique identifier matches the target data identifier in the hot data list. The hot data list includes multiple data identifiers, and the target data identifier is any one of the data identifiers.

[0091] The hot data list represents a dynamic list maintained by the scheduling and processing system that records frequently accessed data. A data identifier is a string or number used to uniquely identify a data block. The target data identifier represents the data identifier of the hot data that needs to be matched and compared. A match indicates that two data identifiers are completely identical in terms of bit length, content, etc. Frequent access refers to an access pattern that is requested more than a preset threshold number of times per unit of time.

[0092] Specifically, after selecting candidate nodes, the scheduling system first checks whether data access requests involve hot data, which helps optimize load balancing. First, the scheduling system retrieves a currently maintained list of hot data from memory or cache. This list typically includes data identifiers that have been accessed the top N (e.g., top 100) times in a recent period (e.g., the past hour). Then, the scheduling system performs an exact match between the unique identifier of the data block to be transmitted and each identifier in the hot data list. The matching rules can be string equality or hash value equality. This matching process typically uses an efficient lookup algorithm (e.g., hash table lookup) to ensure rapid result location.

[0093] S205. If yes, then query the target node corresponding to the target data identifier.

[0094] Specifically, after confirming that a data access request has hit hot data, the scheduling system needs to determine the current storage location of that hot data. The system queries a mapping table that maintains the distribution status of hot data. This table is typically organized in a key-value pair format, where the key is the identifier of the hot data and the value is a list of nodes storing the hot data. The system uses the target data identifier as the query key to retrieve the corresponding target node and its information. The target node information includes necessary parameters for subsequent access, such as the target node's network address and storage path.

[0095] S206. Determine whether there is an intersection between the target node and the candidate node.

[0096] Here, the intersection node represents a stored node that appears simultaneously in both the target node set and the candidate node set. This can be determined using set operations, which refer to performing a mathematical intersection operation on the two node sets. The existence check represents the logical operation of checking whether the intersection set is empty.

[0097] Specifically, after obtaining the target nodes and candidate nodes, the scheduling system needs to determine whether these two sets of nodes overlap. The scheduling system performs a set intersection operation on the target node set and the candidate node set. This can be achieved by traversing the smaller node set and checking whether each node in it exists in the other node set. If the intersection is not empty, it means that there exists a node that simultaneously possesses the advantages of hot data caching and hash proximity; that is, an intersection node.

[0098] S207. If so, the disk queue depth of the intersection node will be reduced according to the preset cache hit weight to reduce the real-time load resistance value of the intersection node.

[0099] The preset cache hit weight refers to a pre-configured coefficient value used to adjust the load assessment under cache hit conditions, typically ranging from 0 to 1. Reduction processing refers to the mathematical operation process of reducing the original value by a specific ratio.

[0100] Specifically, after discovering overlapping nodes, the scheduling system needs to perform special processing on the load status of these nodes. The system first obtains the current disk queue depth of the overlapping nodes, then reads a preset cache hit weight (e.g., 0.6). This preset cache hit weight reflects the reduction in processing load that can be achieved when data is already in the node's cache. The system multiplies the original disk queue depth by this preset cache hit weight to obtain a reduced disk queue depth. For example, if the current disk queue depth of an overlapping node is 100 and the preset cache hit weight is 0.6, the reduced disk queue depth will become 60. This processing method takes into account the characteristic that cache hits for hot data can significantly reduce actual I / O operations, making the load assessment more consistent with reality.

[0101] S208. Query whether there are historical status records of candidate nodes in the local cache.

[0102] The local cache refers to a temporary data storage area maintained internally by the scheduling and processing system. Historical state records refer to the set of node performance metric data previously collected through probing.

[0103] Specifically, before retrieving candidate node status information, the scheduling system first checks its local cache for available historical status records. The scheduling system maintains an efficient local cache system (typically implemented using a hash table or key-value store) that stores the status information of each storage node detected recently. The scheduling system searches its local cache for the data identifier of each candidate node to check if a corresponding historical status record exists.

[0104] S209. If there are historical status records of candidate nodes in the local cache and the update time of the historical status records is within the preset valid time window, then directly read the disk queue depth and available bandwidth in the historical status records.

[0105] The update time represents the timestamp of the last refresh of the historical status record. The preset valid time window refers to the maximum time span during which historical data can be trusted.

[0106] Specifically, upon discovering historical state records, the scheduling system further verifies their timeliness. First, the system obtains the update time of the historical state record, compares it with the current time, and calculates the time difference. Then, it compares this time difference with a preset valid time window (e.g., 30 seconds). If the time difference is less than the preset valid time window, it indicates that the historical data still has reference value. The scheduling system will then directly read the disk queue depth and available bandwidth—two key performance indicators—from the historical state records without real-time probing, significantly reducing unnecessary network probing overhead.

[0107] S210. If there is no historical status record of the candidate node in the local cache and / or the update time of the historical status record is outside the preset valid time window, then the step of sending link probe packets marked with the sending timestamp to the candidate node is executed.

[0108] Specifically, when the scheduling system detects that there are no valid historical state records in its local cache, it needs to initiate a real-time probing process. The scheduling system constructs standard link probe packets, containing millisecond-precision sending timestamps, sequence numbers, and other information. These link probe packets are typically sent using the UDP protocol to reduce connection establishment overhead. The scheduling system sends link probe packets in parallel to all candidate nodes that need to be probed. Each link probe packet carries a unique sequence number for matching subsequent status response packets. In this way, the scheduling system can obtain the latest status information of candidate nodes, ensuring the accuracy of load balancing decisions. This probing process typically includes a timeout mechanism (e.g., 500 milliseconds) to handle possible node unresponsiveness.

[0109] S211. Send link probe packets marked with sending timestamps to candidate nodes, receive status response packets from candidate nodes including disk queue depth and available bandwidth, and record the receiving timestamp of the status response packets.

[0110] For details, please refer to step S104, which will not be repeated here.

[0111] S212. Calculate the network round-trip time of the candidate node based on the difference between the sending timestamp and the receiving timestamp. Calculate the real-time load resistance value of the candidate node based on the disk queue depth and the network round-trip time, and map the real-time load resistance value to the corresponding queuing waiting time.

[0112] For details, please refer to step S105, which will not be repeated here.

[0113] S213. Combining the data volume and the available bandwidth of the candidate nodes, estimate the data transmission time, add the data transmission time to the queuing time, and obtain the expected task completion time of the candidate nodes.

[0114] For details, please refer to step S106, which will not be repeated here.

[0115] S214. Compare the expected task completion times of the candidate nodes, select the candidate node with the shortest expected task completion time as the target execution node, generate a scheduling instruction including the target network address corresponding to the target execution node, and send it to the client to instruct the client to establish a data transmission connection with the target execution node.

[0116] For details, please refer to step S107, which will not be repeated here.

[0117] The scheduling processing system in the embodiments of this invention is described below from a hardware processing perspective. Please refer to [link / reference needed]. Figure 3 This is a schematic diagram of the physical device structure of the scheduling and processing system in this application embodiment.

[0118] It should be noted that, Figure 3 The structure of the scheduling and processing system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0119] like Figure 3 As shown, the scheduling processing system includes a CPU 301, which can perform various appropriate actions and processes based on a program stored in the read-only memory ROM 302 or a program loaded from the storage section 308 into the random access memory RAM 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0120] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0121] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by CPU 301, it performs the various functions defined in the present invention.

[0122] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0124] Specifically, the scheduling system in this embodiment includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the scheduling method for the distributed network disk provided in the above embodiment.

[0125] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the scheduling processing system described in the above embodiments; or it may exist independently and not be assembled into the scheduling processing system. The storage medium carries one or more computer programs, which, when executed by a processor of the scheduling processing system, cause the scheduling processing system to implement the distributed network disk scheduling method provided in the above embodiments.

[0126] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0127] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A scheduling and processing method for a distributed cloud storage system, characterized in that, The method, applied to a scheduling and processing system, includes: Receive a data access request sent by a client, the data access request including a unique identifier of the data block to be transmitted and the size of the data; The unique identifier is calculated based on the consistent hashing algorithm to obtain the target hash value, and the target hash value is mapped onto a virtual hash ring, which is used to represent the logical position of multiple storage nodes in the hash space. Starting from the position of the target hash value on the virtual hash ring, the virtual hash ring is traversed in a clockwise direction, and a preset number of storage nodes are selected as candidate nodes. Send link probe packets marked with a sending timestamp to each of the candidate nodes, receive status response packets from the candidate nodes including disk queue depth and available bandwidth, and record the receiving timestamp of the status response packets; Based on the difference between the sending timestamp and the receiving timestamp, the network round-trip time of the candidate node is calculated. Based on the disk queue depth and the network round-trip time, the real-time load resistance value of the candidate node is calculated, and the real-time load resistance value is mapped to the corresponding queuing waiting time. Based on the data volume and the available bandwidth of the candidate nodes, the data transmission time is estimated, and the data transmission time is added to the queuing time to obtain the expected task completion time of the candidate nodes. The expected task completion time of the candidate nodes is compared, and the candidate node with the shortest expected task completion time is selected as the target execution node. A scheduling instruction including the target network address corresponding to the target execution node is generated and sent to the client to instruct the client to establish a data transmission connection with the target execution node. The step of calculating the real-time load resistance value of the candidate node based on the disk queue depth and the network round-trip time specifically includes: determining the ratio of the network round-trip time to a preset baseline network delay as the transmission impedance exponent; determining the ratio of the disk queue depth to the maximum concurrent I / O threshold of the candidate node as the first I / O saturation; exponentially weighting the first I / O saturation to obtain the second I / O saturation; and linearly summing the transmission impedance exponent and the second I / O saturation to obtain the real-time load resistance value of the candidate node. The step of mapping the real-time load resistance value to the corresponding queuing time specifically includes: obtaining the storage medium type of the candidate node and determining the unit resistance time coefficient corresponding to the storage medium type, wherein the unit resistance time coefficient is used to represent the average time required to process data corresponding to the storage medium type under unit load resistance; multiplying the real-time load resistance value by the unit resistance time coefficient to obtain the basic queuing time; obtaining the average I / O response jitter value of the candidate node within a preset historical time window, and correcting the basic queuing time based on the average I / O response jitter value to obtain the queuing time.

2. The method according to claim 1, characterized in that, The step of combining the data volume and the available bandwidth of the candidate nodes to estimate the data transmission time specifically includes: Determine whether the amount of data is less than a preset TCP slow start threshold; If so, then the bandwidth utilization coefficient is calculated based on the data volume, and the bandwidth utilization coefficient is positively correlated with the data volume. The available bandwidth is corrected using the bandwidth utilization coefficient to obtain the effective transmission bandwidth; The data transmission time is obtained by calculating the ratio of the data volume to the effective transmission bandwidth.

3. The method according to claim 2, characterized in that, Before the step of sending link probe packets marked with a transmission timestamp to the candidate nodes respectively, the method further includes: Check if the candidate node's historical state record exists in the local cache; If the candidate node's historical status record exists in the local cache and the update time of the historical status record is within a preset valid time window, then the disk queue depth and available bandwidth in the historical status record are read directly. If the candidate node's historical state record is not present in the local cache and / or the update time of the historical state record is outside the preset valid time window, then the step of sending link probe packets marked with a sending timestamp to the candidate node is executed.

4. The method according to claim 1, characterized in that, After the step of traversing the virtual hash ring clockwise from the corresponding position of the target hash value on the virtual hash ring and selecting a preset number of storage nodes as candidate nodes, the method further includes: Determine whether the unique identifier matches the target data identifier in the hot data list, wherein the hot data list includes multiple data identifiers, and the target data identifier is any one of the data identifiers. If so, then query the target node corresponding to the target data identifier; Determine whether there is an intersection between the target node and the candidate node; If so, the disk queue depth of the intersection node is reduced according to the preset cache hit weight to reduce the real-time load resistance value of the intersection node.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the total number of link probe packets sent to the candidate node within a preset time period, and the total number of packets successfully received corresponding to the status response packets. Calculate the difference between the total transmit count and the total receive count, and determine the ratio of the difference to the total transmit count as the data packet loss rate; The bandwidth loss coefficient is calculated based on the packet loss rate. The higher the packet loss rate, the larger the bandwidth loss coefficient. The available bandwidth of the candidate node is reduced using the bandwidth loss factor to obtain the actual effective bandwidth of the candidate node; The data transmission time is calculated by dividing the data volume by the actual effective bandwidth.

6. A scheduling and processing system, characterized in that, The scheduling processing system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors invoke the computer instructions to cause the scheduling processing system to perform the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the scheduling processing system, it causes the scheduling processing system to perform the method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, When the computer program product is run on the scheduling processing system, the scheduling processing system performs the method as described in any one of claims 1 to 5.

Images