Data processing method and first network device

Abstract

A data processing method is executed by a network device constituting a hyper-converged infrastructure. When the running resource of the network device is relatively large, the network device can detect whether the data read by a read request is in a high-performance priority volume. If the data read by the read request is in the high-performance priority volume, the read request is queued in the high-performance priority volume, and the data in the local memory is read, so that the stable high IOPS and low latency of the network device can be ensured. If the data read by the read request is not in the high-performance priority volume, the read request can directly read the data in the memory of other network devices, so that the network device resources are prevented from being contended by the low-performance priority volume and the high-performance priority volume, and the latency of the read request reading the data in the low-performance priority volume is avoided.

Description

Technical Field

[0001] This invention relates to the field of data reading technology, and more particularly to a data processing method and a first network device. Background Technology

[0002] Hyperconverged infrastructure (HCI) is an information infrastructure that integrates virtual computing resources and storage devices. HCI is typically deployed in a distributed system consisting of multiple network devices. If a node receives a large number of read input / output (I / O) and write I / O requests in a short period of time, the node may not be able to process each read I / O request in a timely manner, leading to timeouts for some read I / O and write I / O requests. Summary of the Invention

[0003] To address the aforementioned problems, embodiments of this application provide a data processing method that can guarantee stable high IOPS and low latency for network devices. Furthermore, this application also provides a first network device corresponding to this data processing method.

[0004] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0005] In a first aspect, embodiments of this application provide a data processing method executed by a first network device. The method includes: receiving a read request; detecting whether the resources being operated by the first network device exceed a first preset threshold; in response to the first network device operating resources exceeding the first preset threshold, detecting whether the data to be read by the read request is located within a first type of volume; wherein the first type of volume is a high-priority volume, the volume is composed of multiple memories, or a memory includes multiple volumes, and the volume is one of the multiple volumes; in response to the read request being located within a first type of volume, accessing local data of the first network device based on the read request.

[0006] In this embodiment, when the first network device detects that the resources running read requests are relatively large, the number of read requests is also relatively high. The first network device prioritizes read I / O requests corresponding to high-priority first-type volumes to access data in local memory, ensuring stable high IOPS and low latency for the first network device to meet user requirements.

[0007] In one implementation, detecting whether the resources being used by the first network device exceed a first preset threshold specifically includes: detecting whether the CPU's computing resource utilization ratio of the first network device exceeds the first preset threshold, where the first preset threshold is the CPU's computing resource utilization ratio.

[0008] In this embodiment, the higher the CPU's computing resource utilization rate, the more read requests can be executed. The first network device can determine the number of read requests to be executed based on the CPU's computing resource utilization rate, thereby informing the first network device of the maximum number of read requests that can be executed.

[0009] In one implementation, detecting whether the resources running the first network device exceed a first preset threshold specifically includes: detecting whether the amount of data cached in the running memory of the first network device exceeds the first preset threshold, wherein the first preset threshold is the amount of data cached in the running memory.

[0010] In this embodiment, the more data cached in memory, the more read requests can be executed. The first network device can determine the number of read requests to be executed based on the amount of data cached in memory, thereby informing the first network device of the maximum number of read requests that can be executed.

[0011] In one implementation, detecting whether the resources being operated by the first network device exceed a first preset threshold specifically includes: detecting whether the token issuance value of the token bucket of the first network device exceeds the first preset threshold, wherein the first preset threshold is the token issuance value.

[0012] In this embodiment, the larger the token issuance value of the token bucket, the more read requests can be executed. The first network device can determine the number of read requests to be executed based on the token issuance value of the token bucket, thereby letting the first network device know the maximum number of read requests to be executed.

[0013] In one embodiment, the method further includes: in response to the resource being operated by the first network device being less than or equal to the first preset threshold, accessing local data of the first network device based on the read request.

[0014] In this embodiment, when the resources of the first network device to run all read requests are less than or equal to a first set threshold, the resources of the first network device are relatively sufficient to run all read requests to read the data stored in the local memory, thereby improving the stable high IOPS and low latency of the first network device.

[0015] In one implementation, the method further includes: in response to the read request not being read from a volume of the first type, accessing data from other network devices based on the read request.

[0016] In this embodiment, when the first network device determines that the data to be read by the read request is not located in a high-priority volume, the first network device can allow the read request to read data from the memory of other network devices, thereby avoiding contention between low-priority and high-priority volumes for network device resources and avoiding latency in reading data from low-priority volumes.

[0017] In one implementation, if the data read in response to the read request is not located within a volume of the first type, the method further includes: detecting whether the data read by the read request is located within a volume of the second type, wherein the volume of the second type has a lower priority than the volume of the first type; if the data read by the read request is located within a volume of the second type, sending the read request to a second network device, wherein the second network device is a network device with operating resources less than a second set threshold; wherein the second network device is communicatively connected to the first network device; or, if the data read by the read request is not located within a volume of the second type, sending the read request to a third network device, wherein the third network device is a network device with operating resources greater than or equal to the second set threshold, wherein the third network device is communicatively connected to the first network device.

[0018] In this embodiment, the second set threshold is greater than the maximum resource value of a normal read request from the first network device. When the resource value of a normal read request from the first network device is greater than the maximum resource value of a normal read request from the first network device, but less than the second set threshold, the first network device can access the third network device based on read requests for low-priority volumes. This avoids contention for network device resources by low-priority volumes and avoids latency in reading data from low-priority volumes. When the resource value of a normal read request from the first network device is greater than the second set threshold, but less than the first set threshold, the first network device can access the third network device based on read requests for low-priority volumes, and access the second network device based on read requests for medium-priority volumes. This avoids contention for network device resources by low-to-medium priority volumes and avoids latency in reading data from low-to-medium priority volumes.

[0019] Secondly, embodiments of this application provide a first network device, characterized in that it includes: a transceiver and a processor, wherein the transceiver is configured to receive a read request; the processor is configured to detect whether the resources being operated by the first network device exceed a first preset threshold; in response to the first network device operating resources exceeding the first preset threshold, detect whether the data to be read by the read request is located within a first type of volume; wherein the first type of volume is a high-priority volume, the volume is composed of multiple memories, or a memory includes multiple volumes, the volume being one of the multiple volumes; in response to the read request being located within a first type of volume, accessing local data of the first network device based on the read request.

[0020] In one embodiment, the processor is specifically configured to detect whether the CPU's computing resource utilization ratio of the first network device is greater than a first preset threshold, the first preset threshold being the CPU's computing resource utilization ratio; or to detect whether the amount of data cached in the first network device's running memory is greater than the first preset threshold, the first preset threshold being the amount of data cached in the running memory; or to detect whether the token issuance value of the first network device's token bucket is greater than the first preset threshold, the first preset threshold being the token issuance value.

[0021] In one implementation, the processor is configured to access local data of the first network device based on the read request in response to the first network device operating resources being less than or equal to the first preset threshold.

[0022] In one implementation, the processor is configured to, in response to a read request that the data to be read is not in the first type of volume, access data from other network devices based on the read request.

[0023] In one embodiment, the processor is further configured to: detect whether the data to be read by the read request is located in a second type of volume in response to the read request not being located in the first type of volume, wherein the second type of volume has a lower priority than the first type of volume; send the read request to a second network device in response to the read request being located in the second type of volume, wherein the second network device is a network device with operating resources less than a second set threshold, wherein the second network device is communicatively connected to the first network device; or, send the read request to a third network device in response to the read request not being located in the second type of volume, wherein the third network device is a network device with operating resources greater than or equal to the second set threshold, wherein the third network device is communicatively connected to the first network device.

[0024] Thirdly, embodiments of this application provide a hyperconverged infrastructure, including: a network device comprising multiple virtual machines, wherein the virtual machines execute the methods that may be implemented as described in the first aspect.

[0025] Fourthly, embodiments of this application provide a hyperconverged infrastructure, including: multiple network devices, wherein the network devices perform the methods that may be implemented as described in the first aspect.

[0026] Fifthly, embodiments of this application provide a computing device, including: at least one memory and at least one processor, the at least one processor being configured to execute instructions stored in the at least one memory to cause the processor to perform the methods as may be implemented in the first aspect.

[0027] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods that are possible implementations of the first aspect.

[0028] In a seventh aspect, embodiments of this application provide a computer program product, characterized in that the computer program product stores instructions that, when executed by a computer, cause the computer to perform the methods that may be implemented as described in the first aspect. Attached Figure Description

[0029] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0030] Figure 1 This is a schematic diagram of an HCI architecture provided in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram of another HCI architecture provided in the embodiments of this application;

[0032] Figure 3 This is a schematic diagram of the architecture of the processor units provided in the embodiments of this application;

[0033] Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of a data center architecture provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of another data center architecture provided in the embodiments of this application. Detailed Implementation

[0037] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0038] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0039] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0040] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0041] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0042] Traditional data centers primarily consist of servers, network equipment, storage devices, and auxiliary equipment, which are typically supplied by multiple vendors. As business applications and the data they generate grow, the types and quantities of equipment also increase, leading to a rapid increase in operational costs and equipment deployment cycles. This also increases the difficulty of resource scheduling and reduces the resource utilization efficiency of the data center.

[0043] To address the shortcomings of existing data centers, converged systems are offered by the same vendor or several vendors. A converged system integrates servers, network equipment, and storage devices onto a single rack or cabinet to reduce deployment complexity and improve resource utilization. However, converged systems simply assemble servers, network equipment, and storage devices together; these devices are still managed and expanded independently, failing to solve the problems of high data center operation and maintenance costs and difficulties in upgrading and expanding.

[0044] HCI (Hardware-Defined Storage) is a software-defined storage architecture that uses a large number of servers to form a cluster. Through virtualized computing and networking technologies, it integrates local resources distributed across different nodes into a unified resource pool. HCI exposes standard computing, storage, and networking capabilities through a unified management interface, enabling modular and seamless horizontal scaling. HCI's scaling mechanism involves horizontally adding nodes, making data center construction very convenient, increasing data center flexibility, and simplifying deployment and maintenance.

[0045] HCI is typically deployed in distributed systems, which are collections of independent nodes. Taking an HCI consisting of multiple nodes as an example, when a host sends a write request to the target node of the HCI, the target node writes the data not only to its own storage but also to the storage of other nodes to prevent data loss due to target node failure. When a host sends a read request to the target node of the HCI, the target node reads data from its own storage or from the storage of other nodes based on the read request.

[0046] In this context, nodes can be network devices with storage and computing capabilities, such as servers or computers. The following explanation uses nodes as network devices. Network devices typically have at least one memory. This memory can be partitioned based on defined logic to create multiple volumes. A volume can consist of multiple memories. Alternatively, a memory can include multiple volumes; that is, a memory can be partitioned to create multiple volumes. Administrators can assign priorities to each volume based on its importance, classifying them as high-priority, medium-priority, and low-priority volumes. High-priority volumes execute read and write requests first. Medium-priority volumes execute read and write requests later. Low-priority volumes execute read and write requests last. In this embodiment, the first type of volume refers to a high-priority volume, i.e., a volume with high priority. The second type of volume refers to a medium-priority volume. Other types of volumes refer to low-priority volumes. In other embodiments, the priorities of the first type of volume, the second type of volume, and other types of volumes decrease sequentially. The number of volume levels can be increased; this application only uses three levels, but it can use two, four, or more levels, and this application is not limited to these.

[0047] In related network devices, when a network device suddenly receives a large number of read requests, it will forward some of the read requests to other network devices to read data from their memory. However, when the network device's operating load exceeds its capacity, all volumes will compete for network device resources, failing to leverage the capabilities of a distributed cluster. If low-priority volumes experience high read request pressure, they will compete for more resources, leading to increased latency for high-priority volume read requests. Conversely, if high-priority volumes experience high read request pressure, they will compete for more resources, further increasing the latency for low-priority volume read requests, causing low-priority volume read requests to time out and fail.

[0048] To address the shortcomings of existing network devices, embodiments of this application provide an HCI, a network device, and a data processing method.

[0049] Figure 1 This is a schematic diagram of an HCI architecture provided in an embodiment of this application. Figure 1 As shown, the HCI 10 includes multiple network devices 100. These network devices 100 are designated as network device 100-1, network device 100-2, ..., network device 100-n. Each network device 100 is a node in the HCI 10. Network devices 100 can be servers, computers, or other similar devices. The multiple network devices 100 establish communication connections with each other, enabling seamless horizontal scaling of multiple nodes and forming a unified resource pool.

[0050] Each network device 100 includes a transceiver 110, a processor 120, and a memory 130. The transceiver 110 of the network device 100 is used to establish communication connections between the network device 100 and other network devices 100, enabling data transmission between multiple network devices 100. The transceiver 110 can be a wired interface, such as gold wire, pins, a printed circuit board (PCB), or other types of wired interfaces. The transceiver 110 can also be a wireless interface, such as a Bluetooth module, a near field communication (NFC) module, or other wireless communication modules.

[0051] The processor 120 can be a central processing unit (CPU), a microprocessor unit (MCU), a cloud server, or other devices with processing and control functions. The processor 120 is used to receive data stored in the memory 130 and data received through the transceiver 110, and to execute data processing methods.

[0052] The memory 130 may include volatile memory, such as random-access memory (RAM). The memory 130 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. The memory 130 may be a cloud disk or a combination of the various memory types mentioned above. In this embodiment, the memory 130 can serve as a database for storing large amounts of data. The memory 130 can store applications running on the processor 120, allowing the processor 120 to execute data processing methods. The memory 130 can serve as a cache for storing read requests, write requests, and other data. The memory 130 may also have other functions, which are not limited herein.

[0053] Figure 2 This is a schematic diagram of another HCI architecture provided in an embodiment of this application. For example... Figure 2 As shown, HCI is a network device 200. Network device 200 is a node of HCI. Network device 200 includes a processor 210 and a memory 220. The processor 210 can be... Figure 1 The processor 110 of each network device 100. The memory 220 can be... Figure 1 The memory 130 of each network device 100.

[0054] Network device 200 can virtualize multiple virtual machines (VMs) according to a set logic. A VM refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. A VM includes a processing unit and a storage unit. The processing unit is the virtual processor of network device 200, which performs part of the computing functions of processor 210. The storage unit is the virtual memory of network device 200, which provides part of the computing storage space for memory 220. In this embodiment, when HCI is a single network device 200, each VM of network device 200 can serve as... Figure 1 Network device 100 in the middle, executes Figure 1 The functions of network device 100. Each VM's processing unit has Figure 1 The functions of the processor 110 of each network device 100. Each VM's storage unit has... Figure 1 The functions of the memory 130 of each network device 100.

[0055] The following is based on Figure 1 Taking HCI 10 as an example, the implementation process of the technical solution protected in this application is introduced. It is conceivable that the technical solution protected in this application can be implemented through... Figure 2 The network device 200 shown is implemented.

[0056] like Figure 3 As shown, the processor 120 can be divided into a load analysis unit 121, a performance statistics unit 122, and an I / O scheduling unit 123 according to its execution functions. Each unit can be implemented in software or in hardware. For example, the implementation of the load analysis unit 121 will be described below. Similarly, the implementation of the performance statistics unit 122 and the I / O scheduling unit 123 can refer to the implementation of the load analysis unit 121.

[0057] As an example of a software functional unit, the load analysis unit 121 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the load analysis unit 121 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0058] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0059] As an example of a hardware functional unit, the load analysis unit 121 may include at least one computing device, such as a server. Alternatively, the load analysis unit 121 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0060] When the load analysis unit 121 includes multiple computing devices, these devices can be distributed in the same region or in different regions. They can also be distributed in the same Availability Zone (AZ) or in different AZs. Similarly, they can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0061] It should be noted that, in other embodiments, the load analysis unit 121, the performance statistics unit 122, and the I / O scheduling unit 123 can all execute any step in the data processing method. The steps implemented by the load analysis unit 121, the performance statistics unit 122, and the I / O scheduling unit 123 can be specified as needed. By implementing different steps in the data processing method through the load analysis unit 121, the performance statistics unit 122, and the I / O scheduling unit 123, all functions of the network device 100 can be achieved.

[0062] The load analysis unit 121 can detect the resources used by the network device 100 to process received read requests based on the status of its CPU, RAM, token bucket, and other components. The more read requests, the more resources the network device 100 uses, and the more likely it is to operate under high load. Conversely, the fewer read requests, the fewer resources the network device 100 uses, and the more likely it is to operate under low load. In this embodiment, a read request can be a read I / O request. I / O refers to input / output ports. Each device has a dedicated I / O address for processing its own input / output information. A read I / O request is a request to read data through a specified I / O address. Resources can be I / O resources. I / O resources refer to I / O address resources, which are the resources consumed by a series of related operations such as reading data, writing data, and analyzing related requests through a specified I / O address.

[0063] In this embodiment, the load analysis unit 121 can detect the resource status of the network device 100 based on the CPU's computing resource utilization ratio. Here, network device 100 is a first network device. If the CPU's computing resource utilization ratio is greater than a first preset threshold, network device 100 is operating under high load, and it is difficult for network device 100 to allocate additional resources to execute read requests. If the CPU's computing resource utilization ratio is less than or equal to the first preset threshold, network device 100 is operating under low load, and network device 100 can allocate additional resources to execute read requests. The first preset threshold is the CPU's computing resource utilization ratio. The CPU's computing resource utilization ratio refers to the I / O address resource utilization ratio, which is the ratio of used I / O address resources to all I / O address resources.

[0064] In other embodiments, the load analysis unit 121 can detect the resource status of the network device 100 based on the size (data volume) of the cached data in the running memory. If the cached data volume in the running memory is greater than a first preset threshold, the network device 100 is operating under high load, and it is difficult for the network device 100 to call additional resources to execute read requests. If the cached data volume in the running memory is less than or equal to the first preset threshold, the network device 100 is operating under low load, and the network device 100 can call additional resources to execute read requests. The first preset threshold is the cached data volume in the running memory.

[0065] In another embodiment, the load analysis unit 121 can detect the resource status of the network device 100 based on the token issuance value of the token bucket. If the token issuance value of the token bucket is greater than a first preset threshold, the network device 100 is operating under high load, and it is difficult for the network device 100 to allocate additional resources to execute read requests. If the token issuance value of the token bucket is less than or equal to the first preset threshold, the network device 100 is operating under low load, and the network device 100 can allocate additional resources to execute read requests. The first preset threshold is the token issuance value.

[0066] A token bucket is a bucket based on the token bucket algorithm. The token bucket algorithm is one of the most commonly used algorithms in network traffic shaping and rate limiting. It controls the amount of data sent to the network and allows bursty data transmission. A fixed-size token bucket continuously generates tokens at a constant rate. If tokens in the bucket are not consumed, or if the rate of token consumption is less than the rate of token generation, the number of tokens in the bucket will continuously increase until it is full. Once the token bucket is full, tokens will overflow, ensuring that the total number of tokens in the bucket does not exceed its maximum capacity. The token issuance value refers to the rate at which tokens are consumed. A larger token issuance value results in a higher load on network device 100. Conversely, a smaller token issuance value results in a lower load on network device 100.

[0067] The performance statistics unit 122 is used to detect the volume where the data read by each read request is located, and to queue each read request in the corresponding volume according to the time order. In this embodiment, the performance statistics unit 122 can receive the detection results of the load analysis unit 121, and when the network device 100 is running under high load, it queues each read request in the respective volume of the network device 100 according to the time order.

[0068] Based on the detection results of the load analysis unit 121 and the sorting results of the performance statistics unit 122, the I / O scheduling unit 123 allows read requests corresponding to high-priority volumes to access data in local memory, read requests corresponding to medium-priority volumes to access data in the memory of other high-performance network devices, and read requests corresponding to low-priority volumes to access data in the memory of other low-performance network devices. The network device includes multiple volumes. These multiple volumes are high-priority, medium-priority, and low-priority volumes. In one embodiment, the network device includes N volumes. The volumes corresponding to the first i priorities are defined as high-priority volumes. The volumes corresponding to the (i+1)th to jth priorities are defined as medium-priority volumes. The volumes corresponding to the (j+1)th to Nth priorities are defined as low-priority volumes. N is a positive integer greater than 3. i and j are both positive integers greater than 0.

[0069] In this embodiment, when the HCI network device 100 detects that the resources it is running read requests for are relatively large, the number of read requests is also relatively large. The HCI network device 100 can queue multiple read requests according to the volume accessed and the time, allowing each read request to be queued in the order of its request time across different volumes within the network device 100. The HCI network device 100 prioritizes read requests corresponding to high-priority volumes, allowing them to access data in the local storage first, ensuring a stable high input / output operations per second (IOPS) and low latency, thus meeting user requirements. The HCI network device 100 also allows read requests corresponding to medium- and low-priority volumes to access data in the storage of other network devices, leveraging the capabilities of a distributed cluster for processing, resulting in relatively stable performance.

[0070] In the following description of the protection scheme of this application, HCI includes a first network device, a second network device, and a third network device. The first network device includes multiple volumes. Each volume can be divided into high-priority, medium-priority, and low-priority volumes according to priority identifiers. Data backups stored in medium-priority volumes are stored in the second network device. Data backups stored in low-priority volumes are stored in the third network device. It should be noted that the second network device is only an example, and the network devices used for storing data backups in medium-priority volumes can be multiple network devices. The third network device is also only an example, and the network devices used for storing data backups in low-priority volumes can be multiple network devices.

[0071] Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 4 As shown, this method is executed by the processor 120 of the first network device 100, and the specific implementation process is as follows:

[0072] Step S401: The first network device 100 receives a read request.

[0073] When the first network device 100 receives a read request, it can also receive a write request. The read request can originate from a host, other network devices, or other senders. The read request is used to read data stored in the memory of the first network device 100.

[0074] Step S402: The first network device 100 detects whether the resources used by the first network device 100 to process received read requests are greater than a first preset threshold. If the resources used by the first network device 100 are greater than the first preset threshold, step S403 is executed. If the resources used by the first network device 100 are less than or equal to the first preset threshold, step S404 is executed.

[0075] When the first network device 100 receives a read request, it can detect the resources available for processing the received read request. The first network device 100 can detect resources in ways such as checking the CPU's computing resource utilization rate, checking the amount of data cached in the running memory, or checking the number of tokens issued in the token bucket.

[0076] In this embodiment, the first network device 100 can detect its resource usage based on the CPU's computing resource utilization ratio. If the CPU's computing resource utilization ratio is greater than a first preset threshold, the first network device 100 is operating under high load, making it difficult for it to allocate additional resources to execute read requests. If the CPU's computing resource utilization ratio is less than or equal to the first preset threshold, the first network device 100 is operating under low load, allowing it to allocate additional resources to execute read requests. The first preset threshold is the CPU's computing resource utilization ratio. The CPU's computing resource utilization ratio refers to the I / O address resource utilization ratio, which is the ratio of used I / O address resources to all I / O address resources.

[0077] In other embodiments, the first network device 100 can detect its resource status based on the amount of data cached in its running memory. If the amount of data cached in the running memory is greater than a first preset threshold, the first network device 100 is operating under high load, and it is difficult for the first network device 100 to call additional resources to execute read requests. If the amount of data cached in the running memory is less than or equal to the first preset threshold, the first network device 100 is operating under low load, and the first network device 100 can call additional resources to execute read requests. The first preset threshold is the amount of data cached in the running memory.

[0078] In another embodiment, the first network device 100 can detect its resource status by the amount of tokens issued in the token bucket. If the token issuance value in the token bucket is greater than a first preset threshold, the first network device 100 is operating under high load, and it is difficult for the first network device 100 to allocate additional resources to execute read requests. If the token issuance value in the token bucket is less than or equal to the first preset threshold, the first network device 100 is operating under low load, and the first network device 100 can allocate additional resources to execute read requests. The first preset threshold is the token issuance value. Optionally, the token issuance value can be 80% of the maximum token consumption rate.

[0079] In step S403, the first network device 100 detects whether the data to be read by the read request is in a high-priority volume. If the data to be read by the read request is in a high-priority volume, proceed to step S404. If the data to be read by the read request is not in a high-priority volume, proceed to step S405.

[0080] In step S404, the first network device 100 accesses data in its local storage based on a read request.

[0081] The first network device 100 is currently operating at low load, meaning it is not operating at full capacity and can handle additional read requests. The first network device 100 can execute read requests and directly access data in its local memory. That is, the first network device 100 accesses data in its own various memory locations.

[0082] When the first network device 100 is currently operating under high load, it may be difficult for it to allocate additional resources to execute read requests. The first network device 100 can detect whether the data accessed by the read request is on a high-priority volume based on the data being accessed. If the first network device 100 determines that the data accessed by the read request is on a high-priority volume, it can execute the read request and directly access the data in its local storage. The first network device 100 can prioritize processing data on high-priority volumes, striving to ensure stable high IOPS and low latency to meet user requirements.

[0083] Step S405: The first network device 100 detects whether the data to be read by the read request is in a medium-priority volume. If the data to be read by the read request is in a medium-priority volume, proceed to step S406. If the data to be read by the read request is not in a medium-priority volume, proceed to step S407.

[0084] In step S406, the first network device 100 accesses data in the memory of the second network device based on a read request.

[0085] In step S407, the first network device 100 accesses data in the memory of the third network device based on a read request.

[0086] In this application, the second network device is another network device that backs up the data stored on the medium-priority volume of the first network device 100. The third network device is another network device that backs up the data stored on the low-priority volume of the first network device 100. The second network device is communicatively connected to the first network device, and the third network device is communicatively connected to the first network device.

[0087] When the first network device 100 determines that the data accessed by the read request is not located in a high-priority volume, the first network device 100 can execute the read request and access the data stored in the memory of other network devices. The first network device 100 can utilize the capabilities of a distributed cluster to allow read requests corresponding to medium- and low-priority volumes to access other network devices, thus ensuring relatively stable performance of the read requests from network device 100.

[0088] In this embodiment, when network device 100 determines that a read request corresponding to a low-priority volume is accessing another network device, network device 100 can obtain the load operating status of the other network devices and, based on the load operating status, classify the other network devices into high-performance other network devices and low-performance other network devices. High-performance other network devices can be network devices operating under low load. Low-performance other network devices can be network devices operating under high load.

[0089] Network device 100 detects whether the resources requested by other network devices for running read requests are less than a second preset threshold. If network device 100 detects that the resources requested by other network devices for running read requests are less than the second preset threshold, that network device is designated as the second network device. If network device 100 detects that the resources requested by other network devices for running read requests are greater than or equal to the second preset threshold, that network device is designated as the third network device.

[0090] If network device 100 determines that the data requested for the read request is in a medium-priority volume, network device 100 may execute the read request and access the data in the memory of the second network device. If network device 100 determines that the data requested for the read request is not in a medium-priority volume, network device 100 may execute the read request and access the data in the memory of the third network device.

[0091] In other embodiments, when the first network device 100 determines that a read request corresponding to a low-priority volume is accessing another network device, the first network device 100 can classify the other network devices into second and third network devices based on the priority of the volume accessed by the read request. Optionally, the second set threshold is less than the first set threshold to avoid other network devices having to execute read requests from the first network device when they only run read requests for high-priority volumes, thus avoiding resource strain on other network devices.

[0092] A first network device 100 detects whether the resources of a read request from its own device are less than a second preset threshold. If the first network device 100 detects that the resources of a read request from its own device are greater than the second preset threshold, the first network device 100 sends a read request to a second network to read data from a medium-priority volume backup of the second network device. If the first network device 100 detects that the resources of a read request from its own device are equal to or less than the second preset threshold, the first network device 100 sends a read request to a third network to read data from a low-priority volume backup of the third network device.

[0093] It is conceivable that the second network device may store data from medium-priority volume backups of the first network device 100, or it may store data from low-priority volume backups of the first network device 100. When the first network device 100 retrieves low-priority volume backup data stored by the second network device based on a read request, the second network device can be a third network device. Similarly, the third network device can be the second network device.

[0094] In this embodiment, the second set threshold is greater than the maximum value of the resources required for a normal read request by the first network device 100. When the resources required for a normal read request by the first network device 100 are greater than the maximum value of the resources required for a normal read request by the first network device 100, but less than the second set threshold, the first network device 100 can access the third network device based on read requests for accessing lower-priority volumes. This avoids lower-priority volumes from competing for network device resources and avoids latency in reading data from lower-priority volumes.

[0095] When the resources of the read requests of the first network device 100 are greater than the second set threshold and less than the first set threshold, the first network device 100 can access the third network device based on the read requests for accessing low-priority volumes, and access the second network device based on the read requests for accessing medium-priority volumes. This can avoid contention for network device resources between medium and low-priority volumes and avoid the latency of read requests reading data from medium and low-priority volumes.

[0096] In this embodiment, when the first network device 100 determines that the operating resources are relatively large, it can detect whether the data to be read by the read request is in a high-priority volume. If the data to be read by the read request is in a high-priority volume, the read request is queued in the high-priority volume and reads the data from the local memory, which can ensure stable high IOPS and low latency for the network device 100. If the data to be read by the read request is not in a high-priority volume, the read request can directly read the data from the memory of other network devices, avoiding contention for network device resources between low-priority and high-priority volumes, and avoiding latency in reading data from low-priority volumes.

[0097] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 5 As shown, the computing device 500 includes a bus 510, a processor 520, a memory 530, and a communication interface 540. The processor 520, memory 530, and communication interface 540 communicate with each other via the bus 510. The computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 500.

[0098] Bus 510 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 510 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus 510 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 510 may include a path for transmitting information between various components of the computing device 500 (e.g., processor 520, memory 530, and communication interface 540).

[0099] Processor 520 may include any one or more processors such as CPU, MCU, graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0100] Memory 530 may include volatile memory, such as RAM. Processor 520 may also include non-volatile memory, such as ROM, flash memory, HDD, or SSD.

[0101] The memory 530 stores executable program code, which the processor 520 executes to implement the functions of the aforementioned load analysis unit 121, performance statistics unit 122, and I / O scheduling unit 123, thereby realizing the data processing method. In other words, the memory 530 stores instructions for executing the data processing method.

[0102] Alternatively, the memory 530 may store executable code, which the processor 520 executes to implement the functions of the aforementioned computing device, thereby implementing the data processing method. That is, the memory 530 may store instructions for executing the data processing method.

[0103] The communication interface 540 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 500 and other devices or communication networks.

[0104] This application also provides a data center. The data center includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a laptop computer, a smartphone, a switch, or other terminal device.

[0105] like Figure 6 As shown, the data center includes at least one computing device 500. The memory 530 of one or more computing devices 500 in the data center may store the same instructions for performing data processing methods.

[0106] In some possible implementations, the memory 530 of one or more computing devices 500 in the data center may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 500 can jointly execute the instructions for executing data processing methods.

[0107] It should be noted that the memory 530 in different computing devices 500 in the data center can store different instructions, which are used to execute some functions of the computing devices. That is, the instructions stored in the memory 530 in different computing devices 500 can realize the functions of one or more modules in the load analysis unit 121, performance statistics unit 122, and I / O scheduling unit 123.

[0108] In some possible implementations, one or more computing devices 500 in the data center can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 7 One possible implementation is shown. For example... Figure 7 As shown, computing devices 500A and 500B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 530 in computing device 500A stores instructions for performing the functions of the load analysis unit 121. Meanwhile, the memory 530 in computing device 500B stores instructions for performing the functions of the performance statistics unit 122 and the I / O scheduling unit 123.

[0109] Figure 7 The connection method between the data centers shown can be based on the large amount of data storage required by the data processing method provided in this application. Therefore, it is considered that the functions implemented by the load analysis unit 121, the performance statistics unit 122 and the I / O scheduling unit 123 are performed by the computing device 500B.

[0110] It should be understood that Figure 7 The functions of computing device 500A shown can also be performed by multiple computing devices 500. Similarly, the functions of computing device 500B can also be performed by multiple computing devices 500.

[0111] This application also provides another data center. The connection relationships between the computing devices in this data center can be similarly referenced. Figure 6 and Figure 7The connection method of the data center is different in that the memory 530 of one or more computing devices 500 in the data center can store the same instructions for executing data processing methods.

[0112] In some possible implementations, the memory 530 of one or more computing devices 500 in the data center may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 500 can jointly execute the instructions for executing data processing methods.

[0113] It should be noted that the memory 530 in different computing devices 500 within the data center can store different instructions for executing some functions of the data center. That is, the instructions stored in the memory 530 of different computing devices 500 can implement the functions of one or more devices within the computing device.

[0114] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data processing method.

[0115] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data processing method.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, The method is performed by a first network device, and the method includes: Receive read request; Detect whether the resources being used by the first network device exceed a first preset threshold; In response to the fact that the resources being operated by the first network device exceed the first set threshold, it is detected whether the data to be read by the read request is located in a volume of a first type; wherein, the first type of volume is a high-priority volume, the volume is composed of multiple memories, or, a memory includes multiple volumes, and the volume is one of the multiple volumes; The data read in response to the read request is located in the first type of volume, and the read requests in the first type of volume are arranged in chronological order. The data on the local side of the first network device is accessed based on the read request according to the arrangement order. In response to the fact that the data to be read by the read request is not located in the first type of volume, it is detected whether the data to be read by the read request is located in the second type of volume, wherein the second type of volume has a lower priority than the first type of volume; In response to the read request indicating that the data to be read is located within the second type of volume, the read request is sent to a second network device, the second network device being a network device whose operating resources are less than a second set threshold; wherein, the second network device is communicatively connected to the first network device, or, In response to the fact that the data to be read by the read request is not located in the volume of the second type, the read request is sent to a third network device, the third network device being a network device with operating resources greater than or equal to a second set threshold, wherein the third network device is communicatively connected to the first network device.

2. The method according to claim 1, characterized in that, The step of detecting whether the resources being operated by the first network device exceed a first preset threshold specifically includes: The system detects whether the CPU's computing resource utilization ratio of the first network device is greater than the first set threshold, where the first set threshold is the CPU's computing resource utilization ratio.

3. The method according to claim 1, characterized in that, The step of detecting whether the resources being operated by the first network device exceed a first preset threshold specifically includes: The system detects whether the amount of data cached in the running memory of the first network device is greater than the first set threshold, where the first set threshold is the amount of data cached in the running memory.

4. The method according to claim 1, characterized in that, The step of detecting whether the resources being operated by the first network device exceed a first preset threshold specifically includes: The system detects whether the token issuance value of the token bucket of the first network device is greater than the first set threshold, where the first set threshold is the token issuance value.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: In response to the first network device operating resources being less than or equal to the first set threshold, the local data of the first network device is accessed based on the read request.

6. A first network device, characterized in that, include: Transceiver and processor The transceiver is used to receive read requests; The processor is used to detect whether the resources being operated by the first network device exceed a first preset threshold. In response to the fact that the resources being operated by the first network device exceed the first set threshold, it is detected whether the data to be read by the read request is located in a volume of a first type; wherein, the first type of volume is a high-priority volume, the volume is composed of multiple memories, or, a memory includes multiple volumes, and the volume is one of the multiple volumes; In response to the read request being located within a volume of the first type, the read request is queued in chronological order within the first type of volume, and the local data of the first network device is accessed based on the queued order of the read request. In response to the fact that the data to be read by the read request is not located in the first type of volume, it is detected whether the data to be read by the read request is located in the second type of volume, wherein the second type of volume has a lower priority than the first type of volume; In response to the read request indicating that the data to be read is located within a volume of the second type, the read request is sent to a second network device, the second network device being a network device with operating resources less than a second set threshold, wherein the second network device is communicatively connected to the first network device; or, In response to the fact that the data to be read by the read request is not located in the volume of the second type, the read request is sent to a third network device, the third network device being a network device with operating resources greater than or equal to a second set threshold, wherein the third network device is communicatively connected to the first network device.

7. The first network device according to claim 6, characterized in that, The processor is specifically used to detect whether the CPU's computing resource utilization ratio of the first network device is greater than the first set threshold, where the first set threshold is the CPU's computing resource utilization ratio. or Detect whether the amount of data cached in the running memory of the first network device is greater than the first set threshold, where the first set threshold is the amount of data cached in the running memory; or The system detects whether the token issuance value of the token bucket of the first network device is greater than the first set threshold, where the first set threshold is the token issuance value.

8. The network device according to claim 6 or 7, characterized in that, The processor is configured to access local data of the first network device based on the read request in response to the first network device operating resources being less than or equal to the first preset threshold.

Images