Request allocation method and device in distributed storage system, equipment and medium

Abstract

The embodiment of the invention provides a request allocation method and device in a distributed storage system, equipment and a medium, which are applied to the distributed storage system, and the distributed storage system comprises at least one node. Obtaining at least one workload request sent by a preset client; obtaining at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is a capability index for executing the workload request by the node; for any workload request, determining the weight of the evaluation index based on the demand intensity of the workload request for the capability corresponding to the evaluation index; determining an evaluation value of the node based on the weight and the index value of the evaluation index; according to the method, the target node of the workload request is determined based on the evaluation value of the node, and the target node is allocated to the workload request based on the capabilities of different heterogeneous nodes and the demand intensity of the workload request for the capabilities, so that the problem of improper node allocation in the distributed storage system is avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a request allocation method in a distributed storage system, a request allocation device in a distributed storage system, an electronic device and a computer-readable medium. Background Art

[0002] With the development of cloud computing technology, distributed storage systems have become an important part of cloud computing infrastructure. Distributed storage systems usually include multiple nodes. After receiving workload requests from clients, distributed storage systems usually use load balancing methods to evenly distribute workload requests to multiple nodes to maximize resource utilization and improve system performance and reliability.

[0003] However, distributed storage systems are also heterogeneous, and they usually contain nodes of different models, performance, capacity, and lifespan. In the process of using load balancing methods to evenly distribute workload requests to different nodes, distributed storage systems do not take into account the heterogeneity of distributed storage systems, that is, the differences between different nodes in hardware configuration, performance, capacity, etc., which may lead to the following consequences: 1) The load balancing method does not distribute workload requests according to the actual capabilities and characteristics of each node, resulting in improper node allocation in the distributed storage system. Some nodes may not be able to fully utilize their performance, while other nodes may be overloaded, resulting in low overall resource utilization efficiency and operating performance of the distributed storage system. 2) If some nodes fail due to overload, or other reasons cause them to be unable to process workload requests in a timely manner, it may cause instability in the distributed storage system, and even cause storage data loss or service interruption, thereby affecting the security of the system. Summary of the invention

[0004] Embodiments of the present invention provide a request allocation method, device, electronic device and computer-readable storage medium in a distributed storage system to solve the problem that the distributed storage system does not consider the heterogeneity of the distributed storage system when using a load balancing method to evenly distribute workload requests to different nodes, resulting in low overall resource utilization efficiency and operating performance of the distributed storage system and affecting system security.

[0005] An embodiment of the present invention discloses a request allocation method in a distributed storage system, which is applied to a distributed storage system, wherein the distributed storage system includes at least one node; the method includes:

[0006] Obtain at least one workload request sent by a preset client;

[0007] Acquire at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is an index of the ability of the node to execute the workload request;

[0008] For any of the workload requests, determining a weight of the evaluation indicator based on the intensity of demand of the workload request for the capability corresponding to the evaluation indicator;

[0009] Determining the evaluation value of the node based on the weight of the evaluation index and the index value;

[0010] Based on the evaluation value of the node, a target node for the workload request is determined.

[0011] Optionally, the method comprises:

[0012] Obtaining a training workload request from the client;

[0013] obtaining at least one training request characteristic of the training workload request;

[0014] The preset machine learning model is trained using the training workload request and the training request features to obtain a feature extraction model.

[0015] Optionally, the nodes include at least one or more of computing nodes, transmission nodes and storage nodes; the evaluation indicators of the computing nodes include at least one or more of computing resource indicators, memory resource indicators and first network bandwidth indicators; the evaluation indicators of the transmission nodes include at least response delay indicators and / or second network bandwidth indicators; the evaluation indicators of the storage nodes include at least solid-state hard disk indicators and / or third network bandwidth indicators.

[0016] Optionally, the workload request at least includes a computing request; and before the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator, the step includes:

[0017] Inputting the computing request into the feature extraction model to obtain request features of the computing request;

[0018] Classify the computing requests based on the request characteristics of the computing requests to obtain at least one type of the computing requests;

[0019] Based on the type of the computing request, a demand intensity of the capability of the computing request corresponding to the evaluation index of the computing node is determined.

[0020] Optionally, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0021] Based on the demand intensity of the capability corresponding to the evaluation index of the computing node for the computing request, the weights of the computing resource index, the memory resource index and the first network bandwidth index are determined respectively.

[0022] Optionally, the step of determining the evaluation value of the node based on the weight of the evaluation index and the index value includes:

[0023] Based on the weight and index value of the computing resource index, the weight and index value of the memory resource index, and the weight and index value of the first network bandwidth index, an evaluation value of the computing node is determined.

[0024] Optionally, the workload request further includes a transmission request; and before the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator, the step includes:

[0025] Inputting the transmission request into the feature extraction model to obtain request features of the transmission request;

[0026] Classify the transmission request based on the request feature of the transmission request to obtain at least one type of the transmission request;

[0027] Based on the type of the transmission request, a requirement intensity of the transmission request for a capability corresponding to the evaluation indicator of the transmission node is determined.

[0028] Optionally, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0029] Based on the demand intensity of the transmission request for the capability corresponding to the evaluation index of the transmission node, the weights of the response delay index and the second network bandwidth index are determined respectively.

[0030] Optionally, the step of determining the evaluation value of the node based on the weight of the evaluation index and the index value includes:

[0031] Based on the weight and index value of the response delay index and the weight and index value of the second network bandwidth index, an evaluation value of the transmission node is determined.

[0032] Optionally, the workload request further includes a storage request; and before the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator, the step includes:

[0033] Inputting the storage request into the feature extraction model to obtain request features of the storage request;

[0034] Classify the storage request based on the request feature of the storage request to obtain at least one type of the storage request;

[0035] Based on the type of the storage request, the demand intensity of the storage request for the capability corresponding to the evaluation index of the storage node is determined.

[0036] Optionally, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0037] Based on the demand intensity of the storage request for the capacity corresponding to the evaluation index of the storage node, the weights of the solid-state hard disk index and the third network bandwidth index are determined respectively.

[0038] Optionally, the step of determining the evaluation value of the node based on the weight of the evaluation index and the index value includes:

[0039] Based on the weight and index value of the solid state drive index and the weight and index value of the third network bandwidth index, an evaluation value of the storage node is determined.

[0040] Optionally, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0041] Obtaining an importance value of the computing request to the client and time requirement information of the client for the computing request;

[0042] Prioritizing the computing requests based on the importance values ​​and the time requirement information to obtain priority information of the computing requests;

[0043] Based on the priority sorting information, a weight of an evaluation indicator of the computing request is determined.

[0044] Optionally, at least one offline workload request is stored in the distributed storage system; the method includes:

[0045] Obtaining a sending frequency of the client for the offline workload request;

[0046] Based on the sending frequency of the client for the offline workload request, taking the offline workload request whose sending frequency exceeds a preset frequency threshold in the at least one offline workload request as a workload request to be processed;

[0047] The workload request to be processed is input into the feature extraction model to obtain offline request features of the workload request to be processed.

[0048] Optionally, the step of classifying the computing requests based on the request characteristics of the computing requests to obtain at least one type of the computing requests includes:

[0049] Based on the offline request feature and the request feature of the computing request, the computing request is classified to obtain the type of the computing request.

[0050] Optionally, the method comprises:

[0051] executing the workload request using the target node;

[0052] Based on the execution status of the target node on the workload request, an indicator value corresponding to the evaluation indicator of the target node is updated.

[0053] The embodiment of the present invention further discloses a request allocation device in a distributed storage system, which is applied to the distributed storage system. The distributed storage system includes at least one node. The device includes:

[0054] A workload request acquisition module, used to acquire at least one workload request sent by a preset client;

[0055] An evaluation index acquisition module, used to acquire at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is an indicator of the ability of the node to execute the workload request;

[0056] A weight determination module, configured to determine, for any workload request, a weight of the evaluation indicator based on the intensity of demand for the capability corresponding to the evaluation indicator by the workload request;

[0057] An evaluation value determination module, used to determine the evaluation value of the node based on the weight of the evaluation index and the index value;

[0058] The target node determination module is used to determine the target node of the workload request based on the evaluation value of the node.

[0059] Optionally, the device comprises:

[0060] A training workload request acquisition module, used to acquire a training workload request from the client;

[0061] a training request feature acquisition module, configured to acquire at least one training request feature of the training workload request;

[0062] The feature extraction model obtaining module is used to train a preset machine learning model using the training workload request and the training request feature to obtain a feature extraction model.

[0063] Optionally, the nodes include at least one or more of computing nodes, transmission nodes and storage nodes; the evaluation indicators of the computing nodes include at least one or more of computing resource indicators, memory resource indicators and first network bandwidth indicators; the evaluation indicators of the transmission nodes include at least response delay indicators and / or second network bandwidth indicators; the evaluation indicators of the storage nodes include at least solid-state hard disk indicators and / or third network bandwidth indicators.

[0064] Optionally, the workload request at least includes a computing request; and the device includes:

[0065] A first request feature obtaining module, used for inputting the computing request into the feature extraction model to obtain the request feature of the computing request;

[0066] A first category classification module, configured to classify the computing request based on a request feature of the computing request to obtain at least one type of the computing request;

[0067] A first demand intensity determination module is configured to determine, based on the type of the computing request, the demand intensity of the capability corresponding to the evaluation index of the computing node for the computing request.

[0068] Optionally, the weight determination module includes:

[0069] The first weight determination submodule is used to determine the weights of the computing resource indicator, the memory resource indicator and the first network bandwidth indicator respectively based on the demand intensity of the capability corresponding to the evaluation indicator of the computing node for the computing request.

[0070] Optionally, the evaluation value determination module includes:

[0071] The first evaluation value determination submodule is used to determine the evaluation value of the computing node based on the weight and index value of the computing resource index, the weight and index value of the memory resource index, and the weight and index value of the first network bandwidth index.

[0072] Optionally, the workload request further includes a transmission request; and the device includes:

[0073] A second request feature obtaining module, used for inputting the transmission request into the feature extraction model to obtain the request feature of the transmission request;

[0074] A second classification module, configured to classify the transmission request based on the request characteristics of the transmission request to obtain at least one type of the transmission request;

[0075] A second demand strength determination module is configured to determine, based on the type of the transmission request, the demand strength of the capability corresponding to the evaluation index of the transmission node for the transmission request.

[0076] Optionally, the weight determination module includes:

[0077] The second weight determination submodule is used to determine the weights of the response delay indicator and the second network bandwidth indicator respectively based on the demand intensity of the capability corresponding to the evaluation indicator of the transmission node for the transmission request.

[0078] Optionally, the evaluation value determination module includes:

[0079] The second evaluation value determination submodule is used to determine the evaluation value of the transmission node based on the weight and index value of the response delay index and the weight and index value of the second network bandwidth index.

[0080] Optionally, the workload request further includes a storage request; and the apparatus includes:

[0081] A third request feature obtaining module, used for inputting the storage request into the feature extraction model to obtain a request feature of the storage request;

[0082] a third category classification module, configured to classify the storage request based on a request feature of the storage request to obtain at least one type of the storage request;

[0083] A third demand strength determination module is configured to determine, based on the type of the storage request, the demand strength of the capacity of the storage request corresponding to the evaluation index of the storage node.

[0084] Optionally, the weight determination module includes:

[0085] The third weight determination submodule is used to determine the weights of the solid-state drive index and the third network bandwidth index respectively based on the demand intensity of the capacity corresponding to the evaluation index of the storage node for the storage request.

[0086] Optionally, the evaluation value determination module includes:

[0087] The third evaluation value determination submodule is used to determine the evaluation value of the storage node based on the weight and index value of the solid state drive index and the weight and index value of the third network bandwidth index.

[0088] Optionally, the weight determination module includes:

[0089] A time requirement information acquisition submodule, used to acquire the importance value of the computing request to the client, and the time requirement information of the client for the computing request;

[0090] A priority sorting information obtaining submodule, used to prioritize the computing requests based on the importance value and the time requirement information, and obtain priority sorting information of the computing requests;

[0091] The fourth weight determination submodule is used to determine the weight of the evaluation indicator of the calculation request based on the priority sorting information.

[0092] Optionally, at least one offline workload request is stored in the distributed storage system; and the device includes:

[0093] A sending frequency acquisition module, used to acquire the sending frequency of the client for the offline workload request;

[0094] The module for processing workload requests is used to take, based on the sending frequency of the client for the offline workload requests, the offline workload request whose sending frequency exceeds a preset frequency threshold in the at least one offline workload request as the workload request to be processed;

[0095] The offline request feature obtaining module is used to input the workload request to be processed into the feature extraction model to obtain the offline request feature of the workload request to be processed.

[0096] Optionally, the first category classification module includes:

[0097] The type obtaining submodule is used to classify the computing request into categories based on the offline request characteristics and the request characteristics of the computing request to obtain the type of the computing request.

[0098] Optionally, the device comprises:

[0099] An execution module, configured to execute the workload request using the target node;

[0100] An updating module is used to update the indicator value corresponding to the evaluation indicator of the target node based on the execution status of the workload request by the target node.

[0101] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0102] The memory is used to store computer programs;

[0103] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0104] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0105] The embodiment of the present invention further discloses a computer program product, including a computer program, wherein the computer program implements the method described in the embodiment of the present invention when executed by a processor.

[0106] The embodiments of the present invention include the following advantages:

[0107] In an embodiment of the present invention, a distributed storage system includes at least one node. Obtain at least one workload request sent by a preset client; obtain at least one evaluation index of the node, and the index value corresponding to the evaluation index; the evaluation index is an index of the ability of the node to execute the workload request; for any workload request, determine the weight of the evaluation index based on the workload request for the ability corresponding to the evaluation index; determine the evaluation value of the node based on the weight and index value of the evaluation index; determine the target node of the workload request based on the evaluation value of the node. In an embodiment of the present invention, when allocating nodes for workload requests, the heterogeneity of nodes in the distributed storage system is taken into account, and the target node is allocated to the workload request based on the capabilities of different heterogeneous nodes and the intensity of the workload request for the capabilities, thereby avoiding improper node allocation in the distributed storage system, some nodes may fail to fully exert their performance, and other nodes may be overloaded, resulting in the overall resource utilization efficiency and low operating performance of the distributed storage system, and further avoiding some nodes from failing due to overload, or other reasons that cause them to fail to process workload requests in time, which may cause instability of the distributed storage system, and even cause storage data loss or service interruption, thereby affecting the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 is a flowchart of a method for allocating requests in a distributed storage system provided in an embodiment of the present invention;

[0109] Figure 2 is a schematic diagram of a communication connection between a client and a distributed storage system provided in an embodiment of the present invention;

[0110] Figure 3 is a schematic diagram of a multi-level trust mechanism provided in an embodiment of the present invention;

[0111] Figure 4 is a schematic diagram of dividing computing requests into at least one category provided in an embodiment of the present invention;

[0112] Figure 5 is a schematic diagram of dividing a transmission request into at least one category provided in an embodiment of the present invention;

[0113] Figure 6 is a schematic diagram of dividing storage requests into at least one category provided in an embodiment of the present invention;

[0114] Figure 7 is a schematic diagram of extracting offline request features using a feature extraction model provided in an embodiment of the present invention;

[0115] Figure 8 It is a schematic diagram of analyzing a workload request based on offline request characteristics and request characteristics of a workload request provided in an embodiment of the present invention;

[0116] Fig. 9 is a schematic diagram of an indicator value corresponding to an updating evaluation indicator provided in an embodiment of the present invention;

[0117] Fig.10 is a flowchart of steps of another method for allocating requests in a distributed storage system provided in an embodiment of the present invention;

[0118] Fig.11 is a structural block diagram of a request allocation device in a distributed storage system provided in an embodiment of the present invention;

[0119] Fig.12 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0120] Fig.13 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0121] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0122] To facilitate understanding of the technical solutions and technical effects of the embodiments of the present invention, the prior art of the present invention is briefly described below.

[0123] With the development of cloud computing technology, distributed storage systems have become an important part of cloud computing infrastructure. Distributed storage systems usually include a computing node layer, a network transmission layer, and a data storage layer. Among them, the computing node layer includes multiple computing nodes, the network transmission layer includes multiple transmission nodes, and the data storage layer includes multiple storage nodes. It should be noted that these nodes can be physical nodes or virtual nodes; these nodes are also called servers, resources, and devices.

[0124] After receiving workload requests from clients, distributed storage systems usually distribute workload requests evenly to multiple nodes according to the workload request processing flow to maximize resource utilization and improve system performance and reliability. Workload requests usually include computing requests, transmission requests, and storage requests. Distributed storage systems can distribute computing requests to computing nodes, transmission requests to transmission nodes, and storage requests to storage nodes. Among them, the method of evenly distributing workload requests to multiple nodes can be called a load balancing method, which is widely used in various scenarios such as computing, networking, and storage.

[0125] Among them, computing requests include foreground load requests and background load requests. Foreground load requests refer to workload requests that users interact with directly, which usually require immediate response and processing, and have the characteristics of high priority, low latency requirements, and interactivity; for example, when a user is using an application, clicking buttons, entering data, browsing web pages, and other operations will generate foreground load requests. Background load requests refer to workload requests that are executed in the background, which usually do not require immediate response to user requests, and have the characteristics of low priority, high throughput requirements, and non-interactivity; for example, operations such as erasure data recovery, data backup, system updates, and file downloads will generate background load requests.

[0126] Distributed storage systems provide cloud computing with high availability, scalability, and fault tolerance by storing data in multiple nodes. They are widely used in cloud computing, big data processing, and other fields. Their application scenarios include data backup and recovery, big data analysis, and content distribution networks (CDNs). Their technical implementation relies on object storage (such as Amazon S3) and distributed file systems (such as HDFS).

[0127] Distributed storage systems are also heterogeneous. They usually contain nodes of different models, performance, capacity, and lifespans. For example, computing nodes, network transmission nodes, and storage nodes often include different types of storage devices, hardware platforms, network protocols, and software components. They can include different types of CPUs (Central Processing Units), SSDs (Solid State Drives), memories, and network cards, providing cloud computing with higher performance, reliability, and flexibility. In order to respond to and execute workload requests of different granularity and characteristics, distributed storage systems need to select appropriate nodes from these heterogeneous nodes.

[0128] In the process of using load balancing methods to evenly distribute workload requests to different nodes, the distributed storage system does not consider the heterogeneity of the distributed storage system, that is, the differences between different nodes in hardware configuration, performance, capacity, etc., which may lead to the following consequences: 1) The load balancing method does not distribute workload requests according to the actual capabilities and characteristics of each node, resulting in improper node allocation in the distributed storage system. Some nodes may not be able to fully utilize their performance, while other nodes may be overloaded, resulting in low overall resource utilization efficiency and operating performance of the distributed storage system. 2) If some nodes fail due to overload or other reasons cause them to fail to process workload requests in time, it may cause instability in the distributed storage system, and even cause storage data loss or service interruption, thereby affecting the security of the system. Specifically, background load requests may include large-scale erasure stripe data recovery requests. Among them, large-scale erasure refers to a coding method in which the coding parameter k (number of data blocks) of the erasure code is relatively large. This coding method can significantly reduce redundant data ((k+m) / k) while maintaining the same data reliability by increasing the value of k, thereby reducing storage costs. This request requires a large amount of system resources. If the distributed storage system only considers the uniform distribution of workload requests when allocating large-scale erasure stripe data recovery requests and foreground load requests that emphasize high priority, low latency requirements and interactivity, but does not consider the capacity, performance and other characteristics of the nodes, it may cause resource shortages on the nodes allocated to the foreground load requests, and the nodes allocated to large-scale erasure stripe data recovery requests may face the risk of overload. The large-scale erasure stripe data recovery requests occupy node resources, resulting in a significant impact on the performance of the foreground load requests.

[0129] In addition, the storage nodes in the distributed storage system include hard disks. The SMART (Self-Monitoring, Analysis and Reporting Technology) disk monitoring tool in the distributed storage system can provide detailed statistics on the status and performance of the hard disk to help users and system administrators monitor the health of the hard disk and predict possible failures. The detailed statistics provided by the SMART disk monitoring tool mainly include read and write error rates, erase counts, bad block counts, percentage of available reserved space, etc. However, the SMART disk monitoring tool only provides information about the hard disk itself, and cannot perform adaptive scheduling based on the security and capacity characteristics of the storage request to allocate appropriate storage nodes to the storage request, which may cause the distributed storage system to face the risk of downtime and data loss.

[0130] Reference Figure 1 , shows a flowchart of the steps of a request allocation method in a distributed storage system provided in an embodiment of the present invention, which is applied to a distributed storage system, wherein the distributed storage system includes at least one node; specifically, the following steps may be included:

[0131] Step 101, obtaining at least one workload request sent by a preset client;

[0132] In an embodiment of the present invention, the distributed storage system is connected to the client through a network communication. The client can send at least one workload request to the distributed storage system through the network. The workload request may include a computing request, a transmission request, and a storage request. The distributed storage system generally includes a computing node layer, a network transmission layer, and a data storage layer. Among them, the computing node layer includes multiple computing nodes, the network transmission layer includes multiple transmission nodes, and the data storage layer includes multiple storage nodes.

[0133] In the embodiment of the present invention, the distributed storage system can utilize computing nodes to execute computing requests, utilize transmission nodes to execute transmission requests, and utilize storage nodes to execute storage requests.

[0134] Reference Figure 2 , showing a schematic diagram of a communication connection between a client and a distributed storage system provided in an embodiment of the present invention.

[0135] Figure 2 The switches in the network are transmission nodes, and the data nodes are storage nodes. Figure 2There are q clients in total. The distributed storage system and the clients are connected via a network (such as Ethernet, Internet, LAN or WAN), and the computing nodes and storage nodes are connected via transmission nodes. In the distributed storage system, the computing node layer includes m computing nodes, the network transmission layer includes p transmission nodes, and the data storage layer includes n storage nodes.

[0136] Step 102, obtaining at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is an index of the ability of the node to execute the workload request;

[0137] In an embodiment of the present invention, the distributed storage system has heterogeneity, that is, the distributed storage system includes nodes with different models, performances, capacities, and lifespans. Therefore, when executing workload requests, the nodes in the distributed storage system will exhibit different capabilities based on their different characteristics. For the different capabilities of the nodes in the distributed storage system, at least one evaluation index of the node and the index value corresponding to the evaluation index can be obtained. Among them, the evaluation index is an index of the ability of the node to execute the workload request. The higher the index value of the evaluation index, the stronger the ability of the node to execute the workload request.

[0138] Step 103, for any of the workload requests, determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator;

[0139] In an embodiment of the present invention, when executing a workload request, a node in a distributed storage system will exhibit different capabilities based on its different characteristics. Different workload requests require different node capabilities and different strengths of the capabilities required. In an embodiment of the present invention, the weight of the evaluation index can be determined based on the strength of the workload request's requirements for the capabilities corresponding to the evaluation index.

[0140] Step 104, determining the evaluation value of the node based on the weight of the evaluation index and the index value;

[0141] In the embodiment of the present invention, when allocating a node for any workload request, the distributed storage system can determine the target node from multiple nodes of the distributed storage system. For any node, the distributed storage system can obtain the weight and index value of the evaluation index of the node, and then multiply the weight of each evaluation index of the node with the index value to obtain the multiplied result, and then add the multiplied results corresponding to all the evaluation indexes of the node to obtain the evaluation value of the node.

[0142] Step 105: Determine a target node for the workload request based on the evaluation value of the node.

[0143] In an embodiment of the present invention, when allocating a node for any workload request, the distributed storage system may determine a target node from multiple nodes of the distributed storage system. After obtaining the evaluation value of each node, the target node of the workload request may be determined based on the evaluation value of the node. Specifically, some nodes with the highest evaluation value or the evaluation value located upstream are used as the target nodes of the workload request.

[0144] In an embodiment of the present invention, a distributed storage system includes at least one node. Obtain at least one workload request sent by a preset client; obtain at least one evaluation index of the node, and the index value corresponding to the evaluation index; the evaluation index is an index of the ability of the node to execute the workload request; for any workload request, determine the weight of the evaluation index based on the workload request for the ability corresponding to the evaluation index; determine the evaluation value of the node based on the weight and index value of the evaluation index; determine the target node of the workload request based on the evaluation value of the node. In an embodiment of the present invention, when allocating nodes for workload requests, the heterogeneity of nodes in the distributed storage system is taken into account, and the target node is allocated to the workload request based on the capabilities of different heterogeneous nodes and the intensity of the workload request for the capabilities, thereby avoiding improper node allocation in the distributed storage system, some nodes may fail to fully exert their performance, and other nodes may be overloaded, resulting in the overall resource utilization efficiency and low operating performance of the distributed storage system, and further avoiding some nodes from failing due to overload, or other reasons that cause them to fail to process workload requests in time, which may cause instability of the distributed storage system, and even cause storage data loss or service interruption, thereby affecting the security of the system.

[0145] Furthermore, in any of the above embodiments, the method includes:

[0146] Sub-step S11, obtaining a training workload request from the client;

[0147] Sub-step S12, obtaining at least one training request feature of the training workload request;

[0148] Sub-step S13, using the training workload request and the training request features to train a preset machine learning model to obtain a feature extraction model.

[0149] In an embodiment of the present invention, the distributed storage system can obtain a training workload request from a client and obtain at least one training request feature of the training workload request. The training workload request and the training request feature are used to train a preset machine learning model to obtain a feature extraction model.

[0150] In an embodiment of the present invention, the feature extraction model can perform load feature extraction, that is, it can extract request features in workload requests. Load feature extraction refers to extracting useful features from data such as network traffic or system logs, which can help administrators or system monitoring tools better understand the operating status and performance bottlenecks of the system. Common load features include IOPS, throughput, latency, storage capacity utilization, load distribution, data access patterns, etc.

[0151] Furthermore, in any of the above embodiments, the nodes include at least one or more of computing nodes, transmission nodes and storage nodes; the evaluation indicators of the computing nodes include at least one or more of computing resource indicators, memory resource indicators and first network bandwidth indicators; the evaluation indicators of the transmission nodes include at least response delay indicators and / or second network bandwidth indicators; the evaluation indicators of the storage nodes include at least solid-state hard disk indicators and / or third network bandwidth indicators.

[0152] In an embodiment of the present invention, the node includes at least one or more of a computing node, a transmission node, and a storage node. The evaluation index of the computing node includes at least one or more of a computing resource index, a memory resource index, and a first network bandwidth index, the evaluation index of the transmission node includes at least a response delay index and / or a second network bandwidth index, and the evaluation index of the storage node includes at least a solid state drive index and / or a third network bandwidth index.

[0153] When allocating a computing node for any computing request, the distributed storage system can determine the target node from multiple computing nodes. For any computing node, the distributed storage system can obtain the weights and index values ​​of the computing resource index, memory resource index, and first network bandwidth index (i.e., CPU, memory, and network bandwidth index [c, s, n]) of the computing node respectively, and then calculate and define the three-dimensional trust metric of the computing node by normalizing the weights and index values ​​of all evaluation indicators of the computing node. Based on the three-dimensional trust metric of each computing node, the target node of the computing request can be determined.

[0154] When allocating a transmission node for any transmission request, the distributed storage system can determine the target node from multiple transmission nodes. For any transmission node, the distributed storage system can obtain the weights and index values ​​of the response delay index and the second network bandwidth index of the transmission node (i.e., the network bandwidth and response delay index [b, l] of each switch in the network transmission layer) respectively, and then calculate and define the two-dimensional trust metric of the transmission node through normalization for the weights and index values ​​of all evaluation indicators of the transmission node. Based on the two-dimensional trust metric of each transmission node, the target node of the transmission request can be determined.

[0155] When allocating a storage node for any storage request, the distributed storage system can determine the target node from multiple storage nodes. For any storage node, the distributed storage system can obtain the weights and index values ​​of the solid-state drive index and the third network bandwidth index of the storage node respectively, and then the weights and index values ​​of all evaluation indicators of the storage node can be normalized to calculate and define the two-dimensional trust metric of the storage node. Based on the two-dimensional trust metric of each storage node, the target node of the storage request can be determined. The solid-state drive index and the third network bandwidth index of the storage node specifically refer to the SSD-related information of each node in the data storage layer based on SMART statistical information, as well as indicators such as network bandwidth [Issd,nssd].

[0156] Reference Figure 3 , showing a schematic diagram of a multi-level trust mechanism provided in an embodiment of the present invention. The three-dimensional trust measurement of the computing node, the two-dimensional trust measurement of the transmission node, and the two-dimensional trust measurement of the storage node together constitute a multi-level trust mechanism. The multi-level trust mechanism refers to collecting relevant trust measurement information of each layer (such as storage capacity, network bandwidth, CPU and other computing resources, etc.) according to the processing flow of workload requests (such as computing node layer, network transmission layer, and data storage layer), and summing the trust measurement information by weight to obtain a trust value so that tasks at different levels can select appropriate response nodes or devices.

[0157] In an embodiment of the present invention, by constructing a multi-level trust mechanism, the status of the distributed storage system can be analyzed and observed from different task processing levels and multiple dimensions. According to the analysis results, the node allocation plan can be optimized to improve the performance, reliability and availability of the system.

[0158] In an embodiment of the present invention, in a distributed storage system, the present invention defines multi-dimensional trust metrics and constructs a multi-level trust mechanism by sensing the storage, network, computing and other resource information in the task processing flow of the system; in addition, the application load characteristics are extracted, and the demand for resources of different dimensions is estimated based on the application load characteristics; according to the requirements of the foreground application load and the background erasure code data recovery task, the weights and final trust values ​​of the metrics of each dimension in the trust mechanism are calculated, and according to the level of trust, appropriate response nodes and devices are selected for the foreground load and background load.

[0159] Further, in any of the above embodiments, the workload request at least includes a computing request; and before step 103, the steps include:

[0160] Sub-step S21, inputting the computing request into the feature extraction model to obtain request features of the computing request;

[0161] Sub-step S22, classifying the computing request into categories based on the request characteristics of the computing request to obtain at least one type of the computing request;

[0162] Sub-step S23, based on the type of the computing request, determining the demand intensity of the capability corresponding to the evaluation index of the computing node for the computing request.

[0163] In the embodiment of the present invention, the workload request at least includes a computing request. The distributed storage system can input the computing request into a feature extraction model to extract the request feature of the computing request. Specifically, the computing request includes a foreground workload request and a background workload request.

[0164] The distributed storage system can classify computing requests based on the request characteristics of the computing requests, obtain at least one type of computing requests, and determine the demand intensity of the capabilities corresponding to the evaluation indicators of the computing nodes for the computing requests based on the types of computing requests. Evaluation indicators are also called information metrics, and trust metrics refer to indicators or dimensions for evaluating the trust level between systems, entities or individuals. In the fields of computer science and information technology, trust metrics are generally used to evaluate trust or reliability relationships in aspects such as network security and distributed systems. In the present invention, it refers to indicators that constitute the trust of tasks in devices at various levels of processing flows, such as the available network bandwidth of the node, the available space of the node, or information such as disk space, disk performance, and IO load within the node.

[0165] Reference Figure 4 , shows a schematic diagram of dividing computing requests into at least one type provided in an embodiment of the present invention. The types of computing requests include computing intensive, IO intensive, and bandwidth intensive. Based on the type of computing request, the demand intensity of the capability corresponding to the evaluation index of the computing node of the computing request can be determined, and then the target node of the computing request can be determined.

[0166] Further, in any of the above embodiments, step 103 includes:

[0167] Sub-step S31, based on the demand intensity of the capability corresponding to the evaluation index of the computing node for the computing request, respectively determine the weights of the computing resource index, the memory resource index and the first network bandwidth index.

[0168] In an embodiment of the present invention, the distributed storage system can determine the weights of the computing resource indicator, the memory resource indicator, and the first network bandwidth indicator respectively based on the demand intensity of the capability corresponding to the evaluation indicator of the computing node for the computing request.

[0169] Further, in any of the above embodiments, step 104 includes:

[0170] Sub-step S41, determining the evaluation value of the computing node based on the weight and index value of the computing resource index, the weight and index value of the memory resource index, and the weight and index value of the first network bandwidth index.

[0171] In an embodiment of the present invention, the distributed storage system can obtain the weights and index values ​​of the computing resource index, memory resource index and first network bandwidth index of the computing node, and then multiply the weight of each evaluation index of the computing node by the index value to obtain the multiplied result, and then add the multiplied results corresponding to all the evaluation indicators of the node to obtain the evaluation value of the computing node.

[0172] In an embodiment of the present invention, the allocation of computing nodes is performed based on the demand intensity of node capacity by foreground load requests and background load requests, thereby avoiding improper node allocation in a distributed storage system, where some nodes may fail to fully utilize their performance while other nodes may be overloaded, resulting in low overall resource utilization efficiency and operating performance of the distributed storage system.

[0173] Further, in any of the above embodiments, the workload request also includes a transmission request; and before step 103, the steps include:

[0174] Sub-step S51, inputting the transmission request into the feature extraction model to obtain request features of the transmission request;

[0175] Sub-step S52, classifying the transmission request into categories based on the request characteristics of the transmission request to obtain at least one type of the transmission request;

[0176] Sub-step S53: determining, based on the type of the transmission request, the required intensity of the capability corresponding to the evaluation index of the transmission node for the transmission request.

[0177] In an embodiment of the present invention, the workload request also includes a transmission request. The distributed storage system inputs the transmission request into a feature extraction model to obtain a request feature of the transmission request. Based on the request feature of the transmission request, the transmission request is classified to obtain at least one type of the transmission request. Based on the type of the transmission request, the demand intensity of the capability corresponding to the evaluation index of the transmission node for the transmission request can be determined, and then the target node of the transmission node can be determined. Specifically, during the calculation process of the computing node, the computing node needs to perform read and write operations on the relevant data of the storage node. According to the demand for data transmission, the weight of the evaluation index of the transmission node in the network transmission layer trust mechanism is determined. The evaluation value of the transmission request for each transmission node is calculated, and several switches with higher evaluation values ​​max(t 传输 ), and will read and write to the corresponding storage nodes to meet the needs of the transmission request.

[0178] Reference Figure 5 , shows a schematic diagram of dividing a transmission request into at least one type provided in an embodiment of the present invention. The types of transmission requests include delay-sensitive and bandwidth-intensive. Based on the type of transmission request, the demand intensity of the capability corresponding to the evaluation index of the transmission node of the transmission request can be determined, and then the target node of the transmission request can be determined.

[0179] Further, in any of the above embodiments, step 103 includes:

[0180] Sub-step S61, based on the demand intensity of the transmission request for the capability corresponding to the evaluation index of the transmission node, respectively determining the weights of the response delay index and the second network bandwidth index.

[0181] In the embodiment of the present invention, the distributed storage system can determine the weights of the response delay index and the second network bandwidth index respectively based on the demand intensity of the capability corresponding to the evaluation index of the transmission node for the transmission request.

[0182] Further, in any of the above embodiments, step 104 includes:

[0183] Sub-step S71, determining the evaluation value of the transmission node based on the weight and index value of the response delay index and the weight and index value of the second network bandwidth index.

[0184] In an embodiment of the present invention, the distributed storage system can obtain the weight and index value of the response delay index and the second network bandwidth index of the transmission node, and then multiply the weight of each evaluation index of the transmission node by the index value to obtain the multiplied result, and then add the multiplied results corresponding to all the evaluation indicators of the node to obtain the evaluation value of the transmission node.

[0185] Further, in any of the above embodiments, the workload request also includes a storage request; and before step 103, the steps include:

[0186] Sub-step S81, inputting the storage request into the feature extraction model to obtain request features of the storage request;

[0187] Sub-step S82, classifying the storage request into categories based on the request characteristics of the storage request to obtain at least one type of the storage request;

[0188] Sub-step S83, based on the type of the storage request, determining the demand intensity of the capacity of the storage request corresponding to the evaluation index of the storage node.

[0189] In an embodiment of the present invention, the workload request also includes a storage request. The distributed storage system can input the storage request into a feature extraction model to obtain the request feature of the storage request. Then, based on the request feature of the storage request, the storage request is classified to obtain at least one type of storage request. Based on the type of storage request, the distributed storage system can determine the demand intensity of the storage request for the capability corresponding to the evaluation index of the storage node. Specifically, in the data reading and writing process of each storage node, the weight coefficient of the evaluation index of the storage node in the data storage layer trust mechanism is determined according to the demand for data reading and writing. The evaluation value of the storage request for each storage disk (storage node) in the data storage layer is calculated, and several storage disks with higher evaluation values ​​are selected from the storage nodes to meet the demand for storage requests.

[0190] Reference Figure 6 , shows a schematic diagram of dividing storage requests into at least one type provided in an embodiment of the present invention. The types of storage requests include security-sensitive and capacity-intensive. Based on the type of storage request, the demand intensity of the capacity corresponding to the evaluation index of the storage node for the storage request can be determined, and then the target node of the storage request can be determined.

[0191] Further, in any of the above embodiments, step 103 includes:

[0192] Sub-step S91, based on the demand intensity of the capacity corresponding to the evaluation index of the storage node for the storage request, respectively determine the weights of the solid state drive index and the third network bandwidth index.

[0193] In the embodiment of the present invention, the distributed storage system can determine the weights of the solid state drive index and the third network bandwidth index respectively based on the demand intensity of the capacity corresponding to the evaluation index of the storage node for the storage request.

[0194] Further, in any of the above embodiments, step 104 includes:

[0195] Sub-step S101, determining the evaluation value of the storage node based on the weight and index value of the solid state drive index and the weight and index value of the third network bandwidth index.

[0196] In an embodiment of the present invention, the distributed storage system can obtain the weights and index values ​​of the solid-state hard disk index and the third network bandwidth index of the storage node, and then multiply the weight of each evaluation index of the storage node by the index value to obtain the multiplied result, and then add the multiplied results corresponding to all evaluation indicators of the node to obtain the evaluation value of the storage node.

[0197] In an embodiment of the present invention, nodes are allocated according to the measurement results of the multi-level trust mechanism, and appropriate response nodes can be selected for workload requests at the computing layer, transmission layer and storage layer, thereby improving the equipment utilization and work efficiency of the cloud distributed storage system.

[0198] Further, in any of the above embodiments, step 103 includes:

[0199] Sub-step S111, obtaining the importance value of the computing request to the client, and the time requirement information of the client for the computing request;

[0200] Sub-step S112, prioritizing the computing requests based on the importance values ​​and the time requirement information to obtain priority information of the computing requests;

[0201] Sub-step S113, determining the weight of the evaluation index of the computing request based on the priority sorting information.

[0202] In an embodiment of the present invention, when allocating computing nodes to computing requests, the distributed storage system can obtain the importance value of the computing request to the client and the time requirement information of the client for the computing request, and prioritize the computing request based on the importance value and the time requirement information to obtain the priority ranking information of the computing request, and determine the weight of the evaluation index of the computing request based on the priority ranking information. Specifically, the allocation order of the computing requests is the priority ranking order.

[0203] The computing requests include foreground load requests and background load requests. The priority of the foreground load request is higher than the priority of the background load request. Therefore, the node allocation to the foreground load request precedes the node allocation to the background load request.

[0204] In the embodiment of the present invention, computing nodes are allocated according to the demand intensity and priority of foreground load requests and background load requests for node capabilities, which can improve the adaptability of the system to workload requests.

[0205] Further, in any of the above embodiments, at least one offline workload request is stored in the distributed storage system; and the method includes:

[0206] Sub-step S121, obtaining the sending frequency of the client for the offline workload request;

[0207] Sub-step S122: based on the sending frequency of the client for the offline workload request, taking the offline workload request whose sending frequency exceeds a preset frequency threshold in the at least one offline workload request as a workload request to be processed;

[0208] Sub-step S123: inputting the workload request to be processed into the feature extraction model to obtain offline request features of the workload request to be processed.

[0209] Reference Figure 7 , showing a schematic diagram of extracting offline request features using a feature extraction model provided in an embodiment of the present invention. At least one offline workload request is stored in a distributed storage system. The distributed storage system obtains the sending frequency of the client for the offline workload request. Based on the sending frequency of the client for the offline workload request, the offline workload request whose sending frequency exceeds a preset frequency threshold in at least one offline workload request is used as a workload request to be processed, that is, the commonly used offline workload request is used as the workload request to be processed. Then, the workload request to be processed is input into the feature extraction model to obtain the offline request features of the workload request to be processed, and the extracted offline request features are persisted to the memory of the computing node according to the type of workload request.

[0210] If the offline workload request is an offline foreground workload request, the offline request feature of the foreground workload request is the read-write ratio R i , average request size S i , average number of visits C i and the number of unique requests U i If the offline workload request is an offline background workload request, the offline request characteristics of the background workload request are the erasure stripe width Wj and the data block size Sj.

[0211] Further, in any of the above embodiments, sub-step S22 includes:

[0212] Sub-step S131, based on the offline request characteristics and the request characteristics of the computing request, classify the computing request into categories to obtain the type of the computing request.

[0213] In an embodiment of the present invention, when computing requests, transmission requests, and storage requests are classified, offline request features stored in the memory of the computing node can be loaded in advance. Since the offline request features are stored according to the type of workload request, the type of computing request can be obtained by comparing the offline request features with the request features of the computing request; the type of transmission request can be obtained by comparing the offline request features with the request features of the transmission request; and the type of storage request can be obtained by comparing the offline request features with the request features of the storage request.

[0214] Reference Figure 8, shows a schematic diagram of analyzing a workload request based on offline request features and workload request features provided in an embodiment of the present invention. The features of the real load (workload request) are extracted, and then the features of the workload request are compared with the offline request features to determine the type of the workload request.

[0215] Furthermore, in any of the above embodiments, the method includes:

[0216] Sub-step S141, executing the workload request using the target node;

[0217] Sub-step S142: updating the indicator value corresponding to the evaluation indicator of the target node based on the execution status of the workload request by the target node.

[0218] Reference Fig. 9 , a schematic diagram of updating an indicator value corresponding to an evaluation indicator provided in an embodiment of the present invention is shown. The target node is used to execute a workload request, and based on the execution of the workload request by the target node, the indicator value corresponding to the evaluation indicator of the target node is updated, and the offline request feature is updated regularly, so as to optimize the node allocation strategy of each layer.

[0219] Reference Fig.10 , shows a flowchart of the steps of another method for allocating requests in a distributed storage system provided in an embodiment of the present invention, which may specifically include the following steps:

[0220] Step 1001, start;

[0221] Step 1002, matching the client foreground application load and background load characteristics;

[0222] Step 1003, statistics various related indicators and states between nodes and within nodes, define multi-dimensional trust measurement and build a multi-level trust mechanism;

[0223] Step 1004, based on the identified foreground and background load characteristics and priorities, a trust mechanism of the computing node layer is used to select a suitable response node for the computing load;

[0224] Step 1005, selecting a suitable response device for the transmission load according to the network transmission layer trust mechanism;

[0225] Step 1006, according to the storage layer trust mechanism, execute the load task for the storage task load, feedback the completion status to the client, and update various related indicators and status of the computing layer, transport layer and storage layer;

[0226] Step 1007, end.

[0227] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0228] Reference Fig.11 , shows a structural block diagram of a request allocation device in a distributed storage system provided in an embodiment of the present invention, which is applied to a distributed storage system, wherein the distributed storage system includes at least one node; specifically, the following modules may be included:

[0229] The workload request acquisition module 1101 is used to acquire at least one workload request sent by a preset client;

[0230] An evaluation index acquisition module 1102 is used to acquire at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is an index of the ability of the node to execute the workload request;

[0231] A weight determination module 1103 is used to determine the weight of the evaluation indicator for any workload request based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator;

[0232] An evaluation value determination module 1104, configured to determine the evaluation value of the node based on the weight of the evaluation index and the index value;

[0233] The target node determination module 1105 is used to determine the target node of the workload request based on the evaluation value of the node.

[0234] In an optional embodiment of the present invention, the device comprises:

[0235] A training workload request acquisition module, used to acquire a training workload request from the client;

[0236] a training request feature acquisition module, configured to acquire at least one training request feature of the training workload request;

[0237] The feature extraction model obtaining module is used to train a preset machine learning model using the training workload request and the training request feature to obtain a feature extraction model.

[0238] In an optional embodiment of the present invention, the nodes include at least one or more of computing nodes, transmission nodes and storage nodes; the evaluation indicators of the computing nodes include at least one or more of computing resource indicators, memory resource indicators and first network bandwidth indicators; the evaluation indicators of the transmission nodes include at least response delay indicators and / or second network bandwidth indicators; the evaluation indicators of the storage nodes include at least solid-state hard disk indicators and / or third network bandwidth indicators.

[0239] In an optional embodiment of the present invention, the workload request at least includes a computing request; the apparatus includes:

[0240] A first request feature obtaining module, used for inputting the computing request into the feature extraction model to obtain the request feature of the computing request;

[0241] A first category classification module, configured to classify the computing request based on a request feature of the computing request to obtain at least one type of the computing request;

[0242] A first demand intensity determination module is configured to determine, based on the type of the computing request, the demand intensity of the capability corresponding to the evaluation index of the computing node for the computing request.

[0243] In an optional embodiment of the present invention, the weight determination module includes:

[0244] The first weight determination submodule is used to determine the weights of the computing resource indicator, the memory resource indicator and the first network bandwidth indicator respectively based on the demand intensity of the capability corresponding to the evaluation indicator of the computing node for the computing request.

[0245] In an optional embodiment of the present invention, the evaluation value determination module includes:

[0246] The first evaluation value determination submodule is used to determine the evaluation value of the computing node based on the weight and index value of the computing resource index, the weight and index value of the memory resource index, and the weight and index value of the first network bandwidth index.

[0247] In an optional embodiment of the present invention, the workload request also includes a transmission request; the device includes:

[0248] A second request feature obtaining module, used for inputting the transmission request into the feature extraction model to obtain the request feature of the transmission request;

[0249] A second classification module, configured to classify the transmission request based on the request characteristics of the transmission request to obtain at least one type of the transmission request;

[0250] A second demand strength determination module is configured to determine, based on the type of the transmission request, the demand strength of the capability corresponding to the evaluation index of the transmission node for the transmission request.

[0251] In an optional embodiment of the present invention, the weight determination module includes:

[0252] The second weight determination submodule is used to determine the weights of the response delay indicator and the second network bandwidth indicator respectively based on the demand intensity of the capability corresponding to the evaluation indicator of the transmission node for the transmission request.

[0253] In an optional embodiment of the present invention, the evaluation value determination module includes:

[0254] The second evaluation value determination submodule is used to determine the evaluation value of the transmission node based on the weight and index value of the response delay index and the weight and index value of the second network bandwidth index.

[0255] In an optional embodiment of the present invention, the workload request also includes a storage request; the device includes:

[0256] A third request feature obtaining module, used for inputting the storage request into the feature extraction model to obtain a request feature of the storage request;

[0257] a third category classification module, configured to classify the storage request based on a request feature of the storage request to obtain at least one type of the storage request;

[0258] A third demand strength determination module is configured to determine, based on the type of the storage request, the demand strength of the capacity of the storage request corresponding to the evaluation index of the storage node.

[0259] In an optional embodiment of the present invention, the weight determination module includes:

[0260] The third weight determination submodule is used to determine the weights of the solid-state drive index and the third network bandwidth index respectively based on the demand intensity of the capacity corresponding to the evaluation index of the storage node for the storage request.

[0261] In an optional embodiment of the present invention, the evaluation value determination module includes:

[0262] The third evaluation value determination submodule is used to determine the evaluation value of the storage node based on the weight and index value of the solid state drive index and the weight and index value of the third network bandwidth index.

[0263] In an optional embodiment of the present invention, the weight determination module includes:

[0264] A time requirement information acquisition submodule, used to acquire the importance value of the computing request to the client, and the time requirement information of the client for the computing request;

[0265] A priority sorting information obtaining submodule, used to prioritize the computing requests based on the importance value and the time requirement information, and obtain priority sorting information of the computing requests;

[0266] The fourth weight determination submodule is used to determine the weight of the evaluation indicator of the calculation request based on the priority sorting information.

[0267] In an optional embodiment of the present invention, at least one offline workload request is stored in the distributed storage system; the device includes:

[0268] A sending frequency acquisition module, used to acquire the sending frequency of the client for the offline workload request;

[0269] The module for processing workload requests is used to take, based on the sending frequency of the client for the offline workload requests, the offline workload request whose sending frequency exceeds a preset frequency threshold in the at least one offline workload request as the workload request to be processed;

[0270] The offline request feature obtaining module is used to input the workload request to be processed into the feature extraction model to obtain the offline request feature of the workload request to be processed.

[0271] In an optional embodiment of the present invention, the first category classification module includes:

[0272] The type obtaining submodule is used to classify the computing request into categories based on the offline request characteristics and the request characteristics of the computing request to obtain the type of the computing request.

[0273] In an optional embodiment of the present invention, the device comprises:

[0274] An execution module, configured to execute the workload request using the target node;

[0275] An updating module is used to update the indicator value corresponding to the evaluation indicator of the target node based on the execution status of the workload request by the target node.

[0276] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0277] In addition, an embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it includes a processor 1201, a communication interface 1202, a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204.

[0278] Memory 1203, used for storing computer programs;

[0279] The processor 1201 is used to execute the program stored in the memory 1203 to implement the following steps:

[0280] Obtain at least one workload request sent by a preset client;

[0281] Acquire at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is an index of the ability of the node to execute the workload request;

[0282] For any of the workload requests, determining a weight of the evaluation indicator based on the intensity of demand of the workload request for the capability corresponding to the evaluation indicator;

[0283] Determining the evaluation value of the node based on the weight of the evaluation index and the index value;

[0284] Based on the evaluation value of the node, a target node for the workload request is determined.

[0285] In an optional embodiment of the present invention, the method includes:

[0286] Obtaining a training workload request from the client;

[0287] obtaining at least one training request characteristic of the training workload request;

[0288] The preset machine learning model is trained using the training workload request and the training request features to obtain a feature extraction model.

[0289] In an optional embodiment of the present invention, the nodes include at least one or more of computing nodes, transmission nodes and storage nodes; the evaluation indicators of the computing nodes include at least one or more of computing resource indicators, memory resource indicators and first network bandwidth indicators; the evaluation indicators of the transmission nodes include at least response delay indicators and / or second network bandwidth indicators; the evaluation indicators of the storage nodes include at least solid-state hard disk indicators and / or third network bandwidth indicators.

[0290] In an optional embodiment of the present invention, the workload request includes at least a computing request; and before the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator, the step includes:

[0291] Inputting the computing request into the feature extraction model to obtain request features of the computing request;

[0292] Classify the computing requests based on the request characteristics of the computing requests to obtain at least one type of the computing requests;

[0293] Based on the type of the computing request, a demand intensity of the capability of the computing request corresponding to the evaluation index of the computing node is determined.

[0294] In an optional embodiment of the present invention, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0295] Based on the demand intensity of the capability corresponding to the evaluation index of the computing node for the computing request, the weights of the computing resource index, the memory resource index and the first network bandwidth index are determined respectively.

[0296] In an optional embodiment of the present invention, the step of determining the evaluation value of the node based on the weight of the evaluation index and the index value includes:

[0297] Based on the weight and index value of the computing resource index, the weight and index value of the memory resource index, and the weight and index value of the first network bandwidth index, an evaluation value of the computing node is determined.

[0298] In an optional embodiment of the present invention, the workload request further includes a transmission request; and before the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator includes:

[0299] Inputting the transmission request into the feature extraction model to obtain request features of the transmission request;

[0300] Classify the transmission request based on the request feature of the transmission request to obtain at least one type of the transmission request;

[0301] Based on the type of the transmission request, a requirement intensity of the transmission request for a capability corresponding to the evaluation indicator of the transmission node is determined.

[0302] In an optional embodiment of the present invention, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0303] Based on the demand intensity of the transmission request for the capability corresponding to the evaluation index of the transmission node, the weights of the response delay index and the second network bandwidth index are determined respectively.

[0304] In an optional embodiment of the present invention, the step of determining the evaluation value of the node based on the weight of the evaluation index and the index value includes:

[0305] Based on the weight and index value of the response delay index and the weight and index value of the second network bandwidth index, an evaluation value of the transmission node is determined.

[0306] In an optional embodiment of the present invention, the workload request further includes a storage request; and before the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator, the step includes:

[0307] Inputting the storage request into the feature extraction model to obtain request features of the storage request;

[0308] Classify the storage request based on the request feature of the storage request to obtain at least one type of the storage request;

[0309] Based on the type of the storage request, the demand intensity of the storage request for the capability corresponding to the evaluation index of the storage node is determined.

[0310] In an optional embodiment of the present invention, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0311] Based on the demand intensity of the storage request for the capacity corresponding to the evaluation index of the storage node, the weights of the solid-state hard disk index and the third network bandwidth index are determined respectively.

[0312] In an optional embodiment of the present invention, the step of determining the evaluation value of the node based on the weight of the evaluation index and the index value includes:

[0313] Based on the weight and index value of the solid state drive index and the weight and index value of the third network bandwidth index, an evaluation value of the storage node is determined.

[0314] In an optional embodiment of the present invention, the step of determining the weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator includes:

[0315] Obtaining an importance value of the computing request to the client and time requirement information of the client for the computing request;

[0316] Prioritizing the computing requests based on the importance values ​​and the time requirement information to obtain priority information of the computing requests;

[0317] Based on the priority sorting information, a weight of an evaluation indicator of the computing request is determined.

[0318] In an optional embodiment of the present invention, at least one offline workload request is stored in the distributed storage system; the method includes:

[0319] Obtaining a sending frequency of the client for the offline workload request;

[0320] Based on the sending frequency of the client for the offline workload request, taking the offline workload request whose sending frequency exceeds a preset frequency threshold in the at least one offline workload request as a workload request to be processed;

[0321] The workload request to be processed is input into the feature extraction model to obtain offline request features of the workload request to be processed.

[0322] In an optional embodiment of the present invention, the step of classifying the computing requests based on the request characteristics of the computing requests to obtain at least one type of the computing requests includes:

[0323] Based on the offline request feature and the request feature of the computing request, the computing request is classified to obtain the type of the computing request.

[0324] In an optional embodiment of the present invention, the method includes:

[0325] executing the workload request using the target node;

[0326] Based on the execution status of the target node on the workload request, an indicator value corresponding to the evaluation indicator of the target node is updated.

[0327] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0328] The communication interface is used for communication between the above terminal and other devices.

[0329] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0330] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0331] like Fig.13 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1301 is also provided, in which instructions are stored. When the computer-readable storage medium 1301 is run on a computer, the computer executes a request allocation method in a distributed storage system described in the above embodiment.

[0332] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes the request allocation method in a distributed storage system described in the above embodiment.

[0333] In another embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the request allocation method in a distributed storage system described in the above embodiment is implemented.

[0334] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0335] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0336] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0337] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A request allocation method in a distributed storage system, characterized in that: Applied to a distributed storage system, the distributed storage system includes at least one node; the method includes: Obtain at least one workload request sent by a preset client; Acquire at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is an index of the ability of the node to execute the workload request; For any of the workload requests, determining a weight of the evaluation indicator based on the demand intensity of the workload request for the capability corresponding to the evaluation indicator; Determining the evaluation value of the node based on the weight of the evaluation index and the index value; Based on the evaluation value of the node, a target node for the workload request is determined.

2. The method according to claim 1, characterized in that The method comprises: Obtaining a training workload request from the client; obtaining at least one training request characteristic of the training workload request; The preset machine learning model is trained using the training workload request and the training request features to obtain a feature extraction model.

3. The method according to claim 2, characterized in that The nodes include at least one or more of computing nodes, transmission nodes and storage nodes; the evaluation indicators of the computing nodes include at least one or more of computing resource indicators, memory resource indicators and first network bandwidth indicators; the evaluation indicators of the transmission nodes include at least response delay indicators and / or second network bandwidth indicators; the evaluation indicators of the storage nodes include at least solid-state hard disk indicators and / or third network bandwidth indicators.

4. The method according to claim 3, characterized in that The workload request at least includes a computing request; and the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator includes: Inputting the computing request into the feature extraction model to obtain request features of the computing request; Classify the computing requests based on the request characteristics of the computing requests to obtain at least one type of the computing requests; Based on the type of the computing request, a demand intensity of the capability of the computing request corresponding to the evaluation index of the computing node is determined.

5. The method according to claim 4, characterized in that The step of determining the weight of the evaluation indicator based on the workload request for the capability corresponding to the evaluation indicator comprises: Based on the demand intensity of the capability corresponding to the evaluation index of the computing node for the computing request, the weights of the computing resource index, the memory resource index and the first network bandwidth index are determined respectively.

6. The method according to claim 5, characterized in that The step of determining the evaluation value of the node based on the weight of the evaluation index and the index value comprises: Based on the weight and index value of the computing resource index, the weight and index value of the memory resource index, and the weight and index value of the first network bandwidth index, an evaluation value of the computing node is determined.

7. The method according to claim 3, characterized in that The workload request also includes a transmission request; and the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator includes: Inputting the transmission request into the feature extraction model to obtain request features of the transmission request; Classify the transmission request based on the request feature of the transmission request to obtain at least one type of the transmission request; Based on the type of the transmission request, a requirement intensity of the transmission request for a capability corresponding to the evaluation indicator of the transmission node is determined.

8. The method according to claim 7, characterized in that The step of determining the weight of the evaluation indicator based on the workload request for the capability corresponding to the evaluation indicator comprises: Based on the demand intensity of the transmission request for the capability corresponding to the evaluation index of the transmission node, the weights of the response delay index and the second network bandwidth index are determined respectively.

9. The method according to claim 8, characterized in that The step of determining the evaluation value of the node based on the weight of the evaluation index and the index value comprises: Based on the weight and index value of the response delay index and the weight and index value of the second network bandwidth index, an evaluation value of the transmission node is determined.

10. The method according to claim 3, characterized in that: The workload request also includes a storage request; and the step of determining the weight of the evaluation indicator based on the workload request for the demand intensity of the capability corresponding to the evaluation indicator includes: Inputting the storage request into the feature extraction model to obtain request features of the storage request; Classify the storage request based on the request feature of the storage request to obtain at least one type of the storage request; Based on the type of the storage request, the demand intensity of the storage request for the capability corresponding to the evaluation index of the storage node is determined.

11. The method according to claim 10, characterized in that The step of determining the weight of the evaluation indicator based on the workload request for the capability corresponding to the evaluation indicator comprises: Based on the demand intensity of the storage request for the capacity corresponding to the evaluation index of the storage node, the weights of the solid-state hard disk index and the third network bandwidth index are determined respectively.

12. The method according to claim 11, characterized in that The step of determining the evaluation value of the node based on the weight of the evaluation index and the index value comprises: Based on the weight and index value of the solid state drive index and the weight and index value of the third network bandwidth index, an evaluation value of the storage node is determined.

13. The method according to claim 4, characterized in that The step of determining the weight of the evaluation indicator based on the workload request for the capability corresponding to the evaluation indicator comprises: Obtaining an importance value of the computing request to the client and time requirement information of the client for the computing request; Prioritizing the computing requests based on the importance values ​​and the time requirement information to obtain priority information of the computing requests; Based on the priority sorting information, a weight of an evaluation indicator of the computing request is determined.

14. The method according to claim 4, characterized in that At least one offline workload request is stored in the distributed storage system; the method comprises: Obtaining a sending frequency of the client for the offline workload request; Based on the sending frequency of the client for the offline workload request, taking the offline workload request whose sending frequency exceeds a preset frequency threshold in the at least one offline workload request as a workload request to be processed; The workload request to be processed is input into the feature extraction model to obtain offline request features of the workload request to be processed.

15. The method according to claim 14, characterized in that The step of classifying the computing requests based on the request characteristics of the computing requests to obtain at least one type of the computing requests includes: Based on the offline request feature and the request feature of the computing request, the computing request is classified to obtain the type of the computing request.

16. The method according to claim 1, characterized in that The method comprises: executing the workload request using the target node; Based on the execution status of the target node on the workload request, an indicator value corresponding to the evaluation indicator of the target node is updated.

17. A request allocation device in a distributed storage system, characterized in that: Applied to a distributed storage system, the distributed storage system includes at least one node; the device includes: A workload request acquisition module, used to acquire at least one workload request sent by a preset client; An evaluation index acquisition module, used to acquire at least one evaluation index of the node and an index value corresponding to the evaluation index; the evaluation index is an indicator of the ability of the node to execute the workload request; A weight determination module, configured to determine, for any workload request, a weight of the evaluation indicator based on the intensity of demand for the capability corresponding to the evaluation indicator by the workload request; An evaluation value determination module, used to determine the evaluation value of the node based on the weight of the evaluation index and the index value; The target node determination module is used to determine the target node of the workload request based on the evaluation value of the node.

18. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 16 when executed by the processor.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

20. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images