Distributed object storage scheduling method and system

Abstract

The present invention relates to the technical field of distributed object storage and provides a distributed object storage scheduling method and system. The method comprises: receiving a keep-alive message sent by an existing node, and updating internal load information of object storage clusters on the basis of a trigger condition, so as to update node information of all nodes; a scheduling center collecting internal load information of all related nodes and calculating an internal weight of each object storage cluster; when a current request from a user side is received, calculating a user side load weight of each object storage cluster; and on the basis of the calculated internal weight of each object storage cluster and the calculated user side load weight of each object storage cluster, calculating a total weight, so as to select an object storage cluster that is most suitable for the current request, and returning response information back to a corresponding client. According to the present invention, the problem of load imbalance of existing distributed object storage systems is solved and the load capacity of a storage node can be effectively scheduled.

Description

A distributed object storage scheduling method and system Technical Field

[0001] The present invention relates to the technical field of distributed object storage, and in particular to a distributed object storage scheduling method and system. Background Art

[0002] Traditional distributed object storage services are deployed on a single physical machine. As business access gradually increases, the following problems arise: 1) The distributed object storage service has a single point of failure, resulting in load imbalance; 2) Single-machine deployment has performance bottlenecks; 3) Resources cannot be adjusted in real time based on business volume, which easily leads to resource waste and cannot provide more timely and effective storage scheduling services.

[0003] Furthermore, in related technologies, to improve object storage system performance and distribute access pressure across system nodes, an object storage system typically consists of multiple nodes (a distributed object storage system). If user storage requests are simply distributed, this can easily lead to excessive access pressure on a particular node, potentially causing node paralysis. Using simple round-robin or the nearest-recent approach can also lead to excessive access to nodes in popular locations. Therefore, load balancing and intelligent scheduling across multiple nodes require additional systems.

[0004] Therefore, it is necessary to provide a distributed object storage scheduling method that can solve the above problems.

[0005] Summary of the Invention

[0006] The present invention aims to provide a distributed object storage scheduling method and system to solve the technical problems in the existing technology of distributed object storage services, such as load imbalance caused by single point failures, inability to adjust resources in real time according to business volume, which easily leads to resource waste, inability to adjust resources in real time according to business volume, which easily leads to resource waste, and inability to provide more timely and effective storage scheduling services. Some clusters in popular locations are overloaded, resulting in excessive storage pressure, and selecting nodes that are too far away leads to slow upload / download speeds and poor user storage experience. The technical problems to be solved by the present invention are achieved through the following technical solutions.

[0007] A first aspect of the present invention proposes a distributed object storage scheduling method, comprising: receiving keep-alive messages sent by existing nodes, and updating the internal load information of the object storage cluster according to trigger conditions to update the node information of all nodes; the scheduling center collects the internal load information of all relevant nodes and calculates the internal weight of each object storage cluster; when receiving the current request from the user side, calculates the user-side load weight of each object storage cluster; based on the calculated internal weight of each object storage cluster and the calculated user-side load weight of each object storage cluster, calculates the total weight to select the object storage cluster that is most suitable for the current request, and returns response information to the corresponding client.

[0008] According to an optional implementation, updating the internal load information of the object storage cluster according to the trigger condition to update the node information of all nodes includes: updating the node information of all nodes when the heartbeat information of the new node is received through the intelligent gateway, wherein each storage node corresponds to an object storage cluster, and each object storage cluster includes storage sub-nodes corresponding to multiple servers; the node information includes internal weight and user-side load weight.

[0009] According to an optional implementation, the updating of the internal load information of the object storage cluster according to the trigger conditions to update the node information of all nodes includes: when an existing node goes offline or is suddenly disconnected, updating the internal load information of the object storage cluster, and updating the node information of the remaining nodes except the existing node that is offline or suddenly disconnected, wherein each storage node corresponds to an object storage cluster, and each object storage cluster includes storage sub-nodes corresponding to multiple servers; the node information includes internal weights and user-side load weights.

[0010] According to an optional implementation, the scheduling center collects internal load information of all relevant nodes and uses the following expression (1) to calculate the internal weight of each object storage cluster, including:

[0011] Among them, Weight1 iRepresents the internal weight of all influencing factors in the object storage cluster; W11 is the weight coefficient of the CPU usage of the current object storage cluster; UsedCpu represents the CPU usage of the current object storage cluster; CpuTotal represents the total CPU usage of all clusters; W12 is the weight coefficient of the network IO load of the current object storage cluster; IO represents the network IO speed of the current object storage cluster; BandWidth represents the transmission speed of the total broadband of all clusters; W13 represents the weight coefficient of the disk usage of the current object storage cluster; UsedSize represents the total disk usage of the current object storage cluster; Capacity represents the total disk capacity of all clusters; W14 represents the weight coefficient of the memory usage of the current object storage cluster; UsedRam represents the memory usage of the current object storage cluster; RamTotal represents the total memory usage of all clusters; LinkNum represents the total number of connection requests received in the current object storage cluster, and LinkTotal represents the maximum number of connections that the current object storage cluster can tolerate; W1 i Indicates the weight coefficient of the i-th influencing factor in the current object storage cluster. The weight coefficient W1 i It can be freely adjusted according to different business focuses, and can also increase network traffic or other influencing factors.

[0012] According to an optional embodiment, when a user connects to the dispatch center and sends an object storage request, the dispatch center calculates the user-side load weight. Specifically, based on the IP address and location of the user request, the following expression (3) is used to calculate the user-side load weight of the physical distance between the user and each object storage cluster and the connection time:

[0013] Weight2 j =W21*time / 1000+W22*distance / 100000,...,+W2 j *other (3)

[0014] Among them, Weight2 j It represents the user-side load weight of the user and each object storage cluster when the user connects to the scheduling center and sends an object storage request; W21 represents the user-side load weight of the connection time between the user and each object storage cluster; W22 represents the user-side load weight of the physical distance between the user and each object storage cluster; W2 jIndicates the user-side load weight of the jth influencing factor, where j is a positive integer. The urgency of user requests or other user-side load information can be added according to actual conditions. Time indicates the connection time between the user and each object storage cluster, in seconds. Distance indicates the physical distance between the user and each object storage cluster, in km, ranging from 1km to 1000km. Weight2 j and Weight1 i The units keep the same order of magnitude.

[0015] According to an optional implementation, when receiving an object storage request from a user, the following expression (4) is used to calculate the total weight Weight 总 :

[0016] Weight 总 =m*Weight1 i +h*Weight2 j , 1≤i≤n, 1≤j≤n(4)

[0017] Among them, m represents the internal priority coefficient of all influencing factors in the object storage cluster; Weight1 i Represents the internal weight of all influencing factors in the object storage cluster. The value of m is determined according to the business application scenario parameters. h represents the priority coefficient on the user side. The value of h is determined according to the business application scenario parameters. Weight2 j Indicates the user-side load weight of each object storage cluster when a user connects to the scheduling center and sends an object storage request.

[0018] According to an optional embodiment, m and h satisfy expression (5):

[0019] m+h=1 (5).

[0020] According to an optional embodiment, it further includes: when receiving an object storage request sent by a user, first updating the node information of all nodes and calculating the user-side load weight, and then using expression (4) to calculate the total weight Weight 总 .

[0021] According to an optional implementation, the nodes of each object storage cluster are sorted according to the calculated total weight to select a server that best matches the current request, and return a storable address to the corresponding client.

[0022] A second aspect of the present invention provides a distributed object storage scheduling system, which is used to execute the distributed object storage scheduling method described in the present invention. The distributed object storage scheduling system includes: a receiving and processing module, which receives keep-alive messages sent by existing nodes and updates the internal load information of the object storage cluster according to the trigger conditions to update the node information of all nodes; a first calculation module, in which the scheduling center collects the internal load information of all relevant nodes and calculates the internal weight of each object storage cluster; a second calculation module, which calculates the user-side load weight of each object storage cluster when receiving the current request from the user side; and a response determination module, which calculates the total weight based on the calculated internal weight of each object storage cluster and the calculated user-side load weight of each object storage cluster to select the object storage cluster that is most suitable for the current request and return the response information to the corresponding client.

[0023] The embodiments of the present invention include the following advantages:

[0024] Compared with the prior art, the present invention uses a dispatch center to continuously collect various information about the object storage cluster through a heartbeat message mechanism using an intelligent gateway, and updates the internal load information of the object storage cluster according to trigger conditions to update the node information of all nodes to provide a more accurate data basis for subsequent calculations. The dispatch center uses an intelligent gateway to collect updated information of each object storage cluster and continuously updates the internal weight of the object storage cluster through a heartbeat message mechanism. The internal weight is obtained by calculation, which can effectively solve the problem of a node (object storage cluster or its corresponding intelligent gateway) being disconnected / offline due to an excessive number of nodes in the existing distributed object storage system. When a user connects to the dispatch center and sends an object storage request, the dispatch center calculates the user-side load weight based on the IP address and location of the user request. This can effectively avoid the impact of factors such as the physical distance and connection time of each object storage cluster on the scheduling process, and can accurately determine the information status on the user side.

[0025] In addition, the method of the present invention comprehensively considers the load information of the distributed object storage cluster and user-side information, and can determine in real time the information on the scheduling service capability of the distributed object storage cluster. It can prevent certain clusters in popular locations from being overloaded and causing excessive storage pressure, and can effectively avoid problems such as poor user storage experience caused by slow upload / download speeds due to selecting nodes that are too far away. Specifically, by comprehensively considering the dynamic information of the current object storage cluster (current disk space remaining capacity, CPU / memory occupancy, and other influencing factors), and paying more attention to the configuration information of each object storage cluster (number of CPU cores / memory size, etc.), the dynamic load status of the current object storage cluster can be accurately determined and updated in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a flow chart of an example of a distributed object storage scheduling method of the present invention;

[0027] FIG2 is a schematic diagram of an example of an application scenario of the distributed object storage scheduling method of the present invention;

[0028] FIG3 is a schematic diagram of information transmission from another angle of FIG2 ;

[0029] FIG4 is a schematic structural diagram of an example of a distributed object storage scheduling system of the present invention. DETAILED DESCRIPTION

[0030] It should be noted that the embodiments and features of the embodiments in this application can be combined with each other unless otherwise specified. In the present invention, the upper surface of an object in the accompanying drawings is referred to as the upper surface, and the lower surface of an object in the accompanying drawings is referred to as the lower surface. This is merely for the purpose of more clearly illustrating the process of the distributed object storage scheduling method and should not be construed as limiting the present invention. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0031] 1 to 3 , a first aspect of the present invention provides a distributed object storage scheduling method. This method comprehensively considers the load information of the distributed object storage cluster and user-side information, and can determine in real time the information about the distributed object storage cluster's scheduling service capabilities. This method can prevent certain clusters in popular locations from being overloaded and causing excessive storage pressure, and can effectively avoid problems such as poor user storage experience caused by slow upload / download speeds due to selecting nodes that are too far away. The present invention can comprehensively consider the dynamic information of the current object storage cluster (current disk space remaining capacity, CPU / memory usage, and other influencing factors), and pay more attention to the configuration information of each object storage cluster (CPU core number / memory size, etc.), to determine and update the dynamic load of the current object storage cluster in real time. When receiving a current request from the user side (especially an object storage request), the method selects the object storage cluster that best suits the current request for the user and returns response information to the corresponding client.

[0032] Figure 1 is a flow chart of an example of a distributed object storage scheduling method of the present invention. Figure 2 is a diagram of an example of an application scenario of the distributed object storage scheduling method of the present invention.

[0033] In the example of Figure 2, the application scenario includes a client, a scheduling center, and multiple object storage clusters (specifically, object storage cluster A, object storage cluster B, and object storage cluster C). Each object storage cluster includes multiple servers (or multiple storage nodes corresponding to the multiple servers, also referred to as nodes for short), which are used to provide, for example, storage services to the client. Each object storage cluster corresponds to an intelligent gateway connected to the scheduling center (specifically, intelligent gateway a, intelligent gateway b, and intelligent gateway c corresponding to object storage cluster A, object storage cluster B, and object storage cluster C). The client sends the current request (or user request, such as an object storage request) to the scheduling center. The scheduling center selects the object storage cluster that best matches the current request for the user through real-time calculation and returns a response to the corresponding client.

[0034] It should be noted that other implementations may include one, two, four, or more object storage clusters. The present invention is particularly suitable for scenarios such as clusters in popular locations experiencing excessive load or user-side emergency use. The above descriptions are provided as optional examples only and are not to be construed as limiting the present invention.

[0035] As shown in FIG1 , in step S101 , a keep-alive message sent by an existing node is received, and internal load information of the object storage cluster is updated according to a triggering condition to update node information of all nodes.

[0036] In the example of FIG2 , each object storage cluster sends a keep-alive message (a heartbeat keep-alive message as shown in FIG3 ) to the scheduling center through the intelligent gateway, and the scheduling center receives the keep-alive message sent by each object storage cluster through the intelligent gateway.

[0037] Specifically, each object storage cluster corresponds to a storage node, and each server in each object storage cluster corresponds to a storage sub-node.

[0038] In a specific embodiment, the scheduling center receives a keep-alive message from, for example, an existing storage node (also referred to as "existing node") corresponding to the object storage cluster B through the intelligent gateway b.

[0039] It should be noted that in the present invention, the intelligent gateway is responsible for collecting various internal information of the object storage cluster it is responsible for, including the current network 10 speed of the object storage cluster, the current load status of each disk in the object storage cluster, the current number of connections and other information. Other parameter information can be added as a reference for the load according to the needs of different businesses.

[0040] In a specific implementation, the intelligent gateway is bound to the object storage cluster, is responsible for collecting various information of the object storage cluster, and represents the current survival status or current online status by reporting heartbeat keep-alive messages to the scheduling center.

[0041] Specifically, the dispatch center is responsible for communicating with the client and obtaining the specific IP address and location of the current client. The dispatch center communicates with the intelligent gateway, where the communication mechanism of the intelligent gateway is a heartbeat keep-alive message or heartbeat message. The dispatch center obtains various information such as the current load, link status, connection duration, and geographical location of each object storage cluster through the heartbeat message of the intelligent gateway. The object storage cluster is the edge scheduling unit of the dispatch center. The object storage cluster is responsible for providing object storage functions. After being scheduled by the dispatch center, the client actually transmits information data with the server in the object storage cluster. The information data is, for example, the following data: (1) Daily life scenarios: user photos, videos, recordings and other files; (2) Privacy file scenarios: user work records, company contracts and other private internal files; (3) Film and television entertainment scenarios: various game files, movie and TV series and other film and television data files; (4) Mobile backup scenarios: various files stored in the user's mobile phone, text messages, contacts, mobile phone albums and other files that need to be backed up.

[0042] Next, the internal load information of the object storage cluster is updated based on the triggering condition to update the node information of all nodes. The triggering condition is a condition for triggering the update process. Specifically, this includes updating the internal load information of the object storage cluster and updating the node information of all nodes (including updating the node information of the storage node and the node information of the child nodes).

[0043] Specifically, the node information includes internal weight and user-side load weight.

[0044] In a specific embodiment, when the heartbeat information of a new node is received through the intelligent gateway (or "a new object storage cluster is added" as shown in Figure 3), the node information of all nodes is updated, wherein each storage node corresponds to an object storage cluster, and each object storage cluster includes storage sub-nodes corresponding to multiple servers.

[0045] In another optional embodiment, when an existing node goes offline or is suddenly disconnected (as shown in "Existing object storage cluster goes offline or is suddenly disconnected" in Figure 3), the internal load information of the object storage cluster is updated, and the node information of the remaining nodes excluding the offline or suddenly disconnected existing node is updated. Each storage node corresponds to an object storage cluster, and each object storage cluster includes storage child nodes corresponding to multiple servers. The node information includes internal weights and user-side load weights.

[0046] It should be noted that the existing node refers to an online object storage cluster (including storage sub-nodes corresponding to each server) connected to the scheduling center. The above is only provided as an optional example and should not be construed as limiting the present invention.

[0047] The scheduling center uses an intelligent gateway to continuously collect various information about the object storage cluster through the heartbeat message mechanism, and updates the internal load information of the object storage cluster according to the trigger conditions to update the node information of all nodes to provide a more accurate data basis for subsequent calculations.

[0048] Next, in step S102 , the scheduling center collects internal load information of all relevant nodes and calculates the internal weight of each object storage cluster.

[0049] Specifically, the scheduling center collects the internal load information of all relevant nodes and calculates the internal weight of each object storage cluster using the following expression (1), for example, by collecting the internal load information of all relevant nodes in real time or at a specified frequency.

[0050] Among them, Weight1 i Represents the internal weight of all influencing factors in the object storage cluster; W11 is the weight coefficient of the CPU usage of the current object storage cluster; UsedCpu represents the CPU usage of the current object storage cluster; CpuTotal represents the total CPU usage of all clusters; W12 is the weight coefficient of the network IO load of the current object storage cluster; IO represents the network IO speed of the current object storage cluster; BandWidth represents the transmission speed of the total broadband of all clusters; W13 represents the weight coefficient of the disk usage of the current object storage cluster; UsedSize represents the total disk usage of the current object storage cluster; Capacity represents the total disk capacity of all clusters; W14 represents the weight coefficient of the memory usage of the current object storage cluster; UsedRam represents the memory usage of the current object storage cluster; RamTotal represents the total memory usage of all clusters; LinkNum represents the total number of connection requests received in the current object storage cluster, and LinkTotal represents the maximum number of connections that the current object storage cluster can tolerate; W1 i Indicates the weight coefficient of the i-th influencing factor in the current object storage cluster. The weight coefficient W1 i It can be freely adjusted according to different business focuses, and can also increase network traffic or other influencing factors.

[0051] It should be noted that due to the weights within the object storage cluster, all parameters are calculated in percentage form, so there is no need to consider unit conversion, and W1 iIt can be adjusted according to the actual situation of each business, but must meet the following formula:

[0052] in, Weight1 i Both represent the internal weights of all influencing factors in the object storage cluster; W1 i This parameter represents the internal weight of the i-th influencing factor in the object storage cluster. i represents the i-th influencing factor in the object storage cluster and is a positive integer ranging from 1, 2, ..., to n. When a specific parameter is 0, it means that the object storage cluster does not consider the impact of the current internal load.

[0053] In one specific implementation, i=5, as follows.

[0054] Among them, Weight1 i Indicates the internal weights of all influencing factors in the object storage cluster; W11 is the weight coefficient of the CPU usage of the current object storage cluster; UsedCpu indicates the CPU usage of the current object storage cluster; CpuTotal indicates the total CPU usage of all clusters; W12 is the weight coefficient of the network IO load of the current object storage cluster; IO indicates the network IO speed of the current object storage cluster; BandWidth indicates the transmission speed of the total broadband of all clusters; W13 is the weight coefficient of the disk usage of the current object storage cluster; UsedSize indicates the total disk usage of the current object storage cluster; Capacit y represents the total disk capacity of all clusters; W14 represents the weight coefficient of the memory usage of the current object storage cluster; UsedRam represents the memory usage of the current object storage cluster; RamTotal represents the total memory usage of all clusters; W15 represents the weight coefficient of the fifth influencing factor in the current object storage cluster, which is the relationship between the number of connection requests received in the current object storage cluster and the maximum number of connections that can be tolerated; LinkNum represents the total number of connection requests received in the current object storage cluster, and LinkTotal represents the maximum number of connections that can be tolerated in the current object storage cluster.

[0055] It should be noted that in other implementations, the current object storage cluster does not need to consider the impact of disk capacity, and can focus more on the impact of other factors (such as network traffic). Furthermore, there is no specific limit on the number of i, and it can be adaptively increased based on actual needs. In one example, W11, W12, W13, W14, and W15 are all 0.2. However, this is not a limitation. In another example, W11, W12, W13, W14, and W15 are not equal.

[0056] The dispatch center uses an intelligent gateway to collect updated information of each object storage cluster, and continuously updates the internal weight of the object storage cluster through the heartbeat message mechanism. The internal weight is calculated by the above expression (1), which can effectively solve the problem of loss of connection / offline of a node (object storage cluster or its corresponding intelligent gateway) caused by excessive number of nodes in the existing distributed object storage system.

[0057] It should be noted that the above is merely provided as an optional example and should not be construed as a limitation to the present invention.

[0058] Next, in step S103 , when receiving the current request from the user side, the user side load weight of each object storage cluster is calculated.

[0059] Specifically, when a user connects to the dispatch center and sends an object storage request, the dispatch center calculates the user-side load weight. Specifically, based on the IP address and location of the user request, the dispatch center uses the following expression (3) to calculate the user-side load weight of the physical distance between the user and each object storage cluster and the connection time:

[0060] Weight2 j =W21*time / 1000+W22*distance / 100000,...,+W2 j *other (3)

[0061] Among them, Weight2 j It represents the user-side load weight of the user and each object storage cluster when the user connects to the scheduling center and sends an object storage request; W21 represents the user-side load weight of the connection time between the user and each object storage cluster; W22 represents the user-side load weight of the physical distance between the user and each object storage cluster; W2 j Indicates the user-side load weight of the jth influencing factor, where j is a positive integer. The urgency of user requests or other user-side load information can be added according to actual conditions. Time indicates the connection time between the user and each object storage cluster in seconds. Distance indicates the physical distance between the user and each object storage cluster in km, ranging from 1km to 1000km. Weight2 j and Weight1 i The units keep the same order of magnitude.

[0062] For example, when an object storage request from the user side (client 1) is received through the intelligent gateway c, the user side load weight of each object storage cluster is calculated, and the internal weight of the current object storage cluster is calculated.

[0063] Specifically, using the above expression, the user-side load weight of client 1 is calculated (if the influence of delay is only considered without considering the influence of distance): Weight2 j =1*time / 1000.

[0064] When a user connects to the dispatch center and sends an object storage request, the dispatch center calculates the user-side load weight based on the IP and location of the user's request. This can effectively avoid the impact of factors such as the physical distance and connection time of each object storage cluster on the scheduling process, and can accurately determine the information situation on the user side.

[0065] It should be noted that the above is merely provided as an optional example and should not be construed as a limitation to the present invention.

[0066] Next, in step S104, the total weight is calculated based on the calculated internal weights of each object storage cluster and the calculated user-side load weights of each object storage cluster to select the object storage cluster that best suits the current request and return response information to the corresponding client.

[0067] Specifically, the following expression (4) is used to calculate the total weight: 总 :

[0068] Weight 总 =m*Weight1i+h*Weight2 j , 1≤i≤n, 1≤j≤n (4)

[0069] Among them, m represents the internal priority coefficient of all influencing factors in the object storage cluster; Weight1 i Represents the internal weight of all influencing factors in the object storage cluster. The value of m is determined according to the business application scenario parameters. h represents the priority coefficient on the user side. The value of h is determined according to the business application scenario parameters. Weight2 i Indicates the user-side load weight of each object storage cluster when a user connects to the scheduling center and sends an object storage request.

[0070] Among them, m and h satisfy the following weight formula:

[0071] m+h=1 (5)

[0072] Then, when receiving the object storage request from the user, the node information of all nodes is updated and the user-side load weight is calculated. Then, the total weight is calculated using expression (4). 总 .

[0073] In an optional embodiment, upon receiving an object storage request from a user, a priority set of object storage clusters is determined based on the priorities of the object storage clusters determined in real time to provide storage services to the user. When calculating the user-side load weight of the physical distance between the user and each object storage cluster and the connection time using the following expression (3), the object storage clusters in the priority set are used for the calculation.

[0074] For object storage clusters, based on statistical calculations of information such as the number of times each object storage cluster goes offline, the offline time, and the available storage capacity within a specified time period, an evaluation calculation is performed to prioritize the provision of storage services. For example, a machine learning model is used (an evaluation model is established using a neural network algorithm, and a training data set for training the evaluation model is established using each object storage cluster labeled with a frequency label for providing storage services). The number of times each object storage cluster goes offline, the offline time, and the available storage capacity within a specified time period are converted into feature vectors for use as model inputs, which are input into the trained evaluation model to output an evaluation value for the provision of storage services (the frequency with which storage services can be provided). The priority coefficient of each object storage cluster is configured accordingly based on the output evaluation value.

[0075] Specifically, according to the determined priorities of the object storage clusters, all currently online object storage clusters are grouped to obtain a priority set for providing storage services for a certain type of user.

[0076] In addition, a historical trend chart is drawn based on information data such as the number of object storage requests sent by users within a specified historical time period and the time of the requests. The priority processing level of each user and the coefficient of each corresponding user side are determined based on the historical trend chart.

[0077] In one embodiment, the user-side priority coefficient h is first determined, and the internal-side priority coefficient m is determined. For example, based on the historical trend graph, if the priority level of each user and the corresponding user-side priority coefficient h are 0.7, then m is equal to 0.3, that is, m + h = 1.

[0078] In another embodiment, the internal priority coefficient m is first determined, and then the user-side priority coefficient h is determined. For example, all object storage clusters corresponding to the current user request and their related information are obtained. Specifically, the number of offline times, offline time periods, and available storage capacity of each object storage cluster in all obtained object storage clusters are converted into feature vectors for use as model inputs. This is then input into the evaluation model to obtain an evaluation value, and the object storage cluster with the highest evaluation value is determined. The priority coefficient of the determined object storage cluster is then further determined. For example, if the priority coefficient m is determined to be 0.6, the user-side priority coefficient h is 0.4.

[0079] Then, according to the calculated total weight, the nodes of each object storage cluster are sorted to select the server that best matches the current user request (i.e., the best-suited object storage cluster), and return the storage address to the corresponding client. For example, the load calculation formula for the storage address is set to Weight 总 =0.6*Weight1 i +0.4*Weight2 j , indicating that the current focus is on the internal load situation. On this basis, we continue to set that there are currently two object storage clusters A and B, of which Weight1 i =0.7,Weight2 j =0.4, Weight 1 i =0.8,Weight2 j =0.7, then the Weight of object storage cluster A is calculated 总 =0.328, while the Weight of object storage cluster B is 总 =0.472; then through calculation, it is concluded that the object storage cluster Cluster B will be selected as the server to accept the user's request.

[0080] It should be noted that the above is merely provided as an optional example and should not be construed as a limitation to the present invention.

[0081] Compared with the prior art, the present invention uses a dispatch center to continuously collect various information about the object storage cluster through a heartbeat message mechanism using an intelligent gateway, and updates the internal load information of the object storage cluster according to trigger conditions to update the node information of all nodes to provide a more accurate data basis for subsequent calculations. The dispatch center uses an intelligent gateway to collect updated information of each object storage cluster and continuously updates the internal weight of the object storage cluster through a heartbeat message mechanism. The internal weight is obtained by calculation, which can effectively solve the problem of a node (object storage cluster or its corresponding intelligent gateway) being disconnected / offline due to an excessive number of nodes in the existing distributed object storage system. When a user connects to the dispatch center and sends an object storage request, the dispatch center calculates the user-side load weight based on the IP address and location of the user request. This can effectively avoid the impact of factors such as the physical distance and connection time of each object storage cluster on the scheduling process, and can accurately determine the information status on the user side.

[0082] In addition, the method of the present invention comprehensively considers the load information of the distributed object storage cluster and user-side information, and can determine in real time the information on the scheduling service capability of the distributed object storage cluster. It can prevent certain clusters in popular locations from being overloaded and causing excessive storage pressure, and can effectively avoid problems such as poor user storage experience caused by slow upload / download speeds due to selecting nodes that are too far away. Specifically, by comprehensively considering the dynamic information of the current object storage cluster (current disk space remaining capacity, CPU / memory occupancy, and other influencing factors), and paying more attention to the configuration information of each object storage cluster (number of CPU cores / memory size, etc.), the dynamic load status of the current object storage cluster can be accurately determined and updated in real time.

[0083] The following are embodiments of the system of the present invention. The method of the first aspect of the present invention is particularly applicable to the system of the present invention. For details not disclosed in the embodiments of the system of the present invention, please refer to the embodiments of the method of the present invention.

[0084] 4 , a second aspect of the present invention provides a distributed object storage scheduling system 400 , which is used to execute the distributed object storage scheduling method described in the first aspect of the present invention.

[0085] The distributed object storage scheduling system 400 includes a receiving and processing module 410 , a first computing module 420 , a second computing module 430 , and a determining and responding module 440 .

[0086] In a specific embodiment, the receiving and processing module 410 is used to receive keep-alive messages sent by existing nodes, and update the internal load information of the object storage cluster according to the triggering conditions to update the node information of all nodes. The first calculation module 420 scheduling center collects the internal load information of all relevant nodes and calculates the internal weight of each object storage cluster. The second calculation module 430 calculates the user-side load weight of each object storage cluster when receiving the current request from the user side. The determination response module 440 calculates the total weight based on the calculated internal weight of each object storage cluster and the calculated user-side load weight of each object storage cluster to select the object storage cluster that is most suitable for the current request and return the response information to the corresponding client.

[0087] According to an optional implementation, updating the internal load information of the object storage cluster according to the trigger condition to update the node information of all nodes includes: updating the node information of all nodes when the heartbeat information of the new node is received through the intelligent gateway, wherein each storage node corresponds to an object storage cluster, and each object storage cluster includes storage sub-nodes corresponding to multiple servers; the node information includes internal weight and user-side load weight.

[0088] According to an optional implementation, the updating of the internal load information of the object storage cluster according to the trigger conditions to update the node information of all nodes includes: when an existing node goes offline or is suddenly disconnected, updating the internal load information of the object storage cluster, and updating the node information of the remaining nodes except the existing node that is offline or suddenly disconnected, wherein each storage node corresponds to an object storage cluster, and each object storage cluster includes storage sub-nodes corresponding to multiple servers; the node information includes internal weights and user-side load weights.

[0089] According to an optional implementation, the scheduling center collects internal load information of all relevant nodes and uses the following expression (1) to calculate the internal weight of each object storage cluster, including:

[0090] Among them, Weight1 i Indicates the internal weight of all influencing factors in the object storage cluster; W11 is the weight coefficient of the CPU usage of the current object storage cluster; UsedCpu indicates the CPU usage of the current object storage cluster; CpuTotal indicates the total CPU usage of all clusters; W12 is the weight coefficient of the network IO load of the current object storage cluster; IO indicates the network IO speed of the current object storage cluster; BandWidth indicates the transmission speed of the total broadband of all clusters; W13 indicates the weight coefficient of the disk usage of the current object storage cluster; UsedSiz e Indicates the total disk usage of the current object storage cluster; Capacity indicates the total disk capacity of all clusters; W14 indicates the weight coefficient of the memory usage of the current object storage cluster; UsedRam indicates the memory usage of the current object storage cluster; RamTotal indicates the total memory usage of all clusters; LinkNum represents the total number of connection requests received in the current object storage cluster, and LinkTotal indicates the maximum number of connections that the current object storage cluster can tolerate; W1 i Indicates the weight coefficient of the i-th influencing factor in the current object storage cluster. The weight coefficient W1 i It can be freely adjusted according to different business focuses, and can also increase network traffic or other influencing factors.

[0091] According to an optional embodiment, when a user connects to the dispatch center and sends an object storage request, the dispatch center calculates the user-side load weight. Specifically, based on the IP address and location of the user request, the following expression (3) is used to calculate the user-side load weight of the physical distance between the user and each object storage cluster and the connection time:

[0092] Weight2 j=W21*time / 1000+W22*distance / 100000,...,+W2 j *other (3)

[0093] Among them, Weight2 j It represents the user-side load weight of the user and each object storage cluster when the user connects to the scheduling center and sends an object storage request; W21 represents the user-side load weight of the connection time between the user and each object storage cluster; W22 represents the user-side load weight of the physical distance between the user and each object storage cluster; W2 j Indicates the user-side load weight of the jth influencing factor, where j is a positive integer. The urgency of user requests or other user-side load information can be added according to actual conditions. Time indicates the connection time between the user and each object storage cluster in seconds. Distance indicates the physical distance between the user and each object storage cluster in km, ranging from 1km to 1000km. Weight2 j and Weight1 i The units keep the same order of magnitude.

[0094] According to an optional implementation, when receiving an object storage request from a user, the following expression (4) is used to calculate the total weight Weight 总 :

[0095] Weight 总 =m*Weight1 i +h*Weight2 j , 1≤i≤n, 1≤j≤n (4)

[0096] Among them, m represents the internal priority coefficient of all influencing factors in the object storage cluster; Weight1 i Represents the internal weight of all influencing factors in the object storage cluster. The value of m is determined according to the business application scenario parameters. h represents the priority coefficient on the user side. The value of h is determined according to the business application scenario parameters. Weight2 j Indicates the user-side load weight of each object storage cluster when a user connects to the scheduling center and sends an object storage request.

[0097] According to an optional embodiment, m and h satisfy expression (5):

[0098] m+h=1 (5)

[0099] According to an optional embodiment, it further includes: when receiving an object storage request sent by a user, first updating the node information of all nodes and calculating the user-side load weight, and then using expression (4) to calculate the total weight Weight 总 .

[0100] According to an optional implementation, the nodes of each object storage cluster are sorted according to the calculated total weight to select a server that best matches the current request, and return a storable address to the corresponding client.

[0101] It should be noted that the above is merely provided as an optional example and should not be construed as a limitation to the present invention.

[0102] It should be noted that the content of the distributed object storage scheduling method of the first aspect of the present invention is substantially the same as the content of the distributed object storage scheduling method of the second aspect of the present invention, and therefore the description of the identical parts is omitted. In addition, the figures are merely schematic illustrations of the processes included in the method according to the exemplary embodiment of the present invention and are not intended to be limiting. It is easy to understand that the processes shown in the figures do not indicate or limit the chronological order of these processes. It is also easy to understand that these processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0103] It should be noted that the above detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs.

[0104] It should be noted that the terms used herein are intended only to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0105] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0106] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0107] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used herein to describe the spatial positional relationship of a device or feature to other devices or features as shown in the figures. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figures. For example, if the device in the drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be subsequently positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0108] In the above detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless the context dictates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0109] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A distributed object storage scheduling method, characterized in that: include: Receive keep-alive messages sent by existing nodes, and update the internal load information of the object storage cluster according to the trigger conditions to update the node information of all nodes; The scheduling center collects the internal load information of all relevant nodes and calculates the internal weight of each object storage cluster; When receiving the current request from the user side, the user side load weight of each object storage cluster is calculated; According to the calculated internal weights of each object storage cluster and the calculated user-side load weights of each object storage cluster, a total weight is calculated to select the object storage cluster that best matches the current request, and return response information to the corresponding client.

2. The distributed object storage scheduling method according to claim 1, characterized in that: The updating of the internal load information of the object storage cluster according to the trigger condition to update the node information of all nodes includes: When the heartbeat information of the new node is received through the intelligent gateway, the node information of all nodes is updated, where: Each storage node corresponds to an object storage cluster, and each object storage cluster includes storage sub-nodes corresponding to multiple servers; The node information includes internal weight and user-side load weight.

3. The distributed object storage scheduling method according to claim 1, characterized in that: The updating of the internal load information of the object storage cluster according to the trigger condition to update the node information of all nodes includes: When an existing node goes offline or is suddenly disconnected, the internal load information of the object storage cluster is updated, and the node information of the remaining nodes except the existing node that is offline or suddenly disconnected is updated, where: Each storage node corresponds to an object storage cluster, and each object storage cluster includes storage sub-nodes corresponding to multiple servers; The node information includes internal weight and user-side load weight.

4. The distributed object storage scheduling method according to claim 2 or 3, characterized in that: The scheduling center collects the internal load information of all relevant nodes and uses the following expression (1) to calculate the internal weight of each object storage cluster, including: Among them, Weight1 i Represents the internal weights of all influencing factors in the object storage cluster, i is a positive integer; W11 is the weight coefficient of the CPU usage of the current object storage cluster; UsedCpu represents the CPU usage of the current object storage cluster; CpuTotal represents the total CPU of all clusters; W12 is the weight coefficient of the network IO load of the current object storage cluster; IO represents the network IO speed of the current object storage cluster; BandWidth represents the transmission speed of the total bandwidth of all clusters; W13 represents the weight coefficient of the disk usage of the current object storage cluster; UsedSize represents the total disk usage of the current object storage cluster; Capac ity represents the total disk capacity of all clusters; W14 represents the weight coefficient of the memory usage of the current object storage cluster; UsedRam represents the memory usage of the current object storage cluster; W15 represents the weight coefficient of the fifth influencing factor in the current object storage cluster, which is the relationship between the number of connection requests received in the current object storage cluster and the maximum number of connections that can be tolerated; RamTotal represents the total memory usage of all clusters; LinkNum represents the total number of connection requests received in the current object storage cluster, and LinkTotal represents the maximum number of connections that can be tolerated in the current object storage cluster; W1 i Indicates the weight coefficient of the i-th influencing factor in the current object storage cluster. The weight coefficient W1 i It can be freely adjusted according to different business focuses, and can also increase network traffic or other influencing factors.

5. The distributed object storage scheduling method according to claim 4, characterized in that: When a user connects to the dispatch center and sends an object storage request, the dispatch center calculates the user-side load weight. Specifically, based on the IP address and location of the user's request, the dispatch center uses the following expression (3) to calculate the user-side load weight of the physical distance between the user and each object storage cluster and the connection time: Weight2 j =W21*time / 1000+W22*distance / 100000、...、+ W2 j *other (3) Among them, Weight2 j W21 represents the user-side load weight of the user and each object storage cluster when the user connects to the scheduling center and sends an object storage request; W22 represents the user-side load weight of the physical distance between the user and each object storage cluster; W2 j Indicates the user-side load weight of the jth influencing factor, where j is a positive integer. The urgency of user requests or other user-side load information can be added according to actual conditions; time indicates the time it takes to connect the user to each object storage cluster, in seconds; distance indicates the physical distance between the user and each object storage cluster, in km, between 1km and 1000km, and Weight2 j and Weight1 i The units keep the same order of magnitude.

6. The distributed object storage scheduling method according to claim 5, characterized in that: When receiving an object storage request from a user, the following expression (4) is used to calculate the total weight: 总 : Weight 总 =m*Weight1 i +h*Weight2 j ,1≤i≤n,1≤j≤n (4) Where m represents the internal priority coefficient of all influencing factors in the object storage cluster; Weight1 i Represents the internal weight of all influencing factors in the object storage cluster. The value of m is determined based on the business application scenario parameters. h represents the priority coefficient on the user side. The value of h is determined based on the business application scenario parameters. Weight2 j Indicates the user-side load weight of each object storage cluster when a user connects to the scheduling center and sends an object storage request.

7. The distributed object storage scheduling method according to claim 6, characterized in that: m and h satisfy expression (5): m+h=1 (5).

8. The distributed object storage scheduling method according to claim 1, characterized in that: Further including: When receiving an object storage request from a user, the node information of all nodes is updated first and the user-side load weight is calculated. Then, the total weight Weight is calculated using expression (4). 总 .

9. The distributed object storage scheduling method according to claim 6, characterized in that: According to the calculated total weight, the nodes of each object storage cluster are sorted to select the server that best matches the current request and return the storable address to the corresponding client.

10. A distributed object storage scheduling system, characterized in that: The distributed object storage scheduling system includes: A receiving processing module receives keep-alive messages sent by existing nodes, and updates the internal load information of the object storage cluster according to the triggering conditions to update the node information of all nodes; In the first calculation module, the scheduling center collects the internal load information of all relevant nodes and calculates the internal weight of each object storage cluster; The second calculation module calculates the user of each object storage cluster when receiving the current request from the user side side load weight; A response module is determined, and a total weight is calculated according to the calculated internal weights of each object storage cluster and the calculated user-side load weights of each object storage cluster to select the object storage cluster that best matches the current request, and return response information to the corresponding client.

Images