Cluster construction method, apparatus, device, medium, and product

Abstract

This invention relates to the field of distributed storage technology and discloses a cluster construction method, apparatus, device, medium, and product. The method includes: when a target cluster is initialized, calculating the OSD affinity weight of each node based on the performance parameters of each node in the target cluster; determining the unit weight of each node based on the number of data sets to be stored and the OSD affinity weight of each node; for any data set to be stored, selecting the primary OSD from the target node with the highest OSD affinity weight among all nodes, and updating the affinity weight of the target node according to the unit weight of the target node. This scheme achieves comprehensive calculation of factors such as computing power in heterogeneous distributed storage clusters, ensuring that high-performance nodes carry more primary OSDs, thereby improving the overall performance of the cluster.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, specifically to a cluster construction method, apparatus, device, medium, and product. Background Technology

[0002] With the continuous development of information technology, the explosive growth of data volume has brought new challenges to storage systems. How to build high-performance, scalable, and highly stable storage systems to support massive data storage is a key challenge.

[0003] Distributed storage systems based on the Ceph open-source architecture are currently a mainstream type of distributed storage system. In a Ceph distributed storage system, data striping results in multiple objects. These objects are mapped to Groups (PGs) using a hash algorithm, and then further mapped to Operating Devices (OSDs) by the PGs. This mapping is determined by the CRUSH algorithm, with the PG ID as the input. This results in a set containing three (three replicas) OSDs. The first OSD in the set is designated as the master OSD, and the others are designated as slave OSDs in sequence.

[0004] However, in the above scheme, the computing power of chips with different architectures varies greatly. The effect of determining the main OSD selection according to the general OSD affinity algorithm is poor, which will lead to low performance utilization of the distributed storage cluster. Summary of the Invention

[0005] In view of this, this application provides a cluster construction method, apparatus, device, medium and product, which improves the performance of the cluster. The technical solution is as follows.

[0006] Firstly, a cluster construction method is provided, the method comprising:

[0007] When the target cluster is initialized, the OSD affinity weight of each node is calculated based on the performance parameters of each node in the target cluster.

[0008] The unit weight of each node is determined based on the number of data sets of objects to be stored and the OSD affinity weight of each node.

[0009] For any set of object data to be stored, select the primary OSD from the target node with the highest OSD affinity weight among all nodes, and update the affinity weight of the target node according to the unit weight of the target node.

[0010] In one possible implementation, updating the affinity weight of the target node based on the unit weight of the target node includes:

[0011] The difference between the affinity weight of the target node and the unit weight of the target node is used to update the new affinity weight of the target node.

[0012] In one possible implementation, determining the unit weight of each node based on the number of data sets of objects to be stored and the OSD affinity weight of each node includes:

[0013] Based on the number of the data set of objects to be stored and the number of storage nodes in the target cluster, obtain the average number of primary OSDs carried on the storage nodes;

[0014] The ratio of the OSD affinity weight of each node to the average number of master OSDs is determined as the unit weight of each node.

[0015] In one possible implementation, the performance parameters include processor computing power, physical memory, and network bandwidth;

[0016] The step of calculating the OSD affinity weight of each node based on the performance parameters of each node in the target cluster includes:

[0017] Obtain the performance weight corresponding to each performance parameter of each node; the sum of the performance weights of each performance parameter is 1;

[0018] Based on the performance parameters and corresponding performance weights of each node, the OSD affinity weight of each node is obtained.

[0019] In one possible implementation, the method further includes:

[0020] When the first node of the target cluster changes, determine each second node associated with the data of the first node and the first set of object data to be stored on the first node;

[0021] Obtain the current affinity weight of each of the second nodes;

[0022] Based on the current affinity weight of each of the second nodes, the primary OSD corresponding to the first set of objects to be stored is selected from each of the second nodes.

[0023] In one possible implementation, obtaining the current affinity weights of each of the second nodes includes:

[0024] Obtain the processor utilization, memory utilization, and network bandwidth utilization of each second node;

[0025] The current affinity weight of the second node is calculated based on the processor utilization, memory utilization, network bandwidth utilization, and the performance weights corresponding to each performance parameter of each second node.

[0026] Secondly, a cluster building apparatus is provided, the apparatus comprising:

[0027] The affinity weight calculation module is used to calculate the OSD affinity weight of each node based on the performance parameters of each node in the target cluster when the target cluster is initialized.

[0028] The unit weight acquisition module is used to determine the unit weight of each node based on the number of data sets of objects to be stored and the OSD affinity weight of each node.

[0029] The node selection module is used to select the primary OSD from the target node with the highest OSD affinity weight among all nodes for any set of objects to be stored, and update the affinity weight of the target node according to the unit weight of the target node.

[0030] Thirdly, a computer device is provided, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the aforementioned cluster construction method by executing the computer instructions.

[0031] Fourthly, a computer-readable storage medium is provided, on which computer instructions are stored, the computer instructions being used to cause a computer to execute the above-described cluster construction method.

[0032] Fifthly, a computer program product or computer program is provided, including computer instructions for causing a computer to execute the cluster construction method described above.

[0033] The technical solution provided in this application may include the following beneficial effects:

[0034] When the target cluster is initialized, the OSD affinity weight of each node can be calculated based on the performance parameters of each node in the target cluster. Then, based on the number of data sets to be stored and the OSD affinity weight of each node, the unit weight of each node is determined. For any data set to be stored, the primary OSD is selected from the target node with the highest OSD affinity weight among all nodes, and the affinity weight of the target node is updated according to the unit weight of the target node. At this time, the next data set to be stored will select an OSD based on the updated affinity weight of the target node and the affinity weights of other nodes. The above scheme realizes the overall calculation of factors such as computing power in heterogeneous distributed storage clusters, ensuring that high-performance nodes carry more primary OSDs, thereby improving the overall performance of the cluster. Attached Figure Description

[0035] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 This illustration shows a schematic diagram of the mapping relationship between PG and OSD in a heterogeneous distributed storage cluster according to an embodiment of this application.

[0037] Figure 2 This is a flowchart illustrating a cluster construction method according to an exemplary embodiment.

[0038] Figure 3 This is a flowchart illustrating a cluster construction method according to an exemplary embodiment.

[0039] Figure 4 This illustration shows a schematic diagram of a heterogeneous distributed storage cluster variation according to an embodiment of this application.

[0040] Figure 5 This is a schematic diagram of a cluster building device provided in an embodiment of this application.

[0041] Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In the description of the embodiments of this application, the term "correspondence" may indicate that there is a direct or indirect correspondence between two things, or that there is an association between two things, or that there is a relationship of instruction and being instructed, configuration and being configured, etc.

[0044] In Ceph's distributed storage system, data striping results in multiple objects. These objects are mapped to Groups (PGs) using a hash algorithm, and then further mapped to Operating Devices (OSDs) by the PGs. This mapping is determined by the CRUSH algorithm, with the PG ID as input. This results in a set of three (three replicas) OSDs. The first OSD in the set is designated as the master OSD, and the others are designated as slave OSDs. The master OSD handles the majority of IO reads and writes for that PG and has the greatest impact on the overall performance of the distributed storage. In real-world production scenarios, the selection of the master OSD needs to consider factors such as OSD affinity and fault domain. Higher affinity increases the likelihood of it being selected as the master OSD. In heterogeneous computing scenarios, determining OSD affinity in Ceph distributed storage can be more challenging because different types of hardware and storage devices may differ in performance, capacity, and other aspects. In such cases, you need to carefully consider the following factors to determine OSD affinity:

[0045] Hardware Performance: Evaluate the hardware performance of different OSD nodes, including CPU, memory, storage speed, and network bandwidth. Depending on the hardware specifications, you can create different CRUSH rules for different types of hardware devices.

[0046] CRUSH rules: Use CRUSH rules to define OSD affinity. Create different CRUSH rules for different types of hardware devices to ensure that data objects are mapped to OSDs that suit their needs.

[0047] Dynamic adjustment: Monitor cluster performance and data access patterns, and dynamically adjust OSD affinity as needed. This helps optimize performance and balance the load across hardware devices.

[0048] Please refer to Figure 1 This illustrates a schematic diagram of the mapping relationship between PG and OSD in a heterogeneous distributed storage cluster according to an embodiment of this application. Figure 1As shown, this cluster includes several heterogeneous resources, namely several nodes including Node1, Node2, Node3, Node4, etc. In distributed storage, after data striping, it is mapped to PGs as objects. The final data storage of a PG needs to be mapped to OSDs using the CRUSH algorithm. Taking multi-replica storage as an example, a single PG will be mapped to N OSDs. The first OSD in the set will be used as the master OSD, and the other OSDs will be used as slave OSDs in sequence. N is the number of replicas, which is usually 3 in a production environment. The master OSD is the node with the most IO processing in data reading and writing, and requires more resources. Therefore, for distributed storage clusters with different computing power and configuration, computing power weight should be used as an important factor in OSD affinity calculation during initialization. Combined with fault domain factors, PGs are mapped to OSDs with different affinities. The higher the affinity, the greater the probability of being selected as the master OSD, in order to improve the overall performance of the cluster.

[0049] Figure 2 This is a flowchart illustrating a cluster construction method according to an exemplary embodiment. The method is executed by a computer device. Figure 2 As shown, the cluster construction method may include the following steps:

[0050] Step 201: When the target cluster is initialized, calculate the OSD affinity weight of each node based on the performance parameters of each node in the target cluster.

[0051] When initializing the target cluster, the performance parameters of each node need to be considered, including CPU, memory, storage speed, and network bandwidth. The hardware performance of each node directly affects its load capacity and efficiency in handling storage operations.

[0052] Hardware performance is evaluated for each OSD node to obtain its hardware performance metrics (such as CPU performance, memory size, storage media speed, network bandwidth, etc.). Then, an "affinity weight" is assigned to each node based on these performance metrics. This weight reflects the node's suitability for processing specific data. The larger the weight, the stronger the node's performance and the more suitable it is for storing and processing that data.

[0053] Step 202: Determine the unit weight of each node based on the number of data sets of objects to be stored and the OSD affinity weight of each node.

[0054] The unit weight of each node is determined based on the number of object datasets to be stored (i.e., the number of PGs) and the OSD affinity weight of each node, in order to optimize load balancing and ensure the efficiency of each node in processing storage requests.

[0055] Specifically, the number of data sets to be stored represents the total amount of data to be stored. Typically, these data sets may contain data objects of varying sizes, with different storage requirements. Unit weight, by combining the affinity weight of each node with the number of data sets to be stored, determines the node's load capacity under specific conditions. Nodes with higher unit weights can handle more storage requests and have stronger performance in processing this data.

[0056] Step 203: For any set of object data to be stored, select the primary OSD from the target node with the highest OSD affinity weight among all nodes, and update the affinity weight of the target node according to the unit weight of the target node.

[0057] For each dataset to be stored, an OSD is selected as the primary OSD. This primary OSD will handle the majority of read and write loads during data storage and access. Simultaneously, the affinity weights of target nodes are updated to reflect the selection of the primary OSD and the change in its weight. Specifically, for each dataset to be stored, the target node with the highest affinity weight and best performance is selected as the primary OSD based on the affinity weight of each node. The primary OSD will be responsible for the main read and write operations of that dataset. Once the primary OSD is selected and the dataset is allocated, the system dynamically adjusts the affinity weight of that node based on its unit weight. For example, if a node becomes the primary OSD and handles more load, its affinity weight may change (e.g., decrease), helping to distribute subsequent data to nodes with lower loads, thereby maintaining load balance in the storage cluster, avoiding overload of a single node, and maximizing the overall system performance.

[0058] In summary, when the target cluster is initialized, the OSD affinity weight of each node can be calculated based on the performance parameters of each node in the target cluster. Then, based on the number of data sets to be stored and the OSD affinity weight of each node, the unit weight of each node is determined. For any data set to be stored, the primary OSD is selected from the target node with the highest OSD affinity weight among all nodes, and the affinity weight of the target node is updated according to the unit weight of the target node. At this time, the next data set to be stored will select OSDs based on the updated affinity weight of the target node and the affinity weights of other nodes. The above scheme realizes the overall calculation of factors such as computing power in heterogeneous distributed storage clusters, ensuring that high-performance nodes carry more primary OSDs, thereby improving the overall performance of the cluster.

[0059] Figure 3 This is a flowchart illustrating a cluster construction method according to an exemplary embodiment. The method is executed by a computer device and may include the following steps:

[0060] Step 301: When the target cluster is initialized, calculate the OSD affinity weight of each node based on the performance parameters of each node in the target cluster.

[0061] Optionally, the performance parameters include processor computing power (Flops), actual memory (Mem), and network bandwidth (Net); obtain the performance weights corresponding to each performance parameter of each node; the sum of the performance weights of each performance parameter is 1; obtain the OSD affinity weight of each node based on each performance parameter and its corresponding performance weight.

[0062] In a typical distributed storage cluster, the number of Group Publishers (PGs) is M, which is usually an integer power of 2, such as 4096 PGs. Therefore, during cluster initialization, there is a selection of M master OSDs.

[0063] Assuming the cluster has N distributed storage servers, and considering that the resource consumption of host nodes in distributed storage mainly includes three factors: CPU, memory, and network, then the OSD affinity weight value of the node is set as follows:

[0064] W = Flops * W cpu +Mem*W mem +Net*W net

[0065] Among them, computing power (Flops) is calculated based on different CPU computing power, architecture, threads, etc., memory is calculated based on the actual memory value, and network mainly calculates bandwidth. Meanwhile, the computing power weight W... cpu Memory weight W mem Network weight W net Determined based on the actual scenario, W cpu / W mem / W net The sum of the three weight values ​​is 1.

[0066] In distributed storage, data striping, hash value calculation, and the core Crush addressing algorithm all consume significant CPU resources. The computing power of CPUs varies considerably across different architectures, significantly impacting the performance of distributed storage clusters. Therefore, a unified computing power evaluation method is needed to assess node performance, upon which OSD affinity design and distribution can be based. This patent proposes a computing power evaluation method for CPU resources with different architectures. CPU computing power is affected by factors such as CPU architecture, clock speed, number of cores, cache capacity and hierarchy, instruction set parallelism, hyper-threading, heat dissipation, and power consumption. For distributed storage scenarios, the focus is on architecture performance factors, clock speed, number of cores, and hyper-threading. The computing power evaluation function is as follows:

[0067] Flops = F * Core * Thread * Arch * HT

[0068] in:

[0069] Flops represent the final computing power result.

[0070] The architecture factor (Arch) is strongly correlated with the CPU architecture and the degree of instruction set optimization. This parameter varies depending on the architecture and CPU manufacturer. It can be quantitatively evaluated based on the specific CPU model.

[0071] Thread: CPU thread.

[0072] Clock speed F: CPU core frequency, measured in Hertz (Hz).

[0073] Core: The number of CPU cores, representing the parallel processing capability of multiple cores.

[0074] Hyper-threading factor HT: 2 if hyper-threading is supported, 1 if not.

[0075] Step 302: Based on the number of data sets of objects to be stored and the number of storage nodes in the target cluster, obtain the average number of primary OSDs carried on the storage nodes.

[0076] The number of data sets to be stored represents the amount of data the system needs to store. The number of storage nodes refers to the total number of available OSD nodes in the entire cluster. More nodes result in a more even distribution of load, preventing individual nodes from becoming overloaded. The primary OSD is responsible for the main read and write operations; each storage node in the cluster hosts a certain number of primary OSDs. Calculating the average number of primary OSDs hosted by each node helps determine the load on each node. If some nodes host more primary OSDs than the average, load balancing or reallocation may be necessary.

[0077] Step 303: The ratio of the OSD affinity weight of each node to the average number of master OSDs is determined as the unit weight of each node.

[0078] Specifically, after calculating the affinity weights of the nodes, they are W1W2W3...W N To ensure data balance in distributed storage, each storage node typically has the same number of OSDs. Since a total of M PGs need to map to M logical master OSDs, the average number of master OSDs that each storage node needs to support is M / N. Therefore, the affinity weight W for each master OSD is... osd-pg It can be calculated as:

[0079] W osd-pg =W N / (M / N)=W N *N / M

[0080] Step 304: For any set of object data to be stored, select the primary OSD from the target node with the highest OSD affinity weight among all nodes.

[0081] For each dataset, the system will select the node with the highest OSD affinity weight as the master OSD. This selection is based on the node's hardware performance and load balancing requirements. Therefore, this step can select the node with the best performance so that the cluster can handle data read and write operations more effectively and improve the overall system performance.

[0082] Step 305: Update the target node's new affinity weight by the difference between the target node's affinity weight and its unit weight.

[0083] Once a primary OSD is selected, the system needs to update the affinity weights of the target nodes based on the primary OSD selection result. This is to dynamically adjust the affinity weights during the next primary OSD selection, ensuring load balancing. The updated affinity weights should take into account the current node load and differences in unit weights. For example, if a node is handling more primary OSDs, its affinity weight may need to be reduced accordingly to prevent it from becoming a bottleneck for cluster performance.

[0084] Optionally, when the first node of the target cluster changes, determine each second node associated with the data of the first node and the first set of object data to be stored on the first node; obtain the current affinity weight of each second node; and select the master OSD corresponding to the first set of object data to be stored in each second node according to the current affinity weight of each second node.

[0085] Optionally, obtain the processor utilization rate, memory utilization rate, and network bandwidth utilization rate of each second node; calculate the current affinity weight of the second node based on the processor utilization rate, memory utilization rate, network bandwidth utilization rate, and the performance weight corresponding to each performance parameter of each second node.

[0086] When nodes are added or removed from the cluster, some PGs (PGs) will experience data synchronization issues or new OSD (Optical Server) leader elections. Most PGs in the cluster remain unaffected, thus maintaining cluster stability. During cluster changes, OSD affinity is determined based on real-time node computing power assessment factors, thereby enabling the selection of a new OSD leader for the affected PGs.

[0087] The computing power assessment of nodes affected by cluster changes uses real-time CPU utilization, memory utilization, and network bandwidth utilization to calculate the current affinity weight value, as shown in the following formula:

[0088] W = Flops * (1 - Cpu%) * W cpu +Mem*(1-Mem%)*W mem +Net*(1-Net%)*W net

[0089] The calculation focused on the percentage of CPU, memory, and network usage, and then compared the remaining computing power with the newly added nodes based on affinity weight values.

[0090] The OSD affinity algorithm for re-electing the master PG affected by cluster changes is consistent with the aforementioned method. It dynamically compares weight values ​​to determine the OSD with the highest weight affinity as the master OSD, ensuring maximum cluster performance. After all affected PGs have completed master election and data migration or synchronization, the cluster becomes stable again and provides the highest cluster performance.

[0091] Please refer to Figure 4 This illustration shows a schematic diagram of a heterogeneous distributed storage cluster variation according to an embodiment of this application. Figure 4 As shown, when expanding the cluster, some PGs involved in the expansion process are identified. For any PG, the new node NodeX of the expansion is taken into account, and the master OSD of each PG is re-determined according to the affinity weight value.

[0092] In summary, when the target cluster is initialized, the OSD affinity weight of each node can be calculated based on the performance parameters of each node in the target cluster. Then, based on the number of data sets to be stored and the OSD affinity weight of each node, the unit weight of each node is determined. For any data set to be stored, the primary OSD is selected from the target node with the highest OSD affinity weight among all nodes, and the affinity weight of the target node is updated according to the unit weight of the target node. At this time, the next data set to be stored will select OSDs based on the updated affinity weight of the target node and the affinity weights of other nodes. The above scheme realizes the overall calculation of factors such as computing power in heterogeneous distributed storage clusters, ensuring that high-performance nodes carry more primary OSDs, thereby improving the overall performance of the cluster.

[0093] This application also provides a cluster building apparatus for implementing the above embodiments and preferred embodiments, which will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0094] This application provides a cluster building device. Figure 5This is a schematic diagram of a cluster building device provided in an embodiment of this application. The device includes:

[0095] The affinity weight calculation module 501 is used to calculate the OSD affinity weight of each node based on the performance parameters of each node in the target cluster when the target cluster is initialized.

[0096] The unit weight acquisition module 502 is used to determine the unit weight of each node based on the number of data sets of objects to be stored and the OSD affinity weight of each node.

[0097] The node selection module 503 is used to select the primary OSD from the target node with the highest OSD affinity weight among all nodes for any set of objects to be stored, and update the affinity weight of the target node according to the unit weight of the target node.

[0098] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0099] In this embodiment, the cluster building device is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0100] This invention also provides a computer device having the above-described features. Figure 5 The cluster building device shown.

[0101] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information in a graphical user interface on an external input / output device (such as a display device coupled to the interface). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6Take a processor 10 as an example.

[0102] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0103] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0104] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0106] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0107] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0108] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0109] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the invention.

Claims

1. A cluster construction method, characterized in that, The method includes: When the target cluster is initialized, the OSD affinity weight of each node is calculated based on the performance parameters of each node in the target cluster. The unit weight of each node is determined based on the number of data sets of objects to be stored and the OSD affinity weight of each node. For any set of object data to be stored, select the primary OSD from the target node with the highest OSD affinity weight among all the nodes, and update the affinity weight of the target node according to the unit weight of the target node. The step of determining the unit weight of each node based on the number of data sets of objects to be stored and the OSD affinity weight of each node includes: Based on the number of the data set of objects to be stored and the number of storage nodes in the target cluster, obtain the average number of primary OSDs carried on the storage nodes; The ratio of the OSD affinity weight of each node to the average number of master OSDs is determined as the unit weight of each node. The step of updating the affinity weight of the target node based on the unit weight of the target node includes: The difference between the affinity weight of the target node and the unit weight of the target node is used to update the new affinity weight of the target node.

2. The method according to claim 1, characterized in that, The performance parameters include processor computing power, actual memory, and network bandwidth; The step of calculating the OSD affinity weight of each node based on the performance parameters of each node in the target cluster includes: Obtain the performance weight corresponding to each performance parameter of each node; the sum of the performance weights of each performance parameter is 1; Based on the performance parameters and corresponding performance weights of each node, the OSD affinity weight of each node is obtained.

3. The method according to claim 2, characterized in that, The method further includes: When the first node of the target cluster changes, determine each second node associated with the data of the first node and the first set of object data to be stored on the first node; Obtain the current affinity weight of each of the second nodes; Based on the current affinity weight of each of the second nodes, the primary OSD corresponding to the first set of objects to be stored is selected from each of the second nodes.

4. The method according to claim 3, characterized in that, The step of obtaining the current affinity weight of each of the second nodes includes: Obtain the processor utilization, memory utilization, and network bandwidth utilization of each second node; The current affinity weight of the second node is calculated based on the processor utilization, memory utilization, network bandwidth utilization, and the performance weights corresponding to each performance parameter of each second node.

5. A cluster construction device, characterized in that, The device includes: The affinity weight calculation module is used to calculate the OSD affinity weight of each node based on the performance parameters of each node in the target cluster when the target cluster is initialized. The unit weight acquisition module is used to determine the unit weight of each node based on the number of data sets of objects to be stored and the OSD affinity weight of each node. The node selection module is used to select the primary OSD from the target node with the highest OSD affinity weight among all nodes for any set of objects to be stored, and update the affinity weight of the target node according to the unit weight of the target node. Specifically, the unit weight acquisition module is used for: Based on the number of the data set of objects to be stored and the number of storage nodes in the target cluster, obtain the average number of primary OSDs carried on the storage nodes; The ratio of the OSD affinity weight of each node to the average number of master OSDs is determined as the unit weight of each node. The node selection module is specifically used for: The difference between the affinity weight of the target node and the unit weight of the target node is used to update the new affinity weight of the target node.

6. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory storing computer instructions, and the processor executing the computer instructions to perform the cluster construction method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the cluster construction method according to any one of claims 1 to 4.

8. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the cluster construction method according to any one of claims 1 to 4.

Images