Data storage method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data storage method, electronic equipment and a computer readable storage medium. According to the method, the disks are graded according to the obtained disk information of the disks, resource pools of multiple grades are obtained, the resource pool of each grade comprises at least one sub-resource pool, and the sub-resource pools comprise the disks of the same grade; according to the obtained data information of the to-be-stored data, selecting a target-level resource pool from the resource pools of all levels; determining a target resource pool from the target level resource pool according to the number of the sub-resource pools in the target level resource pool; and storing the to-be-stored data in the target resource pool. Therefore, the resource utilization rate is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data storage method, electronic device, and computer-readable storage medium. Background Technology

[0002] In today's society, information is ubiquitous, and one of the key processes in processing information is data storage. Data storage is the process of preserving and managing information through specific technologies and storage media, and the methods used for data storage determine the efficiency and reliability of data processing.

[0003] Current data storage methods typically divide HDD (Hard Disk Drive Pool) and SSD (Solid State Drive) types into separate resource pools, and then store the data in a fixed resource pool. However, this fixed resource pool approach is not flexible enough and has low resource utilization. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a data storage method, electronic device, and computer-readable storage medium that can improve computational efficiency.

[0005] To address the aforementioned technical problems, this application provides a data storage method, comprising: classifying each disk according to acquired disk information to obtain multiple levels of resource pools, each level of resource pool including at least one sub-resource pool, and each sub-resource pool including disks of the same level; selecting a target level resource pool from the resource pools of each level according to acquired data information of the data to be stored; determining a target resource pool from the target level resource pool according to the number of sub-resource pools in the target level resource pool; and storing the data to be stored in the target resource pool.

[0006] In one embodiment, the data information includes the required resource quantity and data type of the data to be stored. The step of selecting a target level resource pool from resource pools of each level based on the acquired data information of the data to be stored includes: selecting a first candidate resource pool from resource pools of each level based on the required resource quantity of the data to be stored; determining a second candidate resource pool from each first candidate resource pool based on the node health status of the nodes where each first candidate resource pool is located; and determining a target level resource pool from each second candidate resource pool based on the data type of the data to be stored.

[0007] In one embodiment, the node health status includes node abnormality and node normality. Before the step of determining the second candidate resource pool from each of the first candidate resource pools based on the obtained node health status of the nodes where each first candidate resource pool is located, the method further includes: obtaining the node status score of the node where the first candidate resource pool is located; in response to the node status score being greater than a preset score threshold, determining the node health status of the corresponding node as node abnormality; in response to the node status score being less than or equal to the preset score threshold, determining the node health status of the corresponding node as node normality.

[0008] In one embodiment, the step of obtaining the node status score of the node where the first candidate resource pool is located includes: obtaining the load information, hardware information, and performance information of the node where the first candidate resource pool is located; and performing weighted processing on the load information, hardware information, and performance information of the node where the first candidate resource pool is located to obtain the node status score of the node where the first candidate resource pool is located.

[0009] In one embodiment, the step of determining the target level resource pool from each of the second candidate resource pools based on the data type of the data to be stored includes: determining the second candidate resource pools of the candidate level from each of the second candidate resource pools based on the data type of the data to be stored; and performing matching processing on the second candidate resource pools of the candidate level based on the erasure coding configuration amount of the data to be stored to obtain the target level resource pool.

[0010] In one embodiment, the candidate levels include a first level and a second level. The step of matching the second candidate resource pool of the candidate levels with the erasure coding configuration amount of the data to be stored to obtain the target level resource pool includes: determining whether the resource configuration amount of the second candidate resource pool of the first level meets the erasure coding configuration amount of the data to be stored, wherein the storage performance of the resource pool of the first level is better than that of the second level; if yes, then the second candidate resource pool of the first level is determined as the target level resource pool; if no, then in response to the resource configuration amount of the second candidate resource pool of the second level meeting the erasure coding configuration amount of the data to be stored, the second candidate resource pool of the second level is determined as the target level resource pool.

[0011] In one embodiment, the step of determining the target resource pool from the target level resource pool based on the number of sub-resource pools in the target level resource pool includes: in response to the existence of a sub-resource pool in the target level resource pool, the sub-resource pool is used as the target resource pool; In response to the existence of at least two sub-resource pools in the target level resource pool, the node status score of the node where each sub-resource pool is located is obtained; the sub-resource pools are sorted according to the node status score of the node where each sub-resource pool is located to obtain sorted sub-resource pools; the sub-resource pool with a preset sorting position is selected from the sorted sub-resource pools as the target resource pool; and the data to be stored is stored in the target resource pool.

[0012] In one embodiment, the disk information includes disk load information and disk health information. The step of classifying each disk according to the acquired disk information to obtain multiple levels of resource pools includes: weighting the disk load information and the disk health information to obtain a health score for each disk; determining the performance level of each disk based on the range of its health score; determining the resource pool level of each disk based on its performance level and type; and allocating each disk to a corresponding level of resource pool according to its resource pool level to obtain multiple levels of resource pools.

[0013] To address the aforementioned technical problems, this application provides an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the aforementioned data storage method.

[0014] To address the aforementioned technical problems, this application provides a computer-readable storage medium, comprising: storing program data, wherein the program data, when executed by a processor, is used to implement the aforementioned data storage method.

[0015] The above scheme categorizes disks based on their acquired disk information, resulting in multiple levels of resource pools. Each level of resource pool includes at least one sub-resource pool, and each sub-resource pool contains disks of the same level. A target level resource pool is selected from these pools based on the acquired data information of the data to be stored. The target resource pool is then determined from the target level resource pool based on the number of sub-resource pools within it. Finally, the data to be stored is stored in the target resource pool. Thus, by categorizing the disks and obtaining multiple levels of resource pools, the resource pools are dynamically divided into different levels, improving the flexibility of storing data and ultimately increasing resource utilization. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a flowchart illustrating an exemplary embodiment of the data storage method shown in this application; Figure 2 This is a schematic diagram of an exemplary embodiment of the conversion process of resource pools of various levels shown in this application; Figure 3 This is a timing diagram illustrating an exemplary embodiment of obtaining a disk health score as shown in this application; Figure 4 yes Figure 1 A schematic flowchart of an exemplary embodiment of step S120 in the data storage method is shown; Figure 5 This is a schematic flowchart illustrating an exemplary embodiment of obtaining a target resource pool as shown in this application; Figure 6 This is a block diagram illustrating a data storage device in an exemplary embodiment of this application; Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Figure 8 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] First, it's important to note that in today's society, information is ubiquitous, and one of the key processes in information processing is data storage. Data storage is the process of preserving and managing information using specific technologies and storage media. The method of data storage determines the efficiency and reliability of data processing. Current data storage methods typically divide HDD (Hard Disk Drive Pool) and SSD (Solid State Drive) types into separate resource pools, and then store the data in a fixed resource pool. However, this fixed resource pool approach is inflexible and has low resource utilization.

[0019] Based on this, this application provides a data storage method, an electronic device, and a computer-readable storage medium. For details, please refer to [link / reference needed]. Figure 1 , Figure 1This is a schematic flowchart of an exemplary embodiment of a data storage method shown in this application.

[0020] The execution entity of a data storage method can be a terminal device, a server, or other processing device. The terminal device can be a computer, mobile device, terminal, computing device, vehicle-mounted device, etc. The execution entity of a data storage method can also be a data storage device. In some possible implementations, the data storage method can be implemented by a processor calling computer-readable instructions stored in memory. The execution entity of a data storage method can also be a big data cluster. A big data cluster is a computer system architecture formed by multiple computers connected through a network. Big data clusters can be deployed on a private cloud built using Kubernetes (a container orchestration engine).

[0021] The data storage method is applied to a data storage system, which includes multiple nodes, and each node includes multiple resource pools. A node can be a physical machine.

[0022] Specifically, one data storage method in this embodiment includes the following steps: Step S110: Based on the obtained disk information of each disk, perform hierarchical processing on each disk to obtain multiple levels of resource pools. Each level of resource pool includes at least one sub-resource pool, and each sub-resource pool includes disks of the same level.

[0023] A disk is a hardware device used to store data. Disk information refers to information related to a disk. Disk information can include static disk attributes and dynamic disk attributes. Static disk attributes include storage media type, total capacity, vendor code, interface protocol, and disk-specific IOPS, etc. Dynamic disk attributes can include disk bad blocks, average latency, current disk I / O utilization, disk wear, etc.

[0024] Specifically, the data storage system obtains the static attribute set of each disk reported when the node starts up, thus obtaining the disk static attributes; it also periodically collects the dynamic attribute set of each disk, thus obtaining the disk dynamic attributes. For example, the disk attribute set is {Type, Capacity, Vendor, Protocol, Firmware, Max_IOPS}, where Type is the storage medium type of the disk, Type∈{HDD, SSD, NVMe (Non-Volatile Memory Express)}; Capacity is the total capacity, in TB (Terabyte); Vendor is the vendor code; Protocol is the interface protocol; Firmware is the firmware version number; and Max_IOPS is the disk's rated IOPS (Input / Output Operations Per Second). For example, the dynamic attribute set of a disk is {Bad_Blockst, Latencyt, IO_Utilizationt, Weart}, where Bad_Blockst is the current number of bad blocks on the disk, Latencyt is the average latency, IO_Utilizationt is the current I / O (Input / Output) utilization of the disk, and Weart is the disk wear level.

[0025] A resource pool comprises multiple sub-resource pools. Each sub-resource pool contains multiple disks of the same priority. Sub-resource pools may also include GPUs (Graphics Processing Units), etc.

[0026] In one embodiment, the multiple levels may include a first level, a second level, and a third level, wherein the resource pool of the first level is a high-performance pool, the resource pool of the second level is a degraded pool, and the resource pool of the third level is a decommissioned pool. The disks in the high-performance pool have superior performance and are suitable for handling low-latency, high-throughput services. The disks in the degraded pool have partially degraded performance or reliability but can still handle non-critical tasks. The disks in the decommissioned pool have the worst performance, only support read-only mode, prohibit rewriting, and can only perform the task of temporarily storing data. The storage performance of the first-level resource pool is better than that of the second-level resource pool, and the storage performance of the second-level resource pool is better than that of the third-level resource pool.

[0027] In another embodiment, the multiple tiers may include a first tier of SSD type, a second tier of SSD type, a third tier of SSD type, a first tier of HDD type, a second tier of HDD type, and a third tier of HDD type. The SSD type resource pool contains only SSD type disks, and the HDD type resource pool contains only HDD type disks.

[0028] The data storage system categorizes each disk based on its acquired disk information, resulting in multiple levels of resource pools. Specifically, the system obtains the health score of each disk based on its disk information; determines the resource pool level of each disk based on its health score; allocates each disk to a corresponding sub-resource pool according to its resource pool level, resulting in multiple levels of sub-resource pools; and combines sub-resource pools of the same level into a single resource pool, thus creating multiple levels of resource pools.

[0029] For example, disk information includes the number of bad blocks, the total number of disks, and disk wear. The data storage system obtains a first ratio between the number of bad blocks and the total number of disks, and a first product term between the first ratio and a preset bad block weight. It also obtains a second ratio between the disk wear and a preset wear threshold, and a second product term between the second ratio and a preset wear weight. The system calculates the sum of the first and second product terms, and determines the difference between the preset scoring threshold and this sum as the health score of each disk. Disks with health scores greater than or equal to the first scoring threshold are identified as disks in the first-level resource pool, disks with health scores less than the first scoring threshold but greater than or equal to the second scoring threshold are identified as disks in the second-level resource pool, and disks with health scores less than the second scoring threshold are identified as disks in the third-level resource pool.

[0030] Step S120: Select the target level resource pool from the resource pools of each level based on the data information of the data to be stored.

[0031] Data information refers to the information about the data to be stored. Data information includes the required resources and data type of the data to be stored.

[0032] The target level resource pool refers to the resource pool at the target level. The target level refers to the level ultimately selected.

[0033] The data storage system selects a target-level resource pool from resource pools of various levels based on the acquired data information of the data to be stored. As an example, the data storage system selects a first candidate resource pool from resource pools of various levels based on the required resource quantity of the data to be stored; and determines the target-level resource pool from each of the first candidate resource pools based on the obtained node health status of the nodes where each first candidate resource pool is located. As another example, the data storage system selects a first candidate resource pool from resource pools of various levels based on the required resource quantity of the data to be stored; and determines the target-level resource pool from each of the first candidate resource pools based on the data type of the data to be stored.

[0034] Step S130: Determine the target resource pool from the target resource pool based on the number of sub-resource pools in the target level resource pool.

[0035] The data storage system determines the target resource pool from the target-level resource pool based on the number of sub-resource pools in the target-level resource pool. Specifically, the data storage system selects a sub-resource pool as the target resource pool if the number of sub-resource pools in the target-level resource pool is a preset value. The preset value is 1.

[0036] Step S140: Store the data to be stored in the target resource pool.

[0037] The data storage system stores the data to be stored on the disk of the target resource pool.

[0038] As can be seen, by classifying the disks according to the acquired disk information, multiple levels of resource pools are obtained. Each level of resource pool includes at least one sub-resource pool, and each sub-resource pool includes disks of the same level. Based on the acquired data information of the data to be stored, a target level resource pool is selected from each level of resource pool. The target resource pool is then determined from the target level resource pool based on the number of sub-resource pools within it. Finally, the data to be stored is stored in the target resource pool. Therefore, by classifying the disks, multiple levels of resource pools are obtained, thereby dynamically dividing the resource pools into different levels, improving the flexibility of storing data in the resource pools, and ultimately improving the resource utilization rate of the resource pools.

[0039] Disk information includes disk load information and disk health information. The data storage system categorizes disks based on the acquired disk information to obtain multiple levels of resource pools. This process includes: weighting the disk load and health information to obtain a health score for each disk; determining the performance level of each disk based on its health score range; determining the resource pool level of each disk based on its performance level and type; and allocating each disk to the corresponding resource pool level, resulting in multiple levels of resource pools.

[0040] Disk load information includes storage utilization, etc.

[0041] Disk health information includes bad blocks and disk wear.

[0042] The data storage system weights the disk bad blocks, disk wear, and storage occupancy of each disk to obtain a health score for each disk. Specifically, the data storage system obtains a first ratio between the number of disk bad blocks and the total number of disks, and a first product term between the first ratio and a preset disk bad block weight; it obtains a second ratio between the disk wear degree and a preset wear threshold, and a second product term between the second ratio and a preset wear weight; it obtains a third ratio between the storage occupancy rate and a preset storage threshold, and a third product term between the third ratio and a preset storage weight; it calculates the sum of the first, second, and third product terms, and the difference between the threshold score and this sum is determined as the health score of each disk.

[0043] For example, the health score of a disk satisfies the following formula: HealthScore=1-(w1×Bad_Blockst / Total_Blocks+w2×Weart / 100+w3×IO_Utilizationt / 100); In the above formula, HealthScore represents the health score of the disk, with a threshold score of 1; w1 represents the preset bad block weight; Bad_Blockst represents the number of bad blocks on the disk; Total_Blocks represents the total number of disks; w2 represents the preset wear weight; Weart represents the degree of disk wear, with a preset wear threshold of 100; w3 represents the preset storage weight; and IO_Utilization represents the storage utilization rate, with 100 being the preset storage threshold. The preset bad block weight can be 0.2, the preset wear weight can be 0.4, and the preset storage weight can be 0.4.

[0044] The data storage system determines the performance level of each disk based on the range of its health score. Specifically, the data storage system classifies disks with health scores greater than a first score threshold as disks of the first performance level, disks with health scores less than or equal to the first score threshold but greater than a second score threshold as disks of the second performance level, and disks with health scores less than or equal to the second score threshold as disks of the third performance level.

[0045] The data storage system determines the resource pool level of each disk based on its performance level and type. Specifically, the corresponding resource pool level is determined from the resource pool level mapping table based on the performance level and type of each disk.

[0046] For example, if a disk's performance level is Level 1 and its type is SSD, then that disk is classified as a Level 1 SSD resource in the resource pool; if its performance level is Level 2 and its type is SSD, then it is classified as a Level 2 SSD resource in the resource pool; and if its performance level is Level 3 and its type is SSD, then it is classified as a Level 3 SSD resource in the resource pool. Similarly, if a disk's performance level is Level 1 and its type is HDD, then it is classified as a Level 1 HDD resource in the resource pool; if its performance level is Level 2 and its type is HDD, then it is classified as a Level 2 HDD resource in the resource pool; and if its performance level is Level 3 and its type is HDD, then it is classified as a Level 3 HDD resource in the resource pool.

[0047] For example, the resource pool levels include levels one through six. The level one resource pool includes disks of the first performance level and SSD type; the level two resource pool includes disks of the second performance level and SSD type; the level three resource pool includes disks of the third performance level and SSD type; the level four resource pool includes disks of the first performance level and HDD type; the level five resource pool includes disks of the second performance level and HDD type; and the level six resource pool includes disks of the third performance level and HDD type.

[0048] The data storage system allocates disks to resource pools of corresponding levels according to their resource pool level, resulting in multiple levels of resource pools. The system then evenly distributes each disk into multiple sub-resource pools of the corresponding resource pool level, grouping sub-resource pools of the same level together as a single resource pool. For example, a first-level resource pool may contain multiple first-level sub-resource pools, and disks of first-performance level are allocated to these first-level sub-resource pools.

[0049] The data storage system further includes the following steps to classify and process each disk according to the disk information obtained, resulting in multiple levels of resource pools: weighting the disk bad blocks, disk wear, and storage occupancy rate of each disk to obtain a health score for each disk; determining the resource pool level of each disk based on the range of its health score; and allocating each disk to the corresponding level of resource pool according to its resource pool level, thus obtaining multiple levels of resource pools.

[0050] Specifically, the data storage system identifies disks with health scores greater than a first score threshold as disks in the first-level resource pool, disks with health scores less than or equal to the first score threshold but greater than a second score threshold as disks in the second-level resource pool, and disks with health scores less than or equal to the second score threshold as disks in the third-level resource pool.

[0051] For example, the first preset score threshold is 0.8, the second preset score threshold is 0.5, and the resource pool rules for determining the disk's level based on the disk's health score are as follows:

[0052] In the above formula, PoolAssignment is the resource pool allocation, high_performance is the high-performance pool, legacy is the degradation pool, and retired is the retirement pool. When the disk's health score is greater than 0.8, it enters the high-performance pool; when the disk's health score is less than or equal to 0.8 but greater than 0.5, it enters the degradation pool; and when the disk's health score is less than or equal to 0.5, it enters the retirement pool.

[0053] Furthermore, the data storage method also includes: periodically obtaining the health scores of disks in each level pool to obtain the current health score and duration of the current health score of each disk; matching the current health score of each disk with the score range of each level resource pool to obtain the matching result; and adjusting the resource pool where the disk is located based on the matching result and the duration of the current health score.

[0054] The scoring range for the first-level resource pool is greater than the first scoring threshold; the scoring range for the second-level pool is less than or equal to the first scoring threshold and greater than the second scoring threshold; and the scoring range for the third-level pool is less than or equal to the second scoring threshold.

[0055] If the disk's current health score falls within the score range of the first-level resource pool, the matching result is first-level; if the disk's current health score falls within the score range of the second-level resource pool, the matching result is second-level; and if the disk's current health score falls within the score range of the third-level resource pool, the matching result is third-level.

[0056] The data storage system adjusts the resource pool in which the disk resides based on the matching results and the duration of the current health score. Specifically, in response to a difference between the current level of the resource pool in which the disk is currently located and the level in the matching results, the data storage system determines whether the duration of the disk's current health score meets the corresponding duration requirement. If so, the disk is moved to the resource pool of the corresponding level in the matching results.

[0057] In one embodiment, such as Figure 2As shown, if the health score of a disk in the high-performance pool is between 0.5 and 0.8 and lasts for more than 2 minutes, the disk is moved to the degraded pool. If the health score of a disk in the high-performance pool is less than 0.5 and lasts for more than 1 minute, the disk is moved to the retirement pool. If the health score of a disk in the degraded pool is greater than 0.8 and lasts for more than 5 minutes, the disk is moved to the high-performance pool. If the health score of a disk in the degraded pool is less than or equal to 0.5 and lasts for more than 2 minutes, the disk is moved to the retirement pool. If the health score of a disk in the retirement pool is greater than 0.5 and lasts for more than 5 minutes, the disk is moved to the degraded pool.

[0058] In one embodiment, combined with Figure 3 As shown, the data storage system includes hardware nodes, a control center, and a database. When a hardware node starts up, it reports the static disk attributes of each disk to the control center. The control center then sends these static disk attributes to the database for storage. After startup, the hardware node periodically reports the dynamic disk attributes of each disk to the control center. The control center calculates the health score of each disk and sends it to the database. The database updates the disks in each resource pool based on their health scores and the duration of the health score.

[0059] As can be seen, by automatically reporting static and dynamic disk attributes from hardware nodes, calculating disk health scores based on these attributes, and then updating the disks in the resource pool according to these health scores, a hardware information automatic reporting and awareness technology can achieve pooled management based on differences in hardware. Compared to traditional static resource pool partitioning strategies, this approach maximizes the performance advantages of hardware and facilitates efficient storage of view data.

[0060] Based on the above embodiments, please refer to Figure 4 , Figure 4 yes Figure 1 The illustrated flowchart shows an exemplary embodiment of step S120 in the data storage method. Specifically, the process of selecting a target level resource pool from resource pools of various levels based on the acquired data information of the data to be stored in step S120 includes the following steps: Step S410: Select the first alternative resource pool from the resource pools of each level according to the required amount of resources for the data to be stored.

[0061] The first alternative resource pool refers to the resource pool that meets the resource quantity requirements.

[0062] The required resources for storing data refer to the total amount of various types of resources needed to store the data. For example, the required resources may include computing resources, storage resources, etc. The required resources may also include video encoding format, image resolution, or EC (Erasure Code) settings.

[0063] A data storage system analyzes the data to be stored to determine the required resource amount. For example, the system parses the file header or analyzes the metadata of the data to be stored to determine the required resource amount.

[0064] The data storage system selects a first candidate resource pool from resource pools of various levels based on the required resource quantity of the data to be stored. Specifically, the data storage system compares the required resource quantity of the data to be stored with the available resource quantity of each level of resource pool to obtain the comparison result; the resource pool whose resource quantity meets the required resource quantity of the data to be stored is determined as the first candidate resource pool.

[0065] The required resource quantity for the data to be stored can also be the tag information of the data to be stored. The first alternative resource pool is selected from the resource pools of each level based on the tag information.

[0066] Tag information can include erasure coding settings, low latency requirements for real-time monitoring, and / or tolerable high latency for historical playback. As one example, a data storage system obtains tag information through explicit declarations of the data to be stored. For instance, the data storage system identifies tags in APIs (Application Programming Interfaces) or configuration files within the data to be stored as tag information. As yet another example, the data storage system obtains tag information by analyzing the data to be stored. For example, tags are automatically derived based on the context of the data to be stored. For yet another example, if the data to be stored is real-time playback video recordings and frequently queried image channels, the tag information determines that an SSD-type resource pool should be prioritized.

[0067] Step S420: Determine the second alternative resource pool from each of the first alternative resource pools based on the node health status of the nodes where each first alternative resource pool is located.

[0068] Node health status includes node abnormality and node normality.

[0069] The second alternative resource pool refers to the resource pool where the resource quantity meets the requirements and the node where it is located is normal.

[0070] Before the step of determining the second alternative resource pool from each of the first alternative resource pools based on the obtained node health status of the nodes where each of the first alternative resource pools is located, the method further includes: obtaining the node health status of the nodes where each of the first alternative resource pools is located.

[0071] The steps for the data storage system to obtain the node health status of each node in the first candidate resource pool include: obtaining the node status score of the node in the first candidate resource pool; determining the node health status of the corresponding node as normal in response to the node status score being greater than a preset score threshold; and determining the node health status of the corresponding node as abnormal in response to the node status score being less than or equal to the preset score threshold.

[0072] The steps for the data storage system to obtain the node status score of the node where the first alternative resource pool is located include: obtaining the load information, hardware information, and performance information of the node where the first alternative resource pool is located; and performing weighted processing on the load information, hardware information, and performance information of the node where the first alternative resource pool is located to obtain the node status score of the node where the first alternative resource pool is located.

[0073] Load information can include disk IOPS utilization, remaining memory utilization, and CPU utilization. Load information can also include cross-data center latency and TCP (Transmission Control Protocol) retransmission rate.

[0074] Hardware information may include CPU (Central Processing Unit) temperature.

[0075] Performance information can include write latency and network throughput.

[0076] The data storage system performs weighted processing on the load information, hardware information, and performance information of the node where the first candidate resource pool is located to obtain a node status score. Specifically, the data storage system quantifies the load information to obtain a load score; quantifies the hardware information to obtain a disk health score; quantifies the performance information to obtain a performance score; obtains the fourth product term between the load score and the preset load weight; obtains the fifth product term between the disk health score and the preset disk health weight; obtains the sixth product term between the performance score and the preset performance weight; and determines the node status score by the sum of the fourth, fifth, and sixth product terms.

[0077] For example, the node status score of the node where the first alternative resource pool is located satisfies the following formula: J=Q1xPerfScore+Q2xHardwareScore+Q3xLoadScore In the above formula, J represents the node status score, Q1 represents the preset performance weight, PerfScore represents the performance score, Q2 represents the preset disk health weight, HardwareScore represents the disk health score, Q3 represents the preset load weight, and LoadScore represents the load score.

[0078] In one embodiment, the preset performance weight can be 0.4, the preset disk health weight can be 0.4, and the preset load weight can be 0.2. Then the formula for the node status score can be J=0.4xPerfScore+0.4xHardwareScore+0.2xLoadScore.

[0079] The performance score satisfies the following formula: PerfScore = 1 - max(0, Latency - 50) / 100; where Latency represents the write latency.

[0080] The hardware health score satisfies the following formula: HardwareScore = 1 - max(0, CPU_Temperature) / 75; where Cpu_Temperature represents the CPU temperature.

[0081] Load sharing satisfies the following formula: LoadScore = 1 - max(0, CPU_Usage - 70) / 30 - max(0, Mem_Usage - 70) / 30; where CPU_Usage is the CPU utilization rate and Mem_Usage is the memory utilization rate.

[0082] The step of the data storage system obtaining the node status score of the node where the first candidate resource pool is located further includes: obtaining at least two of the load information, hardware information, and performance information of the node where the first candidate resource pool is located; performing weighted processing on at least two of the load information, hardware information, and performance information of the node where the first candidate resource pool is located to obtain the node status score of the node where the first candidate resource pool is located. For example, the data storage system performs numerical processing on the load information to obtain a load score; performs numerical processing on the hardware information to obtain a disk health score; performs numerical processing on the performance information to obtain a performance score; obtains a fourth product term between the load score and a preset load weight; obtains a fifth product term between the disk health score and a preset disk health weight; obtains a sixth product term between the performance score and a preset performance weight; and determines the sum of any two of the fourth, fifth, and sixth product terms as the node status score.

[0083] The step of the data storage system obtaining the node status score of the node where the first candidate resource pool is located further includes: obtaining one of the load information, hardware information, and performance information of the node where the first candidate resource pool is located; and using the product of one of the load information, hardware information, and performance information of the node where the first candidate resource pool is located with a preset gain weight as the node status score of the node where the first candidate resource pool is located. For example, the data storage system performs numerical processing on the load information to obtain a load score; performs numerical processing on the hardware information to obtain a disk health score; performs numerical processing on the performance information to obtain a performance score; and uses the product of one of the load score, disk health score, and performance score with a preset gain weight as the node status score of the node where the first candidate resource pool is located.

[0084] In one embodiment, the data storage system compares the status score of each node with a preset scoring threshold. If the node status score is greater than the preset scoring threshold, the node health status of the corresponding node is determined to be normal; if the node status score is less than or equal to the preset scoring threshold, the node health status of the corresponding node is determined to be abnormal.

[0085] The step of the data storage system to obtain the node health status of each first alternative resource pool also includes: obtaining the load information, hardware information and performance information of the node where the first alternative resource pool is located; determining whether the load information, hardware information and performance information meet the indicator requirements; if so, determining the node health status as normal; if not, determining the node health status as abnormal.

[0086] In one embodiment, if the following conditions are met simultaneously: SSD type disk write latency ≤ 50ms, network throughput ≥ 80% of nominal bandwidth, CPU temperature < 75°C, disk IOPS utilization < 70%, remaining memory utilization ≥ 20% of total memory, cross-data center latency ≤ 30ms, and TCP retransmission rate < 0.1%, then the node health status is determined to be normal; otherwise, the node status is determined to be abnormal.

[0087] The data storage system determines the second alternative resource pool from the first alternative resource pools based on the node health status of the nodes where each first alternative resource pool is located. Specifically, the first alternative resource pool at the location of a normal node is determined as the second alternative resource pool.

[0088] Step S430: Determine the target level resource pool from each of the second alternative resource pools according to the data type of the data to be stored.

[0089] The steps of the data storage system to determine the target level resource pool from each of the second alternative resource pools based on the data type of the data to be stored include: determining the second alternative resource pools of the alternative level from each of the second alternative resource pools based on the data type of the data to be stored; and matching the second alternative resource pools of the alternative level with the erasure coding configuration amount of the data to be stored to obtain the target level resource pool.

[0090] In one embodiment, if the data type to be stored is video, then each level of HDD type resource pool is determined from each second alternative resource pool as the second alternative resource pool for the alternative level; if the data type to be stored is image, then each level of SDD type resource pool is determined from each second alternative resource pool as the second alternative resource pool for the alternative level; if the data type to be stored includes both image and video types, then each level of mixed type resource pool is determined from each second alternative resource pool as the second alternative resource pool for the alternative level.

[0091] In another embodiment, if the data to be stored is of image type, then the second alternative resource pools of the first, second, and third levels are determined as the second alternative resource pools of the alternative levels. If the data to be stored is of video type, then the second alternative resource pools of the fourth, fifth, and sixth levels are determined as the second alternative resource pools of the alternative levels. If the data to be stored is of both image and video type, then the second alternative resource pools of the first to sixth levels are determined as the second alternative resource pools of the alternative levels. As an example, the alternative resource levels may include a first level, a second level, and a third level. The data storage system matches the resource allocation of the second alternative resource pool of the alternative levels with the erasure coding configuration of the data to be stored, obtaining the target level resource pool through a process that includes: determining whether the resource allocation of the second alternative resource pool of the first level meets the erasure coding configuration of the data to be stored, wherein the storage performance of the first level resource pool is superior to that of the second level; if so, then the second alternative resource pool of the first level is determined as the target level resource pool; if not, in response to the resource allocation of the second alternative resource pool of the second level meeting the erasure coding configuration of the data to be stored, the second alternative resource pool of the second level is determined as the target level resource pool; if not, in response to the resource allocation of the second alternative resource pool of the second level not meeting the erasure coding configuration of the data to be stored, an early warning is issued, and determining whether the resource allocation of the second alternative resource pool of the third level meets the erasure coding configuration of the data to be stored; if so, the second alternative resource pool of the third level is determined as the target level resource pool; if not, an early warning is issued.

[0092] As another example, the alternative resource levels may include Level 4, Level 5, and Level 6. The data storage system matches the resource allocation of the second alternative resource pool for each alternative level with the erasure coding configuration of the data to be stored, to obtain the target level resource pool. This process includes: determining whether the resource allocation of the second alternative resource pool for Level 4 meets the erasure coding configuration of the data to be stored, and whether the storage performance of the Level 4 resource pool is better than that of Level 5; if yes, then the second alternative resource pool for Level 4 is determined as the target level resource pool; if not, in response to the second alternative resource pool for Level 5 meeting the erasure coding configuration of the data to be stored, the second alternative resource pool for Level 5 is determined as the target level resource pool; if not, in response to the second alternative resource pool for Level 5 not meeting the erasure coding configuration of the data to be stored, an early warning is issued; and determining whether the resource allocation of the second alternative resource pool for Level 6 meets the erasure coding configuration of the data to be stored, if yes, the second alternative resource pool for Level 6 is determined as the target level resource pool, if not, an early warning is issued.

[0093] As another example, the alternative resource levels can include levels one through six. The data storage system sequentially checks whether the resource allocation of the second alternative resource pools at levels one and two meets the erasure coding configuration requirements of the image sub-data in the data to be stored. If neither meets the requirements, it sequentially checks whether the resource allocation of the second alternative resource pools at levels four and five meets the erasure coding configuration requirements of the image sub-data. If they do, the second alternative resource pool that first meets the erasure coding configuration requirements of the image sub-data is selected as the target level resource pool for the image sub-data. If neither meets the requirements, an early warning is issued. Similarly, the data storage system sequentially checks whether the resource allocation of the second alternative resource pools at levels four and five meets the erasure coding configuration requirements of the video sub-data in the data to be stored. If neither meets the requirements, it sequentially checks whether the resource allocation of the second alternative resource pools at levels one and two meets the erasure coding configuration requirements of the video sub-data. If they do, the second alternative resource pool that first meets the erasure coding configuration requirements of the video sub-data is selected as the target level resource pool for the video sub-data. If neither meets the requirements, an early warning is issued.

[0094] The erasure coding configuration can be represented as the number of data fragments of m+n; where m represents the number of original data blocks, i.e. the original data blocks that have not been encoded, and n represents the number of check blocks, i.e. the redundant check blocks generated through encoding calculation.

[0095] As can be seen, by obtaining the node status score of the node where the first alternative resource pool is located, obtaining the node health status based on the node status score, and selecting the resource pool based on the node health status, the system can automatically isolate the resource pool with abnormal nodes, thereby ensuring the high reliability of the accelerated service.

[0096] In one embodiment, combined with Figure 5 As shown, the process by which a data storage system acquires a target resource pool includes the following steps: Step S501: Determine whether the data to be stored uses the SSD high-performance pool. If yes, proceed to step S502; otherwise, proceed to step S506. Step S502: Query the current SSD high-performance pool information list; Step S503: Determine whether the SSD high-performance pools in the current SSD high-performance pool information list meet the erasure coding configuration amount of the data to be stored. If yes, proceed to step S511; otherwise, proceed to step S504. Step S504: Query the current SSD downgrade pool information list; Step S505: Determine whether the SSD degradation pools in the current SSD degradation pool information list meet the erasure coding configuration amount of the data to be stored. If yes, proceed to step S511; otherwise, proceed to step S506. Step S506: Query the current HDD high-performance pool information list; Step S507: Determine whether the HDD high-performance pools in the current HDD high-performance pool information list meet the erasure coding configuration amount of the data to be stored. If yes, proceed to step S511; otherwise, proceed to step S508. Step S508: Query the current HDD downgrade pool information list; Step S509: Determine whether the HDD downgrade pools in the current HDD downgrade pool information list meet the erasure coding configuration amount of the data to be stored. If yes, proceed to step S511; otherwise, proceed to step S510. Step S510: Alarm reporting, process ends; Step S511: Select the second alternative resource pool that meets the erasure coding configuration of the data to be stored as the target level resource pool, determine the target resource pool from the target level resource pool, and update the usage information of the target resource pool. Step S512: Increase the real-time status detection frequency of the selected disks in the target resource pool by 3 minutes; process ends.

[0097] As can be seen, when the data to be stored is of video or image type, it is stored in a resource pool of a single disk type. When the data to be stored is of both video and image types, it is stored in resource pools of multiple disk types. Therefore, flexibly selecting the resource pool type based on the data type maximizes the advantages of storage hardware while also enabling flexible physical isolation of business data. Furthermore, by selecting the target level resource pool based on the required resource amount of the data to be stored, the node health status of the node where the resource pool is located, and the data type, dynamic selection of the resource pool is achieved, which is more conducive to efficient storage of view data in complex business environments.

[0098] The data storage system determines the target resource pool from the target-level resource pool based on the number of sub-resource pools in the target-level resource pool, including: in response to the existence of one sub-resource pool in the target-level resource pool, the sub-resource pool is selected as the target resource pool; in response to the existence of at least two sub-resource pools in the target-level resource pool, the node status score of the node where each sub-resource pool is located is obtained; the sub-resource pools are sorted according to the node status score of the node where each sub-resource pool is located to obtain sorted sub-resource pools; the sub-resource pool with a preset sorting position is selected from the sorted sub-resource pools as the target resource pool; and the data to be stored is stored in the target resource pool.

[0099] In one embodiment, the data storage system determines whether there is only one sub-resource pool in the target-level resource pool. If so, the sub-resource pool in the target-level resource pool is taken as the target resource pool.

[0100] In one embodiment, the data storage system obtains the node status score of the node where each sub-resource pool is located, sorts the sub-resource pools at each node in descending order of node status score, and selects the first sub-resource pool from the sorted sub-resource pools as the target resource pool, storing the data to be stored in the disk of the target resource pool. Alternatively, the data storage system obtains the node status score of the node where each sub-resource pool is located, sorts the sub-resource pools at each node in ascending order of node status score, and selects the last sub-resource pool from the sorted sub-resource pools as the target resource pool, storing the data to be stored in the disk of the target resource pool.

[0101] The data storage system also includes: identifying disks with a performance level of first performance level and disk types of SSD and HDD as disks in a mixed-type resource pool, resulting in a seventh-level resource pool, which includes disks with the first performance level and disk type of HDD and disks with the first performance level and disk type of SSD; identifying disks with a performance level of second performance level and disk types of SSD and HDD as disks in a mixed-type resource pool, resulting in an eighth-level resource pool; and identifying disks with a performance level of third performance level and disk types of SSD and HDD as disks in a mixed-type resource pool, resulting in a ninth-level resource pool.

[0102] When the data to be stored includes image sub-data of image type and video sub-data of video type, it is determined in turn whether the resource configuration of the second alternative resource pool of the seventh level, the eighth level and the ninth level meets the erasure coding configuration of the data to be stored. If yes, the second alternative resource pool that first meets the erasure coding configuration of the data to be stored is determined as the target level resource pool. If not, an early warning is issued.

[0103] Figure 6 This is a block diagram illustrating a data storage device according to an exemplary embodiment of this application. Figure 6 As shown, the exemplary data storage device 600 includes: a hierarchical module 610, a target level resource pool determination module 620, a target resource pool determination module 630, and a data storage module 640. Specifically: The grading module 610 is used to grade each disk according to the disk information of each disk to obtain multiple levels of resource pools. Each level of resource pool includes at least one sub-resource pool, and each sub-resource pool includes disks of the same level.

[0104] The target level resource pool determination module 620 is used to select the target level resource pool from the resource pools of each level based on the data information of the data to be stored.

[0105] The target resource pool determination module 630 is used to determine the target resource pool if a sub-resource pool exists in the target level resource pool.

[0106] The data storage module 640 is used to store the data to be stored into the target resource pool.

[0107] In this exemplary data storage device, the disks are classified according to the acquired disk information to obtain multiple levels of resource pools. Each level of resource pool includes at least one sub-resource pool, and each sub-resource pool includes disks of the same level. A target level resource pool is selected from the resource pools based on the acquired data information of the data to be stored. A target resource pool is determined from the target level resource pool based on the number of sub-resource pools in the target level resource pool. The data to be stored is then stored in the target resource pool. Thus, by classifying the disks to obtain multiple levels of resource pools, the resource pools are dynamically divided into different levels, improving the flexibility of storing data in the resource pools and thereby increasing the resource utilization rate of the resource pools.

[0108] The functions of each module can be found in the data storage method implementation example, and will not be repeated here.

[0109] To implement the data storage method of the above embodiments, this application proposes another electronic device, please refer to [link / reference needed]. Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application.

[0110] Electronic device 700 includes memory 701 and processor 702, wherein memory 701 and processor 702 are coupled together.

[0111] The memory 701 is used to store program data, and the processor 702 is used to execute the program data to implement the data storage method of the above embodiment.

[0112] In this embodiment, processor 702 can also be referred to as CPU (Central Processing Unit). Processor 702 may be an integrated circuit chip with signal processing capabilities. Processor 702 can also be a general-purpose processor, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor, or processor 702 can be any conventional processor.

[0113] This application also provides a computer-readable storage medium, such as Figure 8 As shown, the computer-readable storage medium 800 is used to store program data 801, which, when executed by a processor, is used to implement the data storage method as described in the method embodiments of this application.

[0114] The methods involved in the data storage method embodiments of this application, when implemented as software functional units and sold or used as independent products, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data storage method, characterized in that, The method includes: Based on the disk information obtained from each disk, each disk is classified and processed to obtain multiple levels of resource pools. Each level of resource pool includes at least one sub-resource pool, and each sub-resource pool includes disks of the same level. Based on the data information of the data to be stored, select the target level resource pool from the resource pools of each level; The target resource pool is determined from the target resource pool based on the number of sub-resource pools in the target level resource pool; The data to be stored is stored in the target resource pool.

2. The method according to claim 1, characterized in that, The data information includes the required resource quantity and data type of the data to be stored. The step of selecting a target level resource pool from the resource pools of each level based on the acquired data information of the data to be stored includes: The first candidate resource pool is selected from the resource pools of each level based on the required amount of resources for the data to be stored. The second alternative resource pool is determined from each of the first alternative resource pools based on the node health status of the nodes where each first alternative resource pool is located. The target level resource pool is determined from each of the second alternative resource pools based on the data type of the data to be stored.

3. The method according to claim 2, characterized in that, The node health status includes node abnormality and node normality. Before the step of determining the second alternative resource pool from each of the first alternative resource pools based on the obtained node health status of the nodes where each of the first alternative resource pools is located, the method further includes: Obtain the node status score of the node where the first candidate resource pool is located; In response to the node status score being greater than a preset score threshold, the node health status of the corresponding node is determined to be abnormal. In response to a node status score being less than or equal to a preset score threshold, the node health status of the corresponding node is determined to be normal.

4. The method according to claim 3, characterized in that, The step of obtaining the node status score of the node where the first candidate resource pool is located includes: Obtain the load information, hardware information, and performance information of the node where the first candidate resource pool is located; The load information, hardware information, and performance information of the node where the first candidate resource pool is located are weighted to obtain the node status score of the node where the first candidate resource pool is located.

5. The method according to claim 2, characterized in that, The step of determining the target level resource pool from each of the second candidate resource pools based on the data type of the data to be stored includes: The second alternative resource pool with the alternative level is determined from each second alternative resource pool according to the data type of the data to be stored; The second alternative resource pool of the alternative level is matched with the erasure coding configuration amount of the data to be stored to obtain the target level resource pool.

6. The method according to claim 5, characterized in that, The candidate levels include a first level and a second level. The step of matching the second candidate resource pool of the candidate levels with the erasure coding configuration amount of the data to be stored to obtain the target level resource pool includes: Determine whether the resource configuration of the second alternative resource pool of the first level meets the erasure coding configuration of the data to be stored, and the storage performance of the resource pool of the first level is better than that of the second level. If so, the second alternative resource pool of the first level shall be determined as the resource pool of the target level; If not, then in response to the resource configuration of the second alternative resource pool of the second level satisfying the erasure coding configuration of the data to be stored, the second alternative resource pool of the second level is determined as the target level resource pool.

7. The method according to claim 1, characterized in that, The step of determining the target resource pool from the target level resource pool based on the number of sub-resource pools in the target level resource pool includes: If a sub-resource pool exists in the target resource pool, then the sub-resource pool is used as the target resource pool. In response to the existence of at least two sub-resource pools in the target level resource pool, the node status score of the node where each sub-resource pool is located is obtained; The sub-resource pools are sorted according to the node status score of the node where each sub-resource pool is located, resulting in the sorted sub-resource pools. Select a sub-resource pool with a preset sorting position from the sorted sub-resource pools as the target resource pool; The data to be stored is stored in the target resource pool.

8. The method according to claim 1, characterized in that, The disk information includes disk load information and disk health information. The step of classifying each disk according to the acquired disk information to obtain multiple levels of resource pools includes: The disk load information and the disk health information are weighted to obtain a health score for each disk. The performance level of each disk is determined based on the range of its health score. The resource pool level of each disk is determined based on its performance level and type. Each disk is assigned to a resource pool of the corresponding level according to its resource pool level, resulting in multiple levels of resource pools.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to perform the method as claimed in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, include: The system stores program data, which, when executed by a processor, is used to implement the method as described in any one of claims 1-8.

Images