Data storage method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data storage method, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a cache node queue and a direct storage node queue in a storage system; each type of node queue comprises a plurality of storage nodes, and the storage nodes comprise a mechanical hard disk and a solid state disk; obtaining target data and data information corresponding to the target data, and determining a queue type corresponding to the target data based on the data information; responding to a cache node queue corresponding to the target data, and writing the target data into a solid state disk of at least one storage node in the cache node queue; when the target data in the solid state disk meets a transfer condition, at least part of the target data is transferred to the mechanical hard disk, and the transfer condition is related to the data size and the cache time limit of the target data in the solid state disk; and responding to the direct storage node queue corresponding to the target data, and writing the target data into a mechanical hard disk of at least one storage node in the direct storage node queue. According to the scheme, the utilization rate of storage node resources can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data storage method, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of big data, cloud storage technology has become a mainstream trend. A cloud storage cluster comprises multiple storage nodes, which write data by allocating storage space. Currently, common data storage methods lack fine-grained scheduling of storage resources and specific processing of data characteristics, resulting in low overall resource utilization of storage nodes. Therefore, improving the utilization rate of storage node resources has become an urgent problem to be solved. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a data storage method, electronic device, and computer-readable storage medium that can improve the utilization rate of storage node resources.

[0004] To address the aforementioned technical problems, this application provides a data storage method, comprising: acquiring a cache node queue and a direct storage node queue in a storage system; wherein each type of node queue includes multiple storage nodes, the storage nodes including hard disk drives (HDDs) and solid-state drives (SSDs); acquiring target data and its corresponding data information, and determining the queue type corresponding to the target data based on the data information; in response to the target data corresponding to the cache node queue, writing the target data to the SSD of at least one storage node in the cache node queue; wherein, when the target data in the SSD meets a transfer condition, at least a portion of the target data is transferred to the HDD, the transfer condition being related to the amount of target data in the SSD and the cache duration; and in response to the target data corresponding to the direct storage node queue, writing the target data to the HDD of at least one storage node in the direct storage node queue.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device including a memory and a processor coupled to each other, wherein the memory stores program instructions and the processor executes the program instructions to implement the method described in the first aspect.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the method described in the first aspect.

[0007] The above scheme obtains the cache node queue and direct storage node queue in the storage system. Each cache node queue and direct storage node queue includes multiple storage nodes, each containing both a mechanical hard drive (HDD) and a solid-state drive (SSD). It acquires the target data and the data information carried by the target data, determines the queue type corresponding to the target data based on the data information, and writes the target data to the SSD of at least one storage node in the cache node queue when the target data is determined to be written to the cache node queue. When the target data in the SSD meets the transfer conditions, at least a portion of the target data is transferred to the HDD, and the transfer conditions are related to the amount of target data in the SSD and the cache duration. When the target data is determined to be written to the direct storage node queue, it is written to the HDD of at least one storage node in the direct storage node queue. By utilizing the caching acceleration capability and high-speed read / write characteristics of the SSD for temporary storage, the response time of the initial data write can be significantly shortened, improving storage efficiency. Furthermore, the HDD only performs centralized operations when processing direct storage data or receiving cached transfer data, which helps reduce the overall energy consumption of the storage nodes, thereby improving the utilization rate of storage node resources. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one embodiment of the data storage method of this application; Figure 2 This is a flowchart illustrating another embodiment of the data storage method of this application; Figure 3 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 4 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0011] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the data storage method of this application. The method includes: S101: Obtain the cache node queue and direct storage node queue in the storage system; each type of node queue includes multiple storage nodes, including mechanical hard drives and solid-state drives.

[0012] Specifically, the cache node queue and the direct storage node queue in the storage system are obtained. Each cache node queue and the direct storage node queue includes multiple storage nodes, and each storage node includes a mechanical hard drive and a solid-state drive.

[0013] In one application mode, the disk load information and available capacity of the solid-state drives in each storage node are obtained in real time. When the disk load information of the solid-state drive meets the preset load threshold condition, or the available capacity of the solid-state drive meets the preset capacity threshold condition, the storage node corresponding to the solid-state drive is allocated to the cache node queue; otherwise, it is allocated to the direct storage node queue.

[0014] In another application, the identifiers corresponding to the cache node queue and the direct storage node queue are predefined in the storage system's configuration file. When the storage system starts, the configuration is loaded to divide the cache node queue and the direct storage node queue.

[0015] It should be noted that the disk load information of a solid-state drive (SSD) refers to the amount of data processed by the SSD per unit time, reflecting the SSD's performance, working status, and other conditions.

[0016] S102: Obtain the target data and its corresponding data information, and determine the queue type corresponding to the target data based on the data information.

[0017] Specifically, the target data and the data information carried by the target data are obtained, and the queue type corresponding to the target data is determined based on the data information.

[0018] In one application approach, the queue type corresponding to the node queue to which the target data is written is determined based on the size of the target data.

[0019] In one application scenario, when the amount of target data exceeds a preset threshold, the target data is determined to be a large file, and the queue type corresponding to the write node queue is determined to be a direct storage node queue. When the amount of target data does not exceed the preset threshold, the target data is determined to be a small file, and the queue type corresponding to the write node queue is determined to be a cache node queue.

[0020] In another application, the queue type corresponding to the node queue to which the target data is written is determined based on the data type of the target data.

[0021] In one application scenario, when the target data is text and does not require real-time viewing, the queue type corresponding to the node queue to be written is determined to be a direct storage node queue. When the target data is video and requires real-time viewing, the queue type corresponding to the node queue to be written is determined to be a cache node queue.

[0022] It is understood that the data types of the target data may also include audio data, image data, etc., and this application does not impose specific restrictions here.

[0023] It should be noted that the storage nodes in this application all support the simultaneous use of solid-state drives (SSDs) and hard disk drives (HDDs) as storage media. Typically, the number of SSDs is much smaller than the number of HDDs. For example, a storage node may contain two SSDs and twenty HDDs. One SSD can be bound to multiple HDDs simultaneously, while one HDD can only be bound to one SSD.

[0024] S103: In response to the cache node queue corresponding to the target data, write the target data to the solid-state drive of at least one storage node in the cache node queue; wherein, when the target data in the solid-state drive meets the transfer conditions, at least part of the target data is transferred to the mechanical hard drive, and the transfer conditions are related to the amount of target data in the solid-state drive and the cache time limit.

[0025] Specifically, when the target data is determined to be written to the cache node queue, the target data is written to the solid-state drive (SSD) of at least one storage node in the cache node queue. When the target data in the SSD meets the transfer conditions, at least a portion of the target data will be transferred to the mechanical hard drive (HDD). The transfer conditions are related to the amount of target data in the SSD and the cache duration.

[0026] In one application, when target data is determined to be written to the cache node queue, the access frequency of the target data is obtained, and the corresponding target data is written to the solid-state drive of at least one storage node in the cache node queue according to the access frequency. When the target data in the solid-state drive meets the transfer conditions, at least part of the target data will be transferred to the mechanical hard drive.

[0027] In one application scenario, when target data is determined to be written to the cache node queue, the access frequency of the target data is obtained. High-frequency target data is written to the high-frequency data partition of the solid-state drive, and low-frequency target data is written to the low-frequency data partition of the solid-state drive. When the target data in the solid-state drive meets the transfer conditions, the target data in the low-frequency data partition is transferred to the mechanical hard drive first, while the target data in the high-frequency data partition will be retained in the solid-state drive for a period of time.

[0028] In another application, when the target data is determined to be written to the cache node queue, the priority of the target data is obtained, and the corresponding target data is written to the solid-state drive of at least one storage node in the cache node queue according to the priority. When the target data in the solid-state drive meets the transfer conditions, at least part of the target data will be transferred to the mechanical hard drive.

[0029] In one application scenario, when target data is determined to be written to the cache node queue, the priority of the target data is obtained. The high-priority target data is written to the solid-state drive first, and then the low-priority target data is written to the solid-state drive. When the target data in the solid-state drive meets the transfer conditions, the low-priority target data is transferred to the mechanical hard drive first, while the high-priority target data will be retained in the solid-state drive for a period of time.

[0030] Optionally, the transfer condition is met when the amount of target data in the solid-state drive exceeds the capacity threshold of the solid-state drive's total capacity, such as exceeding 80% of the total capacity of the solid-state drive. Of course, the transfer condition can also be met only when the solid-state drive is full. Alternatively, the transfer condition is met when the cache duration of the target data in the solid-state drive exceeds the cache time limit, such as the cache duration exceeding six hours. Of course, the cache time limit can also be set to eight hours, ten hours, etc. This application does not impose specific restrictions on the numerical settings of the capacity threshold and the cache time limit.

[0031] It should be noted that data storage is only considered complete when the target data is written to the mechanical hard drive.

[0032] S104: In response to the direct storage node queue corresponding to the target data, write the target data to the mechanical hard disk of at least one storage node in the direct storage node queue.

[0033] Specifically, when the target data is determined to be written to the direct storage node queue, the target data is written to the mechanical hard disk of at least one storage node in the direct storage node queue.

[0034] In one application mode, when the target data is determined to be written to the direct storage node queue, write requests are generated according to the enqueue order of the target data, and the mechanical hard disk executes the write in the order of the requests.

[0035] In another application, when the target data is determined to be written to the direct storage node queue, the priority of the target data is obtained, and the target data is written to the mechanical hard disk according to the priority.

[0036] Understandably, the direct-write node queue does not provide cache write functionality for solid-state drives (SSDs), but only provides direct-write functionality for mechanical hard drives (HDDs).

[0037] The above scheme obtains the cache node queue and direct storage node queue in the storage system. Each cache node queue and direct storage node queue includes multiple storage nodes, each containing both a mechanical hard drive (HDD) and a solid-state drive (SSD). It acquires the target data and the data information carried by the target data, determines the queue type corresponding to the target data based on the data information, and writes the target data to the SSD of at least one storage node in the cache node queue when the target data is determined to be written to the cache node queue. When the target data in the SSD meets the transfer conditions, at least a portion of the target data is transferred to the HDD, and the transfer conditions are related to the amount of target data in the SSD and the cache duration. When the target data is determined to be written to the direct storage node queue, it is written to the HDD of at least one storage node in the direct storage node queue. By utilizing the caching acceleration capability and high-speed read / write characteristics of the SSD for temporary storage, the response time of the initial data write can be significantly shortened, improving storage efficiency. Furthermore, the HDD only performs centralized operations when processing direct storage data or receiving cached transfer data, which helps reduce the overall energy consumption of the storage nodes, thereby improving the utilization rate of storage node resources.

[0038] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the data storage method of this application. The data storage method includes: S201: Obtain real-time information on the available capacity and disk load of the solid-state drives in each storage node from the storage system.

[0039] Specifically, the available capacity and disk load information of all solid-state drives in each storage node are obtained from the storage system in real time.

[0040] S202: In response to the fact that the available capacity of the solid-state drive meets the preset capacity threshold condition or the disk load information of the solid-state drive meets the preset load threshold condition, the storage node corresponding to the solid-state drive is allocated to the cache node queue; otherwise, it is allocated to the direct storage node queue.

[0041] Specifically, when the available capacity of the solid-state drive (SSD) meets the preset capacity threshold, or when the SSD's disk load information meets the preset load threshold, the storage nodes corresponding to the SSDs that meet the conditions are allocated to the cache node queue, and the storage nodes corresponding to the SSDs that do not meet the conditions are allocated to the direct storage node queue. By setting capacity and load thresholds, it is ensured that the storage nodes (SSDs) entering the cache node queue have sufficient available space and low load pressure, which can efficiently handle the read and write requirements of cached data, and avoid cache performance degradation due to insufficient SSD capacity or excessive load. This allows limited cache resources to be concentrated on serving high-frequency access data, thereby improving overall cache efficiency.

[0042] Each type of node queue includes multiple storage nodes, and each storage node includes both mechanical hard drives and solid-state drives.

[0043] In a specific implementation scenario, when the available capacity of all solid-state drives in a storage node is less than 40% of the total capacity, or when the disk load information of all solid-state drives in a storage node exceeds 60% of the load limit, the storage node is allocated to the direct storage node queue; otherwise, the storage node is allocated to the cache node queue.

[0044] In other implementation scenarios, the preset capacity threshold and preset load threshold can be set according to the actual situation of the node, and this application does not impose specific restrictions here.

[0045] It should be noted that the storage system can allocate storage nodes from the cache node queue to the direct storage node queue based on the available capacity and disk load information of all solid-state drives in each storage node in real time.

[0046] S203: Obtain the target data and its corresponding data information, and determine the queue type corresponding to the target data based on the data information.

[0047] Specifically, the target data and the data information carried by the target data are obtained, and the queue type corresponding to the target data is determined based on the data information.

[0048] S204: In response to the cache node queue corresponding to the target data, write the target data to the solid-state drive of at least one storage node in the cache node queue.

[0049] Specifically, when the target data is determined to be written to the cache node queue, the target data is written to the solid-state drive (SSD) of at least one storage node in the cache node queue. When the target data in the SSD meets the transfer conditions, at least a portion of the target data will be transferred to the mechanical hard drive (HDD). The transfer conditions are related to the amount of target data in the SSD and the cache duration.

[0050] In one implementation scenario, the storage node includes multiple mechanical hard drives and at least one solid-state drive (SSD). On each storage node in the cache node queue, each SSD is bound to at least a portion of the mechanical hard drives, and the binding relationship between the mechanical hard drives and the SSD is unique. After step S204, the method further includes: in response to the target data in the SSD exceeding a preset data volume threshold, transferring the target data exceeding the preset data volume threshold to at least a portion of the mechanical hard drives bound to the SSD; or, in response to the cache duration of at least a portion of the target data in the SSD exceeding the cache time limit, transferring the target data with the cache duration exceeding the cache time limit to at least a portion of the mechanical hard drives bound to the SSD; wherein, the cache time limit is related to the disk load information of the SSD.

[0051] Specifically, when the amount of target data in the solid-state drive exceeds a preset data amount threshold, the target data exceeding the preset data amount threshold is transferred to at least a portion of the mechanical hard drives bound to the solid-state drive; or when the cache duration of at least a portion of the target data in the solid-state drive exceeds the cache time limit, the target data with the cache duration exceeding the cache time limit is transferred to at least a portion of the mechanical hard drives bound to the solid-state drive.

[0052] It should be noted that after the target data is transferred from the solid-state drive (SSD) to the hard disk drive (HDD), it will still be retained on the SSD for a period of time to reduce the risk of data loss and improve the security of the target data transfer. The original target data will only be completely deleted when new target data needs to be written to the SSD.

[0053] In one implementation scenario, the cache timeout is adjusted based on the following steps: in response to the solid-state drive's disk load information being lower than a first load threshold, the cache timeout is determined to be a first preset value; in response to the solid-state drive's disk load information being between a second load threshold and a third load threshold, the cache timeout is determined to be a second preset value; in response to the solid-state drive's disk load information being higher than a third load threshold, the cache timeout is reset; wherein, the first load threshold is less than the second load threshold, the second load threshold is less than the third load threshold, and the first preset value is greater than the second preset value.

[0054] Specifically, when the disk load information of the solid-state drive is lower than the first load threshold, the cache time limit is set to the first preset value. Since the first load threshold is set to the minimum, in order to make full use of the cache performance of the solid-state drive, the cache time limit is increased and set to the first preset value, such as six hours. Of course, it can also be set to other values, depending on the actual situation. This application does not impose specific restrictions here.

[0055] Furthermore, when the disk load information of the solid-state drive is between the first load threshold and the second load threshold, the first preset value remains unchanged. When the disk load information of the solid-state drive is between the second load threshold and the third load threshold, the cache time limit is reduced to the second preset value, for example, from six hours to two hours, so as to trigger data migration faster and transfer the target data from the solid-state drive to the mechanical hard drive. Of course, the reduced cache time limit can also be set to other values, depending on the actual situation. This application does not impose specific restrictions here.

[0056] Furthermore, when the SSD's disk load information exceeds the third load threshold, the cache timeout is reset, i.e., the cache timeout is lowered to 0, thereby enabling the target data to be immediately transferred to the mechanical hard drive to release the SSD's capacity and reduce pressure.

[0057] The first load threshold is less than the second load threshold, the second load threshold is less than the third load threshold, and the first load threshold, the second load threshold and the third load threshold can be set according to the actual situation. This application does not impose specific restrictions here.

[0058] In one implementation scenario, the data storage method proposed in this application further includes: in response to the fact that the hard disk load information of the solid-state drive is higher than the third load threshold for a period of time exceeding a preset duration threshold, obtaining the hard disk load information of the mechanical hard drive bound to the solid-state drive; and selecting the mechanical hard drive to be unbound from the solid-state drive based on the hard disk load information of all mechanical hard drives bound to the solid-state drive.

[0059] Specifically, when a solid-state drive (SSD) is under high load for an extended period, a warning is sent to the storage nodes of the storage system. The system also retrieves the real-time disk load information of the mechanical hard drives (HDDs) bound to the SSD, sorts the HDDs according to their load information, and prioritizes unbinding the HDD with the lowest load information. At this point, the SSD will no longer provide caching capabilities for the unbound HDD. Furthermore, the system retrieves the target data that would otherwise be transferred to the unbound HDD, writes this target data to the SSD, and then transfers it to other HDDs bound to the SSD. Immediately afterward, the SSD's capacity is released. This unbinding operation quickly reduces the cache pressure on the SSD, preventing performance crashes.

[0060] Understandably, the unbound mechanical hard drive will become a write-through mechanical hard drive until the SSD's load information drops to a normal range before it is rebound to the mechanical hard drive.

[0061] Optionally, other mechanical hard drives can also be selected for unbinding according to the actual situation. For example, mechanical hard drives with poor performance (aging, high failure rate, long response latency, etc.) can be unbound first to avoid the solid-state drive wasting cache resources as an inefficient device, thereby further improving the utilization of node resources. This application does not impose specific restrictions on how to select mechanical hard drives for unbinding.

[0062] In one implementation scenario, the process of writing target data to the solid-state drive of at least one storage node in the cache node queue also includes: in response to a solid-state drive failure in the cache node queue, obtaining the hard disk load information of the mechanical hard drive bound to the failed solid-state drive, and transferring the target data to the mechanical hard drive with the lowest load.

[0063] Specifically, during the process of writing target data to the solid-state drive of at least one storage node in the cache node queue, the method further includes: when the solid-state drive in the cache node queue fails, obtaining the hard disk load information of the mechanical hard drive bound to the failed solid-state drive, transferring the target data to the mechanical hard drive with the lowest load, and disabling the cache acceleration function of the failed solid-state drive.

[0064] In one implementation scenario, the process of transferring target data to at least some mechanical hard drives bound to a solid-state drive (SSD) further includes: in response to a mechanical hard drive failure in the cache node queue, obtaining the data to be backed up from the failed mechanical hard drive and writing it to a fixed hard drive bound to it, and transferring the data to be backed up from the SSD to other bound mechanical hard drives; wherein the data to be backed up includes the target data already written to the failed mechanical hard drive.

[0065] Specifically, the process of transferring target data to at least some mechanical hard drives bound to the solid-state drive also includes: when a mechanical hard drive in the cache node queue fails, retrieving the data to be backed up from the failed mechanical hard drive and writing it to a fixed hard drive bound to that mechanical hard drive, and transferring the data to be backed up from the solid-state drive to other bound mechanical hard drives, thereby reducing the possibility of data loss during hard drive failure and ensuring the integrity and security of data storage.

[0066] It should be noted that repaired solid-state drives or mechanical hard drives are not used in real-world applications. This is because when a hard drive is considered to be in a faulty state, it is usually because there are a large number of write errors on the hard drive, which has been forcibly taken offline by the server. If it is necessary to expand the storage node, a disk can be transferred from another storage node in the same cluster or a new formatted disk can be connected.

[0067] S205: In response to the direct storage node queue corresponding to the target data, write the target data to the mechanical hard disk of at least one storage node in the direct storage node queue.

[0068] Specifically, when the target data is determined to be written to the direct storage node queue, the target data is written to the mechanical hard disk of at least one storage node in the direct storage node queue.

[0069] In one embodiment, after step S203, the method further includes: obtaining node load information for each storage node in the node queue corresponding to the target data; and determining at least one storage node corresponding to the target data based on the node load information.

[0070] Specifically, in the node queue corresponding to the target data, the node load information of each storage node is obtained. Based on the obtained node load information, at least one storage node corresponding to the target data is determined. For example, storage nodes with low load are selected first for storage. This can distribute the data to storage nodes with less pressure, avoid a single node becoming a performance bottleneck due to data clustering, ensure the stable operation of the entire storage cluster, and reduce the risk of service interruption due to overload.

[0071] In one implementation scenario, the steps of obtaining the node load information of each storage node in the node queue corresponding to the target data specifically include: obtaining the storage load, processor load, memory load, network load, and their corresponding preset weight coefficients for each storage node in the node queue corresponding to the target data; wherein, when the target data corresponds to the cache node queue, the storage load is related to the solid-state drives and hard disk drives in the storage node, and when the target data corresponds to the direct storage node queue, the storage load is related to the hard disk drives in the storage node; and based on the storage load, processor load, memory load, network load, and their corresponding preset weight coefficients, a comprehensive load value is obtained.

[0072] Specifically, the formula for calculating the comprehensive load value is as follows: (1) in, For the overall load value, , , , These are the preset weight coefficients for each item, and their sum is 1. They can be adjusted according to actual business needs; for example, they can be increased in read / write I / O intensive scenarios. and , For storage load, For processor load, For memory load, For network load.

[0073] Furthermore, storage load The calculation formula is as follows: (2) in, , , , These are the used capacity and total capacity of the solid-state drive (SSD) and hard disk drive (HDD), respectively. , These represent the requests per second and the maximum capacity of a solid-state drive (SSD). , These are the weights of the storage media.

[0074] in, The calculation method is as follows: (3) in, and The numbers of solid-state drives and hard disk drives, respectively. and satisfy .

[0075] Furthermore, processor load Memory load Network load The calculation method is as follows: (4) (5) (6) Among them, storage load It increases with the increase of the used capacity of the storage medium. It increases with increasing CPU usage. It increases with increasing memory usage. Increases with the increase in network data volume, and various items are related to They are positively correlated, and the final calculation yields... A higher value indicates a higher load on the node.

[0076] It should be noted that for the direct storage node queue, when calculating the overall load value, the indicators corresponding to the solid-state drives are not calculated, only the indicators corresponding to the mechanical hard drives are calculated.

[0077] Furthermore, this application also provides a storage system for implementing the data storage methods described in the above embodiments. This storage system includes the following modules: a business parsing module for parsing storage business requests, including large files, small files, mixed-size files, and the allocated capacity required by the business; a node detection module for real-time acquisition of resource indicators of each storage node in the cluster, detection of node availability, and maintenance of the node queue; a scheduling decision module for determining the node queue based on business write requests, calculating the load based on a set method, prioritizing nodes with low load that meet business volume requirements, and generating scheduling decisions; a cluster scheduling module for executing scheduling decisions, and executing a reduced scheduling strategy when the selected node is unavailable or scheduling fails; and a node adaptation module for dynamically adjusting the distribution of cached data between solid-state drives and hard disk drives in the cache node queue (e.g., the target data to be stored in the cache space for a certain period), avoiding low utilization caused by full load or idling of solid-state drives.

[0078] Please see Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 300 and a processor 302 coupled to each other. The memory 300 stores program data (not shown). The processor 302 calls the program data to implement the method in any of the above embodiments. For related descriptions, please refer to the detailed description of the above method embodiments, which will not be repeated here. Specifically, the electronic device 30 includes: desktop computers, laptops, tablet computers, servers, etc., which are not limited here. In addition, the processor 302 can also be called a CPU (Center Processing Unit). The processor 302 may be an integrated circuit chip with signal processing capabilities. The processor 302 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. In addition, the processor 302 can be implemented by integrated circuit chips.

[0079] Please see Figure 4 , Figure 4This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0080] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data storage method, characterized in that, include: Obtain the cache node queue and direct storage node queue in the storage system; wherein each type of node queue includes multiple storage nodes, and the storage nodes include mechanical hard disks and solid-state drives; Acquire the target data and its corresponding data information, and determine the queue type corresponding to the target data based on the data information; In response to the target data corresponding to the cache node queue, the target data is written to the solid-state drive (SSD) of at least one storage node in the cache node queue; wherein, when the target data in the SSD meets the transfer conditions, at least a portion of the target data is transferred to the mechanical hard drive (HDD), and the transfer conditions are related to the amount of target data in the SSD and the cache duration; In response to the target data corresponding to the direct storage node queue, the target data is written to the hard disk of at least one storage node in the direct storage node queue.

2. The method according to claim 1, characterized in that, The acquisition of the cache node queue and direct storage node queue in the storage system includes: The available capacity and disk load information of the solid-state drives in each storage node are obtained in real time from the storage system. In response to the available capacity of the solid-state drive meeting a preset capacity threshold or the disk load information of the solid-state drive meeting a preset load threshold, the corresponding storage node of the solid-state drive is allocated to the cache node queue; otherwise, it is allocated to the direct storage node queue.

3. The method according to claim 1, characterized in that, The storage node includes multiple mechanical hard drives and at least one solid-state drive. On each storage node in the cache node queue, each solid-state drive is bound to at least a portion of the mechanical hard drives, and the binding relationship between the mechanical hard drives and the solid-state drives is unique. After writing the target data to the solid-state drive of at least one storage node in the cache node queue in response to the target data corresponding to the cache node queue, the method further includes: In response to the target data in the solid-state drive exceeding a preset data volume threshold, at least a portion of the target data exceeding the preset data volume threshold is transferred to the mechanical hard drive bound to the solid-state drive; or, In response to at least a portion of the target data in the solid-state drive exceeding the cache time limit, at least a portion of the target data exceeding the cache time limit is transferred to at least a portion of the mechanical hard drive bound to the solid-state drive; wherein the cache time limit is related to the hard drive load information of the solid-state drive.

4. The method according to claim 3, characterized in that, The cache duration is adjusted based on the following steps: In response to the solid-state drive's disk load information being lower than a first load threshold, the cache duration is determined to be a first preset value; In response to the solid-state drive's disk load information being between a second load threshold and a third load threshold, the cache duration is determined to be a second preset value; In response to the solid-state drive's disk load information being higher than a third load threshold, the cache timeout is reset; wherein the first load threshold is less than the second load threshold, the second load threshold is less than the third load threshold, and the first preset value is greater than the second preset value.

5. The method according to claim 4, characterized in that, Also includes: In response to the fact that the hard disk load information of the solid-state drive is higher than the third load threshold for a period of time exceeding a preset time threshold, the hard disk load information of the mechanical hard drive bound to the solid-state drive is obtained; Based on the disk load information of all mechanical hard drives bound to the solid-state drive, select the mechanical hard drive to be unbound from the solid-state drive.

6. The method according to claim 3, characterized in that, The process of writing the target data to the solid-state drive of at least one storage node in the cache node queue also includes: In response to a solid-state drive failure in the cache node queue, the disk load information of the mechanical hard drive bound to the failed solid-state drive is obtained, and the target data is transferred to the mechanical hard drive with the lowest load. The process of transferring the target data to at least a portion of the mechanical hard drive bound to the solid-state drive also includes: In response to a mechanical hard drive failure in the cache node queue, the system retrieves the data to be backed up from the failed mechanical hard drive and writes it to a fixed hard drive bound to it, and transfers the data to be backed up from the solid-state drive to other bound mechanical hard drives; wherein, the data to be backed up includes target data already written to the failed mechanical hard drive.

7. The method according to claim 1, characterized in that, After acquiring the target data and its corresponding data information, and determining the queue type corresponding to the target data based on the data information, the process further includes: In the node queue corresponding to the target data, obtain the node load information of each storage node; Based on the node load information, at least one storage node corresponding to the target data is determined.

8. The method according to claim 7, characterized in that, The step of obtaining the node load information of each storage node in the node queue corresponding to the target data includes: In the node queue corresponding to the target data, the storage load, processor load, memory load, network load and their corresponding preset weight coefficients of each storage node are obtained; wherein, when the target data corresponds to the cache node queue, the storage load is related to the solid-state drive and the hard disk drive in the storage node, and when the target data corresponds to the direct storage node queue, the storage load is related to the hard disk drive in the storage node; The comprehensive load value is obtained based on the storage load, the processor load, the memory load, the network load, and their corresponding preset weighting coefficients.

9. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the method according to any one of claims 1-8.

Images