A data processing method, device and equipment

Abstract

The present application provides a data processing method, device and equipment, the method comprising: obtaining target decision information corresponding to a capacity physical hard disk; determining a target time corresponding to a garbage collection operation based on the target decision information; performing a garbage collection operation on duplicate data in the capacity physical hard disk at the target time. Through the technical solution of the present application, a garbage collection operation can be performed at an appropriate time to avoid the garbage collection operation occupying the bandwidth of a data reading operation, so that the capacity physical hard disk can meet the demand for fast reading and writing.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method, device and equipment. Background Art

[0002] Cloud computing is a type of distributed computing, which means breaking down data computing programs into countless small programs through the network "cloud", and then analyzing and processing these small programs through a system composed of a large number of servers, and returning the processing results to the user. Cloud computing has the characteristics of virtualized dynamic scalability, on-demand deployment, high flexibility, high reliability, high cost performance, and scalability.

[0003] With the rapid development of cloud computing technology, the performance requirements for physical hard disks are also increasing. In order to support cloud computing technology, various types of physical hard disks can be used to store user data.

[0004] For example, there are two types of mainstream physical hard drives. The first type of physical hard drive has the characteristics of fast reading and writing speed but small capacity. When this type of physical hard drive is used to store user data, it cannot meet large-capacity storage requirements. For example, cloud computing scenarios usually require the storage of a large amount of user data, and physical hard drives cannot store all of this user data at the same time.

[0005] The second type of physical hard disk has the characteristics of large capacity but slow reading and writing speed, such as QLC (Quad Level Cell) physical hard disk. When this type of physical hard disk is used to store user data, it cannot meet the storage requirements of fast reading and writing. For example, cloud computing scenarios usually require fast writing and fast reading of data, and physical hard disks cannot support fast writing and fast reading of data. Summary of the invention

[0006] The present application provides a data processing method, which is applied to a storage device, wherein the storage device includes a high-speed physical hard disk and a capacity physical hard disk, user data is stored in the high-speed physical hard disk by an append write mode, and when the high-speed physical hard disk meets a migration condition, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by an append write mode, the method comprising:

[0007] Obtain target decision information corresponding to the physical hard disk of the capacity;

[0008] Determine a target time corresponding to a garbage collection operation based on the target decision information;

[0009] At the target time, a garbage collection operation is performed on duplicate data in the capacity physical hard disk.

[0010] Exemplarily, determining a target time corresponding to the garbage collection operation based on the target decision information includes: determining a target garbage collection threshold corresponding to the garbage collection operation based on the target decision information;

[0011] Determine a time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold, and determine the time as a target time corresponding to the garbage collection operation.

[0012] Exemplarily, the target decision information includes the remaining storage space of the capacity physical hard disk; and determining the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information includes:

[0013] If the remaining storage space is less than the preset first storage space threshold, the configured initial garbage collection threshold is reduced to obtain the target garbage collection threshold; or,

[0014] If the remaining storage space is greater than the preset second storage space threshold, then the configured initial garbage collection threshold is increased to obtain the target garbage collection threshold; or,

[0015] If the remaining storage space is not less than a preset first storage space threshold and not greater than a preset second storage space threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

[0016] Exemplarily, the target decision information includes a garbage ratio of the capacity physical hard disk, where the garbage ratio indicates a ratio of the amount of duplicate data to the total amount of data; and determining a target garbage collection threshold corresponding to the garbage collection operation based on the target decision information includes:

[0017] If the garbage ratio is less than a preset first garbage ratio threshold, then increasing the configured initial garbage collection threshold to obtain the target garbage collection threshold; or,

[0018] If the garbage ratio is greater than a preset second garbage ratio threshold, the configured initial garbage collection threshold is lowered to obtain the target garbage collection threshold; or,

[0019] If the garbage ratio is not less than a preset first garbage ratio threshold and not greater than a preset second garbage ratio threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

[0020] Exemplarily, the statistical period is divided into a plurality of time periods, and the target decision information includes the total number of IOs corresponding to each time period; for the total number of IOs corresponding to each time period, the total number of IOs is the sum of the number of write operations and the number of read operations for the high-speed physical hard disk and the number of read operations for the capacity physical hard disk within the time period;

[0021] The determining a target time corresponding to the garbage collection operation based on the target decision information includes:

[0022] Based on the total number of IOs corresponding to each time period, if the total number of IOs corresponding to the time period is greater than the preset IO number threshold, the target time corresponding to the garbage collection operation is determined based on the time period;

[0023] The target time is before the start time of the time period.

[0024] Exemplarily, when the high-speed physical hard disk meets the migration condition, migrating the user data in the high-speed physical hard disk to the capacity physical hard disk by appending write includes:

[0025] Obtaining hotspot data distribution information corresponding to the high-speed physical hard disk;

[0026] Determine a target migration threshold corresponding to the migration operation based on the hot data distribution information;

[0027] When the amount of user data in the high-speed physical hard disk reaches the target migration threshold, it is determined that the high-speed physical hard disk meets the migration condition, and the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending write.

[0028] Exemplarily, the hot data distribution information includes the total number of hot data, which is the sum of the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk; the target migration threshold corresponding to the migration operation is determined based on the hot data distribution information, including: if the total number of hot data is greater than a preset first number threshold, then increasing the configured initial migration threshold to obtain the target migration threshold; or,

[0029] If the total amount of the hotspot data is less than the preset second amount threshold, the configured initial migration threshold is lowered to obtain the target migration threshold; or,

[0030] If the total quantity of the hotspot data is not greater than a preset first quantity threshold and not less than a preset second quantity threshold, the initial migration threshold is determined as the target migration threshold.

[0031] Exemplarily, at the target time, performing a garbage collection operation on the duplicate data in the capacity physical hard disk includes: determining a storage block area where the duplicate data in the capacity physical hard disk is located, the storage block area includes a plurality of storage blocks, and the plurality of storage blocks includes a storage block having duplicate data;

[0032] If the plurality of storage blocks further include a target storage block without duplicate data, storing the user data in the target storage block to the capacity physical hard disk by appending write;

[0033] The user data in each storage block in the storage block area is deleted.

[0034] Exemplarily, the storage device uses a write shaping cache WSC architecture to store user data;

[0035] The read and write speed of the high-speed physical hard disk is greater than a preset speed threshold, and the storage capacity of the high-speed physical hard disk is less than a preset capacity threshold; the read and write speed of the capacity physical hard disk is not greater than the preset speed threshold, and the storage capacity of the capacity physical hard disk is not less than the preset capacity threshold.

[0036] The present application provides a data processing device, which is applied to a storage device, wherein the storage device includes a high-speed physical hard disk and a capacity physical hard disk, and the device includes:

[0037] A storage module, configured to store the user data in the high-speed physical hard disk by appending write after obtaining the user data; and to migrate the user data in the high-speed physical hard disk to the capacity physical hard disk by appending write when the high-speed physical hard disk meets the migration condition;

[0038] An acquisition module, used for acquiring target decision information corresponding to the capacity physical hard disk;

[0039] A determination module, configured to determine a target time corresponding to a garbage collection operation based on the target decision information;

[0040] The processing module is used to perform garbage collection operation on duplicate data in the capacity physical hard disk at the target time.

[0041] Exemplarily, when the determination module determines the target time corresponding to the garbage collection operation based on the target decision information, it is specifically used to: determine the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information; determine the time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold, and determine the time as the target time corresponding to the garbage collection operation;

[0042] Wherein, the target decision information includes the remaining storage space of the physical hard disk of the capacity; when the determination module determines the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, it is specifically used to: if the remaining storage space is less than a preset first storage space threshold, then reduce the configured initial garbage collection threshold to obtain the target garbage collection threshold; or, if the remaining storage space is greater than a preset second storage space threshold, then increase the initial garbage collection threshold to obtain the target garbage collection threshold; or, if the remaining storage space is not less than the preset first storage space threshold and not greater than the preset second storage space threshold, then determine the initial garbage collection threshold as the target garbage collection threshold;

[0043] Among them, the target decision information includes the garbage ratio of the capacity physical hard disk, and the garbage ratio represents the ratio of the amount of duplicate data to the total amount of data; the determination module determines the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, and is specifically used for: if the garbage ratio is less than a preset first garbage ratio threshold, then the configured initial garbage collection threshold is increased to obtain the target garbage collection threshold; or, if the garbage ratio is greater than a preset second garbage ratio threshold, then the initial garbage collection threshold is lowered to obtain the target garbage collection threshold; or, if the garbage ratio is not less than the preset first garbage ratio threshold and not greater than the preset second garbage ratio threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

[0044] Exemplarily, the statistical period is divided into multiple time periods, and the target decision information includes the total number of IOs corresponding to each time period; for the total number of IOs corresponding to each time period, the total number of IOs is the sum of the number of write operations and the number of read operations for the high-speed physical hard disk within the time period, and the number of read operations for the capacity physical hard disk; when the determination module determines the target time corresponding to the garbage collection operation based on the target decision information, it is specifically used to: based on the total number of IOs corresponding to each time period, if the total number of IOs corresponding to the time period is greater than a preset IO number threshold, determine the target time corresponding to the garbage collection operation based on the time period; wherein, the target time is before the start time of the time period.

[0045] Exemplarily, when the high-speed physical hard disk meets the migration condition, the storage module migrates the user data in the high-speed physical hard disk to the capacity physical hard disk by appending write, and is specifically used to: obtain the hot data distribution information corresponding to the high-speed physical hard disk; determine the target migration threshold corresponding to the migration operation based on the hot data distribution information; when the data volume of the user data in the high-speed physical hard disk reaches the target migration threshold, determine that the high-speed physical hard disk meets the migration condition, and migrate the user data in the high-speed physical hard disk to the capacity physical hard disk by appending write;

[0046] Among them, the hot data distribution information includes the total number of hot data, and the total number of hot data is the sum of the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk; the storage module determines the target migration threshold corresponding to the migration operation based on the hot data distribution information, which is specifically used for: if the total number of hot data is greater than the preset first number threshold, then increase the configured initial migration threshold to obtain the target migration threshold; or, if the total number of hot data is less than the preset second number threshold, then reduce the configured initial migration threshold to obtain the target migration threshold; or, if the total number of hot data is not greater than the preset first number threshold and not less than the preset second number threshold, then determine the initial migration threshold as the target migration threshold.

[0047] Exemplarily, when the processing module performs a garbage collection operation on the duplicate data in the capacity physical hard disk at a target time, it is specifically used to: determine a storage block area where the duplicate data in the capacity physical hard disk is located, the storage block area includes multiple storage blocks, and the multiple storage blocks include storage blocks with duplicate data; if the multiple storage blocks also include a target storage block without duplicate data, then store the user data in the target storage block to the capacity physical hard disk by appending writes; and delete the user data in each storage block in the storage block area.

[0048] The present application provides a storage device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the data processing method disclosed in the above example of the present application.

[0049] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions to implement the data processing method disclosed in the above example of the present application.

[0050] As can be seen from the above technical solutions, in the embodiment of the present application, the storage device includes a high-speed physical hard disk and a capacity physical hard disk, the high-speed physical hard disk is a physical hard disk with fast reading and writing speed but small capacity, and the capacity physical hard disk is a physical hard disk with large capacity but slow reading and writing speed. When storing user data, the user data is stored in the high-speed physical hard disk by appending write mode, so as to meet the storage requirements of fast reading and writing, that is, it can support fast writing data and fast reading data. When the high-speed physical hard disk meets the migration conditions, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending write mode, so as to meet the storage requirements of large capacity, that is, a large amount of user data can be stored at the same time through the capacity physical hard disk. The target time corresponding to the garbage collection operation can be determined, and the garbage collection operation is performed on the duplicate data in the capacity physical hard disk at the target time, so as to perform the garbage collection operation at the appropriate time, avoid the garbage collection operation to seize the bandwidth of the data reading operation, so that the capacity physical hard disk can meet the requirements of fast reading and writing, and the user's read traffic can meet the product requirements, avoid the traffic occupation and data loss problems caused by insufficient storage space, ensure that the user requirements are met in the mixed scene of large traffic reading and writing, and make the storage device in a dynamic balance steady state. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of a data writing process and a data reading process in one embodiment of the present application;

[0052] Figure 2 It is a flowchart of a data processing method in one embodiment of the present application;

[0053] Figure 3 is a schematic diagram of a data writing process and a data reading process in one embodiment of the present application;

[0054] Figure 4 is a schematic diagram of the structure of a fuzzy controller in one embodiment of the present application;

[0055] Figure 5 is a schematic diagram of the structure of a data processing device in one embodiment of the present application;

[0056] Figure 6 It is a hardware structure diagram of a storage device in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0058] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0059] With the rapid development of cloud computing technology, the performance requirements for physical hard disks are also increasing. In order to support cloud computing technology, various types of physical hard disks can be used to store user data. In order to meet the needs of low latency and high throughput, a solution for storing user data using a WSC (Write Shaping Cache) architecture is proposed in the embodiment of the present application. The WSC architecture is a cache technology.

[0060] In the embodiment of the present application, the storage device includes a high-speed physical hard disk and a capacity physical hard disk, and the storage device uses the WSC architecture to store user data, that is, the high-speed physical hard disk and the capacity physical hard disk are used to store user data. When the high-speed physical hard disk and the capacity physical hard disk are used to store user data, the high-speed physical hard disk is used as a cache of the capacity physical hard disk to accelerate the read and write speed of the capacity physical hard disk.

[0061] Among them, the high-speed physical hard disk is a physical hard disk with fast reading and writing speed and small capacity. It has the characteristics of fast reading and writing speed but small capacity. The high-speed physical hard disk adopts ultra-high-speed memory technology to realize data storage. This embodiment does not limit the type of this high-speed physical hard disk, as long as it has the characteristics of fast reading and writing speed but small capacity.

[0062] For example, the read and write speed of the high-speed physical hard disk is greater than the preset speed threshold, and the storage capacity of the high-speed physical hard disk is less than the preset capacity threshold. The preset speed threshold can be configured based on experience, and when the read and write speed of the high-speed physical hard disk is greater than the preset speed threshold, it indicates that the read and write speed is fast. The preset capacity threshold can be configured based on experience, and when the storage capacity of the high-speed physical hard disk is less than the preset capacity threshold, it indicates that the capacity is small.

[0063] Among them, the capacity physical hard disk is a physical hard disk with slow reading and writing speed and large capacity, and has the characteristics of large capacity but slow reading and writing speed, such as QLC physical hard disk, etc. QLC physical hard disk is a large-capacity SSD (Solid State Disk, solid-state hard disk), and the voltage of QLC physical hard disk can be changed from 0000 to 1111 in a total of 16 kinds. Of course, QLC physical hard disk is only an example of capacity physical hard disk. In this embodiment, there is no restriction on the type of this capacity physical hard disk, as long as it has the characteristics of large capacity but slow reading and writing speed.

[0064] For example, the read and write speed of the capacity physical hard disk is not greater than the preset speed threshold, and the storage capacity of the capacity physical hard disk is not less than the preset capacity threshold. The preset speed threshold is configured based on experience. When the read and write speed of the capacity physical hard disk is not greater than the preset speed threshold, it indicates that the read and write speed is slow. The preset capacity threshold is configured based on experience. When the storage capacity of the capacity physical hard disk is not less than the preset capacity threshold, it indicates that the capacity is large.

[0065] See also Figure 1 , which is a schematic diagram of the data writing process and the data reading process under the WSC architecture.

[0066] In the data writing process, user data is stored in the high-speed physical hard disk by appending write, that is, the user data is appended to the data written to the high-speed physical hard disk. When the high-speed physical hard disk meets the migration conditions, the user data in the high-speed physical hard disk can be migrated to the capacity physical hard disk by appending write, that is, the user data is appended to the data written to the capacity physical hard disk.

[0067] For example, for user data A to be written, user data A can be stored in a high-speed physical hard disk through an append write method, and the physical address of user data A in the high-speed physical hard disk can be determined, recorded as physical address A1, and the logical address corresponding to user data A can be determined, recorded as logical address A2.

[0068] Exemplarily, an L2P (Logic to Physis) table may be stored, such as in a memory, and the L2P table may be persisted to a sector of a high-speed physical hard disk, and the L2P table is used to record the mapping relationship between the logical address and the logical physics. On this basis, for user data A, the mapping relationship between the logical address A2 and the physical address A1 may be recorded in the L2P table, as shown in Table 1.

[0069] For another example, for user data B to be written, user data B can be stored in a high-speed physical hard disk through an append write method, the physical address B1 of user data B in the high-speed physical hard disk is determined, and the logical address B2 corresponding to user data B is determined, and the mapping relationship between logical address B2 and physical address B1 is recorded in the L2P table. Similarly, each user data can be stored in a high-speed physical hard disk.

[0070] Table 1

[0071] Logical address Physical Address A2 A1 B2 B1 ... ...

[0072] In one possible implementation, an initial migration threshold can be configured for the high-speed physical hard disk. When the amount of user data in the high-speed physical hard disk reaches the initial migration threshold, it is determined that the high-speed physical hard disk meets the migration conditions, and the user data in the high-speed physical hard disk (which may be all or part of the user data in the high-speed physical hard disk) is migrated to the capacity physical hard disk through an append write.

[0073] For example, the initial migration threshold m1 can be pre-configured. The initial migration threshold m1 can be configured based on experience and is not limited. The initial migration threshold m1 is used to represent the data volume threshold. For each user data, after the user data is stored in the high-speed physical hard disk, it is determined whether the data volume of the user data in the high-speed physical hard disk reaches the initial migration threshold m1. If not, the new user data continues to be stored in the high-speed physical hard disk. If yes, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk through an append write mode (append mode), and the migrated user data is deleted from the high-speed physical hard disk. In addition, for new user data, the new user data continues to be stored in the high-speed physical hard disk.

[0074] Among them, after migrating the user data in the high-speed physical hard disk to the capacity physical hard disk, it is also necessary to determine the physical address of the user data in the capacity physical hard disk, such as user data A corresponds to physical address A3 in the capacity physical hard disk, user data B corresponds to physical address B3 in the capacity physical hard disk, and so on.

[0075] On this basis, the physical addresses in the high-speed physical hard disk in the L2P table may also be updated to the physical addresses in the capacity physical hard disk. See Table 2 for an example of the updated L2P table.

[0076] Table 2

[0077]

[0078]

[0079] Among them, when migrating user data in the high-speed physical hard disk to the capacity physical hard disk by appending write, the user data in the high-speed physical hard disk can be deduplicated, and the user data after the deduplication operation can be migrated to the capacity physical hard disk. For example, since the user data is stored in the high-speed physical hard disk by appending write, there may be multiple duplicate user data in the high-speed physical hard disk, that is, duplicate user data generated at different times. For these duplicate user data, during data migration, these duplicate user data can be deduplicated, and the user data after the deduplication operation can be migrated to the capacity physical hard disk, such as migrating the last user data to the capacity physical hard disk, deleting the user data from the high-speed physical hard disk, and the remaining user data is directly deleted from the high-speed physical hard disk and no longer migrated to the capacity physical hard disk.

[0080] When user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending write, the user data in the high-speed physical hard disk can be migrated to the capacity physical hard disk based on the granularity of band (storage block area). Figure 1 As shown, the storage block area may include multiple storage blocks. When migrating user data, multiple user data are migrated to the multiple storage blocks in the storage block area at one time.

[0081] For the data reading process, the logical address corresponding to the user data can be determined, and the L2P table can be queried through the logical address to obtain the physical address corresponding to the user data, and the user data can be read based on the physical address. For example, the user data may be read from a high-speed physical hard disk based on the physical address, or the user data may be read from a capacity physical hard disk based on the physical address. For example, for user data A, the logical address A2 corresponding to user data A is determined, and the L2P table is queried through the logical address A2. If user data A is currently in a high-speed physical hard disk, the L2P table is shown in Table 1, and the physical address A1 is queried from the L2P table, and the user data A corresponding to the physical address A1 is read from the high-speed physical hard disk. If user data A is currently in a capacity physical hard disk, the L2P table is shown in Table 2, and the physical address A3 is queried from the L2P table, and the user data A corresponding to the physical address A1 is read from the capacity physical hard disk.

[0082] In a possible implementation, since user data in a high-speed physical hard disk is migrated to a capacity physical hard disk by an append write method, multiple migration processes may migrate multiple duplicate user data to the capacity physical hard disk, that is, the capacity physical hard disk may have multiple duplicate user data, that is, duplicate user data generated at different times. In order to save storage resources of the capacity physical hard disk, a GC (Garbage Collection) operation may be performed on the capacity physical hard disk, that is, a deduplication operation is performed on the duplicate user data in the capacity physical hard disk, and the user data after the deduplication operation is retained in the capacity physical hard disk, while the remaining user data is deleted from the capacity physical hard disk. For example, the last user data may be retained in the capacity physical hard disk, while the remaining user data may be deleted from the capacity physical hard disk.

[0083] For the garbage collection process, garbage collection operations are usually performed on the capacity physical hard disk at a regular interval, such as performing garbage collection operations on the capacity physical hard disk every 10 hours to complete data deduplication.

[0084] In summary, data operations on the capacity physical hard disk involve three aspects: migrating user data in the high-speed physical hard disk to the capacity physical hard disk (recorded as data write operation), reading user data from the capacity physical hard disk (recorded as data read operation), and performing garbage collection operations on duplicate user data in the capacity physical hard disk (recorded as garbage collection operations). Since the total bandwidth of the capacity physical hard disk remains unchanged, the data write operation, data read operation and garbage collection operation jointly occupy the total bandwidth of the capacity physical hard disk.

[0085] Obviously, when performing garbage collection operations and data reading operations at the same time, the larger the bandwidth occupied by the garbage collection operation, the smaller the bandwidth occupied by the data reading operation. That is, the garbage collection operation will affect the bandwidth occupied by the data reading operation, which greatly reduces the bandwidth occupied by the data reading operation and may not be able to meet the needs of the data reading operation, that is, the data reading operation cannot be completed in time, and the user data cannot be returned to the user in time.

[0086] In response to the above problems, in an embodiment of the present application, the target time corresponding to the garbage collection operation can be determined, and the garbage collection operation can be performed on the duplicate data in the capacity physical hard disk at the target time, so that the garbage collection operation can be performed at the appropriate time to avoid the garbage collection operation occupying the bandwidth of the data reading operation, so that the capacity physical hard disk can meet the needs of the data reading operation, complete the data reading operation in a timely manner, and return the user data to the user in a timely manner, so that the user's read traffic can meet the product requirements, ensuring that user requirements are met in a large-traffic mixed read and write scenario, so that the garbage collection operation and the data reading operation do not affect each other.

[0087] The technical solution of the embodiments of the present application is described below in conjunction with specific embodiments.

[0088] In the embodiment of the present application, a data processing method is proposed, which can be applied to a storage device. The storage device may include a high-speed physical hard disk and a capacity physical hard disk. The high-speed physical hard disk is a physical hard disk with a fast read and write speed and a small capacity, and the capacity physical hard disk is a physical hard disk with a slow read and write speed and a large capacity. Among them, user data is stored in the high-speed physical hard disk by appending writes. When the high-speed physical hard disk meets the migration conditions, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending writes. In the above application scenario, see Figure 2 FIG. 1 is a flow chart of a data processing method, which may include:

[0089] Step 201: Obtain target decision information corresponding to a capacity physical hard disk.

[0090] Step 202: Determine a target time corresponding to the garbage collection operation based on the target decision information.

[0091] Step 203: At the target time, garbage collection is performed on duplicate data in the capacity physical hard disk.

[0092] In one possible implementation, determining the target time corresponding to the garbage collection operation based on the target decision information may include but is not limited to: determining the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information; determining the time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold, and determining this time as the target time corresponding to the garbage collection operation.

[0093] Exemplarily, when determining the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, the target decision information may include the remaining storage space of the capacity physical hard disk. If the remaining storage space is less than the preset first storage space threshold, the configured initial garbage collection threshold (which can be configured based on experience and is not limited to this) is lowered to obtain the target garbage collection threshold; or, if the remaining storage space is greater than the preset second storage space threshold, the initial garbage collection threshold is increased to obtain the target garbage collection threshold; or, if the remaining storage space is not less than the preset first storage space threshold and not greater than the preset second storage space threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

[0094] Exemplarily, when determining the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, the target decision information may include the garbage ratio of the capacity physical hard disk, and the garbage ratio is used to represent the ratio of the amount of duplicate data to the total amount of data. On this basis, if the garbage ratio is less than the preset first garbage ratio threshold, the configured initial garbage collection threshold is increased to obtain the target garbage collection threshold; or, if the garbage ratio is greater than the preset second garbage ratio threshold, the initial garbage collection threshold is lowered to obtain the target garbage collection threshold; or, if the garbage ratio is not less than the preset first garbage ratio threshold and not greater than the preset second garbage ratio threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

[0095] In another possible implementation, the statistical period is divided into multiple time periods, and the target decision information may include the total number of IOs corresponding to each time period; for the total number of IOs corresponding to each time period, the total number of IOs is the sum of the number of write operations and the number of read operations for the high-speed physical hard disk and the number of read operations for the capacity physical hard disk within the time period. On this basis, the target time corresponding to the garbage collection operation is determined based on the target decision information, which may include but is not limited to: based on the total number of IOs corresponding to each time period, if the total number of IOs corresponding to the time period is greater than a preset IO number threshold, then the target time corresponding to the garbage collection operation is determined based on the time period, and the target time is before the start time of the time period.

[0096] In one possible implementation, when the high-speed physical hard disk meets the migration conditions, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by an additional write method, which may include but is not limited to the following methods: obtaining hot data distribution information corresponding to the high-speed physical hard disk, and determining the target migration threshold corresponding to the migration operation based on the hot data distribution information; on this basis, when the data volume of the user data in the high-speed physical hard disk reaches the target migration threshold, it is determined that the high-speed physical hard disk meets the migration conditions, and the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by an additional write method.

[0097] Exemplarily, the hotspot data distribution information may include the total number of hotspot data, which is the sum of the number of hotspot data in the high-speed physical hard disk and the number of hotspot data in the capacity physical hard disk; based on this, the target migration threshold corresponding to the migration operation is determined based on the hotspot data distribution information, which may include but is not limited to: if the total number of hotspot data is greater than a preset first number threshold, then the configured initial migration threshold is increased (it can be configured based on experience and there is no restriction on this) to obtain the target migration threshold; or, if the total number of hotspot data is less than a preset second number threshold, then the initial migration threshold is lowered to obtain the target migration threshold; or, if the total number of hotspot data is not greater than the preset first number threshold and not less than the preset second number threshold, then the initial migration threshold is determined as the target migration threshold.

[0098] In a possible implementation, at the target time, garbage collection operations are performed on duplicate data in the capacity physical hard disk, which may include but is not limited to: determining a storage block area where duplicate data in the capacity physical hard disk is located, the storage block area includes multiple storage blocks, and the multiple storage blocks include storage blocks with duplicate data; if the multiple storage blocks also include a target storage block without duplicate data, then storing the user data in the target storage block to the capacity physical hard disk by appending writes; deleting the user data in each storage block in the storage block area (storage blocks with duplicate data and target storage blocks without duplicate data). From the above, it can be seen that the user data in the storage block with duplicate data is not appended to the capacity physical hard disk, and the user data in the target storage block without duplicate data is appended to the capacity physical hard disk.

[0099] As can be seen from the above technical solutions, in the embodiment of the present application, the storage device includes a high-speed physical hard disk and a capacity physical hard disk, the high-speed physical hard disk is a physical hard disk with fast reading and writing speed but small capacity, and the capacity physical hard disk is a physical hard disk with large capacity but slow reading and writing speed. When storing user data, the user data is stored in the high-speed physical hard disk by appending write mode, so as to meet the storage requirements of fast reading and writing, that is, it can support fast writing data and fast reading data. When the high-speed physical hard disk meets the migration conditions, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending write mode, so as to meet the storage requirements of large capacity, that is, a large amount of user data can be stored at the same time through the capacity physical hard disk. The target time corresponding to the garbage collection operation can be determined, and the garbage collection operation is performed on the duplicate data in the capacity physical hard disk at the target time, so as to perform the garbage collection operation at the appropriate time, avoid the garbage collection operation to seize the bandwidth of the data reading operation, so that the capacity physical hard disk can meet the requirements of fast reading and writing, and the user's read traffic can meet the product requirements, avoid the traffic occupation and data loss problems caused by insufficient storage space, ensure that the user requirements are met in the mixed scene of large traffic reading and writing, and make the storage device in a dynamic balance steady state.

[0100] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0101] See also Figure 3 As shown, the present application involves the WSC architecture layer, the system detection layer and the flow control decision layer.

[0102] For the WSC architecture layer, it may involve data write operations for high-speed physical hard disks (such as writing user data to high-speed physical hard disks), data read operations for high-speed physical hard disks (such as reading user data from high-speed physical hard disks), data write operations for capacity physical hard disks (such as migrating user data in high-speed physical hard disks to capacity physical hard disks), data read operations for capacity physical hard disks (such as reading user data from capacity physical hard disks), and garbage collection operations for capacity physical hard disks (such as garbage collection of duplicate user data in capacity physical hard disks, i.e., deleting duplicate user data).

[0103] For the system detection layer, the high-speed physical hard disk and the capacity physical hard disk can be detected to obtain system information, and the system information is provided to the flow control decision layer. The system information may include but is not limited to target decision information and / or hot data distribution information. Of course, the above information is only an example and is not limited to this.

[0104] The target decision information may include but is not limited to at least one of the following: remaining storage space of the physical hard disk; garbage ratio of the physical hard disk; total number of IOs corresponding to each time period. Of course, the above are just a few examples of target decision information, and there is no limitation on this target decision information.

[0105] With respect to the remaining storage space of the capacity physical hard disk, the system detection layer can periodically detect the remaining storage space of the capacity physical hard disk, that is, the size of the unoccupied storage space in the capacity physical hard disk. The larger the remaining storage space, the larger the unoccupied storage space in the capacity physical hard disk, that is, the larger the storage space used to store new user data. Conversely, the smaller the remaining storage space, the smaller the unoccupied storage space in the capacity physical hard disk, that is, the smaller the storage space used to store new user data.

[0106] With respect to the garbage ratio of the capacity physical hard disk, the system detection layer can periodically detect the garbage ratio of the capacity physical hard disk, that is, the ratio of the amount of duplicate data (the amount of duplicate data in the capacity physical hard disk) to the total amount of data (the amount of all user data in the capacity physical hard disk). For example, assuming that the amount of duplicate data in the capacity physical hard disk is k1, and the amount of all user data in the capacity physical hard disk is k2, the garbage ratio is k1 / k2. The larger the garbage ratio, the more user data in the capacity physical hard disk needs to be garbage collected, and the longer the garbage collection time is. Conversely, the smaller the garbage ratio, the less user data in the capacity physical hard disk needs to be garbage collected, and the shorter the garbage collection time is.

[0107] For the total number of IOs corresponding to each time period, the statistical cycle can be divided into multiple time periods, and the target decision information can include the total number of IOs corresponding to each time period. Among them, for the total number of IOs corresponding to each time period, the total number of IOs can be the sum of the number of write operations and the number of read operations for the high-speed physical hard disk and the number of read operations for the capacity physical hard disk within the time period. The system detection layer can periodically detect the number of write operations and the number of read operations for the high-speed physical hard disk, and the number of read operations for the capacity physical hard disk. Based on the above information, the total number of IOs corresponding to each time period can be determined.

[0108] For example, assuming that the statistical cycle is 24 hours, the statistical cycle can be divided into 24 time periods, each time period corresponding to 1 hour. Of course, the statistical cycle can also be divided into 48 time periods, 12 time periods, etc., without limitation. The following will take time periods 1 to 24 as an example for explanation.

[0109] For time period 1, the total IO quantity p11 corresponding to time period 1 of statistical cycle 1 (such as the first day) can correspond to the total IO quantity p12 corresponding to time period 1 of statistical cycle 2 (such as the second day), and so on. The total IO quantity p11 can be the number of write operations on the high-speed physical hard disk in time period 1 of statistical cycle 1, plus the number of read operations on the high-speed physical hard disk, plus the number of read operations on the capacity physical hard disk, that is, the sum of the above three values. The total IO quantity p12 can be the number of write operations on the high-speed physical hard disk in time period 1 of statistical cycle 2, plus the number of read operations on the high-speed physical hard disk, plus the number of read operations on the capacity physical hard disk, that is, the sum of the above three values, and so on.

[0110] In summary, the total IO quantity corresponding to time period 1 can be obtained, such as the total IO quantity p11, the total IO quantity p12, etc. Similarly, the total IO quantity corresponding to time period 2 to time period 24 can be obtained. So far, the total IO quantity corresponding to each time period can be obtained, and each time period can correspond to multiple total IO quantities.

[0111] The hotspot data distribution information may include but is not limited to the total number of hotspot data. Of course, the total number of hotspot data is only an example of the hotspot data distribution information, and there is no limitation on this hotspot data distribution information.

[0112] The total number of hot data may be the sum of the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk. For example, the system detection layer may periodically detect the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk. Based on the above two numbers of hot data, the total number of hot data may be determined. Hot data refers to user data that is frequently read and written. For example, for a certain user data, if the number of read and write operations for the user data is greater than a preset threshold, the user data may be regarded as hot data.

[0113] For the flow control decision layer, the target time corresponding to the garbage collection operation can be determined based on the system information, and then the WSC architecture layer performs garbage collection operations on duplicate data in the capacity physical hard disk at the target time. For example, the target time corresponding to the garbage collection operation can be determined based on the target decision information. And / or, the target migration threshold corresponding to the migration operation can be determined based on the system information, and then the WSC architecture layer migrates user data in the high-speed physical hard disk to the capacity physical hard disk based on the target migration threshold. For example, the target migration threshold corresponding to the migration operation can be determined based on the hot data distribution information.

[0114] The following combination Figure 3 The application scenario shown explains the data write operation for a high-speed physical hard disk, the data read operation for a high-speed physical hard disk, the data write operation for a capacity physical hard disk, the data read operation for a capacity physical hard disk, and the garbage collection operation for a capacity physical hard disk.

[0115] First, data writing operation for high-speed physical hard disk. For this data writing operation, user data can be stored in the high-speed physical hard disk by appending writing, that is, the user data is appended to the data written in the high-speed physical hard disk. After the user data is stored in the high-speed physical hard disk, the physical address of the user data in the high-speed physical hard disk can also be determined, and the logical address corresponding to the user data can be determined, and the mapping relationship between the logical address and the physical address can be recorded in the L2P table.

[0116] The WSC architecture layer may implement data writing operations on high-speed physical hard disks.

[0117] Second, for the data write operation of the capacity physical hard disk, for the data write operation, an initial migration threshold value, also called the basic migration water level of the high-speed physical hard disk, can be configured for the high-speed physical hard disk.

[0118] The system detection layer can obtain the hot data distribution information corresponding to the high-speed physical hard disk, such as the total number of hot data, which is the sum of the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk, and send the total number of hot data to the flow control decision layer.

[0119] After obtaining the total amount of hot data, the flow control decision layer can adjust the initial migration threshold based on the total amount of hot data to obtain the target migration threshold. The target migration threshold can also be called the target migration water level of the high-speed physical hard disk, or the dynamic migration water level, or the incremental migration water level.

[0120] For example, if the total number of hotspot data is greater than the preset first quantity threshold (which can be configured based on experience), the initial migration threshold is increased to obtain the target migration threshold. When the total number of hotspot data is greater than the preset first quantity threshold, it means that the hotspot distribution is relatively high, and the data failure rate is relatively high at this time. Therefore, the initial migration threshold can be increased to increase the basic migration water level, thereby reducing the data failure rate.

[0121] For another example, if the total number of hotspot data is less than a preset second number threshold (which can be configured based on experience), the initial migration threshold is lowered to obtain a target migration threshold. When the total number of hotspot data is less than the preset second number threshold, it means that the hotspot distribution is relatively low, and the data failure rate is relatively low at this time. Therefore, the initial migration threshold can be lowered to reduce the basic migration water level, thereby completing the data migration operation in a timely manner.

[0122] For another example, if the total number of hotspot data is not greater than the preset first number threshold and not less than the preset second number threshold, the initial migration threshold is determined as the target migration threshold. Wherein, when the total number of hotspot data is not greater than the preset first number threshold and not less than the preset second number threshold, it indicates that the hotspot distribution is relatively appropriate, and therefore, the initial migration threshold can be kept unchanged, so that the basic migration water level remains unchanged.

[0123] In the above embodiment, the preset first quantity threshold may be greater than the preset second quantity threshold.

[0124] After obtaining the target migration threshold, the flow control decision layer may send the target migration threshold to the WSC architecture layer, and the WSC architecture layer implements the data write operation for the capacity physical hard disk based on the target migration threshold.

[0125] Specifically, when the amount of user data in the high-speed physical hard disk reaches the target migration threshold (the target migration threshold is used to indicate the data amount threshold), it is determined that the high-speed physical hard disk meets the migration condition, and the user data in the high-speed physical hard disk (which may be all or part of the user data in the high-speed physical hard disk) is migrated to the capacity physical hard disk by appending. For example, for each user data, after the user data is stored in the high-speed physical hard disk, it is determined whether the amount of user data in the high-speed physical hard disk reaches the target migration threshold. If not, the new user data continues to be stored in the high-speed physical hard disk. If yes, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending, and the migrated user data is deleted from the high-speed physical hard disk.

[0126] Among them, after migrating the user data in the high-speed physical hard disk to the capacity physical hard disk, it is also necessary to determine the physical address of the user data in the capacity physical hard disk, and update the physical address in the high-speed physical hard disk in the L2P table to the physical address in the capacity physical hard disk to obtain the updated L2P table.

[0127] Among them, when migrating user data in the high-speed physical hard disk to the capacity physical hard disk by appending write, the user data in the high-speed physical hard disk can be deduplicated, and the user data after the deduplication operation can be migrated to the capacity physical hard disk. For example, since the user data is stored in the high-speed physical hard disk by appending write, that is, there may be multiple duplicate user data in the high-speed physical hard disk, for these duplicate user data, when migrating data, it is necessary to migrate the user data after the deduplication operation to the capacity physical hard disk, such as migrating the last user data to the capacity physical hard disk, deleting the user data from the high-speed physical hard disk, and directly deleting the remaining user data from the high-speed physical hard disk, and no longer migrating to the capacity physical hard disk.

[0128] When user data in a high-speed physical hard disk is migrated to a capacity physical hard disk by appending write, the user data in the high-speed physical hard disk can be migrated to the capacity physical hard disk based on the granularity of band (storage block area). For example, a storage block area can include multiple storage blocks, and when user data is migrated, multiple user data are migrated to multiple storage blocks in the storage block area at one time.

[0129] Third, data reading operations for high-speed physical hard disks and data reading operations for capacity physical hard disks. For the data reading operation, the logical address corresponding to the user data can be determined, and the L2P table can be queried through the logical address to obtain the physical address corresponding to the user data, and the user data can be read based on the physical address. For example, if the physical address is a physical address in a high-speed physical hard disk, then based on the physical address queried in the L2P table, the user data corresponding to the physical address is read from the high-speed physical hard disk. Alternatively, if the physical address is a physical address in a capacity physical hard disk, then based on the physical address queried in the L2P table, the user data corresponding to the physical address is read from the capacity physical hard disk.

[0130] The WSC architecture layer may implement data reading operations for high-speed physical hard disks, and the WSC architecture layer may implement data reading operations for capacity physical hard disks.

[0131] Fourth, garbage collection operation for the capacity physical hard disk. For the garbage collection operation, target decision information can be obtained, and the target time corresponding to the garbage collection operation can be determined based on the target decision information. At the target time, garbage collection operation, i.e., GC operation, is performed on the duplicate data in the capacity physical hard disk.

[0132] The following describes the process of determining the “target moment” in combination with several specific methods.

[0133] Method 1: The target decision information may include the remaining storage space of the capacity physical hard disk. For Method 1, an initial garbage collection threshold may be configured for the capacity physical hard disk, also known as the basic garbage collection water level of the capacity physical hard disk. The system detection layer may obtain the remaining storage space of the capacity physical hard disk and send the remaining storage space to the flow control decision layer. The flow control decision layer adjusts the initial garbage collection threshold based on the remaining storage space to obtain a target garbage collection threshold, which may also be referred to as the target garbage collection water level of the capacity physical hard disk, or the dynamic garbage collection water level, or the incremental garbage collection water level.

[0134] For example, if the remaining storage space is less than the preset first storage space threshold (which can be configured based on experience), the initial garbage collection threshold is lowered to obtain the target garbage collection threshold. When the remaining storage space is less than the preset first storage space threshold, it means that the remaining storage space is relatively small. Therefore, the initial garbage collection threshold is lowered to reduce the basic garbage collection water level and release more storage space to prevent sudden bursts (i.e., sudden increases in IO traffic), i.e., to avoid garbage collection operations from occupying bandwidth when IO traffic suddenly increases.

[0135] For another example, if the remaining storage space is greater than a preset second storage space threshold (which can be configured based on experience), the initial garbage collection threshold is increased to obtain a target garbage collection threshold. When the remaining storage space is greater than the preset second storage space threshold, it means that there is more remaining storage space. Therefore, the initial garbage collection threshold is increased to increase the basic garbage collection water level, thereby reducing the frequency of garbage collection operations, which is beneficial to reducing the number of garbage collection operations, thereby saving bandwidth for garbage collection operations.

[0136] For another example, if the remaining storage space is not less than the preset first storage space threshold and not greater than the preset second storage space threshold, the initial garbage collection threshold is determined as the target garbage collection threshold. Wherein, when the remaining storage space is not less than the preset first storage space threshold and not greater than the preset second storage space threshold, it means that the remaining storage space is relatively appropriate, therefore, the initial garbage collection threshold can be kept unchanged, so that the basic garbage collection water level remains unchanged, that is, the initial garbage collection threshold is used as the target garbage collection threshold.

[0137] In the above embodiment, the preset first storage space threshold may be smaller than the preset second storage space threshold.

[0138] After obtaining the target garbage collection threshold, the flow control decision layer can send the target garbage collection threshold to the WSC architecture layer, and the WSC architecture layer determines the target time corresponding to the garbage collection operation based on the target garbage collection threshold. For example, the time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold is determined, and this time is determined as the target time corresponding to the garbage collection operation.

[0139] For example, the amount of duplicate data in the physical hard disk of the capacity can be counted periodically, and it can be determined whether the amount of data reaches the target garbage collection threshold. If not, the amount of duplicate data in the physical hard disk of the capacity can be counted again in the next cycle. If yes, the current time can be used as the target time.

[0140] At the target moment, the WSC architecture layer can perform garbage collection operations on duplicate data in the capacity physical hard disk, that is, deduplicate the duplicate user data in the capacity physical hard disk, retain the user data after the deduplication operation in the capacity physical hard disk, and delete the remaining user data from the capacity physical hard disk, such as retaining the last user data in the capacity physical hard disk and deleting the remaining user data.

[0141] Method 2: The target decision information may include the garbage ratio of the capacity physical hard disk, which indicates the ratio of the amount of duplicate data to the total amount of data. For method 2, the initial garbage collection threshold, also called the basic garbage collection water level, may be configured for the capacity physical hard disk. The system detection layer may obtain the garbage ratio of the capacity physical hard disk and send the garbage ratio to the flow control decision layer. The flow control decision layer adjusts the initial garbage collection threshold based on the garbage ratio to obtain the target garbage collection threshold of the capacity physical hard disk.

[0142] For example, if the garbage ratio is less than the preset first garbage ratio threshold (which can be configured based on experience), the initial garbage collection threshold is increased to obtain the target garbage collection threshold. When the garbage ratio is less than the preset first garbage ratio threshold, it means that the garbage ratio is relatively low, that is, there are fewer duplicate data in the capacity physical hard disk, and more valid data in the capacity physical hard disk. Therefore, the initial garbage collection threshold can be increased to increase the basic garbage collection water level, thereby reducing the frequency of garbage collection operations, which is conducive to reducing the number of garbage collection operations, thereby saving bandwidth for garbage collection operations.

[0143] For another example, if the garbage ratio is greater than the preset second garbage ratio threshold (which can be configured based on experience), the initial garbage collection threshold is lowered to obtain the target garbage collection threshold. When the garbage ratio is greater than the preset second garbage ratio threshold, it means that the garbage ratio will be relatively high, that is, there are more duplicate data in the capacity physical hard disk, and less valid data in the capacity physical hard disk. Therefore, the initial garbage collection threshold can be lowered to reduce the basic garbage collection water level, thereby releasing more storage space to prevent sudden bursts (i.e., sudden increases in IO traffic), that is, to avoid garbage collection operations from occupying bandwidth when IO traffic suddenly increases.

[0144] For another example, if the garbage ratio is not less than the preset first garbage ratio threshold and not greater than the preset second garbage ratio threshold, the initial garbage collection threshold is determined as the target garbage collection threshold. Wherein, when the garbage ratio is not less than the preset first garbage ratio threshold and not greater than the preset second garbage ratio threshold, it means that the garbage ratio is appropriate, and therefore, the initial garbage collection threshold can be kept unchanged, so that the basic garbage collection water level remains unchanged.

[0145] In the above embodiment, the preset first garbage ratio threshold may be smaller than the preset second garbage ratio threshold.

[0146] After obtaining the target garbage collection threshold, the flow control decision layer can send the target garbage collection threshold to the WSC architecture layer, and the WSC architecture layer determines the target time corresponding to the garbage collection operation based on the target garbage collection threshold. For example, the time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold is determined, and this time is determined as the target time corresponding to the garbage collection operation.

[0147] At the target moment, the WSC architecture layer can perform garbage collection operations on duplicate data in the capacity physical hard disk, that is, deduplicate the duplicate user data in the capacity physical hard disk, retain the user data after the deduplication operation in the capacity physical hard disk, and delete the remaining user data from the capacity physical hard disk, such as retaining the last user data in the capacity physical hard disk and deleting the remaining user data.

[0148] Method 3: The target decision information may include the total number of IOs corresponding to each time period, that is, the sum of the number of write operations and the number of read operations for the high-speed physical hard disk and the number of read operations for the capacity physical hard disk, such as the total number of IOs p11 and p12 corresponding to time period 1, the total number of IOs p21 and p22 corresponding to time period 2, and so on. For method 3, the system detection layer can obtain the total number of IOs corresponding to each time period, and send the total number of IOs corresponding to each time period to the flow control decision layer. The flow control decision layer determines the target time period based on the total number of IOs corresponding to each time period.

[0149] If the total number of IOs corresponding to a time period is greater than a preset IO number threshold, the time period may be used as a target time period; otherwise, the time period is not used as a target time period.

[0150] For example, for each time period, taking time period 1 as an example, the period number r can be determined, where r is the threshold of the periodic distribution. Assuming that time period 1 corresponds to a total of 100 IOs, r can be 100*s, and s can be configured based on experience, and is a value greater than 0 and less than or equal to 1, such as 0.7, 0.8, 0.9, etc. On this basis, if the total number of r IOs corresponding to time period 1 is greater than the preset IO number threshold, it means that the total number of IOs in time period 1 is periodically distributed, and time period 1 is a high-traffic time period, so time period 1 is used as the target time period. Otherwise, if there are no r total numbers of IOs in time period 1 that are greater than the preset IO number threshold, it means that the total number of IOs in time period 1 is not periodically distributed, and / or time period 1 is not a high-traffic time period, so time period 1 is not used as the target time period.

[0151] In summary, for each time period, it can be determined whether the time period is a target time period. For the target time period, it is a time period in which the total number of IOs is greater than the preset IO number threshold. After obtaining the target time period, the flow control decision layer can send the target time period to the WSC architecture layer, and the WSC architecture layer determines the target time corresponding to the garbage collection operation based on the target time period. For example, the target time can be before the start time of the target time period. For example, assuming that the garbage collection operation takes 100 seconds to complete, the difference between the target time and the start time of the target time period can be greater than or equal to 100 seconds.

[0152] At the target moment, the WSC architecture layer can perform garbage collection operations on duplicate data in the capacity physical hard disk, that is, deduplicate the duplicate user data in the capacity physical hard disk, retain the user data after the deduplication operation in the capacity physical hard disk, and delete the remaining user data from the capacity physical hard disk, such as retaining the last user data in the capacity physical hard disk and deleting the remaining user data.

[0153] Obviously, since the garbage collection operation is performed at the target time, the garbage collection operation can be completed before the target time period, thereby completing the garbage collection operation in advance and avoiding the garbage collection operation and a large amount of IO from occupying the bandwidth. The above method can also be called an implementation method based on user portraits, that is, identifying the user's IO pattern (Pattern), which is the total number of IOs corresponding to each time period. Based on the IO pattern, it can be known whether the total number of IOs is periodically distributed and whether there is a time period with high traffic, and then find the target time period with periodic distribution and high traffic, so as to complete the garbage collection operation in advance and free up storage space.

[0154] In method 1, method 2 and method 3, garbage collection operation is performed on duplicate data in the capacity physical hard disk at the target time. The process includes: determining a storage block area where duplicate data in the capacity physical hard disk is located, the storage block area includes multiple storage blocks, and the multiple storage blocks include storage blocks with duplicate data; if the multiple storage blocks also include a target storage block without duplicate data, then storing user data in the target storage block to the capacity physical hard disk by appending write; deleting user data in each storage block (storage blocks with duplicate data and target storage blocks without duplicate data) in the storage block area.

[0155] For example, since user data in a high-speed physical hard disk is migrated to a capacity physical hard disk by appending writes, multiple migration processes may migrate multiple duplicate user data to the capacity physical hard disk, that is, the capacity physical hard disk may contain multiple duplicate user data, and therefore, duplicate user data can be deduplicated. For example, the last user data can be retained in the capacity physical hard disk (that is, the last user data is not deduplicated), and the remaining user data is deleted from the capacity physical hard disk.

[0156] Regarding the remaining user data, in order to delete the remaining user data from the capacity physical hard disk, the following method can be adopted: determine the storage block area where the remaining user data is located, divide the storage block area into a storage block with the remaining user data and a target storage block without the remaining user data, and on this basis, store the user data in the target storage block to the capacity physical hard disk through an append write method, delete the user data in the target storage block, and directly delete the remaining user data in the storage block.

[0157] In the above embodiments, it involves lowering the initial garbage collection threshold to obtain the target garbage collection threshold, such as subtracting the first adjustment value from the initial garbage collection threshold to obtain the target garbage collection threshold. The first adjustment value may be obtained by a value configured according to experience or by a value obtained by a fuzzy control algorithm, and there is no restriction on this. It involves increasing the initial garbage collection threshold to obtain the target garbage collection threshold, such as adding the second adjustment value to the initial garbage collection threshold to obtain the target garbage collection threshold. The second adjustment value may be obtained by a value configured according to experience or by a value obtained by a fuzzy control algorithm, and there is no restriction on this. It involves increasing the initial migration threshold to obtain the target migration threshold, such as adding the third adjustment value to the initial migration threshold to obtain the target migration threshold. The third adjustment value may be obtained by a value configured according to experience or by a value obtained by a fuzzy control algorithm, and there is no restriction on this. It involves lowering the initial migration threshold to obtain the target migration threshold, such as subtracting the fourth adjustment value from the initial migration threshold to obtain the target migration threshold. The fourth adjustment value may be obtained by a value configured according to experience or by a value obtained by a fuzzy control algorithm, and there is no restriction on this.

[0158] When the fuzzy control algorithm is used to obtain the first adjustment value, the second adjustment value, the third adjustment value and the fourth adjustment value, the structural diagram of the fuzzy controller can be seen in Figure 4 As shown, it includes a fuzzification interface, an inference engine, a defuzzification interface and a knowledge base, the knowledge base includes a rule base and a database, and the first adjustment value, the second adjustment value, the third adjustment value and the fourth adjustment value can be obtained through the fuzzy controller.

[0159] Taking obtaining the first adjustment value as an example, the remaining storage space (or garbage ratio) of the capacity physical hard disk can be used as input data, and the remaining storage space can be input into the fuzzification interface. The fuzzification interface fuzzifies the remaining storage space, and the fuzzified remaining storage space is input into the inference engine.

[0160] The rule base is used to provide the following reasoning strategies to the inference engine: when the remaining storage space is less than the preset first storage space threshold, the initial garbage collection threshold is reduced; when the remaining storage space is greater than the preset second storage space threshold, the initial garbage collection threshold is increased; when the remaining storage space is not less than the preset first storage space threshold and not greater than the preset second storage space threshold, the initial garbage collection threshold is kept unchanged. The database is used to provide the initial garbage collection threshold, the preset first storage space threshold and the preset second storage space threshold to the inference engine.

[0161] Based on the above-mentioned reasoning strategy, the initial garbage collection threshold, the preset first storage space threshold and the preset second storage space threshold, the inference engine can determine the first adjustment value after obtaining the remaining storage space after fuzzification processing, and subtract the first adjustment value from the initial garbage collection threshold to obtain the target garbage collection threshold, and input the target garbage collection threshold to the defuzzification interface.

[0162] The defuzzification interface can perform defuzzification processing on the target garbage collection threshold, and use the defuzzified target garbage collection threshold as output data, and output the target garbage collection threshold externally.

[0163] The implementation methods of the second adjustment value, the third adjustment value and the fourth adjustment value are similar to the implementation method of the first adjustment value, and can all be implemented by a fuzzy controller, which will not be repeated here.

[0164] It can be seen from the above technical solutions that in the embodiment of the present application, the target time corresponding to the garbage collection operation can be determined, and the garbage collection operation can be performed on the duplicate data in the capacity physical hard disk at the target time, so that the garbage collection operation can be performed at the appropriate time, avoiding the garbage collection operation from occupying the bandwidth of the data reading operation, so that the capacity physical hard disk can meet the needs of fast reading and writing, and the user's read traffic can meet the product requirements, avoiding the traffic occupation and data loss problems caused by insufficient storage space, ensuring that the user requirements are met in the mixed read and write scenario with large traffic, and making the storage device in a dynamic balanced steady state. The garbage collection operation tries not to affect the bandwidth occupied by the data reading operation, so that the bandwidth occupied by the data reading operation increases, meets the needs of the data reading operation, and completes the data reading operation in time, so that the garbage collection operation and the data reading operation do not affect each other. An intelligent water level algorithm based on the WSC architecture is proposed. By adjusting the dynamic water level threshold, the capacity physical hard disk completes the garbage collection action at the appropriate time, so that the system as a whole is in a dynamic balanced steady state, so that the user's read traffic can meet the product requirements. For users' burst IO, avoid the risk of user data loss due to lack of storage space on the physical hard disk. In other words, by completing user profiling, garbage collection can be performed in advance to free up enough storage space for users' burst scenarios.

[0165] Based on the same application concept as the above method, a data processing device is proposed in the embodiment of the present application, which is applied to a storage device, wherein the storage device includes a high-speed physical hard disk and a capacity physical hard disk, see Figure 5 FIG. 1 is a schematic diagram of the structure of the data processing device, which may include:

[0166] The storage module 51 is used to store the user data to the high-speed physical hard disk by appending write after obtaining the user data; when the high-speed physical hard disk meets the migration condition, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending write; the acquisition module 52 is used to obtain the target decision information corresponding to the capacity physical hard disk; the determination module 53 is used to determine the target time corresponding to the garbage collection operation based on the target decision information; the processing module 54 is used to perform garbage collection operation on the duplicate data in the capacity physical hard disk at the target time.

[0167] Exemplarily, when the determination module 53 determines the target time corresponding to the garbage collection operation based on the target decision information, it is specifically used to: determine the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information; determine the time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold, and determine the time as the target time corresponding to the garbage collection operation.

[0168] Exemplarily, the target decision information includes the remaining storage space of the physical hard disk of the capacity; the determination module 53 determines the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, and is specifically used to: if the remaining storage space is less than the preset first storage space threshold, then reduce the configured initial garbage collection threshold to obtain the target garbage collection threshold; or, if the remaining storage space is greater than the preset second storage space threshold, then increase the initial garbage collection threshold to obtain the target garbage collection threshold; or, if the remaining storage space is not less than the preset first storage space threshold and not greater than the preset second storage space threshold, then determine the initial garbage collection threshold as the target garbage collection threshold.

[0169] Exemplarily, the target decision information includes the garbage ratio of the capacity physical hard disk, and the garbage ratio represents the ratio of the amount of duplicate data to the total amount of data; the determination module 53 determines the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, and is specifically used for: if the garbage ratio is less than a preset first garbage ratio threshold, then increasing the configured initial garbage collection threshold to obtain the target garbage collection threshold; or, if the garbage ratio is greater than a preset second garbage ratio threshold, then lowering the initial garbage collection threshold to obtain the target garbage collection threshold; or, if the garbage ratio is not less than the preset first garbage ratio threshold and not greater than the preset second garbage ratio threshold, determining the initial garbage collection threshold as the target garbage collection threshold.

[0170] Exemplarily, the statistical period is divided into multiple time periods, and the target decision information includes the total number of IOs corresponding to each time period; for the total number of IOs corresponding to each time period, the total number of IOs is the sum of the number of write operations and the number of read operations for the high-speed physical hard disk within the time period, and the number of read operations for the capacity physical hard disk; the determination module 53 determines the target time corresponding to the garbage collection operation based on the target decision information, and is specifically used for: based on the total number of IOs corresponding to each time period, if the total number of IOs corresponding to the time period is greater than a preset IO number threshold, then determining the target time corresponding to the garbage collection operation based on the time period; wherein, the target time is before the start time of the time period.

[0171] Exemplarily, when the high-speed physical hard disk meets the migration conditions, the storage module 51 is specifically used to migrate the user data in the high-speed physical hard disk to the capacity physical hard disk through an additional write method, and is specifically used to: obtain hot data distribution information corresponding to the high-speed physical hard disk; determine a target migration threshold corresponding to the migration operation based on the hot data distribution information; when the data volume of the user data in the high-speed physical hard disk reaches the target migration threshold, determine that the high-speed physical hard disk meets the migration conditions, and migrate the user data in the high-speed physical hard disk to the capacity physical hard disk through an additional write method.

[0172] Exemplarily, the hot data distribution information includes the total number of hot data, which is the sum of the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk; the storage module 51 determines the target migration threshold corresponding to the migration operation based on the hot data distribution information, and is specifically used for: if the total number of hot data is greater than a preset first number threshold, then increasing the configured initial migration threshold to obtain the target migration threshold; or, if the total number of hot data is less than a preset second number threshold, then reducing the configured initial migration threshold to obtain the target migration threshold; or, if the total number of hot data is not greater than the preset first number threshold and not less than the preset second number threshold, then determining the initial migration threshold as the target migration threshold.

[0173] Exemplarily, when the processing module 54 performs a garbage collection operation on the duplicate data in the capacity physical hard disk at the target time, it is specifically used to: determine a storage block area where the duplicate data in the capacity physical hard disk is located, the storage block area includes multiple storage blocks, and the multiple storage blocks include storage blocks with duplicate data; if the multiple storage blocks also include a target storage block without duplicate data, then store the user data in the target storage block to the capacity physical hard disk by appending writes; and delete the user data in each storage block in the storage block area.

[0174] Based on the same application concept as the above method, a storage device is proposed in the embodiment of the present application, see Figure 6 As shown, the storage device includes a processor 61 and a machine-readable storage medium 62, and the machine-readable storage medium 62 stores machine-executable instructions that can be executed by the processor 61; the processor 61 is used to execute the machine-executable instructions to implement the data processing method disclosed in the above example of this application.

[0175] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor. When the machine-executable instructions are executed by the processor, the data processing method disclosed in the above example of the present application can be implemented.

[0176] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0177] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0178] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0179] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0180] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: Applied to a storage device, the storage device includes a high-speed physical hard disk and a capacity physical hard disk, user data is stored in the high-speed physical hard disk by appending write, and when the high-speed physical hard disk meets the migration condition, the user data in the high-speed physical hard disk is migrated to the capacity physical hard disk by appending write; wherein, when the data amount of the user data in the high-speed physical hard disk reaches the target migration threshold, it is determined that the high-speed physical hard disk meets the migration condition, and the target migration threshold is determined based on the total number of hot data, and the total number of hot data is the sum of the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk; wherein, if the total number of hot data is greater than a preset first number threshold, the configured initial migration threshold is increased to obtain the target migration threshold; or, if the total number of hot data is less than a preset second number threshold, the configured initial migration threshold is reduced to obtain the target migration threshold; or, if the total number of hot data is not greater than the preset first number threshold and not less than the preset second number threshold, the initial migration threshold is determined as the target migration threshold; the method includes: Obtain target decision information corresponding to the physical hard disk of the capacity; Determine a target time corresponding to a garbage collection operation based on the target decision information; At the target time, a garbage collection operation is performed on duplicate data in the capacity physical hard disk.

2. The method according to claim 1, characterized in that The determining a target time corresponding to the garbage collection operation based on the target decision information includes: Determining a target garbage collection threshold corresponding to the garbage collection operation based on the target decision information; Determine a time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold, and determine the time as a target time corresponding to the garbage collection operation.

3. The method according to claim 2, characterized in that The target decision information includes the remaining storage space of the capacity physical hard disk; and determining the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information includes: If the remaining storage space is less than the preset first storage space threshold, the configured initial garbage collection threshold is reduced to obtain the target garbage collection threshold; or, If the remaining storage space is greater than the preset second storage space threshold, then the configured initial garbage collection threshold is increased to obtain the target garbage collection threshold; or, If the remaining storage space is not less than a preset first storage space threshold and not greater than a preset second storage space threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

4. The method according to claim 2, characterized in that: The target decision information includes a garbage ratio of the capacity physical hard disk, where the garbage ratio indicates a ratio of the amount of duplicate data to the total amount of data; The determining a target garbage collection threshold corresponding to the garbage collection operation based on the target decision information includes: If the garbage ratio is less than a preset first garbage ratio threshold, then increasing the configured initial garbage collection threshold to obtain the target garbage collection threshold; or, If the garbage ratio is greater than a preset second garbage ratio threshold, the configured initial garbage collection threshold is lowered to obtain the target garbage collection threshold; or, If the garbage ratio is not less than a preset first garbage ratio threshold and not greater than a preset second garbage ratio threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

5. The method according to claim 1, characterized in that: The statistical period is divided into a plurality of time periods, and the target decision information includes the total number of IOs corresponding to each time period; the total number of IOs corresponding to each time period is the sum of the number of write operations and the number of read operations for the high-speed physical hard disk and the number of read operations for the capacity physical hard disk within the time period; The determining a target time corresponding to the garbage collection operation based on the target decision information includes: Based on the total number of IOs corresponding to each time period, if the total number of IOs corresponding to the time period is greater than the preset IO number threshold, the target time corresponding to the garbage collection operation is determined based on the time period; The target time is before the start time of the time period.

6. The method according to any one of claims 1 to 5, characterized in that: The performing of garbage collection operation on duplicate data in the capacity physical hard disk at the target time includes: Determine a storage block area where duplicate data is located in the capacity physical hard disk, the storage block area includes a plurality of storage blocks, and the plurality of storage blocks include a storage block having duplicate data; If the plurality of storage blocks further include a target storage block without duplicate data, storing the user data in the target storage block to the capacity physical hard disk by appending write; The user data in each storage block in the storage block area is deleted.

7. The method according to any one of claims 1 to 5, characterized in that: The storage device uses a write shaping cache WSC architecture to store user data; The read and write speed of the high-speed physical hard disk is greater than a preset speed threshold, and the storage capacity of the high-speed physical hard disk is less than a preset capacity threshold; the read and write speed of the capacity physical hard disk is not greater than the preset speed threshold, and the storage capacity of the capacity physical hard disk is not less than the preset capacity threshold.

8. A data processing device, characterized in that: Applied to a storage device, the storage device includes a high-speed physical hard disk and a capacity physical hard disk, the device includes: A storage module, used for storing the user data to the high-speed physical hard disk by appending write after obtaining the user data; when the high-speed physical hard disk meets the migration condition, migrating the user data in the high-speed physical hard disk to the capacity physical hard disk by appending write; wherein, when the data amount of the user data in the high-speed physical hard disk reaches the target migration threshold, it is determined that the high-speed physical hard disk meets the migration condition, and the target migration threshold is determined based on the total number of hot data, and the total number of hot data is the sum of the number of hot data in the high-speed physical hard disk and the number of hot data in the capacity physical hard disk; wherein, if the total number of hot data is greater than a preset first number threshold, the configured initial migration threshold is increased to obtain the target migration threshold; or, if the total number of hot data is less than a preset second number threshold, the configured initial migration threshold is reduced to obtain the target migration threshold; or, if the total number of hot data is not greater than the preset first number threshold and not less than the preset second number threshold, the initial migration threshold is determined as the target migration threshold; An acquisition module, used for acquiring target decision information corresponding to the capacity physical hard disk; A determination module, configured to determine a target time corresponding to a garbage collection operation based on the target decision information; The processing module is used to perform garbage collection operation on duplicate data in the capacity physical hard disk at the target time.

9. The device according to claim 8, characterized in that The determination module determines the target time corresponding to the garbage collection operation based on the target decision information and is specifically used to: determine the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information; determine the time when the amount of duplicate data in the capacity physical hard disk reaches the target garbage collection threshold, and determine the time as the target time corresponding to the garbage collection operation; Wherein, the target decision information includes the remaining storage space of the physical hard disk of the capacity; when the determination module determines the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, it is specifically used to: if the remaining storage space is less than a preset first storage space threshold, then reduce the configured initial garbage collection threshold to obtain the target garbage collection threshold; or, if the remaining storage space is greater than a preset second storage space threshold, then increase the initial garbage collection threshold to obtain the target garbage collection threshold; or, if the remaining storage space is not less than the preset first storage space threshold and not greater than the preset second storage space threshold, then determine the initial garbage collection threshold as the target garbage collection threshold; Among them, the target decision information includes the garbage ratio of the capacity physical hard disk, and the garbage ratio represents the ratio of the amount of duplicate data to the total amount of data; the determination module determines the target garbage collection threshold corresponding to the garbage collection operation based on the target decision information, and is specifically used for: if the garbage ratio is less than a preset first garbage ratio threshold, then the configured initial garbage collection threshold is increased to obtain the target garbage collection threshold; or, if the garbage ratio is greater than a preset second garbage ratio threshold, then the initial garbage collection threshold is lowered to obtain the target garbage collection threshold; or, if the garbage ratio is not less than the preset first garbage ratio threshold and not greater than the preset second garbage ratio threshold, the initial garbage collection threshold is determined as the target garbage collection threshold.

10. The device according to claim 8, characterized in that The statistical period is divided into a plurality of time periods, and the target decision information includes the total number of IOs corresponding to each time period; the total number of IOs corresponding to each time period is the sum of the number of write operations and the number of read operations for the high-speed physical hard disk and the number of read operations for the capacity physical hard disk within the time period; The determination module determines the target time corresponding to the garbage collection operation based on the target decision information, and is specifically used to: determine the target time corresponding to the garbage collection operation based on the time period if the total number of IOs corresponding to each time period is greater than a preset IO number threshold; The target time is before the start time of the time period.

11. A storage device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method steps described in any one of claims 1-7.

12. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions that can be executed by a processor; wherein the processor is configured to execute the machine-executable instructions to implement the method steps described in any one of claims 1-7.

Images