Data processing method, system, device and medium based on shingled magnetic recording disk

Abstract

The application provides a data processing method, system, device and medium based on a shingled magnetic recording disk, and belongs to the technical field of data storage. The method comprises the following steps: obtaining a storage period and a time limit attribute of to-be-stored data; sequentially writing the to-be-stored data into a target partition, wherein the target partition is a shingled storage partition with the same storage period and the same time limit attribute as the to-be-stored data; judging whether there is a to-be-recycled partition in the shingled magnetic recording disk, wherein the to-be-recycled partition is a shingled storage partition meeting a garbage recycling condition; when the to-be-recycled partition exists, moving valid data in the to-be-recycled partition to a garbage recycling partition according to the order of the last modification time of the valid data, and releasing the storage space of the to-be-recycled partition after the moving is completed. The application effectively reduces the garbage recycling probability of the shingled magnetic recording disk, thereby improving the storage performance.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and specifically to a data processing method, system, device, and medium based on shingled magnetic recording disks. Background Technology

[0002] Shingled magnetic recording disks divide the storage space into traditional magnetic recording areas and shingled magnetic recording areas. The shingled magnetic recording area is further divided into multiple independent shingled storage partitions, each of which supports only sequential writes and is reclaimed on a per-partition basis. In existing storage systems, user data with different storage periods (such as permanent storage, 30-day, 7-day, etc.) and different time-sensitivity attributes (real-time data, historical data) are often mixed and written to the same shingled storage partition.

[0003] When some data within a partition becomes invalid due to exceeding its storage period or becoming historical data, garbage collection must be performed on the shingled storage partition because shingled storage partitions do not support overwrite and random deletion. This involves moving the valid data to other partitions and then releasing the space in the original partition.

[0004] However, hybrid storage leads to a scattered distribution of valid data within partitions. Garbage collection requires migrating large amounts of data, and the temporal order of valid data becomes disordered after multiple collections, easily triggering repeated collections. This severely impacts normal business read / write bandwidth and degrades overall disk performance. Because shingled storage partitions only support sequential writes and cannot be partially erased, when data within a partition partially becomes invalid due to differences in storage cycles, the garbage collection process must migrate all remaining valid data. If the valid data is scattered and its temporal order becomes disordered after multiple collections, the same data will be repeatedly migrated in multiple garbage collections, further exacerbating performance degradation. Summary of the Invention

[0005] In view of this, it is necessary to provide a data processing method, system, device and medium based on shingled magnetic recording disks to solve the technical problems of frequent garbage collection and low efficiency of effective data migration in the prior art, which affect the performance of the storage system.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a data processing method based on a shingled magnetic recording disk, comprising: Obtain the storage period and expiration attribute of the data to be stored; The data to be stored is sequentially written to the target partition, which is a shingled storage partition with the same storage period and the same timeliness attributes as the data to be stored. Determine whether there is a partition to be recycled in the shingled magnetic recording disk, wherein the partition to be recycled is a shingled storage partition that meets the conditions for garbage collection; When the partition to be reclaimed exists, the valid data in the partition to be reclaimed is moved to the garbage collection partition in the order of the last modification time of each valid data in the partition to be reclaimed, and the storage space of the partition to be reclaimed is released after the relocation is completed.

[0007] In one possible implementation, sequentially writing the data to be stored to the target partition includes: Determine whether there exists a shingled storage partition that has the same storage period and timeliness attributes as the data to be stored and is in a writable state; If it exists, the shingled storage partition is used as the target partition and the data to be stored is written sequentially. If no free partition exists, allocate a free partition, set the storage period and expiration attributes of the free partition to be the same as the data to be stored and mark it as writable, use the free partition as the target partition and write the data to be stored sequentially. After the write operation is completed, the index information of the target partition is updated to record the file identifier and location information of the written data, and the metadata database is updated to record the file identifier, storage period, partition identifier, and last modification time of the written data.

[0008] In one possible implementation, the method further includes: when there are multiple shingled storage partitions that have the same storage period and the same timeliness attributes as the data to be stored and are in a writable state, selecting the shingled storage partition with the earliest write time as the target partition.

[0009] In one possible implementation, determining whether a partition to be reclaimed exists in the shingled magnetic recording disk includes: Traverse all shingled storage partitions in the shingled magnetic recording disk, calculate the effective data percentage of the current shingled storage partition, and detect the duration for which the effective data percentage is lower than a preset percentage threshold. If the percentage of valid data is lower than the preset percentage threshold and the duration exceeds the preset duration threshold, the current shingled storage partition is determined to be the partition to be reclaimed.

[0010] In one possible implementation, the step of migrating the valid data to the garbage collection partition according to the order of the last modification time of each valid data in the partition to be reclaimed includes: Obtain the last modification time corresponding to each valid data in the partition to be reclaimed; Read all data from the target file to which each valid data belongs from the partition to be reclaimed in order of last modification time from earliest to latest, and write the read data into the garbage collection partition; After the target file migration is completed, update the index information of the garbage collection partition and the file storage location information in the metadata database, and delete the data corresponding to the target file in the partition to be collected.

[0011] In one possible implementation, releasing the storage space of the partition to be reclaimed includes: performing a reset operation on the partition to be reclaimed, resetting the write pointer of the partition to be reclaimed, and marking all storage space of the partition to be reclaimed as reusable.

[0012] In one possible implementation, obtaining the storage period and timeliness attributes of the data to be stored includes: Read the preset storage period configuration of the data to be stored to determine the storage period; Obtain the write time of the data to be stored, and compare the time difference between the write time and the current time; When the time difference is less than a preset duration threshold, the timeliness attribute of the data to be stored is determined to be real-time data; When the time difference is greater than the preset duration threshold, the timeliness attribute of the data to be stored is determined to be historical data.

[0013] Secondly, the present invention also provides a data processing system based on a shingled magnetic recording disk, comprising: A shingled magnetic recording disk, comprising a conventional magnetic recording area and a shingled magnetic recording area, wherein the shingled magnetic recording area includes multiple shingled storage partitions; The acquisition module is used to obtain the storage period and timeliness attribute of the data to be stored. The write module is used to sequentially write the data to be stored into the target partition, wherein the target partition is a shingled storage partition with the same storage period and the same timeliness attributes as the data to be stored; The judgment module is used to determine whether there is a partition to be recycled in the shingled magnetic recording disk, wherein the partition to be recycled is a shingled storage partition that meets the conditions for garbage collection; The recycling module is used to, when the partition to be recycled exists, move the valid data to the garbage recycling partition in the order of the last modification time of each valid data in the partition to be recycled, and release the storage space of the partition to be recycled after the relocation is completed.

[0014] Thirdly, the present invention also provides an electronic device, comprising: Shingled magnetic recording disks are used to store data; At least one sensor or image acquisition device is used to acquire the data to be stored and the write time of the data to be stored; Memory, which stores computer programs; A processor, coupled to the memory, is configured to implement the steps of the method described in any of the above implementations when executing the computer program; wherein the write time is used to determine the timeliness attribute of the data to be stored; and... The processor, the memory, the shingled magnetic recording disk, and the sensor or image acquisition device are all connected to the communication interface to achieve data communication.

[0015] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps of the method described in any of the above implementations.

[0016] The beneficial effects of this invention are as follows: Firstly, by aggregating data according to the same storage cycle and the same timeliness attributes and writing it into dedicated shingled memory partitions, this invention ensures that the data within each partition has a consistent lifecycle and timeliness characteristics. This avoids frequent garbage collection triggers for the entire partition due to premature data expiration, effectively reducing the total number of garbage collections. Furthermore, during garbage collection, data is migrated according to the last modification time order of valid data. This allows data with similar modification times to be reintegrated into the garbage collection partition, maintaining the temporal continuity of data, preventing multiple collections, and significantly improving the efficiency of garbage collection and the overall business performance of the disk. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of an embodiment of the data processing method based on shingled magnetic recording disk provided by the present invention; Figure 2 For the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S103; Figure 4 For the present invention Figure 1 A schematic diagram of an embodiment of S104; Figure 5 For the present invention Figure 1 A schematic diagram of an embodiment of S101; Figure 6 A schematic diagram of the disk layout for a file system based on shingled magnetic recording disks provided by the present invention; Figure 7 A schematic diagram of the file system formatting process provided by the present invention; Figure 8 This is a schematic diagram of the data storage process provided by the present invention; Figure 9 This is a schematic diagram illustrating the waste recycling effect provided by the present invention; Figure 10 This is a schematic diagram of the waste recycling process provided by the present invention; Figure 11 This is a schematic diagram of an embodiment of the data processing system based on shingled magnetic recording disks provided by the present invention; Figure 12 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0021] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] Before demonstrating the embodiments, the following terms will be explained.

[0024] SMR (Shingled Magnetic Recording) is a magnetic recording technology that increases storage density by partially overlapping magnetic tracks. Writing to SMR must be done sequentially and does not support in-situ overwrite.

[0025] CMR (Conventional Magnetic Recording) refers to a conventional magnetic recording technology where tracks do not overlap and support random write and overwrite operations.

[0026] SMR (Shingled Magnetic Recording) disks are a type of hard drive that uses shingled magnetic recording technology. Unlike conventional magnetic recording (CMR) where tracks are independent, SMR tracks are arranged in a partially overlapping manner, like roof tiles, thus significantly increasing storage density. However, this overlapping structure means that data cannot be directly overwritten or randomly modified; it can only be written sequentially by zone, and space reclamation must be performed on a whole-zone basis. Typically, an SMR disk allocates about 99% of its space to SMR areas that only support sequential writes, and the remaining about 1% is conventional magnetic recording (CMR) areas that support random writes, used to store critical metadata.

[0027] ZONE (partition): refers to an independent storage unit in a shingled magnetic recording disk where the SMR area is divided into fixed sizes (e.g., 256MB). Each ZONE only supports strict sequential writing, and space is reclaimed on a unit basis for the entire ZONE.

[0028] GC (Garbage Collection) refers to the process of automatically reclaiming space occupied by invalid data in a storage system. It releases contiguous available space by organizing valid data. In shingled magnetic disks, it must be performed sequentially in units of ZONE.

[0029] DB (Database): refers to a database used to store file metadata information, including key information such as file identifier, file last modified time, and file storage location.

[0030] This invention provides a data processing method, system, device, and medium based on shingled magnetic recording disks, which are described below.

[0031] Figure 1 This is a schematic flowchart of an embodiment of the data processing method based on shingled magnetic recording disks provided by the present invention, as shown below. Figure 1 As shown, the data processing method based on shingled magnetic recording disks includes: S101. Obtain the storage period and timeliness attribute of the data to be stored.

[0032] It's important to note that storage period refers to the expected length of time data will be retained on disk, such as permanent storage, 180 days, 30 days, 7 days, or 1 day. This information is typically determined by the business strategy or user configuration at the time the data is generated. The timeliness attribute distinguishes between real-time and historical data. These two attributes are obtained by reading the metadata carried by the data to be stored (such as file tags, parameters in the write request, or a preset storage strategy table), thus providing a classification basis for subsequent data layout.

[0033] S102. The data to be stored is sequentially written to the target partition, which is a shingled magnetic recording zone (SMR ZONE) with the same storage period and the same timeliness attributes as the data to be stored.

[0034] It's important to note that after determining the storage period and timeliness of the data to be stored, the data is sequentially written to a matching SMR zone, called the target partition. Sequential writing means adhering to the requirements of shingled magnetic recording disks, strictly appending data in the order of increasing logical addresses within the partition, prohibiting random overwriting or skipping writes. The target partition has the exact same storage period as the data to be stored (e.g., both for permanent storage or both for 30 days) and the exact same timeliness (both for real-time data or both for historical data), ensuring that all data within the same target partition has a consistent lifecycle and timeliness characteristics, thus avoiding the mixing of data with different retention times or timeliness.

[0035] S103. Determine whether there is a partition to be recycled in the shingled magnetic recording disk, wherein the partition to be recycled is an SMR ZONE that meets the conditions for garbage collection.

[0036] It should be noted that during data writing or in subsequent operation, the storage system will actively check whether there are one or more SMR zones in the current shingled magnetic recording disk that meet the preset garbage collection conditions. The partition that meets the garbage collection conditions is defined as a partition to be reclaimed.

[0037] S104. When the partition to be reclaimed exists, the valid data is moved to the garbage collection partition (GC ZONE) in the order of the last modification time of each valid data in the partition to be reclaimed, and the storage space of the partition to be reclaimed is released after the relocation is completed.

[0038] It's important to note that after confirming the existence of a partition to be reclaimed, the system identifies all still valid data within that partition and obtains the last modification time for each valid piece of data. Subsequently, the system processes each valid piece of data sequentially according to the order of these last modification times (from earliest to latest): the currently processed valid data is read from the original partition to be reclaimed and written to a pre-prepared garbage collection zone (GC ZONE). This GC ZONE is a dedicated SMR ZONE for receiving valid data from the partition to be reclaimed, and it is independent of the ordinary data storage partition. After all valid data from the partition to be reclaimed has been moved to the GC ZONE in the above order, the system performs a storage space release operation on the original partition to be reclaimed, allowing its entire capacity to be reused by subsequent write requests, thus completing a full garbage collection cycle.

[0039] The execution entity for data processing methods based on shingled magnetic recording disks can be a processor (e.g., central processing unit, microcontroller, digital signal processor), storage controller (e.g., disk array controller, solid-state drive controller), processing unit in a host device (e.g., CPU in a server, processing chip in an embedded system), or application-specific integrated circuit / field-programmable gate array (ASIC). These hardware modules execute computer program instructions in memory to complete steps such as data storage cycle acquisition, partition matching, sequential writing, garbage collection condition judgment, data migration, and space release. In actual products, this execution entity is usually integrated into a storage server, network-attached storage device, disk array controller, or computer host, and is directly or indirectly connected to the shingled magnetic recording disk.

[0040] In summary, the data processing method based on shingled magnetic recording disks provided in this invention firstly aggregates data according to the same storage period and the same timeliness attributes and writes it into a dedicated SMR ZONE. This ensures that the data in each partition has a consistent lifecycle and timeliness characteristics, thereby avoiding frequent garbage collection triggered by some data prematurely becoming invalid, effectively reducing the total number of garbage collections. Furthermore, during garbage collection, data is migrated according to the last modification time of valid data, which can reintegrate data with similar times into the garbage collection partition (GC ZONE), maintaining the temporal continuity of data, preventing multiple collections, and significantly improving the efficiency of garbage collection and the overall business performance of the disk.

[0041] In some embodiments of the present invention, such as Figure 2 As shown, step S102 includes: S201. Determine whether there exists an SMR ZONE that has the same storage period and the same timeliness attribute as the data to be stored and is in a writable state.

[0042] It should be noted that: First, all currently created SMR ZONEs are traversed. Based on the storage period, time-limited attributes, and current status (such as whether it is full) recorded in the disk index area for each partition, those SMR ZONEs that have the same storage period and time-limited attributes as the data to be stored and are not yet full (i.e., in a writable state) are selected.

[0043] S202. If it exists, use the SMR ZONE as the target partition and write the data to be stored sequentially. S203. If it does not exist, allocate a free partition, set the storage period and expiration attribute of the free partition to be the same as the data to be stored and mark it as writable, use the free partition as the target partition and write the data to be stored sequentially.

[0044] It should be noted that: If an SMR zone with the same storage period, the same timeliness attributes, and a writable state as the data to be stored is found in the shingled magnetic recording disk, this SMR zone is used as the target partition. Subsequently, the data to be stored is appended sequentially starting from the current write pointer position of this target partition, ensuring that the write operation strictly follows the sequential write constraints of shingled magnetic recording. If no SMR zone meeting the conditions (same storage period, same timeliness attributes, and writable state as the data to be stored) is found in the shingled magnetic recording disk, an unused SMR zone (i.e., a free partition) is allocated from the free partition pool of the shingled magnetic recording disk. After this free partition is initialized, the storage period field in its index area is set to the storage period of the data to be stored, the timeliness attribute field is set to the timeliness attribute of the data to be stored, and its status is marked as "writable" (i.e., not full). Afterward, this free partition is used as the target partition, and the data to be stored is written sequentially starting from the write pointer position of the target partition.

[0045] S204. After writing is completed, update the index information of the target partition to record the file identifier and location information of the written data, and update the metadata database to record the file identifier, storage period, partition identifier and last modification time of the written data.

[0046] It should be noted that after writing data through step S202 or step S203, two metadata entries are updated synchronously: First, the index entry of the target partition itself in the ZONE index area is updated, recording the file identifier (such as file ID) corresponding to the data written this time and the exact location information (starting offset and length) of the data written in the partition; Second, the independent metadata database (DB) is updated, which records the file identifier, storage period, identifier of the partition (i.e., ZONE ID) of the data written, and the last modification time of the data written (usually the write time).

[0047] In this embodiment, by first determining whether there exists a homogeneous partition with the same storage period and timeliness attributes as the data to be stored, and which is writable, if such a partition exists, it is directly reused; otherwise, a new dedicated partition is created and its corresponding attributes are set. This ensures that data with the same storage period and timeliness attributes are aggregated and written to the same or similar partitions, avoiding the mixing of data with different lifecycles and timeliness, thereby significantly reducing the probability of garbage collection being triggered due to premature data expiration. Furthermore, by updating the partition index and metadata database promptly after each write operation, recording key information such as file identifier, location, partition, and last modification time, accurate retrieval criteria are provided for relocating valid data in chronological order during garbage collection. This enables the subsequent collection process to efficiently locate and process data sequentially, improving the overall performance and stability of the storage system.

[0048] In some embodiments of the present invention, the method further includes: when there are multiple SMR ZONEs that have the same storage period and the same timeliness attributes as the data to be stored and are in a writable state, the SMR ZONE with the earliest write time is selected as the target partition.

[0049] It should be noted that during the process of writing data to the shingled magnetic recording disk, all eligible SMR zones that are writable are first selected based on the data's storage period and timeliness attributes. When two or more candidate partitions (i.e., eligible SMR zones that are writable) exist in the selection results, the SMR zone with the earliest write time is selected as the target partition. Specifically, when an SMR zone is created or used for the first time, a "first write time" or "partition creation time" field is recorded in the SMR zone's index information to identify the earliest time data was written to that SMR zone. When it is necessary to select a target partition from multiple candidate partitions, the index entries of these candidate partitions are traversed, the recorded time value is read, and these timestamps are compared. The candidate partition with the smallest timestamp, i.e., the candidate partition with the earliest write time, is selected as the target partition for this write operation. If multiple candidate partitions have the same write time (e.g., multiple partitions are created simultaneously within a very short period of time), an auxiliary strategy (such as selecting the partition with the smallest partition ID) can be used as the final target partition.

[0050] In this embodiment, by selecting the partition with the earliest write time from all eligible partitions as the target partition, the system avoids a large number of partitions remaining in a half-full state for extended periods due to frequent partition creation, thereby improving the overall utilization of disk storage space. Furthermore, by aggregating new data into older partitions as early as possible, the allocation and creation of new partitions can be delayed, reducing the management overhead of the partition index area. At the same time, the data within each partition becomes more concentrated in the time dimension, which is beneficial for more efficient and effective data migration in chronological order during subsequent garbage collection.

[0051] In some embodiments of the present invention, such as Figure 3 As shown, step S103 includes: S301. Traverse all SMR zones in the shingled magnetic recording disk, calculate the effective data percentage of the current SMR zone, and detect the duration for which the effective data percentage is lower than a preset threshold.

[0052] It should be noted that garbage collection condition checks are initiated periodically or triggered by events (such as a partition being full or after data deletion). Once the check begins, each SMR zone (from ID1 to IDn) on the disk is scanned sequentially. For the currently checked SMR zone, the total amount of still valid data (i.e., the number of bytes of data that have not been deleted or expired) in the SMR zone is calculated by reading the file list and validity flags of each file recorded in the zone's index information (ZONE index). Then, the total amount of valid data is divided by the total capacity of the SMR zone (e.g., 256MB) to obtain the percentage of valid data. Simultaneously, a timer is maintained. When the percentage of valid data in a SMR zone first falls below a preset threshold (e.g., 70%), the duration of this period is recorded to obtain the duration for which the percentage of valid data falls below the preset threshold.

[0053] S302. If the percentage of valid data is lower than the preset percentage threshold and the duration exceeds the preset duration threshold, the current SMR ZONE is determined to be the partition to be recycled.

[0054] It should be noted that for the current SMR zone being monitored, if the percentage of valid data is lower than a preset percentage threshold, and this low percentage state has persisted for more than a preset duration threshold (e.g., 8 hours), then the SMR zone is determined to meet the garbage collection criteria and is marked as a partition to be reclaimed. Once a partition to be reclaimed is identified, other SMR zones can be traversed to find all partitions to be reclaimed, or the traversal can be terminated and the currently found partition to be reclaimed can be processed first (the specific strategy is not limited). The preset percentage threshold and the preset duration threshold can be pre-configured according to disk load and business characteristics; this implementation does not impose specific numerical limitations.

[0055] In this embodiment, by traversing all SMR zones and detecting the duration for which the effective data ratio in each SMR zone is below a threshold, partitions with numerous voids due to massive data corruption and which have not been reclaimed for extended periods can be accurately identified. This avoids accidental garbage collection triggering due to momentary fluctuations, thereby reducing invalid migration operations. Furthermore, by setting a duration condition, it ensures that only partitions that are indeed in an inefficient state for a long period are included in the reclamation process, making garbage collection triggering more robust and preventing frequent interruptions to normal read and write operations due to short-term business peaks or temporary data deletion, effectively guaranteeing the overall business performance of the shingled magnetic recording disk.

[0056] In some embodiments of the present invention, such as Figure 4 As shown, step S104 includes: S401. Obtain the last modification time corresponding to each valid data in the partition to be reclaimed.

[0057] It should be noted that: First, the index information (ZONE index) of the partition to be reclaimed is accessed to retrieve a list of identifiers for all files contained in the partition and the storage location of each file within the partition. Then, the metadata database (DB) is queried to obtain the last modification time (usually the file write time or the timestamp of the last update) for each file based on its file identifier. Finally, the system obtains a set of records: (file identifier, last modification time, data location information in the partition to be reclaimed).

[0058] S402. Read all the data of the target file to which each valid data belongs from the garbage collection partition in order of the last modification time from earliest to latest, and write the read data into the garbage collection partition (GC ZONE).

[0059] It should be noted that the file records obtained in step S401 are sorted in ascending order according to their last modification time (earliest modified file first). For each sorted target file, all data of the target file is sequentially read from the garbage collection partition based on its position information in the garbage collection partition. Then, the read data is written to a pre-allocated garbage collection partition (GC ZONE). The garbage collection partition (GC ZONE) is a dedicated partition independent of the ordinary data storage partition, and the write operation also follows the sequential write requirement of the SMR ZONE. The newly written position of each target file in the garbage collection partition (GC ZONE) is recorded for subsequent index updates.

[0060] S403. After the target file is moved, update the index information of the garbage collection zone (GC ZONE) and the file storage location information in the metadata database, and delete the data corresponding to the target file in the zone to be reclaimed.

[0061] It should be noted that after all the data of a target file is successfully written to the garbage collection zone, the index information of the garbage collection zone is updated to record the file identifier of the target file and its new position (starting offset, length) in the garbage collection zone. Additionally, the metadata database is updated to update the storage location information of the target file (partition identifier, data offset, etc.) to the new address in the garbage collection zone. Afterward, the system marks the original data corresponding to the target file in the garbage collection zone as invalid (or directly deletes its index entry). Once all target files in the garbage collection zone have been migrated and updated, the garbage collection zone no longer contains any valid data, and the entire partition becomes completely free.

[0062] In this embodiment, by obtaining the last modification time of each valid data in the garbage collection zone and migrating them in ascending order, data with similar timestamps can be aggregated and stored consecutively in the garbage collection zone, avoiding temporal disorder caused by multiple collections and thus reducing the probability of triggering garbage collection again. Furthermore, by updating the index of the garbage collection zone and the file location in the metadata database in real time after each migration, and deleting the corresponding data in the original garbage collection zone, the consistency of metadata and data is ensured. This allows for a smooth switch of the data access path to the new location, while the original garbage collection zone can be quickly reset, improving the overall efficiency of garbage collection and data security.

[0063] In some embodiments of the present invention, releasing the storage space of the partition to be reclaimed includes: performing a reset operation on the partition to be reclaimed, resetting the write pointer of the partition to be reclaimed, and marking all storage space of the partition to be reclaimed as reusable.

[0064] It's important to note that after migrating all valid data from the partition to be reclaimed and updating the corresponding indexes and metadata, the storage space originally occupied by the partition is released for subsequent writes. The specific operation for releasing storage space involves performing a reset operation (RESET ZONE) on the partition to be reclaimed. The core of the reset operation includes two actions: First, the write pointer of the partition to be reclaimed is reset to the beginning of the partition. The SMR ZONE internally maintains a write pointer indicating the starting address of the next data write. After the partition is continuously written to until it is full, the write pointer points to the end of the partition. Through the reset operation, the write pointer is reset to the beginning of the partition, allowing subsequent writes to resume sequential appending from the beginning of the partition. Second, all storage space of the partition to be reclaimed is marked as reusable, changing the partition's status from "full" or "awaiting reclamation" to "free" or "allocable" in the partition index (ZONE index), and clearing all file identifiers and location information recorded in the partition index (or treating them as invalid). Afterward, the partition to be reclaimed is reclaimed into the free partition pool, where it can be reassigned to store real-time data, historical data, or used as a new garbage collection partition (GCZONE).

[0065] In this embodiment, by performing a reset operation on the partition to be reclaimed to reset the write pointer, the SMR ZONE can resume sequential writing like a new partition, avoiding the situation where it cannot be used due to the write pointer reaching the end, thereby achieving the recycling of storage space. Furthermore, by marking the partition storage space as reusable and reclaiming it to the free partition pool, the capacity of the entire partition can be quickly released for subsequent allocation without time-consuming physical erasure, significantly reducing the overhead of space reclamation and improving the write efficiency and overall throughput performance of the shingled magnetic recording disk.

[0066] In some embodiments of the present invention, such as Figure 5 As shown, step S101 includes: S501. Read the preset storage period configuration of the data to be stored to determine the storage period.

[0067] It should be noted that upon receiving data to be stored, the storage period is first extracted from the data's metadata. The storage period is typically determined by the business strategy or user configuration at the time the data was generated. For example, some important logs need to be stored permanently, ordinary monitoring data is stored for 30 days, and temporary cached data is stored for 7 days. The storage period can be obtained by reading the tags in the data request header, the filename extension, or querying a preset data classification table.

[0068] S502. Obtain the write time of the data to be stored, and compare the time difference between the write time and the current time.

[0069] It should be noted that the process involves obtaining the write time of the data to be stored (i.e., the moment the data was generated or submitted, which can be obtained from the metadata of the data itself or the system's receiving time) and the current real-time of the system. The difference between the two is calculated, which is the length of time elapsed from the data's generation to the present. For example, if the write time is 2026-05-10 10:00:00 and the current time is 2026-05-14 10:00:00, then the time difference is 4 days.

[0070] S503. When the time difference is less than a preset duration threshold, the timeliness attribute of the data to be stored is determined to be real-time data. S504. When the time difference is greater than the preset duration threshold, the timeliness attribute of the data to be stored is determined to be historical data.

[0071] It's important to note that the preset duration threshold is a configurable time limit, such as one day. If the time difference is less than the preset duration threshold, it indicates that the data has been generated for a short time and still has high timeliness value, classifying it as real-time data. Real-time data typically needs to be retained first and may be stored together with other recently generated real-time data for quick access and subsequent processing. If the time difference is greater than or equal to the preset duration threshold, it indicates that the data has been generated for a longer period, reducing its timeliness value, and classifying it as historical data. Historical data may be used for archiving, backtracking, or retransmission, and its access frequency is relatively low. It can be aggregated and stored with similar historical data to reduce the space occupied by real-time data storage.

[0072] In this embodiment, by comparing the time difference between the write time and the current time and comparing it with a threshold, the data is automatically classified as real-time or historical, so that real-time data and historical data can be written into different SMR ZONEs. This effectively prevents the real-time data partition from being polluted after the historical data expires and becomes invalid, significantly reduces the frequency of garbage collection triggered by hybrid storage, and improves the performance stability of shingled magnetic recording disks under high hybrid load scenarios.

[0073] The layout of a shingled magnetic recording disk is as follows: Figure 6 As shown, the shingled magnetic recording disk is divided into two main areas: the conventional magnetic recording (CMR) area and the shingled magnetic recording (SMR) area. The CMR area includes a start area, a zone index area, a reserved area, and a start area backup. The start area records critical disk information, including the zone index size and other key information. Two or more copies are typically used to ensure data security. The zone index area stores the index information for each SMR zone. To ensure data security, two copies can be used. The zone index area consists of multiple zone indexes arranged sequentially. The reserved area is located to the right of the zone index area and is reserved for expansion or internal management. The start area backup is located at the end of the CMR area (after the reserved area).

[0074] SMR zones occupy the majority of disk space and consist of multiple SMR zones arranged sequentially. Each SMR zone is a fixed-size sequential write unit, and its metadata is stored entirely in the corresponding zone index within the CMR zone. Figure 1 In the diagram, each zone index points to its corresponding SMR zone via an arrow, forming a one-to-one mapping. Each SMR zone has the following attributes (recorded in the zone index of the CMR area): ZONE ID: A unique identifier (starting from 1, based on disk partitioning).

[0075] Storage period: refers to the storage period of data. When the storage period is exceeded, data can be deleted through configuration policies.

[0076] ZONE Types: SMR ZONEs are divided into three types: Real-time Data ZONE, Historical Data ZONE, and GC ZONE. The Real-time Data ZONE stores data generated in the current time. The Historical Data ZONE stores past data, such as historical data retransmissions or data reconstruction performed due to disk anomalies / offline events. The GC ZONE stores valid data from the SMR ZONE awaiting garbage collection.

[0077] Status: This mainly includes incomplete and full states. Zones in an incomplete state can continue to store data. Zones in a full state cannot store data, and if the percentage of valid data falls below a threshold, data reclamation will be triggered.

[0078] User file information: File identifiers and data location mappings stored within this zone.

[0079] In addition, a separate database (DB) is set up to record global information such as the file ID, last modification time, ZONE ID, and file size of each file, which works in conjunction with the ZONE index area to achieve metadata management.

[0080] Based on the disk layout described above, the file system needs to be formatted, and the process is as follows: Figure 7 As shown, basic information is retrieved from the disk, including zone size, number of SMR zones, and number of CMR zones. The information for each key metadata element is confirmed according to preset rules: the size of the CMR zone is equal to the zone size multiplied by the number of CMR zones; the boot sector uses a two-replica design (for high availability), located at the beginning and end of the CMR zone, with a size of 1MB; the index information for a single SMR zone is 256KB by default, and the size of the zone index area is equal to 256KB multiplied by the number of SMR zones. Finally, the calculated key metadata information is stored in the boot sector.

[0081] After formatting, the storage system starts. During startup, the ZONE index needs to be loaded into memory to improve allocation efficiency. Data storage is the process of aggregating and storing data according to storage period and time attributes. Storage period refers to the data's storage lifespan, including permanent storage, 180 days, 30 days, 7 days, 1 day, etc. Time attributes are used to distinguish between real-time and historical data: the judgment rule is the difference between the data's write time and the current time. If the difference is less than a threshold (e.g., 1 day), it is considered real-time data; otherwise, it is historical data. Based on the above rules, the data storage process is as follows: Figure 8 As shown, the specific explanation is as follows: The storage system receives user data, confirms the storage period of the user data, and determines whether it is real-time data or historical data according to time rules. It first searches for an SMR zone that has the same storage period and time attributes as the data to be stored and is in a writable state (not full). If such a partition exists, it is used as the target partition, and the data is written to that partition sequentially. If it does not exist, a free SMR zone is allocated, its storage period and time attributes are set to be the same as the data to be stored, and it is marked as not full. Then, this free partition is used as the target partition, and the data is written to it sequentially. After writing is complete, the user file information (including file ID and data storage location information) is updated in the ZONE index, and the file information (including file size, data storage ZONE ID, and last modification time) is updated in the database.

[0082] After data writing is complete, the system periodically checks and executes the garbage collection (GC) process. Garbage collection is the process of moving valid data from a full SMRZONE (including real-time data ZONE, historical data ZONE, and GC ZONE) to the GC ZONE. Triggering GC requires meeting two rules: the data void rule (the percentage of valid data is below a threshold, such as 70%) and the time rule (the state that meets the data void rule continues for more than the threshold, such as 8 hours).

[0083] In this solution, GC is also a process of consolidating storage based on data storage time. The core idea is to move the valid data in the SMR ZONE to be GC'd, file by file, according to the order of the files' last modification time, to the GC ZONE. The overall effect is as follows: Figure 9 As shown, the GC process of sequentially storing data into the GC ZONE is as follows: Figure 10 As shown. First, it is determined whether there is a partition to be reclaimed in the shingled magnetic recording disk (i.e., an SMR ZONE that meets the above garbage collection conditions). If it exists, for that partition, a list of file IDs is obtained from the ZONE index, and the last modification time of each file is queried in the database. The files are sorted from earliest to latest according to their last modification time to determine the order of data migration. Then, the valid data of each file is read from the partition to be reclaimed and written to the GC ZONE in turn. After the file migration is completed, the index information of the GC ZONE is updated first (recording the file ID and new location), and then the SMR ZONE information of the file stored in the database is updated (changed to the new partition ID and location). The original data is not deleted during the migration process, but the data corresponding to the file in the partition to be reclaimed is deleted after the migration is completed. After all files have been migrated, a RESET ZONE operation (write pointer reset, similar to formatting) is performed on the partition to be reclaimed to release its storage space.

[0084] This invention effectively reduces the probability of garbage collection and improves storage performance in data storage of shingled magnetic recording disks, especially in scenarios involving large-scale data reconstruction such as historical data supplementation, node anomalies, or disk anomalies. Storing real-time and historical data in SMR zones with different attributes avoids frequent garbage collection triggers due to the low percentage of valid data in partitions caused by expired and deleted historical data. Furthermore, garbage collection is performed on a file-by-file basis and ordered according to the file's last modification time, further integrating valid data chronologically and effectively reducing the probability of triggering garbage collection again after data deletion, thus avoiding a snowball effect.

[0085] To better implement the data processing method based on shingled magnetic recording disks in the embodiments of the present invention, based on the data processing method based on shingled magnetic recording disks, correspondingly, as follows: Figure 11 As shown, this embodiment of the invention also provides a data processing system 1100 based on a shingled magnetic recording disk, the data processing system 1100 based on a shingled magnetic recording disk comprising: A shingled magnetic recording disk, comprising a conventional magnetic recording area and a shingled magnetic recording area, wherein the shingled magnetic recording area contains multiple SMR zones; The acquisition module is used to obtain the storage period and timeliness attribute of the data to be stored. The writing module is used to sequentially write the data to be stored into the target partition, wherein the target partition is an SMR ZONE with the same storage period and the same timeliness attributes as the data to be stored. The judgment module is used to determine whether there is a partition to be recycled in the shingled magnetic recording disk, wherein the partition to be recycled is an SMR ZONE that meets the conditions for garbage collection. The recycling module is used to move the valid data in the partition to be recycled to the garbage collection zone (GC ZONE) in the order of the last modification time of each valid data in the partition to be recycled when the partition to be recycled exists, and release the storage space of the partition to be recycled after the migration is completed.

[0086] The data processing system 1100 based on shingled magnetic recording disks provided in the above embodiments can realize the technical solutions described in the above embodiments of the data processing method based on shingled magnetic recording disks. The specific implementation principles of each module or unit can be found in the corresponding content in the above embodiments of the data processing method based on shingled magnetic recording disks, and will not be repeated here.

[0087] like Figure 12 As shown, the present invention also provides an electronic device 1200. The electronic device 1200 includes a shingled magnetic recording disk, at least one sensor or image acquisition device, a communication interface, a processor 1201, a memory 1202, and a display 1203. Figure 12Only a portion of the components of the electronic device 1200 are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented alternatively. A shingled magnetic recording disk is used to store data; at least one sensor or image acquisition device is used to acquire the data to be stored and the write time of the data to be stored; wherein the write time is used to determine the timeliness attribute of the data to be stored; the processor, the memory, the shingled magnetic recording disk, and the sensor or image acquisition device are all connected to the communication interface to achieve data communication.

[0088] In some embodiments, processor 1201 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 1202 or process data, such as the data processing method based on shingled magnetic recording disk in this invention.

[0089] In some embodiments, processor 1201 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 1201 may be local or remote. In some embodiments, processor 1201 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0090] In some embodiments, memory 1202 may be an internal storage unit of electronic device 1200, such as a hard disk or memory of electronic device 1200. In other embodiments, memory 1202 may also be an external storage device of electronic device 1200, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 1200.

[0091] Furthermore, the memory 1202 may include both internal storage units of the electronic device 1200 and external storage devices. The memory 1202 is used to store application software and various types of data installed on the electronic device 1200.

[0092] In some embodiments, display 1203 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 1203 is used to display information from electronic device 1200 and to display a visual user interface. Components 1201-1203 of electronic device 1200 communicate with each other via a system bus.

[0093] In one embodiment, when the processor 1201 executes a data processing program based on a shingled magnetic recording disk in the memory 1202, the following steps can be implemented: Obtain the storage period and expiration attribute of the data to be stored; The data to be stored is sequentially written to the target partition, which is an SMR ZONE with the same storage period and the same timeliness attribute as the data to be stored. Determine whether there is a partition to be recycled in the shingled magnetic recording disk, wherein the partition to be recycled is an SMR ZONE that meets the conditions for garbage collection; When the partition to be reclaimed exists, the valid data in the partition to be reclaimed is moved to the garbage collection zone according to the order of the last modification time of each valid data in the partition to be reclaimed, and the storage space of the partition to be reclaimed is released after the relocation is completed.

[0094] It should be understood that when the processor 1201 executes the data processing program based on the shingled magnetic recording disk in the memory 1202, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0095] Furthermore, the embodiments of the present invention do not specifically limit the type of the electronic device 1200 mentioned. The electronic device 1200 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1200 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0096] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the data processing methods based on shingled magnetic recording disks provided in the above-described method embodiments.

[0097] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0098] The data processing method, system, device, and medium based on shingled magnetic recording disks provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method based on a shingled magnetic recording disk, applied to a shingled magnetic recording disk, wherein the shingled magnetic recording disk includes a conventional magnetic recording area and a shingled magnetic recording area, and the shingled magnetic recording area contains multiple shingled storage partitions; characterized in that, The method includes: Obtain the storage period and expiration attribute of the data to be stored; The data to be stored is sequentially written to the target partition, which is a shingled storage partition with the same storage period and the same timeliness attributes as the data to be stored. Determine whether there is a partition to be recycled in the shingled magnetic recording disk, wherein the partition to be recycled is a shingled storage partition that meets the conditions for garbage collection; When the partition to be reclaimed exists, the valid data in the partition to be reclaimed is moved to the garbage collection partition in the order of the last modification time of each valid data in the partition to be reclaimed, and the storage space of the partition to be reclaimed is released after the relocation is completed.

2. The method according to claim 1, characterized in that, The step of sequentially writing the data to be stored into the target partition includes: Determine whether there exists a shingled storage partition that has the same storage period and timeliness attributes as the data to be stored and is in a writable state; If it exists, the shingled storage partition is used as the target partition and the data to be stored is written sequentially. If no free partition exists, allocate a free partition, set the storage period and expiration attributes of the free partition to be the same as the data to be stored and mark it as writable, use the free partition as the target partition and write the data to be stored sequentially. After the write operation is completed, the index information of the target partition is updated to record the file identifier and location information of the written data, and the metadata database is updated to record the file identifier, storage period, partition identifier, and last modification time of the written data.

3. The method according to claim 2, characterized in that, Also includes: When multiple shingled storage partitions exist that have the same storage period and the same timeliness attributes as the data to be stored and are in a writable state, the shingled storage partition with the earliest write time is selected as the target partition.

4. The method according to claim 1, characterized in that, The determination of whether there is a partition to be reclaimed in the shingled magnetic recording disk includes: Traverse all shingled storage partitions in the shingled magnetic recording disk, calculate the effective data percentage of the current shingled storage partition, and detect the duration for which the effective data percentage is lower than a preset percentage threshold. If the percentage of valid data is lower than the preset percentage threshold and the duration exceeds the preset duration threshold, the current shingled storage partition is determined to be the partition to be reclaimed.

5. The method according to claim 1, characterized in that, The step of migrating the valid data to the garbage collection partition according to the order of the last modification time of each valid data in the partition to be collected includes: Obtain the last modification time corresponding to each valid data in the partition to be reclaimed; Read all data from the target file to which each valid data belongs from the partition to be reclaimed in order of last modification time from earliest to latest, and write the read data into the garbage collection partition; After the target file migration is completed, update the index information of the garbage collection partition and the file storage location information in the metadata database, and delete the data corresponding to the target file in the partition to be collected.

6. The method according to claim 1, characterized in that, Releasing the storage space of the partition to be reclaimed includes: performing a reset operation on the partition to be reclaimed, resetting the write pointer of the partition to be reclaimed, and marking all storage space of the partition to be reclaimed as reusable.

7. The method according to any one of claims 1 to 6, characterized in that, The storage period and timeliness attributes of the data to be stored include: Read the preset storage period configuration of the data to be stored to determine the storage period; Obtain the write time of the data to be stored, and compare the time difference between the write time and the current time; When the time difference is less than a preset duration threshold, the timeliness attribute of the data to be stored is determined to be real-time data; When the time difference is greater than the preset duration threshold, the timeliness attribute of the data to be stored is determined to be historical data.

8. A data processing system based on shingled magnetic recording disks, characterized in that, include: A shingled magnetic recording disk, comprising a conventional magnetic recording area and a shingled magnetic recording area, wherein the shingled magnetic recording area includes multiple shingled storage partitions; The acquisition module is used to obtain the storage period and timeliness attribute of the data to be stored. The write module is used to sequentially write the data to be stored into the target partition, wherein the target partition is a shingled storage partition with the same storage period and the same timeliness attributes as the data to be stored; The judgment module is used to determine whether there is a partition to be recycled in the shingled magnetic recording disk, wherein the partition to be recycled is a shingled storage partition that meets the conditions for garbage collection; The recycling module is used to, when the partition to be recycled exists, move the valid data to the garbage recycling partition in the order of the last modification time of each valid data in the partition to be recycled, and release the storage space of the partition to be recycled after the relocation is completed.

9. An electronic device, characterized in that, include: Shingled magnetic recording disks are used to store data; At least one sensor or image acquisition device is used to acquire the data to be stored and the write time of the data to be stored; Memory, which stores computer programs; A processor, coupled to the memory, which, when executing the computer program, implements the method of any one of claims 1 to 7, wherein the write time is used to determine the timeliness attribute of the data to be stored; and The processor, the memory, the shingled magnetic recording disk, and the sensor or image acquisition device are all connected to the communication interface to achieve data communication.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can perform the steps of the method described in any one of claims 1 to 7.

Images