A method and apparatus for redirecting data aggregation

Abstract

This invention relates to the field of storage system IO performance optimization technology, and provides a method and device for redirecting data aggregation, comprising: identifying random small IO data from written IO requests, aggregating and writing to backend storage objects, and constructing a storage mapping relationship in an index table; traversing backend storage objects storing random small IO data, and allocating data originating from the same logical address space to the same target backend storage object using a hash algorithm; sequentially writing the data allocated to the same target backend storage object into the contiguous physical space of the target backend storage object, and updating the mapping relationship in the index table; repeating steps S2 and S3 before reaching a preset termination trigger condition, using a different hash algorithm each time step S2 is repeated; when reading data, retrieving the storage mapping relationship from the index table, and aggregating the read requests corresponding to data located in the same backend storage object before reading them all at once.

Description

Technical Field

[0001] This invention relates to the field of storage system I / O performance optimization technology, and in particular to a method and apparatus for redirecting data aggregation. Background Technology

[0002] In the field of computer data storage, especially in hard disk drive (HDD) storage systems, input / output (I / O) performance is largely limited by its physical mechanical structure. HDD data reading and writing relies on the read / write head's seek operation on the disk platters, which is accomplished by moving the drive arm. When data is written and read sequentially in large blocks (large I / O), the drive arm can move continuously in one direction, resulting in extremely high read / write efficiency. Conversely, when faced with a large number of random small block (small I / O) read / write requests, the drive arm must frequently swing back and forth to seek between different tracks. This leads to significant seek time overhead, severely degrading the storage system's throughput and response latency, and accelerating mechanical wear, thus shortening the hard drive's lifespan.

[0003] To optimize write performance, especially to address the issues caused by random small I / O writes, Redirect-on-Write (ROW) and data aggregation techniques have emerged. The basic principle of this technique is to aggregate multiple random small I / O data received within a certain period into a large data block (large I / O) in the cache, then allocate a contiguous physical space for this large I / O data block and write it sequentially to disk. Simultaneously, the system maintains a metadata index table (usually called the ROW index table), which precisely records the mapping between the logical address (such as the logical unit number LUN ID and offset) of each original small I / O data and its actual physical address (such as the backend storage object ID and internal offset) in the aggregated block.

[0004] While the aforementioned existing technologies successfully transformed random writing into sequential writing, significantly improving write performance, they introduced a new and more severe performance bottleneck: a sharp decline in read performance. Their inherent defects and shortcomings are mainly reflected in the following aspects: 1. Mismatch between data layout and read pattern: The aggregation strategy of this technology is based on time order rather than logical address contiguity. That is, it aggregates data written at the same time but with completely unrelated logical addresses together. As a result, although the writes are sequential large I / Os, from the perspective of physical storage, data that is originally contiguous in the logical address space may actually be scattered in multiple completely different and far apart aggregated blocks, becoming highly discrete.

[0005] 2. Read Amplification: When an application needs to read a contiguous logical address space (e.g., a 1MB sequential read request), the storage system must first query the ROW index table to break it down into numerous discrete physical addresses. Due to the discontinuous physical distribution of the required data, a large logical read request can be amplified into dozens or even hundreds of smaller physical read requests. For example, a 1MB block of data, ideally written by aggregating 256 small 4KB I / O operations, could be read in one go. However, with this technique, if these 256 blocks are physically scattered, up to 256 independent disk read operations are required. This amplification of requests by hundreds of times can instantly fill the disk I / O queue, causing a surge in response latency and, in severe cases, directly paralyzing the storage system service.

[0006] 3. Lack of effective post-processing and optimization mechanisms: Existing ROW technology mainly focuses on optimization during the write phase. For data already stored on disk in a fragmented form, it lacks effective dynamic processing and re-aggregation mechanisms. Once data is written in a certain layout, its inefficient read characteristics will persist, and it cannot self-optimize based on access patterns, thus failing to fundamentally solve the read performance problem.

[0007] Chinese patent CN106776967A discloses a method and apparatus for real-time storage of massive small files based on a time-series aggregation algorithm. This solution improves the storage efficiency of massive small files through time-series aggregation and secondary indexing. However, in practical applications, it has the following significant drawbacks: 1. Read performance is still limited by time-series layout: Data is aggregated in chronological order, but files with the same logical relationship may be scattered across different merged files due to different write times. Reading requires multiple index queries and accesses to multiple merged files, making it impossible to achieve a one-time sequential read of logically continuous data; the read amplification problem remains unresolved.

[0008] 2. Insufficient optimization of data locality: The aggregation strategy only relies on the time dimension and does not consider the continuity of the logical address space (such as the directory to which the file belongs and access correlation), which results in the data of the same logical volume still being fragmented in physical storage, and cannot take advantage of spatial locality to improve read efficiency.

[0009] 3. Lack of dynamic reorganization mechanism: The aggregation space is only reset by triggering a state threshold, but it lacks the ability to dynamically reorganize the stored data. Once the data is written, its physical layout is fixed and cannot adaptively optimize the data distribution according to the access pattern. After long-term operation, the read performance will still degrade.

[0010] 4. Limited aggregation effect due to the reliance on a single hash algorithm: Data distribution is based solely on consistent hashing, and the hashing strategy is fixed. Data within the same logical space may fail to aggregate sufficiently due to hash collisions or the distribution strategy, remaining scattered even after multiple writes, making deep aggregation difficult.

[0011] 5. Index maintenance is complex and costly: It relies on secondary indexes and Bloom filters, and requires the maintenance of a large amount of metadata (such as mapping databases and local indexes), which increases system complexity and memory overhead. Furthermore, the false positive rate of Bloom filters may lead to additional query overhead.

[0012] Therefore, how to effectively solve the read performance degradation problem caused by redirected aggregation write technology while retaining its write performance advantages, and how to achieve large IO sequential reads during reading, ultimately achieving dual optimization of write and read performance, and further improving the overall efficiency and reliability of the storage system, has become an urgent technical problem to be solved. Summary of the Invention

[0013] In view of this, in order to overcome the shortcomings of the prior art, the present invention aims to provide a method and apparatus for redirecting data aggregation.

[0014] According to a first aspect of the present invention, a method for redirecting data aggregation is provided, characterized in that the method comprises: Step S1: Identify random small IO data from the written IO requests, aggregate and write to the backend storage object, and build a storage mapping relationship in the index table; Step S2: Traverse the backend storage objects that store random small IO data, and use a hash algorithm to assign data from the same logical address space to the same target backend storage object; Step S3: Write the data that are allocated to the same target backend storage object into the contiguous physical space of the target backend storage object in sequence, and update the mapping relationship in the index table; Step S4: Before the preset termination trigger condition is reached, repeat steps S2 and S3, and use a different hash algorithm each time step S2 is repeated. Step S5: When reading data, retrieve the storage mapping relationship from the index table, aggregate the read requests corresponding to data located in the same backend storage object, and read them all at once.

[0015] Optionally, in the redirected data aggregation method of the present invention, step S1, constructing a storage mapping relationship in the index table includes: in the ROW index table, generating an index key by combining the volume ID of the front-end logical volume of the data with the logical offset within the volume, generating an index value by combining the identifier of the initial back-end storage object to which the data is written and its physical offset within the initial back-end storage object, establishing a mapping relationship between the index key and the index value, and storing it.

[0016] Optionally, in the redirection data aggregation method of the present invention, step S2, which assigns data originating from the same logical address space to the same target backend storage object using a hash algorithm, includes: reading data from an initial backend storage object, extracting the logical address space identifier of the data, using the extracted logical address space identifier as the input value of the hash algorithm, calculating the output value, and determining the target backend storage object from the available storage resource pool based on the output value, wherein the logical address space identifier is the volume ID of the frontend logical volume to which the data belongs.

[0017] Optionally, in the redirected data aggregation method of the present invention, in step S3, the data allocated to the same target backend storage object are sorted according to the logical address space identifier and the volume logical offset, and the sorted data is written sequentially into the continuous physical space of the target backend storage object.

[0018] Optionally, in the redirection data aggregation method of the present invention, step S3, sorting data allocated to the same target backend storage object according to the logical address space identifier and the volume logical offset includes: grouping data with the same logical address space identifier together using the logical address space identifier as the primary sorting key; and within the same logical address space identifier group, arranging the data in ascending or descending order using the volume logical offset as the secondary sorting key to form a continuous data sequence on the logical address.

[0019] Optionally, in the redirected data aggregation method of the present invention, step S3, updating the mapping relationship in the index table includes: for each piece of data that is moved, finding the entry corresponding to its original logical address in the index table, and updating the physical address information of the entry to the new target backend storage object identifier and the physical offset of the data within the object.

[0020] Optionally, in the redirected data aggregation method of the present invention, the preset termination trigger condition in step S4 is: after N updates of the mapping relationship in the index table, the logical address continuity of the data in the target backend storage object has reached a preset optimization value, or the number of updates of the mapping relationship in the index table has reached the maximum number M, where N and M are integers greater than 1.

[0021] Optionally, in the redirected data aggregation method of the present invention, step S4 involves using a different hash algorithm than all previous executions when step S2 is executed repeatedly, including: pre-storing multiple different hash functions, and selecting a previously unused hash function from the pre-stored multiple hash functions each time step S2 is executed.

[0022] Optionally, in the redirection data aggregation method of the present invention, step S5 includes: Parse the data read request to obtain the logical address list of the data to be read; The index table is queried to convert the logical address list of the data to be read into the corresponding physical address list. The physical address includes the identifier of the backend storage object and the physical offset inside the object. The physical addresses are grouped according to the identifier of the backend storage object. For multiple physical addresses in the same group, it is analyzed whether the physical offsets inside the object form a continuous interval or whether they are within a preset proximity threshold range. If the conditions are met, calculate the smallest contiguous interval covering these offsets, and generate an identifier for the backend storage object and a single read operation instruction for that smallest contiguous interval.

[0023] According to a second aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect of the present invention.

[0024] The redirection data aggregation method of the present invention has the following beneficial technical effects: 1. Significantly improves data reading performance and eliminates read amplification effect. Through multiple iterations of data processing using different hash algorithms, data that was originally stored discretely but belonged to the same logical address space (such as the same LUN) is physically re-aggregated into the same or contiguous backend storage objects. When processing read requests, a large number of small I / O read requests targeting discrete physical addresses can be merged into a few or even one large sequential read request, thereby fundamentally eliminating the troublesome "read amplification" phenomenon, significantly reducing read latency, and improving the throughput of the storage system.

[0025] 2. Achieve dual optimization of write and read performance. Inheriting the advantages of traditional Redirected Write (ROW) technology, it still aggregates random small I / Os into large I / Os for sequential writing during the write phase, ensuring excellent write performance. At the same time, through a data compaction mechanism, it overcomes the performance defects in the read phase, ultimately achieving the ideal state of "large I / Os during writing and large I / Os during reading," achieving bidirectional performance optimization for read and write, and making the overall performance of the storage system more balanced and efficient.

[0026] 3. High efficiency in resource utilization and controllable system overhead. By reusing the storage system's inherent background processes (such as garbage collection or space reclamation) to perform data compaction and migration operations, there is no need to introduce additional, resource-intensive independent processes. Data migration is performed gradually in the background, minimizing the impact on front-end business I / O. At the same time, by setting iteration termination conditions (such as reaching a predetermined aggregation degree or maximum number of migrations), the consumption of computing and I / O resources can be intelligently controlled, avoiding unlimited compaction operations and ensuring controllable and efficient system overhead.

[0027] 4. Enhance data locality and improve caching efficiency By storing logically contiguous data in a centralized manner at the physical level, spatial locality of data is greatly enhanced, significantly improving the effectiveness of caching strategies such as read-ahead. Once a cache hit occurs, the probability of subsequent hits also increases significantly, thereby further improving data access speed and reducing the actual I / O pressure on the backend physical disk.

[0028] 5. Optimize hard drive access modes to extend hardware lifespan. Because read requests are efficiently aggregated, the hard drive's read / write head no longer needs to perform frequent and long-distance seek operations, but can instead perform more continuous, sequential read operations. This optimized access mode not only reduces the average response time of a single read request, but also significantly reduces wear on the mechanical arm, thereby helping to extend the lifespan of the hard drive and improve the reliability and stability of the entire storage system.

[0029] 6. Enhanced system scalability and stability By addressing the read amplification issue, the system's ability to handle high-concurrency read loads is greatly enhanced. This avoids the risks of IO queue saturation, latency spikes, and even system crashes caused by the accumulation of massive random small IO read requests. As a result, the storage system can operate more stably in I / O-intensive application scenarios and has better scalability and business support capabilities. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the steps of a redirected data aggregation method according to an embodiment of the present invention; Figure 2 This is an example diagram of the IO write process in existing technology; Figure 3 This is an example diagram of the data layout obtained after IO writing according to existing technology; Figure 4 This is an example diagram of the data layout obtained according to the redirected data aggregation method of this embodiment; Figure 5 This is a schematic diagram of the structure of the device provided by the present invention. Detailed Implementation

[0032] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0034] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0035] The technical principle of this embodiment is as follows: Identify random small I / O operations and store them separately from other I / O operations in different types of physical storage spaces. Implement a space reuse and reclamation mechanism to move physical storage spaces created by I / O writes. Hash the data according to its logical address space name to ensure that data with the same logical address space name is written to the same physical object storage space. During each move, data is sorted by logical address space and logical offset before being written. Different hash algorithms are used for the first, second, and third moves to ensure that each move aggregates data with the same logical space. When reading data, if the read position information obtained from the ROW index is within the same physical storage object space, aggregate the read request and perform a single read.

[0036] The method of this invention can improve the sequentiality and IO size of data read by users, while ensuring that data is still written in a large IO order model.

[0037] Figure 1 This is a flowchart illustrating the steps of a redirected data aggregation method according to an embodiment of the present invention. Figure 1 As shown, the redirected data aggregation method in this embodiment includes the following steps: Step S1: Identify random small IO data from the written IO requests, aggregate and write to the backend storage object, and build a storage mapping relationship in the index table.

[0038] As an optional example, in the ROW index table, an index key is generated by combining the volume ID of the front-end logical volume of the data with the logical offset within the volume, and an index value is generated by combining the identifier of the initial back-end storage object to which the data is written with its physical offset within the initial back-end storage object. The mapping relationship between the index key and the index value is established and stored.

[0039] Step S2: Traverse the backend storage objects that store random small I / O data, and use a hash algorithm to assign data from the same logical address space to the same target backend storage object.

[0040] As an optional example, this embodiment reads the data in the initial backend storage object, extracts the logical address space identifier of the data, uses the extracted logical address space identifier as the input value of the hash algorithm, calculates the output value, and determines the target backend storage object from the available storage resource pool based on the output value. The logical address space identifier is the volume ID of the frontend logical volume to which the data belongs.

[0041] Step S3: Write the data that are assigned to the same target backend storage object into the contiguous physical space of the target backend storage object in sequence, and update the mapping relationship in the index table.

[0042] As an optional example, this embodiment sorts the data allocated to the same target backend storage object according to the logical address space identifier and the volume logical offset, and writes the sorted data sequentially into the contiguous physical space of the target backend storage object.

[0043] For example, in this embodiment, data allocated to the same target backend storage object is sorted according to the logical address space identifier and the volume logical offset. This includes: grouping data with the same logical address space identifier together using the logical address space identifier as the primary sorting key; and within the same logical address space identifier group, arranging the data in ascending or descending order using the volume logical offset as the secondary sorting key to form a continuous data sequence on the logical address.

[0044] As an optional example, in this embodiment, when updating the mapping relationship in the index table, for each piece of data that is moved, the entry corresponding to its original logical address is found in the index table, and the physical address information of the entry is updated to the new target backend storage object identifier and the physical offset of the data within the object.

[0045] Step S4: Before the preset termination trigger condition is reached, repeat steps S2 and S3, and use a different hash algorithm each time step S2 is repeated.

[0046] As an optional example, the preset termination trigger condition in this embodiment is: after N updates to the mapping relationship in the index table, the logical address continuity of the data in the target backend storage object has reached a preset optimization value, or the number of updates to the mapping relationship in the index table has reached the maximum number M, where N and M are integers greater than 1.

[0047] As an optional example, in this embodiment, a different hash algorithm is used each time step S2 is executed, including: pre-storing multiple different hash functions, and selecting a hash function that has not been used before from the pre-stored multiple hash functions each time step S2 is executed.

[0048] Step S5: When reading data, retrieve the storage mapping relationship from the index table, aggregate the read requests corresponding to data located in the same backend storage object, and read them all at once.

[0049] As an optional example, in this embodiment, when responding to a data read request, the data read request is parsed to obtain a list of logical addresses of the data to be read; an index table is queried to convert the list of logical addresses of the data to be read into a corresponding list of physical addresses, where the physical address includes the identifier of the backend storage object and the physical offset within the object; the physical addresses are grouped according to the identifier of the backend storage object, and for multiple physical addresses in the same group, it is analyzed whether the physical offsets within the object constitute a continuous interval or whether they are within a preset proximity threshold range; if the conditions are met, the minimum continuous interval covering these offsets is calculated, and a single read operation instruction is generated for the identifier of the backend storage object and the minimum continuous interval.

[0050] The redirection data aggregation method of this embodiment will be further described below in a specific scenario.

[0051] Figure 2 Here is an example diagram of the IO write process in existing technology, such as... Figure 2 As shown, when data from one or more clients is received by a storage server, the server aggregates this written data into a large data block. Simultaneously, it records the correspondence information between the original requests and the aggregated data space. The data block information and the correspondence information are packaged together and written to the backend storage object Y. The starting address of the data block write is Z, and the relative offset of each original data request within the aggregated data block is X. Thus, the information actually written to the backend storage object for each request is (Y, X+Z). The logical volume ID + volume offset of each request is used as the key, and the actual storage information is recorded as the value in the ROW index table. When reading data, the required data can be retrieved by looking up the ROW index table.

[0052] Figure 3 Here is an example diagram of the data layout obtained after IO write in existing technology, such as Figure 3 As shown, after random small I / O writes to the backend storage object, the data of the three LUNs are positioned in the backend storage object according to time order. Since data aggregation writes data from the same point in time to a centralized location, the data in the backend storage object is arranged in chronological order. Taking LUN1 as an example, when a request needs to read the five data blocks shown in the figure, since the data is stored in (A,4), (B,7), (A,1), (B,4), and (A,8), these data are not contiguous on the storage object. Therefore, during data reading, what was originally one read request will be amplified into five requests. If it's a 4K small I / O write followed by a 1M I / O read, the request will be amplified by 256 times, which would cause storage system failure in actual use.

[0053] Figure 4This is an example diagram of the data layout obtained according to the redirected data aggregation method of this embodiment. The final data storage layout will show that the data has good ordering, specifically: data from the same front-end volume can be stored in the same back-end storage object, and the offset in the back-end storage object is basically the same as the order in the front-end volume. When a request needs to read the 5 data blocks shown in the figure, since the data is stored in (A,1), (A,2), (A,3), (A,4), (A,5), the data in object A is originally a contiguous data block, which can be directly merged into a single consecutive read request. In practical applications, as mentioned above, with a small 4K IO write followed by a 1M IO read, it is still possible to achieve only one read request, and the system can run normally.

[0054] In practical applications, the redirection data aggregation method of this embodiment has the following beneficial technical effects: 1. Significantly improves data reading performance and eliminates read amplification effect. Through multiple iterations of data processing using different hash algorithms, data that was originally stored discretely but belonged to the same logical address space (such as the same LUN) is physically re-aggregated into the same or contiguous backend storage objects. When processing read requests, a large number of small I / O read requests targeting discrete physical addresses can be merged into a few or even one large sequential read request, thereby fundamentally eliminating the troublesome "read amplification" phenomenon, significantly reducing read latency, and improving the throughput of the storage system.

[0055] 2. Achieve dual optimization of write and read performance. Inheriting the advantages of traditional Redirected Write (ROW) technology, it still aggregates random small I / Os into large I / Os for sequential writing during the write phase, ensuring excellent write performance. At the same time, through a data compaction mechanism, it overcomes the performance defects in the read phase, ultimately achieving the ideal state of "large I / Os during writing and large I / Os during reading," achieving bidirectional performance optimization for read and write, and making the overall performance of the storage system more balanced and efficient.

[0056] 3. High efficiency in resource utilization and controllable system overhead. By reusing the storage system's inherent background processes (such as garbage collection or space reclamation) to perform data compaction and migration operations, there is no need to introduce additional, resource-intensive independent processes. Data migration is performed gradually in the background, minimizing the impact on front-end business I / O. At the same time, by setting iteration termination conditions (such as reaching a predetermined aggregation degree or maximum number of migrations), the consumption of computing and I / O resources can be intelligently controlled, avoiding unlimited compaction operations and ensuring controllable and efficient system overhead.

[0057] 4. Enhance data locality and improve caching efficiency By storing logically contiguous data in a centralized manner at the physical level, spatial locality of data is greatly enhanced, significantly improving the effectiveness of caching strategies such as read-ahead. Once a cache hit occurs, the probability of subsequent hits also increases significantly, thereby further improving data access speed and reducing the actual I / O pressure on the backend physical disk.

[0058] 5. Optimize hard drive access modes to extend hardware lifespan. Because read requests are efficiently aggregated, the hard drive's read / write head no longer needs to perform frequent and long-distance seek operations, but can instead perform more continuous, sequential read operations. This optimized access mode not only reduces the average response time of a single read request, but also significantly reduces wear on the mechanical arm, thereby helping to extend the lifespan of the hard drive and improve the reliability and stability of the entire storage system.

[0059] 6. Enhanced system scalability and stability By addressing the read amplification issue, the system's ability to handle high-concurrency read loads is greatly enhanced. This avoids the risks of IO queue saturation, latency spikes, and even system crashes caused by the accumulation of massive random small IO read requests. As a result, the storage system can operate more stably in I / O-intensive application scenarios and has better scalability and business support capabilities.

[0060] like Figure 5 As shown, the present invention also provides a device including a processor 210, a communication interface 220, a memory 230 for storing a processor-executable computer program, and a communication bus 240. The processor 210, communication interface 220, and memory 230 communicate with each other via the communication bus 240. The processor 210 implements the aforementioned redirected data aggregation method by running the executable computer program.

[0061] The computer program in memory 230, when implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected based on actual needs to achieve the purpose of this embodiment. Those skilled in the art can understand and implement this without any creative effort.

[0063] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for redirecting data aggregation, characterized in that, The method includes: Step S1: Identify random small IO data from the written IO requests, aggregate and write to the backend storage object, and build a storage mapping relationship in the index table; Step S2: Traverse the backend storage objects that store random small IO data, and use a hash algorithm to assign data from the same logical address space to the same target backend storage object; Step S3: Write the data that are assigned to the same target backend storage object into the contiguous physical space of the target backend storage object in sequence, and update the mapping relationship in the index table; Step S4: Before the preset termination trigger condition is reached, repeat steps S2 and S3, and use a different hash algorithm each time step S2 is repeated. Step S5: When reading data, retrieve the storage mapping relationship from the index table, aggregate the read requests corresponding to data located in the same backend storage object, and read them all at once.

2. The redirected data aggregation method according to claim 1, characterized in that, In step S1, the storage mapping relationship is constructed in the index table, including: in the ROW index table, an index key is generated by combining the volume ID of the front-end logical volume of the data with the logical offset within the volume; an index value is generated by combining the identifier of the initial back-end storage object to which the data is written and its physical offset within the initial back-end storage object; the mapping relationship between the index key and the index value is established and stored.

3. The redirected data aggregation method according to claim 1, characterized in that, In step S2, data originating from the same logical address space are assigned to the same target backend storage object using a hash algorithm. This includes: reading data from the initial backend storage object, extracting the logical address space identifier of the data, using the extracted logical address space identifier as the input value of the hash algorithm, calculating the output value, and determining the target backend storage object from the available storage resource pool based on the output value. The logical address space identifier is the volume ID of the frontend logical volume to which the data belongs.

4. The redirected data aggregation method according to claim 1, characterized in that, In step S3, the data allocated to the same target backend storage object are sorted according to the logical address space identifier and the volume logical offset, and the sorted data is written sequentially into the contiguous physical space of the target backend storage object.

5. The redirected data aggregation method according to claim 4, characterized in that, In step S3, the data allocated to the same target backend storage object are sorted according to the logical address space identifier and the volume logical offset. This includes: grouping data with the same logical address space identifier together using the logical address space identifier as the primary sorting key; and arranging the data in ascending or descending order within the same logical address space identifier group using the volume logical offset as the secondary sorting key to form a continuous data sequence on the logical address.

6. The redirected data aggregation method according to claim 1, characterized in that, In step S3, the mapping relationship in the index table is updated, including: for each piece of data that is moved, the entry corresponding to its original logical address is found in the index table, and the physical address information of the entry is updated to the new target backend storage object identifier and the physical offset of the data within the object.

7. The redirected data aggregation method according to claim 1, characterized in that, In step S4, the preset termination trigger condition is: after N updates of the mapping relationship in the index table, the logical address continuity of the data in the target backend storage object has reached the preset optimization value, or the number of updates of the mapping relationship in the index table has reached the maximum number M, where N and M are integers greater than 1.

8. The redirected data aggregation method according to claim 1, characterized in that, In step S4, a different hash algorithm is used each time step S2 is repeated, including: pre-storing multiple different hash functions, and selecting a previously unused hash function from the pre-stored multiple hash functions each time step S2 is executed.

9. The redirected data aggregation method according to claim 1, characterized in that, Step S5 includes: Parse the data read request to obtain the logical address list of the data to be read; The index table is queried to convert the logical address list of the data to be read into the corresponding physical address list. The physical address includes the identifier of the backend storage object and the physical offset inside the object. The physical addresses are grouped according to the identifier of the backend storage object. For multiple physical addresses in the same group, it is analyzed whether the physical offsets inside the object form a continuous interval or whether they are within a preset proximity threshold range. If the conditions are met, calculate the smallest contiguous interval covering these offsets, and generate an identifier for the backend storage object and a single read operation instruction for that smallest contiguous interval.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method according to any one of claims 1-9.

Images