A data storage method and related apparatus based on a storage array

Abstract

A data storage method based on a storage array is disclosed to improve the storage performance of the storage array. In the target storage device storing parity data within the storage array, a write buffer supporting in-place data updates and a persistent storage medium supporting append-only writes are configured. When data needs to be written to other storage devices in the storage array, and parity data needs to be written to the target storage device, the parity data is first written to the write buffer of the target storage device, and can be updated within the write buffer. Furthermore, when all data on a stripe has been written to other storage devices, rendering the parity data no longer requiring updates, the write buffer then writes the no-follower parity data to the persistent storage medium. This resolves the conflict between the persistent storage medium's lack of in-place data updates and the storage array's requirement for in-place parity data updates, enabling the storage array to deploy open storage protocols and improve storage performance.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data storage method and related apparatus based on a storage array. Background Technology

[0002] In the field of storage technology, since storage devices such as solid-state drives (SSDs) and hard disk drives (HDDs) only support append-only write semantics, when managing storage devices based on traditional block storage technology, garbage collection operations often occur during the data writing process, resulting in write amplification and impacting the overall system software performance.

[0003] Open storage protocols such as Zone Namespaces (ZNS) and Open-Channel are solutions to some inherent problems in traditional block storage technologies. Open storage protocols employ sequential writes and zoned management, offloading data placement strategies and garbage collection to the host software. By reducing the complexity of garbage collection, they improve the performance and durability of storage devices.

[0004] However, current open storage protocols are typically designed for managing individual storage devices, making them difficult to adapt to storage arrays composed of multiple storage devices. This results in low storage performance and slow recovery speeds for existing storage arrays. Summary of the Invention

[0005] This application provides a data storage method and related apparatus based on a storage array, which is used to improve the storage performance of the storage array.

[0006] In a first aspect, a data storage method based on a storage array is provided, applied to an execution device connected to a storage array. The storage array-based data storage method includes: the execution device acquiring a first data write request, the first data write request being used to request that first data be written to the storage array, the storage array including multiple storage devices.

[0007] Based on the first data write request, the execution device sends a second data write request and a checksum write request to the storage array. The second data write request instructs the first data to be segmented and written to multiple data blocks included in the first stripe, and the checksum write request instructs the first checksum to be written to the checksum block included in the first stripe. The first checksum is calculated based on the data to be written from multiple data blocks, and the multiple data blocks and checksum blocks included in the first stripe are evenly distributed on multiple storage devices.

[0008] Among them, the target storage device including the check block includes a write buffer that supports in-situ data updates and a persistent storage medium that supports data appending. The check block is distributed on the persistent storage medium, and the length of the check block is less than the length of the write buffer. The write buffer is used to continuously cache and update the check data to be written to the check block, and to store the check data that will no longer be updated in the persistent storage medium.

[0009] In this solution, the target storage device storing the verification data in the storage array is equipped with a write buffer that supports in-place data updates and a persistent storage medium that supports append-only writes. When data needs to be written to other storage devices in the storage array, and verification data needs to be written to the target storage device, the verification data is first written to the write buffer of the target storage device, and can be updated within the write buffer. Furthermore, when all data on a stripe has been written to other storage devices, and the verification data will no longer be updated, the write buffer then writes the no-longer-update verification data to the persistent storage medium. This resolves the conflict between the persistent storage medium not supporting in-place data updates and the requirement for in-place updates to the verification data in the storage array, enabling the storage array to deploy open storage protocols and improve storage performance.

[0010] In one possible implementation, the second data write request is used to instruct the first data to be segmented and written into the first portion of a plurality of data blocks.

[0011] The storage array-based data storage method further includes: an execution device acquiring a third data write request, the third data write request being used to request that second data be written to the storage array. Based on the third data write request, the execution device sends a fourth data write request and a verification data update request to the storage array. The fourth data write request is used to instruct that the second data be written in segments to a second part of a plurality of data blocks, and the verification data update request is used to instruct that the data in the verification block be updated to second verification data, the second verification data being calculated based on the existing data in the plurality of data blocks and the data to be written.

[0012] The write buffer is used to update the cached first verification data to the second verification data after receiving a verification data update request.

[0013] In other words, for the second check data that needs to be written to the same check block, the execution device can know that the second check data does not need to follow the append write rule, so it will send a check data update request to the storage array to instruct the storage array to perform an in-place update of the check data.

[0014] In one possible implementation, the data storage method based on the storage array further includes: the execution device acquiring a Zone Random Write Area (ZRWA) maintained for the target storage device, the length of which is equal to the length of the write buffer, and the ZRWA sliding on the address space corresponding to the target storage device as the data writing progress of the target storage device progresses; in response to the address corresponding to the parity block being located on the ZRWA, the execution device generating a parity data update request.

[0015] In other words, in order to enable the execution device to be aware of the write buffer on the target storage device, so as to ensure that the execution device can instruct the target storage device to perform the update of the verification data, this solution proposes that the execution device maintain ZRWA and determine the data that can be updated in place based on ZRWA.

[0016] In one possible implementation, the write buffer is used to store the second checksum data to persistent storage when the cached data overflows.

[0017] That is, when the write buffer receives a new data write request, causing the data cached in the write buffer to overflow (i.e., it can no longer cache existing data and new data at the same time), the write buffer will trigger the storage of the previously cached second check data to the persistent storage medium.

[0018] In one possible implementation, the logical space corresponding to multiple storage devices includes a first stripe and a second stripe.

[0019] After completing the writing of the second verification data, the device executes a fifth data write request, which is used to request the third data to be written to the storage array.

[0020] Based on the fifth data write request, the execution device sends a sixth data write request to the storage array. The sixth data write request instructs the third data to be segmented and written into multiple data blocks included in the second stripe. The target storage device contains the data blocks in the second stripe. The write buffer is used to store the second parity data in the persistent storage medium when a segment of the third data is obtained and the remaining space cannot accommodate the segment.

[0021] In this scheme, when new data is acquired in the write buffer and the write buffer cannot simultaneously hold both the old and new data, the old data is written to the persistent storage medium, thereby freeing up space to cache the new data. In this way, whenever the execution device writes data to a new address in the target storage device, it can trigger a write buffer overflow, thus storing the no-longer-updated checksum data to the persistent storage medium.

[0022] In one possible implementation, the target storage device records the last write address of the persistent storage medium and the length of valid data cached in the write buffer, wherein the last write address and the length of valid data are used to enable recovery of the data write process.

[0023] In this solution, by simultaneously recording the last write address of the persistent storage medium and the length of the valid data cached in the write buffer on the target storage device, the execution device can determine the actual data writing status based on the last write address of the persistent storage medium and the length of the valid data cached in the write buffer when power is lost, thereby ensuring the recovery of the data writing process.

[0024] In one possible implementation, after the execution device recovers from a power outage, it sends a query request to the target storage device to retrieve data write information. The execution device receives the last write address and the length of valid data from the target storage device. Based on the last write address and the length of valid data, the execution device determines a write pointer for the target storage device, which indicates the end address of the data previously written to the target storage device.

[0025] In one possible implementation, the target storage device records the length of valid data through an out-of-band area (OOB).

[0026] In one possible implementation, OOB uses multiple bits to represent the length of valid data cached in the write buffer, and one of the multiple bits is used to indicate whether a logical block in the write buffer has valid data cached.

[0027] In one possible implementation, the storage array is managed via an open storage protocol based on sequential writes.

[0028] In a second aspect, a data storage device based on a storage array is provided, comprising: an acquisition module for acquiring a first data write request, the first data write request being used to request writing first data into a storage array, the storage array including multiple storage devices; and a sending module for sending a second data write request and a verification data write request to the storage array based on the first data write request, the second data write request being used to instruct the first data to be segmented and written into multiple data blocks included in a first stripe, and the verification data write request being used to instruct the first verification data to be written into a verification block included in the first stripe, the first verification data being calculated based on the data to be written from multiple data blocks, the multiple data blocks and verification blocks included in the first stripe being evenly distributed on multiple storage devices; wherein, the target storage device including the verification block among the multiple storage devices includes a write buffer that supports in-situ data updates and a persistent storage medium that supports data appending, the verification block being distributed on the persistent storage medium, and the length of the verification block being less than the length of the write buffer, the write buffer being used to continuously cache and update the verification data to be written to the verification block, and to store the verification data that will no longer be updated into the persistent storage medium.

[0029] In one possible implementation, a second data write request is used to instruct the first data to be segmented and written into a first portion of a plurality of data blocks; the acquisition module is further used to acquire a third data write request, which requests the second data to be written into the storage array; the sending module is further used to send a fourth data write request and a verification data update request to the storage array based on the third data write request, the fourth data write request being used to instruct the second data to be segmented and written into a second portion of a plurality of data blocks, and the verification data update request being used to instruct the data in the verification block to be updated to second verification data, which is calculated based on the existing data and the data to be written in the plurality of data blocks; wherein, the write buffer is used to update the cached first verification data to the second verification data after the verification data update request is acquired.

[0030] In one possible implementation, the apparatus further includes: a processing module; an acquisition module, further configured to acquire the Region Random Write Area (ZRWA) maintained by the target storage device, wherein the length of the ZRWA is equal to the length of the write buffer, and the ZRWA slides across the address space corresponding to the target storage device as the data writing progress of the target storage device; and a processing module, configured to generate a verification data update request in response to the address corresponding to the verification block being located on the ZRWA.

[0031] In one possible implementation, the write buffer is used to store the second checksum data to persistent storage when the cached data overflows.

[0032] In one possible implementation, the logical space corresponding to multiple storage devices includes a first stripe and a second stripe; the acquisition module is further configured to acquire a fifth data write request, which requests that third data be written to the storage array; the sending module is further configured to send a sixth data write request to the storage array based on the fifth data write request, which instructs that the third data be written in segments to multiple data blocks included in the second stripe, and the target storage device has data blocks distributed in the second stripe; wherein, the write buffer is configured to store the second verification data in the persistent storage medium when the segmented data belonging to the third data is acquired and the remaining space cannot accommodate the segmented data.

[0033] In one possible implementation, the target storage device records the last write address of the persistent storage medium and the length of valid data cached in the write buffer, wherein the last write address and the length of valid data are used to enable recovery of the data write process.

[0034] In one possible implementation, the apparatus is applied to an execution device connected to a storage array, and the apparatus further includes: a processing module; after the execution device recovers from a power outage, a sending module is further configured to send a query request to the target storage device, the query request being used to request data write status; an acquisition module is further configured to receive the last write address and the length of valid data sent by the target storage device; based on the last write address and the length of valid data, the processing module is configured to determine the write pointer corresponding to the target storage device, the write pointer being used to indicate the end address of the data previously written to the target storage device.

[0035] In one possible implementation, the target storage device records the length of valid data via OOB.

[0036] In one possible implementation, OOB uses multiple bits to represent the length of valid data cached in the write buffer, and one of the multiple bits is used to indicate whether a logical block in the write buffer has valid data cached.

[0037] In one possible implementation, the storage array is managed via an open storage protocol based on sequential writes.

[0038] Thirdly, a data storage device based on a storage array is provided, comprising: a processor and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the data storage device based on the storage array to perform the methods described above.

[0039] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0040] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0041] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0042] In a seventh aspect, a computing device is provided, comprising a memory array-based data storage device of the third aspect or a chip system of the sixth aspect, wherein the memory array-based data storage device or chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0043] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0044] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0045] Figure 1 A schematic diagram of data distribution on RAID 5 provided in this application;

[0046] Figure 2 A schematic diagram of a system architecture provided for this application;

[0047] Figure 3 A schematic diagram illustrating a data storage method based on a storage array provided in this application;

[0048] Figure 4 A schematic diagram of the structure of a target storage device provided in this application;

[0049] Figure 5 A schematic diagram of a storage array service provided in this application;

[0050] Figure 6 A schematic diagram of a system architecture provided for this application;

[0051] Figure 7 A schematic diagram of a data writing process based on ZRWA provided for this application;

[0052] Figure 8 A schematic diagram illustrating the location of a real WP device, as provided in this application;

[0053] Figure 9 This application provides a schematic diagram of an application architecture based on a key-value storage database.

[0054] Figure 10 A schematic diagram of a data writing process provided for this application;

[0055] Figure 11 A schematic diagram illustrating a Zebra data writing process provided in this application;

[0056] Figure 12 A schematic diagram of a data storage device based on a storage array provided in this application;

[0057] Figure 13 A schematic diagram of the structure of a computing device provided in this application;

[0058] Figure 14 This application provides a schematic diagram of the structure of a computing device cluster;

[0059] Figure 15 This application provides a schematic diagram of another computing device cluster structure.

[0060] Figure 16 This is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will recognize that, with the emergence of new application scenarios, the technical solutions provided by this application are also applicable to similar technical problems.

[0062] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0063] For ease of understanding, some technical terms used in this application will be introduced below.

[0064] (1) Waste recycling

[0065] Garbage collection is an important background management process in storage devices. When a user deletes data or the file system marks certain data as invalid, the flash memory blocks containing this data are not immediately erased. Instead, garbage collection is performed later to clean up the deleted or no longer used data, making room for new data to be written.

[0066] (2) Write Enlarged

[0067] Write amplification refers to the phenomenon where the amount of data actually written to a storage device (such as an SSD) is much larger than the amount of data the user intends to write. This is because persistent storage media (such as flash memory) can only be erased and written to entire blocks. Therefore, when updating data, it is often necessary to first read the entire block containing the original data, modify the original data, write the modified data to the entire new block, and mark the old block containing the original data as invalid. This process results in additional write operations, thus amplifying the write volume.

[0068] (3) Redundant Array of Independent Disks (RAID)

[0069] RAID (Rapid Allocation) is a system that combines many independent disks into a large-capacity disk group. It leverages the additive effect of data provided by individual disks to improve the overall performance of the disk system. RAID allows data to be divided into multiple segments and stored on different disks.

[0070] (4) Strip

[0071] In RAID, striping divides a continuous block of data into many smaller parts and stores them on different disks. This allows multiple processes to access different parts of the data simultaneously without disk conflicts, and it also provides maximum input / output (I / O) parallelism when sequential access to this data is required, resulting in excellent performance.

[0072] For example, please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating data distribution on a RAID 5 system, as provided in this application. Figure 1 As shown, taking RAID 5 as an example, a stripe includes multiple data blocks and one parity block, and these data blocks and parity block are evenly distributed across the multiple disks in the RAID array. For example, stripe A includes data blocks A1, A2, A3, and parity block Ap, where data block A1 is located on disk 1, data block A2 on disk 2, data block A3 on disk 3, and parity block Ap on disk 4. Furthermore, RAID is divided into multiple stripes, and the parity blocks on these stripes are distributed as evenly as possible across the disks to ensure fault tolerance. For example, parity block Bp in stripe B is located on disk 3, parity block Cp in stripe C is located on disk 2, and parity block Dp in stripe D is located on disk 1.

[0073] (5) Parity check

[0074] Parity checking is a simple error detection and correction mechanism. It can detect errors in a single bit and correct them under certain configurations. For example, in RAID 5, each bit in the parity data stored in the parity block is used to indicate whether the number of 1s in the same bit across the corresponding data blocks is odd or even.

[0075] Currently, open storage protocols such as ZNS and Open-Channel offer solutions to some inherent problems in traditional block storage technologies. Open storage protocols employ sequential writes and regional management, offloading data placement strategies and garbage collection to the host software. By reducing the complexity of garbage collection, they improve the performance and durability of storage devices. However, storage devices supporting open storage protocols differ from traditional block devices in terms of storage architecture and management. Therefore, adapting to multiple aspects is often necessary when using storage devices that support open storage protocols. In particular, the design and optimization of storage system technologies (such as operating systems and file systems) must be adapted to the sequential writes and regional management of open storage protocols.

[0076] However, current open storage protocols are typically designed for managing individual storage devices, making them difficult to adapt to storage arrays composed of multiple storage devices, resulting in low storage performance of existing storage arrays.

[0077] Specifically, taking RAID as an example, RAID uses striping technology to segment data and store it across different storage devices, while storing corresponding parity data on one or more other storage devices. Since data belonging to a single stripe may be written to the storage devices multiple times, the parity data corresponding to that stripe may frequently change, requiring multiple updates. However, under open storage protocols, data is written to persistent storage media with append-only semantics. This means that data in persistent storage media no longer supports in-place updates. Therefore, a conflict arises where persistent storage media does not support in-place data updates, while parity data in the storage array needs to be updated in-place. This prevents the storage array from properly deploying open storage protocols, impacting its storage performance.

[0078] Based on this, this application provides a data storage method based on a storage array. In the target storage device storing verification data within the storage array, a write buffer supporting in-place data updates and a persistent storage medium supporting append-only writes are configured. When data needs to be written to other storage devices in the storage array, and verification data needs to be written to the target storage device, the verification data is first written to the write buffer of the target storage device, and can be updated within the write buffer. Furthermore, when all data on a stripe has been written to other storage devices, rendering the verification data no longer updatable, the write buffer then writes the no-longer-updated verification data to the persistent storage medium. This resolves the conflict between the persistent storage medium's lack of in-place data updates and the storage array's requirement for in-place updates of verification data, enabling the storage array to deploy open storage protocols and improve storage performance.

[0079] Please see Figure 2 , Figure 2 This is a schematic diagram of a system architecture provided for this application. (For example...) Figure 2 As shown, in this system architecture, the execution device 20 can be implemented by a single physical host (personal computer, computer workstation, smartphone, tablet, laptop, or server) or multiple physical hosts (a cluster of computing devices). The execution device is used to execute the storage array-based data storage method provided in this application to achieve data storage. Optionally, the execution device 10 can cooperate with other computing devices, such as data storage devices, load balancers, etc.; the execution device 10 can be deployed on a single physical site or distributed across multiple physical sites.

[0080] In addition, the system architecture also includes a storage array 21, which is used to store data. The storage array 21 may include multiple storage devices, such as multiple SSDs or multiple HDDs.

[0081] An application program may run in execution device 20, which requests data to be stored in storage array 21. At this time, execution device 20 executes the data storage process based on the storage array-based data storage method provided in this application to store the data in storage array 21.

[0082] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a data storage method based on a storage array, as provided in this application. Figure 3 As shown, the execution flow of this data storage method based on a storage array includes the following steps 301-303.

[0083] Step 301: Obtain a first data write request. The first data write request is used to request that first data be written to the storage array, which includes multiple storage devices.

[0084] In this application, the data storage method based on a storage array can be applied to an execution device connected to the storage array. For example, the data storage method based on a storage array can be executed by storage management software on the execution device. When an application running on the execution device or an application running on another device outside the execution device needs to store data in the storage array, the execution device can obtain a first data write request, which comes from the application that needs to store first data in the storage array.

[0085] Optionally, the storage array is managed using an open storage protocol based on sequential writes (such as ZNS or Open-Channel) to ensure that valid data written to the storage array (i.e., data other than checksum data) is written using append-write semantics.

[0086] Step 302: Based on the first data write request, generate a second data write request and a verification data write request. The second data write request is used to instruct the first data to be written in segments to multiple data blocks included in the first stripe. The verification data write request is used to instruct the first verification data to be written to the verification block included in the first stripe. The first verification data is calculated based on the data to be written from multiple data blocks. The multiple data blocks and verification blocks included in the first stripe are evenly distributed on multiple storage devices.

[0087] Specifically, the physical partitions of multiple storage devices on a storage array are uniformly mapped to execution devices, thus forming a logical space. This logical space can consist of data blocks included on multiple stripes; that is, the logical space is actually composed of contiguous data blocks. For example, the logical space can consist of multiple data blocks arranged sequentially on stripe 1, multiple data blocks arranged sequentially on stripe 2, and so on, up to multiple data blocks arranged sequentially on stripe n.

[0088] After receiving the first data write request, the execution device needs to determine how to write the first data to the storage array based on the data write status of the stripes in the logical space. Specifically, if the length of the first data is greater than the length of a data block in the stripe, the execution device will segment the first data to facilitate writing the first data segments into multiple data blocks included in the first stripe. For example, assuming the first stripe includes data blocks 1-5, and the length of the first data is equal to the combined length of the three data blocks, the execution device can divide the first data into data segments 1-3 and generate a second data write request to request that data segments 1-3 be written to data blocks 1-3 respectively. Of course, if the length of the first data is less than or equal to the length of a data block in the stripe, the execution device may not actually need to segment the first data again, but can directly store the first data into one of the data blocks in the stripe. However, if the length of the first data is less than or equal to the length of a data block in the strip, and if a data block in the strip has already stored some data, making it impossible to use a single data block to store the first data, the execution device will also segment the first data so that the first data can be stored in different data blocks.

[0089] In summary, the execution device can generate a second data write request based on the data storage status of the stripes in the logical space corresponding to the storage array and the length of the first data, instructing the first data to be segmented and written to multiple data blocks included in the first stripe. The first stripe is a specific stripe in the logical space corresponding to the storage array.

[0090] In addition, the first stripe also includes one or more check blocks, which are used to store check data to facilitate the subsequent verification or recovery of data stored in multiple data blocks of the first stripe. For ease of description, the following will use the example of the first stripe including one check block. Generally, the check data stored on the check block included in the first stripe needs to be calculated based on the data stored in multiple data blocks. Therefore, when it is determined that the first data will be written to multiple data blocks of the first stripe, the execution device can calculate the first check data to be stored on the check block based on the data to be written to multiple data blocks (i.e., the first data), and then generate a check data write request to instruct the first check data to be written to the check block included in the first stripe.

[0091] Step 303: Send a second data write request and a verification data write request to the storage array. The target storage device, which includes a verification block, includes a write buffer that supports in-situ data updates and a persistent storage medium that supports data appending. The verification block is distributed on the persistent storage medium, and the length of the verification block is less than the length of the write buffer. The write buffer is used to continuously cache and update the verification data to be written to the verification block, and to store the verification data that will no longer be updated in the persistent storage medium.

[0092] Specifically, since the second data write request is used to instruct the first data to be segmented and written into multiple data blocks of the first stripe, and these multiple data blocks are distributed across different storage devices in the storage array, the execution device can be a centralized controller of the storage array that sends the second data write request to facilitate the segmented writing of the first data. Alternatively, the execution device can split the second data write request into multiple sub-requests and distribute them to different storage devices in the storage array to achieve the segmented writing of the first data.

[0093] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a target storage device provided in this application. Figure 4 As shown, for a target storage device with distributed parity blocks, the target storage device includes a write buffer and a persistent storage medium. The write buffer uses a different storage medium than the persistent storage medium. Therefore, the write buffer supports in-place data updates (i.e., new data can overwrite existing data), while the persistent storage medium only supports append-only writes (i.e., new data can only be written after old data, not overwrite old data). For example, the write buffer may use Random Access Memory (RAM), while the persistent storage medium may be flash memory.

[0094] After the target storage device in the storage array receives the verification data write request sent by the execution device, it first writes the first verification data to the write buffer instead of directly writing it to the verification block in the persistent storage medium. Furthermore, during subsequent data storage, the target storage device can update the verification data in the write buffer at any time, and only writes the no-longer-updated verification data to the persistent storage medium when the verification data is no longer being updated (i.e., all data blocks included in the first stripe are full). This enables in-situ updates of the verification data.

[0095] Specifically, when an execution device writes data to a data block in a stripe, it may not completely fill that stripe. Therefore, when the execution device writes data to the same stripe again, the parity data corresponding to that stripe will also change due to changes in the data on multiple data blocks. In this case, the parity data already written to the storage device needs to be updated. Since the parity block resides on persistent storage media that does not support in-situ updates, the target storage device temporarily stores the parity data in a write buffer. This allows for updates to the parity data in the write buffer at any time, and the parity data is only stored in persistent storage media when no further updates are needed.

[0096] It should be noted that the above description describes the target storage devices in the storage array including write buffers that support in-place data updates and persistent storage media that support append-only writes. However, in some storage arrays, such as RAID 5, each storage device has a parity block distributed across it. That is, each storage device may store parity data corresponding to a certain stripe. Based on this, in this application, each storage device in the storage array can include a write buffer that supports in-place data updates and persistent storage media that supports append-only writes, so that any storage device can implement the updating and writing of parity data.

[0097] To facilitate understanding, the following will describe the process of updating verification data based on a write buffer.

[0098] For example, in step 303 above, the second data write request is used to instruct the first data to be segmented and written into a first portion of a plurality of data blocks. Here, the first data block refers to a portion of the plurality of data blocks. For instance, the plurality of data blocks include data blocks 1-5, and the first portion of the data blocks consists of data blocks 1-3.

[0099] After sending the second data write request and the verification data write request to the storage array, the execution device can also obtain a third data write request, which is used to request that the second data be written to the storage array. This third data write request can also originate from an application that needs to write data to the storage array.

[0100] Based on the third data write request, the execution device determines the data block that needs to be written with the second data, and then generates a fourth data write request and a verification data update request, and sends the fourth data write request and the verification data update request to the storage array.

[0101] The fourth data write request instructs the second data to be written in segments to the second portion of multiple data blocks. That is, since the first data cannot completely occupy multiple data blocks in the first stripe, the execution device requests that the second data continue to be written to the multiple data blocks in the first stripe. For example, the multiple data blocks in the first stripe include data blocks 1-5, the first portion of data blocks for writing the first data consists of data blocks 1-3, and the second portion of data blocks for writing the second data consists of data blocks 4-5. The verification data update request instructs the data in the verification block to be updated with second verification data, which is calculated based on the existing data (i.e., the first data) and the data to be written (i.e., the second data) in the multiple data blocks.

[0102] In other words, for the second check data that needs to be written to the same check block, the execution device can know that the second check data does not need to follow the append write rule, so it will send a check data update request to the storage array to instruct the storage array to perform an in-place update of the check data.

[0103] Furthermore, to achieve in-place updates of the verification data, the write buffer in the target storage device is used to update the cached first verification data with the second verification data after receiving a verification data update request. That is, the first verification data is actually still cached in the write buffer and is not written to the persistent storage medium, so the write buffer can achieve in-place updates of the first verification data.

[0104] In order to enable the execution device to be aware of the write buffer on the target storage device, so as to ensure that the execution device can instruct the target storage device to perform the update of the verification data, this application proposes that the execution device maintain a Zone Random Write Area (ZRWA) and determine the data that can be updated in place based on the ZRWA.

[0105] For example, after receiving a third data write request, the execution device can acquire the ZRWA maintained for the target storage device, where the length of the ZRWA is equal to the length of the write buffer, and the ZRWA slides across the address space corresponding to the target storage device (i.e., the address space of the persistent storage medium) as the data write progresses. That is, the data corresponding to the addresses covered by the ZRWA is actually data cached in the write buffer and not yet stored in the persistent storage medium. At this time, in the address space corresponding to the target storage device, data located in the address space before the ZRWA can be considered as data already stored in the persistent storage medium (i.e., data that cannot be updated); data located in the addresses covered by the ZRWA can be considered as data temporarily stored in the write buffer (i.e., data that can be updated).

[0106] Since the address corresponding to the check block is located on ZRWA, the execution device can assume that the first check data that has been requested to be written by the storage array can still be updated, and therefore generates a check data update request to request that the data of the check block be updated to the second check data.

[0107] Optionally, in this application, in order to enable the write buffer to automatically write the no-longer-updated verification data to the persistent storage medium, the write buffer can be used to store the second verification data to the persistent storage medium when the cached data overflows.

[0108] In other words, when the write buffer receives a new data write request, causing the data cached in the write buffer to overflow (i.e., it can no longer cache existing data and new data simultaneously), the write buffer will trigger the transfer of the previously cached second parity data to the persistent storage medium. It can be understood that when the write buffer receives a new data write request, it often means that the execution device has completed writing the data on the first stripe. Therefore, the data on the parity block on the target storage device no longer needs to be updated; instead, the updated data needs to be stored on the target storage device. At this time, the write buffer can transfer the second parity data, which no longer needs updating, to the persistent storage medium, thereby achieving persistent storage of the second parity data.

[0109] For example, suppose that the logical space corresponding to multiple storage devices includes a first stripe and a second stripe.

[0110] After completing the writing of the second verification data, the execution device obtains the fifth data write request, which is used to request the third data to be written to the storage array.

[0111] Based on the fifth data write request, the execution device sends a sixth data write request to the storage array. The sixth data write request instructs the third data to be segmented and written to multiple data blocks included in the second stripe. The target storage device contains data blocks from the second stripe. That is, the multiple data blocks on the first stripe are already completely filled with data, so the execution device requests to continue writing data to the data blocks in the second stripe following the first stripe.

[0112] In this scenario, the target storage device contains a data block included in the second stripe. Therefore, the target storage device will acquire the data (i.e., the segmented data of the third data) that needs to be stored in the data block of the second stripe. At this time, the write buffer is used to store the second check data into the persistent storage medium when the segmented data belonging to the third data is acquired and the remaining space cannot accommodate the segmented data.

[0113] In other words, when new data is acquired in the write buffer, and the write buffer cannot simultaneously hold both the old and new data, the old data is written to the persistent storage medium, thus freeing up space to cache the new data. In this way, whenever the execution device writes data to a new address in the target storage device, it can trigger a write buffer overflow, thereby storing the no-longer-updated checksum data to the persistent storage medium.

[0114] Generally, the execution device records the last address of data written to the target storage device in real time, so as to continuously perform append writes on the target storage device and thus achieve sequential data writing. However, if the execution device fails and loses power, it will lose the recorded last address of data written to the target storage device.

[0115] Optionally, in order to facilitate data recovery after the device is powered off, the target storage device records the last write address of the current data in the target storage device and the length of the valid data cached in the write buffer, in order to facilitate the recovery of the data writing process.

[0116] Generally, storage devices record the address where data was last written to persistent storage. However, when a storage device uses a write buffer to cache data, the write buffer also caches some data, and it may not be completely full. In this case, by simultaneously recording the last write address of the persistent storage medium and the length of the valid data cached in the write buffer on the target storage device, it is possible for the execution device to determine the actual data write status based on the last write address of the persistent storage medium and the length of the valid data cached in the write buffer when power is lost, thus ensuring the recovery of the data write process.

[0117] For example, after the execution device recovers from a power outage, it sends a query request to the target storage device to request information on data write operations. Upon receiving the query request, the target storage device then reports its recorded data write status back to the execution device.

[0118] In this way, the execution device can receive the last write address of the persistent storage medium and the length of the valid data cached in the write buffer from the target storage device. Based on the last write address and the length of the valid data, the execution device determines the write pointer corresponding to the target storage device, which is used to indicate the end address of the data written to the target storage device.

[0119] For example, suppose that before the power outage, the executing device successfully sent a data write request to the target storage device to write data to addresses 0-2. Furthermore, the target storage device has already stored some data in the persistent storage medium at addresses 0-1, while caching the remaining data in the write buffer. Thus, based on the last write address (address 1) of the persistent storage medium returned by the target storage device, and the length of the valid data cached in the write buffer (i.e., the length between address 2 and address 1), the executing device can determine that the end address of the data written to the target storage device is address 2.

[0120] Optionally, the target storage device records the length of valid data using an out-of-band area (OOB). The OOB provides a free area for each flash memory page to store the page's metadata. In this application, the OOB can be used to record the length of valid data written to the write buffer.

[0121] For example, OOB can represent the length of valid data cached in the write buffer using multiple bits, with one bit used to indicate whether a logical block in the write buffer contains valid data. Here, a logical block is the smallest unit of writing in the write buffer, and its size can be 4 kilobytes (KB). Therefore, with a write buffer size of 64KB, only 16 bits are needed in OOB to record whether 16 logical blocks in the write buffer contain valid data.

[0122] To facilitate understanding, the data storage method based on storage arrays provided in this application will be described in detail below with specific examples.

[0123] Please see Figure 5 , Figure 5 This is a schematic diagram of a storage array service provided in this application. Figure 5As shown, the data storage method based on a storage array provided in this application can be specifically applied to a storage array server. The storage array server includes a RAID controller and a storage array composed of multiple storage devices. The storage array server can provide reliable storage and efficient read / write operations via RAID mechanisms based on the storage devices. Taking ZNS-compatible storage devices in the storage array as an example, the storage array server can provide more efficient and reliable storage services to ZNS-adapted file systems or key-value storage systems (such as RocksDB based on Zenfs). The storage array server provides logical space for storage to the upper-layer system or application, and each logical space consists of several physical regions. When a write request for a logical space is received, it writes to the physical regions uniformly at a certain granularity, and simultaneously writes the corresponding parity check code (i.e., check data).

[0124] exist Figure 5 The RAID controller also includes a Zebra control module, which is used to execute the data storage method based on the storage array provided in this application. This Zebra control module can be implemented in software or integrated into the RAID controller card to provide functionality in a low-latency hardware processing manner.

[0125] Furthermore, in the storage array, storage devices can utilize ZNS-based SSDs, which offer faster access speeds, higher reliability, and lower power consumption, making them suitable for applications requiring high-speed read / write, high reliability, and low power consumption. Meanwhile, for scenarios where data access speed requirements are not high, ZNS-based HDDs can be used, offering lower costs, larger capacity, and longer storage lifespan.

[0126] Please see Figure 6 , Figure 6 This is a schematic diagram of a system architecture provided for this application. (For example...) Figure 6 As shown, the system architecture can specifically be the architecture of Zebra, a RAID array system based on ZNS.

[0127] Zebra is a general-purpose RAID framework that exposes a region interface to applications. Zebra merges physical partitions from different storage devices into a single logical space, with data blocks evenly distributed across the storage devices within each logical space. Furthermore, a logical space can be divided into multiple contiguous stripes, each containing multiple data blocks. For example, in... Figure 6In this structure, the logical space is divided into stripe 0 and stripe 1. Stripe 0 includes data blocks D0-D2, and stripe 1 includes data blocks D3-D5. Furthermore, each stripe also corresponds to a parity block, which stores the parity data corresponding to the data blocks within the stripe. For example, stripe 0 corresponds to parity block P0, and stripe 1 corresponds to parity block P1.

[0128] Furthermore, data blocks and parity blocks on the same stripe are evenly distributed across each storage device in the storage array. For example, for stripe 0, parity block P0 is distributed on storage device 0, data block D0 is distributed on storage device 1, data block D1 is distributed on storage device 2, and data block D2 is distributed on storage device 3. For stripe 1, data block D3 is distributed on storage device 0, data block D4 is distributed on storage device 1, data block D5 is distributed on storage device 2, and parity block P1 is distributed on storage device 3.

[0129] For storage devices, each physical partition on a storage device is used to continuously distribute data blocks or parity blocks for each stripe. For example, on the physical partition of storage device 0, there are parity block P0 and data block D3.

[0130] In this application, each storage device may include a write buffer and a persistent storage medium. Data written to the write buffer can be updated in place at any time, while data written to the persistent storage medium can only be appended and cannot be updated in place.

[0131] Based on this, on the host software side, each physical partition on the storage device can be considered to consist of an immutable region preceding it and a Zone Random Written Area (ZRWA) following it. The immutable region can be considered as the area where data has already been written to persistent storage, therefore data in the immutable region cannot be updated. The ZRWA can be considered as the area where data has been written to the write buffer, therefore data in the ZRWA can be updated. The ZRWA can be viewed as a sliding window that slides across the corresponding address space of the storage device as the data storage progresses. As the ZRWA slides, the addresses preceding it become immutable regions. For each physical partition, a write pointer (WP) at the end of the immutable region data and a real write pointer at the end of the ZRWA data need to be maintained. The write pointer is automatically maintained by the storage device hardware, while the real write pointer is maintained by the software layer and recovered from the out-of-bounds (OOB) data in the event of a system crash.

[0132] Specifically, ZRWA is a fixed-size contiguous logical block address (LBA) that can accept random concurrent overwrite requests. In ZRWA mode, the region write pointer points to the beginning of the ZRWA (i.e., the end of the immutable region). If the storage device supports power loss protection, the data on the ZRWA (i.e., the data in the write buffer) can be persistently preserved in the event of a machine failure. The random write characteristics of ZRWA within a sliding window are well-suited for partial parity updates (PPUs) in RAID. PPUs have the characteristic of repeatedly overwriting within a block and are persistent even after power loss.

[0133] ZRWA is an instance of a write buffer within a storage device, used to aggregate I / O requests destined for the device and distribute them in batches to the specific media (such as flash memory or hard disks). Currently, the front write buffer size of most flash memory devices is generally no less than 64KB. Since the parity data in the parity block needs to be updated in-place, this solution requires that the write buffer of the storage device be larger than the parity block size of the stripes in the storage array. Currently, the performance of the storage array is optimal when the parity block size is 16KB, thus satisfying this condition.

[0134] The Out-of-Band (OOB) provides a free area for each flash page to store page metadata. In this application, the OOB can be used to record the length of valid data written to the write buffer. For example, when a user writes data to a flash page, metadata can be appended to the OOB area, and the metadata written to the OOB area is consistent with the data written to the corresponding flash page. At the software level, Zebra maintains the actual write pointer and state of ZRWA in host memory.

[0135] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating a data writing process based on ZRWA, as provided in this application. Figure 7 As shown, assuming each storage device has a write buffer and persistent storage medium, a ZRWA can be maintained for the address space corresponding to each storage device, and the area before the ZRWA is considered an immutable area. The striped blocks are evenly distributed across all storage devices, indicating that each storage device contains both data blocks and parity blocks. Zebra supports different RAID configurations (e.g., RAID 5 or RAID 6). Figure 7 This section will specifically use RAID 5 as an example. RAID 5 sets up multiple data blocks and one parity block on a single stripe, while RAID 6 sets up multiple data blocks and multiple parity blocks on a single stripe.

[0136] Figure 7 The image shows the execution process of five write requests in Zebra and the corresponding ZRWA movement. The five write requests are named W0, W1, W2, W3, and W4. Assuming a data block / parity block size is 16 LBAs and an LBA is 4 KB, then write request W2 is equal to the block size (16 LBAs), and the other four write requests are equal to half the block size.

[0137] In the initial state, data is written to parity block P0 and data blocks D0-D2. At this time, the real write pointer on storage device 0 points to the beginning position of data block D3 (i.e. the beginning position of the second stripe).

[0138] When processing the W0 request, Zebra appends data to data block D3 and adds partial parity data to the first half of parity block P1. Simultaneously, it moves the tail of the ZRWA in storage devices 0 and 3 to the midpoint of data block D3 and parity block P1, respectively. Because of this ZRWA movement, in storage device 0, the area preceding the ZRWA (i.e., the first half of parity block P0) becomes immutable; in storage device 3, the area preceding the ZRWA (i.e., the first half of data block D2) becomes immutable.

[0139] Then, after write request W1 arrives, Zebra continues to append the data carried by write request W1 to data block D3, and appends part of the parity data to parity block P1. At the same time, it moves the tail of ZRWA in storage device 0 to the end of data block D3, and moves the tail of ZRWA in storage device 3 to the end of parity block P1.

[0140] When write request W2 arrives, Zebra appends the data carried by write request W2 to data block D4, moving the tail of ZRWA in storage device 1 to the end of data block D4. At this time, because the data in data block D4 has been updated, a parity data update is triggered. Since parity block P1 is already in ZRWA, Zebra can trigger an in-place update of the parity data of parity block P1 in storage device 3 without moving the ZRWA in storage device 3.

[0141] Finally, when write requests W3 and W4 arrive, Zebra writes the data carried by write requests W3 and W4 into data block D5, and sequentially performs two check data updates on check block P1, ultimately achieving the state where the tail of ZRWA on all storage devices moves to the end of the block in the current stripe.

[0142] The data writing process described above utilizes ZRWA's overwrite capability, allowing the parity block to be modified as new data is written to the stripe without frequent append writes, thus avoiding the complex garbage collection and location issues caused by append writes.

[0143] However, the mismatch between ZRWA's movement granularity and ZRWA's minimum single write granularity also introduces the problem of locating the true write window.

[0144] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the location of a real WP device, as provided in this application. Figure 8 As shown, in the non-ZRWA area, newly written data is moved to the WP (Write Buffer) in a minimum unit of 1 LBA (4KB), equivalent to the write granularity of an SSD. That is, the write granularity of the write buffer corresponding to ZRWA is 4KB. Unlike the non-ZRWA area, ZRWA typically moves data in units of flash memory pages within an SSD. In other words, ZRWA movement means that the original ZRWA data needs to be written to persistent storage. In this case, the minimum move unit of ZRWA is often the minimum write granularity of the persistent storage medium, such as 4 LBA (16KB).

[0145] like Figure 8 As shown in (a), before a new write request arrives, ZRWA is the region between LBA0 and LBA16. If the write request requests to write data of size 2 LBA, then ZRWA will shift to the right by the smallest unit (4 LBA), thus changing ZRWA to the region between LBA4 and LBA20. Before power loss, the host software maintains the Real Write Pointer (RealWP) at position LBA18.

[0146] like Figure 8 As shown in (b), if the host experiences a power outage, the Real WP maintained by the host software will be lost. In this case, if the default position of the end of ZRWA (LBA20) is the Real WP, an incorrect Real WP will be located. Therefore, ZRWA is not completely filled with data at this time; the Real WP is actually located at LBA18, not LBA20.

[0147] like Figure 8As shown in (c), Zebra utilizes the Out-of-Band (OOB) to address this process in order to locate the Real WP. Specifically, Zebra uses the metadata on the OOB to determine whether data has been written. The initial state of the OOB is filled with 0s. When a write request of size 2LBA is sent to LBA16, the corresponding OOB is written to 1, indicating that a write request has occurred on the flash page. If the host fails at this time and then the machine is powered on again, Zebra needs to query the WP from the storage device. At the same time, using the OOB, Zebra traverses the OOB metadata from the flash pages on ZRWA and places the real WP at the end of the consecutive pages with OOB area 1, i.e., LBA18. Therefore, Zebra successfully recovers the WP in this area after a power outage.

[0148] Specifically, the Zebra data recovery process is described below.

[0149] Step 1: Query the region status. Zebra sends a command to the storage device to retrieve region management information and receives the status and working title (WP) of the physical region.

[0150] Step 2: Calculate the Real WP. Since data blocks belonging to the same stripe on multiple storage devices are written simultaneously, Zebra's core task is to find the last valid Real WP for the same stripe across multiple storage devices. Starting with the WP of the physical region found in Step 1 for each storage device, Zebra scans the Out-of-Band (OOB) of each storage device to locate the Real WP on each device. Because full-strip append writes can be split and distributed in parallel across multiple storage devices, Zebra needs to synchronize the Real WPs on these storage devices to determine the Real WP for the logical space. The synchronization process is straightforward: starting with the first data block on the stripe, Zebra scans the Real WP of the last data block on the same stripe. Zebra retrieves the first Real WP that has not yet reached the end of its data block (i.e., the data block is not full) and uses that Real WP as the Real WP for the logical space.

[0151] Step 3: Data synchronization and parity checking. Zebra discards invalid data by overwriting the LBA outside the Real WP in the logical space with zeros, as this data represents incomplete requests that violate the semantics of append-only writes. Finally, Zebra recalculates the parity block based on the recovered data blocks.

[0152] The only exception is when storage device failure and host failure occur simultaneously. Storage device failure prevents Zebra from recovering the Real WP of the physical partition in the failed storage device by scanning the OOB. Therefore, in step 2, Zebra can use the OOB of the parity block in the unfailed storage device to recover the Real WP in the failed storage device. Specifically, the OOB records the parity block corresponding to the update in-situ overwritten on the ZRWA, and Zebra recovers the Real WP by tracking the data block updated by the write request. For this purpose, Zebra expands the OOB area used for the parity block from 1 bit to 1 byte, which is still a modest space consumption.

[0153] Please see Figure 9 , Figure 9 This application provides a schematic diagram of an application architecture based on a key-value storage database. Taking ZNS as an example of a storage device, this solution mainly provides storage services for ZNS-based data storage components, supporting file systems or key-value storage systems. Therefore, in Figure 9 In this solution, the open-source key-value storage database RocksDB has been adapted, and the data storage part of RocksDB has been added to the RAID storage array based on ZNS devices in this solution.

[0154] ZenFS is a file system designed for ZNS devices that provides optimized support for log-structured merge tree storage engines such as RocksDB. It optimizes RocksDB's read / write performance by managing the sequential write characteristics of ZNS devices, reducing SSD write amplification and improving data storage and access efficiency.

[0155] Zebra is a key component of this solution, providing logical spaces to the upper layers. ZenFS writes the data to these logical spaces. Zebra then uses the method provided in this application to write the data from the logical spaces to the physical areas of each ZNS device. During the data writing process, Zebra calculates parity blocks and, leveraging the characteristics of ZRWA, simultaneously overwrites a specific location in the physical area of ​​the corresponding ZNS device, ensuring reliable data storage and efficient read / write operations.

[0156] Please see Figure 10 , Figure 10 This is a schematic diagram illustrating a data writing process provided in this application. Figure 10 As shown, in the RocksDB scenario, the data writing process includes the following steps 1001-1005.

[0157] Step 1001: RocksDB issues a write request.

[0158] First, to ensure no data is lost in the in-memory MemTable, RocksDB stores the Write-Ahead Log (WAL) file on disk. Second, once the MemTable reaches a certain capacity, data is transferred from the MemTable to SST files, which are also eventually stored on disk. Finally, RocksDB periodically merges the stored SST files; the merged data is written to a new SST file. Identical key-value pairs are merged from multiple SST files into a single file, which is then written to disk. In summary, all of these operations trigger write requests to ZenFS for processing.

[0159] Step 1002: ZenFS organizes the write order, writing the data into the logical space in an orderly manner.

[0160] After receiving a write request from RocksDB, ZenFS allocates the data to be written according to the logical space and transforms the application's random write requests into sequential writes. ZenFS aggregates the RocksDB write requests into memory and then sequentially writes the data to the corresponding segments of the SSD. Therefore, the SST files and WAL data in RocksDB are written sequentially by ZenFS into the logical space provided by Zebra.

[0161] Step 1003: Zebra performs striping and writes the data into data blocks.

[0162] For ZenFS requests to write data in an ordered manner into the logical space, Zebra can perform striping and group the data by stripe to write the data in an ordered manner into the data blocks of each stripe.

[0163] Step 1004: Zebra calculates the parity data and writes the parity data into the parity block.

[0164] In addition, during the process of writing data to the data blocks of the stripe, Zebra calculates parity data by combining the data already stored in each data block on the stripe with the data to be written, and writes the parity data into the parity block.

[0165] Step 1005: Determine if the data has been written.

[0166] If the data has been completely written, Zebra ends the data processing; if the data has not been completely written, Zebra continues to execute step 1004.

[0167] Please see Figure 11 , Figure 11 This is a schematic diagram illustrating a Zebra data writing process provided in this application. Figure 11As shown, the Zebra data writing process includes the following steps 1101-1107.

[0168] Step 1101: Determine whether the stripe containing the data to be written is full.

[0169] For data that needs to be written to data blocks, Zebra groups it by stripe, and each stripe is divided into blocks according to the RAID configuration. First, Zebra checks whether all data blocks in the previous stripe have been filled with data.

[0170] Step 1102: Obtain the disk location of the data block and check block in the stripe.

[0171] If a stripe is not full, Zebra obtains the disk location of each data block and parity block in the stripe, that is, which storage device each data block and parity block is located on.

[0172] Step 1103: Read the verification data from the verification block.

[0173] In addition, Zebra reads the parity data already stored in the parity block of the current stripe to facilitate the subsequent calculation of new parity data.

[0174] Step 1104: Calculate new check data based on the new data block.

[0175] Specifically, Zebra calculates the new checksum based on the data to be written to the new data block and the old checksum data.

[0176] Step 1105: Based on the disk location, perform append writing on the data block and overwrite writing on the check block.

[0177] That is, data to be written to the data block is written using append-only semantics; new parity data is directly overwritten onto the parity block. At this time, the old parity data is invalid and will be overwritten by the new parity data in ZRWA, thus resolving the conflict between ZNS sequential append-only writing and the need for in-place updating of parity data.

[0178] Step 1106: After writing is complete, mark it in OOB.

[0179] After the data is written, Zebra marks the write position (OOB) of the corresponding flash page so that it can locate the actual write pointer when performing data recovery.

[0180] Step 1107: Write the new stripe and determine the disk location for each data block and check block.

[0181] When data needs to be written, if the stripe is full, Zebra can allocate the data to be written to a new stripe, determine the disk location of each data block and parity block, and then proceed to step 1104 above.

[0182] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0183] Please see Figure 12 , Figure 12 This is a schematic diagram of a data storage device based on a storage array, as provided in this application. Figure 12 As shown, the data storage device based on the storage array includes: an acquisition module 1201, used to acquire a first data write request, the first data write request being used to request that first data be written to the storage array, the storage array including multiple storage devices; and a sending module 1202, used to send a second data write request and a verification data write request to the storage array based on the first data write request, the second data write request being used to instruct the first data to be written in segments to multiple data blocks included in the first stripe, and the verification data write request being used to instruct the first verification data to be written to the verification block included in the first stripe, the first verification data being calculated based on the data to be written to multiple data blocks, the multiple data blocks and verification blocks included in the first stripe being evenly distributed on multiple storage devices; wherein, the target storage device including the verification block among the multiple storage devices includes a write buffer that supports in-situ data updates and a persistent storage medium that supports data appending, the verification block being distributed on the persistent storage medium, and the length of the verification block being less than the length of the write buffer, the write buffer being used to continuously cache and update the verification data to be written to the verification block, and to store the verification data that will no longer be updated into the persistent storage medium.

[0184] In one possible implementation, the second data write request is used to instruct the first data to be segmented and written into a first part of a plurality of data blocks; the acquisition module 1201 is further used to acquire a third data write request, which is used to request the second data to be written into the storage array; the sending module 1202 is further used to send a fourth data write request and a verification data update request to the storage array based on the third data write request, the fourth data write request being used to instruct the second data to be segmented and written into a second part of a plurality of data blocks, and the verification data update request being used to instruct the data in the verification block to be updated to second verification data, which is calculated based on the existing data and the data to be written in the plurality of data blocks; wherein, the write buffer is used to update the cached first verification data to the second verification data after the verification data update request is acquired.

[0185] In one possible implementation, the apparatus further includes: a processing module 1203; an acquisition module 1201, further configured to acquire the Region Random Write Area (ZRWA) maintained by the target storage device, wherein the length of the ZRWA is equal to the length of the write buffer, and the ZRWA slides on the address space corresponding to the target storage device as the data writing progress of the target storage device progresses; and the processing module 1203, configured to generate a verification data update request in response to the address corresponding to the verification block being located on the ZRWA.

[0186] In one possible implementation, the write buffer is used to store the second checksum data to persistent storage when the cached data overflows.

[0187] In one possible implementation, the logical space corresponding to multiple storage devices includes a first stripe and a second stripe; the acquisition module 1201 is further configured to acquire a fifth data write request, which requests that third data be written to the storage array; the sending module 1202 is further configured to send a sixth data write request to the storage array based on the fifth data write request, which instructs that the third data be written in segments to multiple data blocks included in the second stripe, and the target storage device has data blocks distributed in the second stripe; wherein, the write buffer is configured to store the second verification data in the persistent storage medium when the segmented data belonging to the third data is acquired and the remaining space cannot accommodate the segmented data.

[0188] In one possible implementation, the target storage device records the last write address of the persistent storage medium and the length of valid data cached in the write buffer, wherein the last write address and the length of valid data are used to enable recovery of the data write process.

[0189] In one possible implementation, the apparatus is applied to an execution device connected to a storage array, and the apparatus further includes: a processing module 1203; after the execution device recovers from a power outage, a sending module 1202 is further configured to send a query request to the target storage device, the query request being used to request the acquisition of data write status; an acquisition module 1201 is further configured to receive the last write address and the length of valid data sent by the target storage device; based on the last write address and the length of valid data, the processing module 1203 is configured to determine the write pointer corresponding to the target storage device, the write pointer being used to indicate the end address of the data previously written to the target storage device.

[0190] In one possible implementation, the target storage device records the length of valid data via OOB.

[0191] In one possible implementation, OOB uses multiple bits to represent the length of valid data cached in the write buffer, and one of the multiple bits is used to indicate whether a logical block in the write buffer has valid data cached.

[0192] In one possible implementation, the storage array is managed via an open storage protocol based on sequential writes.

[0193] The acquisition module 1201, processing module 1203, and sending module 1202 can all be implemented in software or in hardware. For example, the implementation of processing module 1203 will be described below. Similarly, the implementation of acquisition module 901 can be referenced to that of acquisition module 1201.

[0194] As an example of a software functional unit, processing module 1203 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1203 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0195] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0196] As an example of a hardware functional unit, the processing module 1203 may include at least one computing device, such as a server. Alternatively, the processing module 1203 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0197] The processing module 1203 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1203 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1203 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0198] Please see Figure 13 , Figure 13 A schematic diagram of the structure of a computing device provided in this application. Figure 13 The computing device 1300 shown can be used to execute the data storage method based on a storage array provided in this embodiment. For example... Figure 13 As shown, the computing device 1300 includes a bus 1302, a processor 1304, a memory 1306, and a communication interface 1308. The processor 1304, the memory 1306, and the communication interface 1308 communicate with each other via the bus 1302. The computing device 1300 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0199] Bus 1302 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 13 The bus 1302 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1302 may include a path for transmitting information between various components of the computing device 1300 (e.g., memory 1306, processor 1304, communication interface 1308).

[0200] The processor 1304 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0201] The memory 1306 may include volatile memory, such as random access memory (RAM). The processor 1304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0202] The memory 1306 stores executable program code, and the processor 1304 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the aforementioned data storage method based on the memory array. That is, the memory 1306 stores instructions for executing the data storage method based on the memory array.

[0203] The communication interface 1308 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1300 and other devices or communication networks.

[0204] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0205] Please see Figure 14 , Figure 14 This is a schematic diagram of a computing device cluster provided in this application. Figure 14 As shown, the computing device cluster includes at least one computing device 1300. The memory 1306 of one or more computing devices 1300 in the computing device cluster may store the same instructions for executing a memory array-based data storage method.

[0206] In some possible implementations, the memory 1306 of one or more computing devices 1300 in the computing device cluster may also store partial instructions for executing the memory array-based data storage method. In other words, a combination of one or more computing devices 1300 can jointly execute instructions for executing the memory array-based data storage method.

[0207] It should be noted that the memory 1306 in different computing devices 1300 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data storage method apparatus based on the memory array. That is, the instructions stored in the memory 1306 of different computing devices 1300 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0208] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 15 One possible implementation method is shown. Figure 15 This is a schematic diagram of another computing device cluster structure provided in this application. Figure 15 As shown, in the computing device cluster 1500, two computing devices 1300A and 1300B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 1306 in computing device 1300A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1306 in computing device 1300B stores instructions for executing the functions of the processing module.

[0209] It should be understood that Figure 15The functions of computing device 1300A shown can also be performed by multiple computing devices 1300. Similarly, the functions of computing device 1300B can also be performed by multiple computing devices 1300.

[0210] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0211] Please refer to Figure 16 , Figure 16 This is a schematic diagram of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 3 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0212] Figure 16 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0213] In one embodiment, the computer-readable storage medium 1600 is provided using a signal bearer medium 1601. The signal bearer medium 1601 may include one or more program instructions 1602, which, when executed by one or more processors, can provide the above-mentioned... Figure 3 The described function or part of the function.

[0214] In some examples, the signal carrying medium 1601 may include a computer-readable medium 1603, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0215] In some embodiments, the signal-bearing medium 1601 may comprise a computer-recordable medium 1604, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, and so on. In some embodiments, the signal-bearing medium 1601 may comprise a communication medium 1605, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, and so on). Therefore, for example, the signal-bearing medium 1601 may be transmitted by a wireless communication medium 1605 (e.g., a wireless communication medium conforming to the IEEE 1302.X standard or other transmission protocols).

[0216] One or more program instructions 1602 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1602 conveyed to the computing device via a computer-readable medium 1603, a computer-recordable medium 1604, and / or a communication medium 1605.

[0217] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0218] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0219] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0220] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data storage method based on a storage array, characterized in that, include: A first data write request is obtained, the first data write request being used to request that first data be written to a storage array, the storage array including multiple storage devices; Based on the first data write request, a second data write request and a verification data write request are sent to the storage array. The second data write request is used to instruct the first data to be written in segments to multiple data blocks included in the first stripe. The verification data write request is used to instruct the first verification data to be written to the verification block included in the first stripe. The first verification data is calculated based on the data to be written in the multiple data blocks. The multiple data blocks and the verification block included in the first stripe are evenly distributed on the multiple storage devices. The target storage device including the verification block in the plurality of storage devices includes a write buffer that supports in-situ data updates and a persistent storage medium that supports data appending. The verification block is distributed on the persistent storage medium, and the length of the verification block is less than the length of the write buffer. The write buffer is used to continuously cache and update the verification data to be written to the verification block, and to store the verification data that will no longer be updated in the persistent storage medium.

2. The method according to claim 1, characterized in that, The second data write request is used to instruct the first data to be written in segments to a first portion of the plurality of data blocks; The method further includes: Obtain a third data write request, the third data write request being used to request that the second data be written to the storage array; Based on the third data write request, a fourth data write request and a verification data update request are sent to the storage array. The fourth data write request is used to instruct the second data to be written in segments to the second part of the data blocks in the plurality of data blocks. The verification data update request is used to instruct the data in the verification block to be updated to the second verification data. The second verification data is calculated based on the existing data and the data to be written in the plurality of data blocks. The write buffer is used to update the cached first verification data to the second verification data after receiving the verification data update request.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the Region Random Write Area (ZRWA) maintained for the target storage device. The length of the ZRWA is equal to the length of the write buffer. The ZRWA slides across the address space corresponding to the target storage device as the data writing progress of the target storage device continues. In response to the fact that the address corresponding to the verification block is located on the ZRWA, a verification data update request is generated.

4. The method according to claim 2 or 3, characterized in that, The write buffer is used to store the second verification data into the persistent storage medium when the cached data overflows.

5. The method according to any one of claims 2-4, characterized in that, The logical space corresponding to the plurality of storage devices includes the first strip and the second strip; The method further includes: Obtain a fifth data write request, the fifth data write request being used to request that the third data be written to the storage array; Based on the fifth data write request, a sixth data write request is sent to the storage array. The sixth data write request is used to instruct the third data to be segmented and written into multiple data blocks included in the second stripe. The target storage device has data blocks distributed in the second stripe. The write buffer is used to store the second verification data into the persistent storage medium when a segment of data belonging to the third data is obtained and the remaining space cannot accommodate the segment of data.

6. The method according to any one of claims 1-5, characterized in that, The target storage device records the last write address of the persistent storage medium and the length of the valid data cached in the write buffer, wherein the last write address and the length of the valid data are used to realize the recovery of the data writing process.

7. The method according to claim 6, characterized in that, The method is applied to an execution device connected to the storage array, and the method further includes: After the execution device recovers from a power outage, the execution device sends a query request to the target storage device. The query request is used to request information on data writing. The execution device receives the last write address and the length of the valid data sent by the target storage device; Based on the last written address and the length of the valid data, the execution device determines the write pointer corresponding to the target storage device, and the write pointer is used to indicate the end address of the data written to the target storage device.

8. The method according to claim 6 or 7, characterized in that, The target storage device records the length of the valid data through an out-of-band (OOB) region.

9. The method according to claim 8, characterized in that, The OOB uses multiple bits to represent the length of valid data cached in the write buffer, and one of the multiple bits is used to indicate whether a logical block in the write buffer caches valid data.

10. The method according to any one of claims 1-9, characterized in that, The storage array is managed using an open storage protocol based on sequential writes.

11. A data storage device based on a storage array, characterized in that, include: An acquisition module is used to acquire a first data write request, the first data write request being used to request that first data be written to a storage array, the storage array including multiple storage devices; The sending module is configured to send a second data write request and a verification data write request to the storage array based on the first data write request. The second data write request is configured to instruct the first data to be segmented and written into multiple data blocks included in the first stripe. The verification data write request is configured to instruct the first verification data to be written into the verification block included in the first stripe. The first verification data is calculated based on the data to be written in the multiple data blocks. The multiple data blocks and the verification block included in the first stripe are evenly distributed on the multiple storage devices. The target storage device including the verification block in the plurality of storage devices includes a write buffer that supports in-situ data updates and a persistent storage medium that supports data appending. The verification block is distributed on the persistent storage medium, and the length of the verification block is less than the length of the write buffer. The write buffer is used to continuously cache and update the verification data to be written to the verification block, and to store the verification data that will no longer be updated in the persistent storage medium.

12. The apparatus according to claim 11, characterized in that, The second data write request is used to instruct the first data to be written in segments to a first portion of the plurality of data blocks; The acquisition module is further configured to acquire a third data write request, the third data write request being used to request that the second data be written to the storage array; The sending module is further configured to send a fourth data write request and a verification data update request to the storage array based on the third data write request. The fourth data write request is used to instruct the second data to be written in segments to the second part of the data blocks in the plurality of data blocks. The verification data update request is used to instruct the data in the verification block to be updated to the second verification data. The second verification data is calculated based on the existing data and the data to be written in the plurality of data blocks. The write buffer is used to update the cached first verification data to the second verification data after receiving the verification data update request.

13. The apparatus according to claim 12, characterized in that, The device further includes: a processing module; The acquisition module is further configured to acquire the Region Random Write Area (ZRWA) maintained by the target storage device, wherein the length of the ZRWA is equal to the length of the write buffer, and the ZRWA slides on the address space corresponding to the target storage device as the data writing progress of the target storage device progresses. The processing module is configured to generate the verification data update request in response to the address corresponding to the verification block being located on the ZRWA.

14. The apparatus according to claim 12 or 13, characterized in that, The write buffer is used to store the second verification data into the persistent storage medium when the cached data overflows.

15. The apparatus according to any one of claims 12-14, characterized in that, The logical space corresponding to the plurality of storage devices includes the first strip and the second strip; The acquisition module is further configured to acquire a fifth data write request, which is used to request that the third data be written to the storage array. The sending module is further configured to send a sixth data write request to the storage array based on the fifth data write request. The sixth data write request is used to instruct the third data to be segmented and written into multiple data blocks included in the second stripe. The target storage device has data blocks distributed in the second stripe. The write buffer is used to store the second verification data into the persistent storage medium when a segment of data belonging to the third data is obtained and the remaining space cannot accommodate the segment of data.

16. The apparatus according to any one of claims 11-15, characterized in that, The target storage device records the last write address of the persistent storage medium and the length of the valid data cached in the write buffer, wherein the last write address and the length of the valid data are used to realize the recovery of the data writing process.

17. The apparatus according to claim 16, characterized in that, The apparatus is applied to an execution device connected to the storage array, and the apparatus further includes: a processing module; After the execution device recovers from a power outage, the sending module is further configured to send a query request to the target storage device, the query request being used to request information on data writing status; The acquisition module is further configured to receive the last written address and the length of the valid data sent by the target storage device; Based on the last written address and the length of the valid data, the processing module is used to determine the write pointer corresponding to the target storage device, and the write pointer is used to indicate the end address of the data written to the target storage device.

18. The apparatus according to claim 16 or 17, characterized in that, The target storage device records the length of the valid data using OOB (Out-of-Browse) technology.

19. The apparatus according to claim 18, characterized in that, The OOB uses multiple bits to represent the length of valid data cached in the write buffer, and one of the multiple bits is used to indicate whether a logical block in the write buffer caches valid data.

20. The apparatus according to any one of claims 11-19, characterized in that, The storage array is managed using an open storage protocol based on sequential writes.

21. A computing device, characterized in that, It includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, and when the code is executed, the computing device performs the method as claimed in any one of claims 1 to 10.

22. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; A processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device to cause a cluster of computing devices to perform operational steps of the method as claimed in any one of claims 1 to 10.

23. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method of any one of claims 1 to 10.

24. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 10.

Images