Data processing method, computing device, storage medium, and program product

Abstract

Embodiments of the present application provide a data processing method, a computing device, a storage medium and a program product. In the method, read requests corresponding to at least two request tasks are triggered in sequence according to a task execution sequence, and in the case that responses to the read requests corresponding to the multiple request tasks belonging to the same valid data segment are completed, write requests corresponding to the multiple request tasks are triggered in sequence to ensure sequential writing; and whether an abnormal event exists is detected, and in the case that the abnormal event exists, the triggering of the read requests and the write requests that have not been triggered is stopped. Through parallel processing of the at least two read requests, the data processing efficiency of the valid data is improved, the performance and availability of the storage device are improved, and at the same time that parallel processing is implemented, an abnormal processing function is added, and the availability and reliability of the storage device are ensured.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data processing method, computing device, storage medium, and program product. Background Technology

[0002] To ensure the storage performance of storage devices, it is necessary to perform garbage collection on the storage devices to clean up junk data, i.e. invalid data, in order to free up storage space.

[0003] One type of current storage device typically divides data into multiple fixed-size, contiguous storage regions. Each storage region only supports strict sequential writes, and erasure is performed on a region-by-region basis. Garbage collection is usually performed by a storage engine running in user space. The process involves first identifying the first storage region that needs garbage collection, then writing at least one valid data segment from the first storage region to a second storage region, and finally erasing the first storage region.

[0004] However, there is currently a problem of low garbage collection efficiency, which cannot keep up with the rate of storage space consumption, resulting in less available storage space and poor performance of storage devices. Summary of the Invention

[0005] This application provides a data processing method, computing device, storage medium, and program product to solve the technical problem of low waste recycling efficiency in the prior art.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] Determine the first storage region and the second storage region in the storage device;

[0008] Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments;

[0009] According to the task execution order, trigger the read requests corresponding to at least two untriggered request tasks respectively; the read requests are used to read the corresponding data segments and write them into memory; the at least two request tasks correspond to one or more valid data segments;

[0010] When the read request responses for multiple request tasks belonging to the same valid data segment have ended, the write requests corresponding to the multiple request tasks are triggered sequentially; the write requests are used to read the corresponding data segments from the memory and write the data segments into the second storage area;

[0011] Detect for any abnormal events;

[0012] In the event of an abnormal event, stop triggering any read and write requests that have not yet been triggered.

[0013] Optionally, triggering the read requests corresponding to at least two untriggered request tasks according to the task execution order includes:

[0014] In the absence of any abnormal events, the read requests corresponding to at least two untriggered request tasks will be triggered according to the task execution order.

[0015] Optionally, it also includes:

[0016] If any read request response ends, determine whether a read error exists based on the execution result of the read request;

[0017] If any write request response ends, determine whether a write error exists based on the execution result of the write request;

[0018] The exception event is generated in the event of a read error or write error.

[0019] Optionally, it also includes:

[0020] Before triggering any read request or any write request, perform at least one of the following detection operations: whether the first storage area has an abnormal state, whether the second storage area has an abnormal state, whether the remaining space in the second storage area meets the write conditions, and whether any valid data segment has an abnormal state.

[0021] The abnormal event is generated when there is an abnormal state in the first storage area, an abnormal state in the second storage area, the remaining space in the second storage area does not meet the write conditions, or any valid data segment has an abnormal state.

[0022] Optionally, detecting whether an abnormal event exists includes:

[0023] Before triggering any read request, before triggering any write request, after the response to any read request has ended, or after the response to any write request has ended, check for any abnormal events.

[0024] Optionally, generating the exception event in the event of a read error or write error includes:

[0025] In the event of a read error or write error, stop triggering any untriggered read or write requests corresponding to the currently processed valid data segment;

[0026] After the triggered write request and triggered read request corresponding to the currently processed valid data segment have finished responding, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended.

[0027] The step of stopping the triggering of untriggered read and write requests in the event of an abnormal event includes:

[0028] In the event of an abnormal event, stop triggering read and write requests corresponding to unprocessed valid data segments.

[0029] Optionally, it also includes:

[0030] In response to the abnormal event, and after the processing of the currently processed valid data segment has ended, a recycling end command is generated; the recycling end command is used to instruct the resource release operation.

[0031] Optionally, it also includes:

[0032] In response to the abnormal event, and after the processing of the currently processed valid data segment has ended, an abnormal handling operation is performed.

[0033] Optionally, the exception handling operation includes one or more of the following implementation methods:

[0034] If the anomaly type of the abnormal event is that the second storage area has an abnormal state, an anomaly flag is set for the second storage area, and the second storage area corresponding to the first storage area is reselected. Then, the step of dividing at least one valid data segment in the first storage area into multiple data segments is returned to continue execution.

[0035] If the anomaly type of the abnormal event is that the first storage area has a state abnormality, an anomaly flag is set for the first storage area;

[0036] If the anomaly type of the abnormal event is that there is an abnormal state in the valid data segment, an anomaly flag is set for the valid data segment;

[0037] If the exception type of the abnormal event is that the remaining space in the second storage area does not meet the write condition, the state of the second storage area will be switched to the closed state.

[0038] Optionally, determining the first storage region and the second storage region in the storage device includes:

[0039] A first storage area in the storage device that is not marked with an anomaly flag is identified, and a second storage area in the storage device that is not marked with an anomaly flag and is in an open state is identified.

[0040] Optionally, the step of sequentially triggering the write requests corresponding to the plurality of request tasks includes:

[0041] If the write request response for any triggered request task has ended, then the write request for the next triggered request task will be triggered.

[0042] Optionally, triggering the read requests corresponding to at least two untriggered request tasks according to the task execution order includes:

[0043] Based on the number of parallel requests and the order of task execution, trigger the read requests corresponding to at least two untriggered request tasks respectively;

[0044] The method further includes:

[0045] If the write request response for any triggered request task has ended, then the read request for the next untriggered request task will be triggered.

[0046] Optionally, it also includes:

[0047] If the read request response for any triggered request task ends, then the read request for the next untriggered request task will be triggered.

[0048] Optionally, triggering the read request corresponding to the next untriggered request task after the read request response for any triggered request task has ended includes:

[0049] If the read request response for any triggered request task ends, and the number of request tasks that have triggered read requests and have not triggered write requests is less than a preset value, then the read request for the next untriggered request task will be triggered.

[0050] Optionally, triggering the read requests corresponding to at least two untriggered request tasks according to the task execution order includes:

[0051] Based on the number of parallel requests and the order of task execution, trigger the read requests corresponding to at least two untriggered request tasks respectively;

[0052] The method further includes:

[0053] Count the number of currently triggered read request tasks;

[0054] If the read request response for any triggered request task ends, update the number of read request tasks.

[0055] Detect whether the number of read request tasks has reached the number of parallel requests;

[0056] If the number of read request tasks has not reached the number of parallel requests, the next read request corresponding to the untriggered request task will be triggered according to the task execution order.

[0057] Optionally, it also includes:

[0058] Count the number of currently triggered write request tasks;

[0059] If the write request response for any triggered request task ends, update the number of write request tasks.

[0060] Check if the number of write request tasks is equal to zero;

[0061] If the number of write request tasks is zero, the next write request corresponding to the already triggered request task will be triggered according to the task execution order.

[0062] Optionally, it also includes:

[0063] When the write request response for any triggered request task ends, the resources occupied by the request task are released.

[0064] Secondly, this application provides a computing device including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the data processing method as described in the first aspect above.

[0065] Thirdly, this application provides a computer storage medium storing a computer program, which, when executed by a computer, implements the data processing method described in the first aspect above.

[0066] Fourthly, this application provides a computer program product, including a computer program / instruction, which, when executed by a computer, implements the data processing method as described in any of the first aspects above.

[0067] This embodiment of the application, following the task execution order, can trigger read requests corresponding to at least two request tasks respectively. After the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, write requests corresponding to multiple request tasks are triggered sequentially to ensure sequential writing. Here, at least two request tasks can correspond to one or more valid data segments. The application detects the presence of abnormal events; if an abnormal event exists, triggering untriggered read and write requests is stopped. This embodiment of the application improves the data processing efficiency of valid data and enhances the performance and availability of the storage device through the parallel processing of at least two read requests. Furthermore, while implementing parallel processing, it adds an exception handling function to ensure that garbage collection can be processed normally, thereby guaranteeing the availability and reliability of the storage device.

[0068] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 A flowchart of one embodiment of a data processing method provided in this application is shown;

[0071] Figure 2 A flowchart of yet another embodiment of a data processing method provided in this application is shown;

[0072] Figure 3 This illustration shows a read / write diagram of an embodiment of this application in a practical application;

[0073] Figure 4 A flowchart of yet another embodiment of a data processing method provided in this application is shown;

[0074] Figure 5 This illustration shows a scenario interaction diagram of an embodiment of this application in a practical application;

[0075] Figure 6 This invention provides a schematic diagram of the structure of one embodiment of a data processing apparatus.

[0076] Figure 7 A schematic diagram of one embodiment of a computing device provided in this application is shown. Detailed Implementation

[0077] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0078] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0079] The technical solution of this application embodiment is applied to the garbage collection scenario of storage devices, especially the garbage collection scenario of distributed storage systems. A distributed storage system can be composed of multiple storage servers, and each storage server can deploy multiple storage devices and can deploy a storage engine to manage the multiple storage devices, such as being responsible for garbage collection in the multiple storage devices.

[0080] For ease of understanding, the technical concepts that may be involved in the embodiments of this application are explained below:

[0081] Distributed storage system: refers to a storage method that distributes data across multiple storage servers in a network. This storage solution can provide high availability, fault tolerance, and scalability.

[0082] Storage devices: Physical hardware devices used to store data. These devices can be, for example, traditional hard disk drives (HDDs), solid-state drives (SSDs), or other forms of non-volatile storage media. In a distributed storage system, multiple storage devices can be deployed in each storage server to collectively provide storage services.

[0083] Zoned storage: A type of storage device whose address space is divided into zones with write constraints different from those of conventional storage devices. Typical zoned storage devices can include two types: ZNS SSD (Zoned Namespace Solid State Drive) and SMR HDD (Shingled Magnetic Recording Hard Disk Drive).

[0084] Zone: The smallest unit of allocation and erasure on zoned storage.

[0085] Storage area: A storage space on a storage device that is divided into fixed-size, contiguous blocks of memory. Each storage area only supports strict sequential writes and erasure on a per-storage-area basis. In Zoned Storage, a storage area is also known as a Zone space.

[0086] Garbage collection (GC) is the operation of reading valid data from the first storage area and writing it to the second storage area, while releasing the first storage area.

[0087] User mode and kernel mode: Kernel mode refers to the environment in which the operating system kernel runs; user mode refers to the environment in which applications and most user-level programs run. Storage devices run in kernel mode and can be garbage collected by the storage engine running in user mode, thus reducing the burden on the operating system kernel and improving the overall system efficiency.

[0088] Storage engine: An application that runs in user space.

[0089] As described in the background section, there is currently a technical problem of low garbage collection efficiency, which makes it impossible for garbage collection efficiency to keep up with the rate of storage area consumption, resulting in less available storage area and poor performance of storage devices.

[0090] The inventors discovered that due to limitations in garbage collection efficiency, garbage collection bandwidth can easily reach a bottleneck. Meanwhile, front-end throughput continuously consumes storage space, causing the garbage collection speed to be slower than the storage space consumption rate, resulting in less available storage space. Currently, the only solution is to limit the front-end throughput, which affects storage performance and reduces the storage experience.

[0091] Through a series of studies, the inventors discovered that in traditional methods, when a stored procedure performs garbage collection, valid data is first divided into multiple data segments, and corresponding request tasks are generated. Then, for each data segment, a read request is triggered first to read the valid data from the first storage area and write it into memory. After the read request is completed, a write request is triggered to write the data from memory into the second storage area. This process continues until all valid data has been read into the second storage area, at which point the first storage area is erased. The inventors found that this implementation uses a serial processing method for reading valid data. If a request has a high latency, it will cause the entire process to stall, affecting the final garbage collection efficiency.

[0092] To improve garbage collection efficiency and storage performance, the inventors have proposed the technical solution of this application after a series of studies. In the embodiments of this application, at least two read requests corresponding to different request tasks can be triggered according to the task execution order. When the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, write requests corresponding to multiple request tasks are triggered sequentially to ensure sequential writing. The system also detects whether there are any abnormal events. If there are abnormal events, the triggering of untriggered read and write requests is stopped. The embodiments of this application realize the parallel processing of at least two read requests. Since at least two request tasks can correspond to one or more valid data segments, parallel processing of request tasks across data segments can be realized. Compared with the serial processing method, the data processing efficiency of valid data is improved, the throughput can be increased, and processing resources can be fully utilized, thereby ensuring garbage collection efficiency. This can improve the utilization rate of the storage device, improve the performance and availability of the storage device, and while realizing parallel processing, an exception handling function is added to ensure that garbage collection can be processed normally, thereby ensuring the availability and reliability of the storage device.

[0093] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0095] The implementation details of the technical solutions in the embodiments of this application are described in detail below.

[0096] Figure 1 A flowchart illustrating an embodiment of a data processing method provided in this application. The technical solution of this embodiment can be executed by a storage engine, which can run in user mode. The method may include the following steps:

[0097] 101: Determine the first storage area and the second storage area in the storage device.

[0098] Storage devices can be, for example, zoned storage, such as ZNS SSDs or SMR HDDs.

[0099] Storage devices can divide storage space into multiple fixed-size, contiguous storage regions. Each storage region only supports writing data strictly in sequence. Once a region is full, no new data can be written unless all data in that region is erased. Storage regions have four states: Free, Open, Closed, and Garbage. When a storage region is in the Free state, it means it is unused, has no written data, and is available to receive write operations immediately. Once writing begins to a storage region, it transitions from the Free state to the Open state. In the Open state, new data can continue to be appended to the region until its maximum capacity is reached or it is closed. When a storage region is full or closed via command, it transitions from the Open state to the Closed state. In the Closed state, the region no longer accepts new write requests but can still be read from. The storage area can perform garbage collection operations, which means migrating valid data to other storage areas. When valid data in a storage area has been deleted or marked as invalid, the storage area may be marked as garbage. Then, the entire storage area can be erased to free up space, and the storage area will be reset to the free state and become available again.

[0100] In this embodiment of the application, a first storage area and a second storage area in the storage device that meet the garbage collection requirements may be determined. The garbage collection requirements may be, for example, that the proportion of invalid data in the first storage area reaches a predetermined proportion and the second storage area is in an idle state. That is, a storage area with a predetermined proportion of invalid data may be determined as the first storage area and an idle storage area may be determined as the second storage area.

[0101] 102: Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments.

[0102] Here, a request task can refer to a task that reads data from a first storage area and writes it to a second storage area. Therefore, each request task can correspond to a read request that triggers the reading of data from the first storage area and the writing of it to memory, and a write request that triggers the reading of data from memory and the writing of it to the second storage area. Since executing a request task involves data reading and writing, a request task can also refer to an I / O (Input / Output) task.

[0103] The first storage area may include at least one valid data segment. If the valid data in the first storage area is contiguous in address, then the first storage area may contain one valid data segment. If the valid data in the first storage area is non-contiguous in address, then the first storage area may contain multiple valid data segments.

[0104] To avoid network congestion and significant memory pressure caused by directly transmitting excessively large valid data segments, which would negatively impact performance, this embodiment of the application can divide at least one valid data segment into multiple data segments. Optionally, each valid data segment can be divided into multiple data segments according to a predetermined size. Each valid data segment can be divided into one or more data segments based on its size. The predetermined size can be determined based on actual IO requirements, such as 64KB (kilobytes).

[0105] For each data segment, a corresponding request task can be generated. Each request task can include a read request and a write request. The read request is used to trigger the reading of the corresponding data segment and write it into memory. The write request is used to trigger the reading of the corresponding data segment from memory and write the data segment into the second storage area, thereby realizing effective data transfer.

[0106] The generated multiple request tasks can be saved to a task list. The task list stores the request information for each request task, which may include, for example, the identifier of the second storage area, the offset of the data segment, and the length.

[0107] 103: In accordance with the task execution order, trigger the read requests corresponding to at least two untriggered request tasks.

[0108] The read request is used to read the corresponding data segment and write it into memory.

[0109] At least two request tasks can correspond to one or more valid data segments.

[0110] For example, since data is written sequentially to the storage area, the execution order of the request tasks corresponding to multiple data segments can be determined based on the storage addresses of the data segments. The storage addresses of the data segments can be determined based on the offset and length of the data segments. Specifically, triggering the read requests corresponding to at least two untriggered request tasks according to the task execution order can be achieved by triggering the read requests corresponding to at least two untriggered request tasks according to the number of concurrent requests and the task execution order. The number of parallel requests indicates the maximum number of read requests that can be triggered simultaneously. This number of parallel requests can, for example, refer to the IO Queue Depth (input / output queue depth), which indicates the maximum number of requests that can be triggered simultaneously. Optionally, this number of parallel requests is greater than 1.

[0111] In this embodiment of the application, an untriggered request task refers to a request task that has not triggered a read request or a write request, while a triggered request task refers to a request task that has triggered a read request or a write request.

[0112] 104: When multiple request tasks belonging to the same valid data segment have completed their respective read request responses, the write requests corresponding to the multiple request tasks are triggered sequentially.

[0113] Write requests are used to read the corresponding data segments from memory and write the data segments into the second storage area.

[0114] The write requests corresponding to multiple request tasks belonging to the same valid data segment can be triggered sequentially according to the task execution order. Optionally, sequential triggering can mean that after the write request response of any triggered request task has ended, the write request corresponding to the next triggered request task is triggered to ensure that the data is written sequentially in the second storage area according to the task execution order.

[0115] In this process, triggering the read requests corresponding to at least two non-triggered request tasks can achieve parallel processing of at least two read requests. Since the response completion order of at least two read requests may be different from the task execution order, the write requests corresponding to multiple request tasks belonging to the same valid data segment can be triggered sequentially after all the read requests corresponding to multiple request tasks belonging to the same valid data segment have completed their responses. This ensures that the valid data segment is complete, so that even if the completion order of the read requests corresponding to multiple tasks belonging to the same valid data segment triggered in parallel is different from the task execution order, it can still be guaranteed that after the valid data segment is completely read into memory, at least one data segment included in it can be written to the second storage area sequentially according to the task execution order.

[0116] 105: Detect whether there are any abnormal events.

[0117] 106: In the event of an abnormal event, stop triggering any untriggered read and write requests.

[0118] In practical applications, many abnormal issues may arise during garbage collection, such as abnormal states of the first or second storage area, abnormal states of valid data segments, read / write errors, or insufficient storage space in the second storage area. These issues can stall the garbage collection process and affect its normal operation. This embodiment of the application addresses this by performing anomaly detection and promptly stopping the triggering of untriggered read and write requests in the event of an anomaly, thus achieving anomaly handling during garbage collection. It should be noted that the execution of step 104 is not limited to the order of operations specified in this embodiment. In practical applications, anomaly detection can be performed at one or more of the following times: before triggering any read request, before triggering any write request, after the response to any read request has ended, and after the response to any write request has ended, etc., to ensure normal garbage collection.

[0119] Optionally, step 103 above may involve triggering the read requests corresponding to at least two untriggered request tasks according to the task execution order in the absence of any abnormal events, so as to ensure the timeliness of exception handling.

[0120] In this embodiment, multiple read requests corresponding to different request tasks can be triggered according to the task execution order without waiting for the previous request task to finish, allowing multiple read requests to be processed in parallel. After the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, write requests corresponding to multiple request tasks are triggered sequentially to ensure data integrity and sequential writing. Furthermore, by detecting the presence of abnormal events, triggering untriggered read and write requests is stopped if an abnormal event exists. This embodiment of the application achieves parallel processing of multiple read requests. Since at least two request tasks can correspond to one or more valid data segments, parallel processing of request tasks across data segments can be achieved. Compared to serial processing, this can improve throughput, fully utilize processing resources, thereby improving garbage collection efficiency, storage device utilization, storage device performance and availability. Furthermore, abnormal handling ensures normal garbage collection processing, guaranteeing the availability and reliability of the storage device.

[0121] In some embodiments, the method may further include: releasing the resources occupied by the request task when the write request response corresponding to any triggered request task ends.

[0122] For example, since the request information of a request task is stored in memory, the end of the write request response of any triggered request task indicates that the execution of the request task has ended. At this time, the memory resources occupied by the request information of the request task, as well as the memory resources occupied by the data segments read by the read request corresponding to the request task, can be released.

[0123] To enable garbage collection anomaly detection for parallel processing of multiple read requests, some embodiments may further include: determining whether a read error exists based on the execution result of the read request when any read request response ends; determining whether a write error exists based on the execution result of the write request when any write request response ends; and generating the anomaly event if either a read error or a write error exists.

[0124] Specifically, determining whether a read error exists based on the execution result of a read request can involve checking the return value or status code returned after the read request response, the logs generated after the read request response, whether any exceptions were thrown during execution, and / or verifying the completeness of the read data after the read request response. Similarly, determining whether a write error exists based on the execution result of a write request can involve checking the return value or status code returned after the write request response, the logs generated after the write request response, whether any exceptions were thrown during execution, and / or verifying the completeness of the written data after the write request response.

[0125] In the process of developing this application, the inventors discovered that in traditional serial processing methods, to ensure correct garbage collection, anomaly detection is typically performed after the read and write operations corresponding to a request task are completed. Since serial processing only triggers one request at a time, garbage collection on the first storage area can be terminated directly if an anomaly occurs. However, in the technical solution of this application, multiple read requests are triggered in parallel. Write requests are triggered after the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding. That is, multiple requests are triggered simultaneously. After all multiple requests have finished responding, anomaly detection is performed, and processing such as stopping the triggering of new requests is executed based on the anomaly detection results. This can lead to a situation where if one request is judged to have an error after its response, other requests will still be triggered, resulting in unnecessary resource waste. Furthermore, waiting for responses from other requests increases the total response time, causing delays.

[0126] To reduce latency and resource waste, the inventors conceived of stopping the triggering of new requests if an error is detected at the end of any of the concurrently triggered requests. Therefore, in some embodiments, generating an exception event in the event of a read or write error may include: stopping the triggering of any untriggered read or write requests corresponding to the currently processed valid data segment in the event of a read or write error; and generating an exception event after the responses of the triggered write and read requests corresponding to the currently processed valid data segment have ended, and determining that the processing of the currently processed valid data segment has concluded.

[0127] The currently processed valid data segment can refer to a valid data segment that has triggered a request task. A valid data segment may be divided into multiple data sub-segments, and each data sub-segment corresponds to a request task. Each valid data segment can also correspond to a data segment task. According to the task execution order, at least two triggered but not triggered request tasks may span one or more valid data segments. That is, at least two triggered but not triggered request tasks may include request tasks corresponding to one or more valid data segments. Furthermore, there may be untriggered request tasks within the valid data segments of triggered request tasks. These request tasks are in a waiting state, and the remaining valid data segments that have not been triggered by any request task are also in a waiting state.

[0128] In the event of a read error or write error, stopping the triggering of untriggered read and write requests corresponding to the currently processed valid data segment can refer to read and write requests that are in a waiting state within the currently processed valid data segment.

[0129] In the event of an abnormal event, stopping the triggering of untriggered read and write requests can include: stopping the triggering of read and write requests corresponding to unprocessed valid data segments. These unprocessed valid data segments can also refer to valid data segments that are in a waiting state.

[0130] Assume the first storage area includes four valid data segments (valid data segment 0, valid data segment 1, valid data segment 2, and valid data segment 3). These four valid data segments are divided into multiple data sub-segments according to a predetermined size, generating task lists corresponding to each data sub-segment. Each task list includes the request tasks corresponding to each data sub-segment. For example, valid data segment 0 can be divided into four data sub-segments, corresponding to request tasks IO0, IO1, IO2, and IO3 respectively; valid data segment 1 can be divided into three data sub-segments, corresponding to request tasks IO4, IO5, and IO6 respectively; valid data segment 2 can be divided into three data sub-segments, corresponding to request tasks IO7, IO8, and IO9 respectively; and valid data segment 3 can be divided into three data sub-segments, corresponding to request tasks IO10, IO11, and IO12 respectively. For instance, the current processing involves read requests corresponding to IO5-IO6 of valid data segment 1, read requests corresponding to IO7-IO9 of valid data segment 2, and write requests corresponding to IO3 of valid data segment 0. If any read request response to IO5-IO9 and any write request response to IO3 is determined to have a read error upon completion, or a write request response to IO3 is determined to have a write error upon completion (e.g., the execution result of the read request response to IO5 is determined to have a read error), then triggering of waiting read and write requests in the currently processed valid data segment will cease. This includes write requests corresponding to IO4-IO6 in valid data segment 1 and write requests corresponding to IO5-IO9 in valid data segment 2. After all read requests corresponding to IO5-IO9 and write requests corresponding to IO3 have been completed, an exception event is generated, and it is determined that processing of valid data segment 0, valid data segment 1, and valid data segment 2 has ended. Furthermore, triggering of read and write requests corresponding to the unprocessed valid data segment, i.e., valid data segment 3, will cease, which also means stopping the triggering of read and write requests corresponding to IO10, IO11, and IO12.

[0131] In this embodiment, if any read request has a read error or any write request has a write error, the triggering of untriggered read and write requests can be stopped immediately, without waiting for the detection of an abnormal event. This allows for timely response to error situations, reduces latency, avoids error accumulation, prevents further resource waste, and improves resource utilization.

[0132] In some embodiments, the method may further include: responding to the abnormal event and generating a recycling end command after the processing of the currently processed valid data segment has ended. The recycling end command may be used to instruct a resource release operation.

[0133] The end command for resource reclamation can be an interrupt signal, etc. Resource release operations can include releasing the memory resources occupied, such as the memory resources occupied by the request information of the requesting task and the memory resources occupied by the data segments read into memory.

[0134] For example, if the execution result of the read request response corresponding to IO5 is determined to have a read error, then triggering of waiting read and write requests in the currently processed valid data segment will be stopped, such as the write requests corresponding to IO4-IO6 in valid data segment 1 and the write requests corresponding to IO5-IO9 in valid data segment 2. After all read requests corresponding to IO5-IO9 and write requests corresponding to IO3 have been responded to, an exception event is generated. Responding to this exception event can generate a recycling termination instruction, instructing the release of memory resources occupied by the request information of the read requests corresponding to IO5-IO9, the request information of the write request corresponding to IO3, and the memory resources occupied by the data segments read into memory by the read requests corresponding to IO5-IO9.

[0135] In cases where a read error occurs upon the completion of any read request response or a write error occurs upon the completion of any write request response, immediately terminating the garbage collection process without waiting for other triggered read and write requests to finish would interrupt these requests and cause further problems. For example, after terminating the garbage collection process, resources are typically released, but if other triggered read and write requests have not yet completed their responses, relying on already released resources during callbacks would lead to errors. Therefore, this embodiment generates an exception event only after all triggered write and read requests corresponding to the currently processed valid data segment have completed their responses. This confirms the completion of processing for the currently processed valid data segment and then instructs for resource release. This ensures that allocated resources are correctly released only after all request responses are completed, avoiding errors and resource management problems caused by premature termination of the garbage collection process.

[0136] In some embodiments, the method may further include: before triggering any read request or before triggering any write request, performing at least one of the following detection operations: whether the first storage area has a state abnormality, whether the second storage area has a state abnormality, whether the remaining space of the second storage area meets the write conditions, and whether any valid data segment has a state abnormality; generating the abnormal event if the first storage area has a state abnormality, the second storage area has a state abnormality, the remaining space of the second storage area does not meet the write conditions, or any valid data segment has a state abnormality.

[0137] The connection status of a storage area can include open, closed, and offline states. An abnormal status for the first storage area can mean that the first storage area is in a closed or offline state, disallowing read and write operations. An abnormal status for the second storage area can mean that the second storage area is in a closed or offline state, disallowing read and write operations. An abnormal status for a valid data segment can mean that data corruption and / or data loss have occurred in a valid data segment. Insufficient remaining space in the second storage area may mean that the second storage area is full and has no remaining space, or that the remaining space is below the minimum storage capacity threshold required to perform a write operation.

[0138] This embodiment can detect potential problems in advance by performing a pre-check before triggering any read request or any write request, thereby avoiding unnecessary read and write operations and saving processing resources.

[0139] As described above, abnormal events may be generated before any read request is triggered, before any write request is triggered, after any read request response ends, or after any write request response ends. Therefore, in some embodiments, detecting the existence of abnormal events may include: detecting whether an abnormal event exists before any read request is triggered, before any write request is triggered, after any read request response ends, or after any write request response ends.

[0140] In some embodiments, the method may further include: responding to an abnormal event and performing an exception handling operation after the processing of the currently processed valid data segment has ended.

[0141] The exception handling operation can include one or more of the following implementation methods:

[0142] As one implementation, if the exception type of the abnormal event is that there is a state abnormality in the second storage area, an exception flag is set for the second storage area, and the second storage area corresponding to the first storage area is reselected. Then, the step of dividing at least one valid data segment in the first storage area into multiple data segments is returned to continue execution.

[0143] For example, the exception flag can indicate that the second storage area is in an abnormal state, which may include states such as "closed" or "offline". For instance, if the second storage area is closed, the exception flag for the second storage area can be set to the "closed" state.

[0144] As another implementation, if the exception type of the abnormal event is that there is a state abnormality in the first storage area, an exception flag is set for the first storage area;

[0145] For example, the exception flag can indicate that the first storage area is in an abnormal state, which may include states such as closed or offline. For instance, if the first storage area is offline, the exception flag for the first storage area can be set to the offline state.

[0146] As another implementation method, when the exception type of the abnormal event is that there is an abnormal state in the valid data segment, an exception flag is set for the valid data segment.

[0147] For example, the anomaly flag can indicate that the valid data segment is unavailable. Unavailable states could include data loss or data corruption. For instance, if the valid data segment is corrupted, the anomaly flag can be set to "data corrupt." Conversely, if the valid data segment is in a normal state, it can be marked as "normal."

[0148] As another implementation, if the exception type of the abnormal event is that the remaining space of the second storage area does not meet the write condition, the state of the second storage area is switched to the closed state.

[0149] In some embodiments, the method may set an anomaly mark for a first storage area and / or a second storage area with an abnormal state. To exclude storage areas with abnormal states, determining the first storage area and the second storage area in the storage device may include: determining a first storage area in the storage device that is not marked with an anomaly mark, and determining a second storage area in the storage device that is not marked with an anomaly mark and is in an open state.

[0150] In this embodiment, by setting an anomaly marker for storage areas with abnormal status and selecting storage areas without anomaly markers as the first or second storage area, storage areas with abnormal status can be excluded, ensuring that data is stored in a stable storage environment and avoiding the risk of data loss or corruption due to storage area anomalies. Setting anomaly markers allows developers to quickly identify and locate storage areas with abnormal status, thereby simplifying the maintenance and troubleshooting process of the storage system.

[0151] In some embodiments, dividing at least one valid data segment in the first storage area into multiple data segments and generating request tasks corresponding to each of the multiple data segments may include: determining at least one valid data segment in the first storage area; dividing the at least one valid data segment into multiple data segments according to a predetermined size; generating a task list corresponding to the multiple data segments; the task list includes request tasks corresponding to each of the multiple data segments.

[0152] In this embodiment, the valid data in the first storage area may be distributed across one or more valid data segments. However, a single valid data segment might be too large. Directly transmitting such a large segment could lead to network congestion and significant memory pressure. Furthermore, an excessively large valid data segment might not be fully loaded into memory, causing program crashes. This embodiment divides the valid data segments in the first storage area into multiple fixed-length data segments of a predetermined size. Each data segment corresponds to an IO request task. Therefore, the amount of data involved in each IO operation is the same, resulting in a more even IO load. The read and write operations for each IO request task will have relatively consistent time requirements. This consistency helps reduce operation time fluctuations caused by varying valid data segment sizes. Moreover, compared to transmitting larger valid data segments, this reduces latency and improves overall system performance.

[0153] In this application embodiment, there are multiple ways to implement parallel processing of read requests.

[0154] As an optional implementation, as described above, triggering the write requests corresponding to multiple request tasks in sequence can be done by: triggering the write request corresponding to the next triggered request task after the write request response of any triggered request task has ended.

[0155] Optionally, the above-mentioned method of triggering read requests corresponding to at least two untriggered request tasks according to the task execution order can be: triggering read requests corresponding to at least two untriggered request tasks according to the number of parallel requests and the task execution order. The method may further include: triggering the read request corresponding to the next untriggered request task after the write request response for any triggered request task has ended. Here, the next untriggered request task is the first request task among the remaining untriggered request tasks determined according to the task execution order.

[0156] For ease of understanding, such as Figure 2 The illustrated embodiment shows another data processing method provided in this application, which may include the following steps:

[0157] 201: Determine the first storage region and the second storage region in the storage device.

[0158] 202: Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments.

[0159] For detailed instructions on steps 201 to 202, please refer to [link / reference]. Figure 1 Steps 101 to 102 in the illustrated embodiment will not be repeated here.

[0160] 203: Based on the number of parallel requests and the order of task execution, trigger the read requests corresponding to at least two of the untriggered request tasks.

[0161] Based on the number of parallel requests, it can refer to the read requests corresponding to at least two request tasks that have not yet been triggered, which can be triggered simultaneously. The number of parallel requests also indicates the maximum number of read requests that can be triggered at the same time.

[0162] 204: If multiple request tasks belonging to the same valid data segment have completed their respective read request responses, write requests corresponding to the multiple request tasks will be triggered sequentially according to the task execution order.

[0163] In other words, when all read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, the write requests corresponding to each request task are triggered one by one according to the task execution order, starting from the first request task among the multiple request tasks. That is, when the write request corresponding to any one of the triggered request tasks finishes responding, the write request corresponding to the next triggered request task is triggered.

[0164] 205: If the write request response for any triggered request task has ended, trigger the read request for the next untriggered request task.

[0165] 206: Detect whether there are any abnormal events.

[0166] 207: In the event of an abnormal event, stop triggering any untriggered read and write requests.

[0167] In this optional implementation, after dividing the valid data segment into multiple data segments, at least two read requests corresponding to each request task can be triggered first, based on the number of parallel requests. Once all read requests corresponding to the same valid data segment have responded and completed, write requests corresponding to the various request tasks are then triggered sequentially. That is, if a write request corresponding to any one of the triggered request tasks has finished responding, the next triggered write request is then triggered. Furthermore, to further improve processing efficiency and performance, while triggering the next triggered write request corresponding to the next triggered request task, a read request corresponding to the next untriggered request task can also be triggered simultaneously, thus achieving parallel processing of read and write requests.

[0168] As described above, detecting the existence of abnormal events can include: detecting the existence of abnormal events before triggering any read request, before triggering any write request, after the response to any read request has ended, or after the response to any write request has ended.

[0169] Pre-detection can be performed before triggering any read request or any write request. For example, it can include at least one of the following detection operations: whether the first storage area has an abnormal state, whether the second storage area has an abnormal state, whether the remaining space of the second storage area meets the write conditions, and whether any valid data segment has an abnormal state.

[0170] If a read request response ends, the system can determine if a read error exists based on the execution result of the read request; if a write request response ends, the system can determine if a write error exists based on the execution result of the write request. If either a read or write error exists, triggering any untriggered read or write requests corresponding to the currently processed valid data segment is stopped. After the triggered write and read requests corresponding to the currently processed valid data segment have finished responding, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Furthermore, if an exception event is detected, triggering any read or write requests corresponding to the unprocessed valid data segment is stopped.

[0171] In this implementation, a pre-detection can be performed before triggering the read requests corresponding to at least two untriggered request tasks, based on the number of parallel requests and the task execution order. If an anomaly exists, triggering the read requests corresponding to the at least two untriggered request tasks is stopped; otherwise, the read requests corresponding to the at least two untriggered request tasks are triggered.

[0172] If multiple read requests corresponding to the same valid data segment have finished responding, a check can be performed to determine if a read error occurred at the end of any one of the read requests. If a read error exists, triggering pending write requests is stopped. After all triggered read and write requests have finished responding, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Triggering read and write requests corresponding to unprocessed valid data segments is then stopped. If no read error occurs, a pre-check is performed after all read requests corresponding to the same valid data segment have finished responding. If an exception exists, triggering pending write requests is stopped, and it is determined that the processing of the currently processed valid data segment has ended. Triggering unprocessed valid data segments is then stopped. If everything is normal, untriggered write requests can be triggered.

[0173] Upon completion of a write request response for any given task, a check can be performed to determine if a write error occurred. If a write error is found, triggering the next pending write request and the next untriggered read request will cease. After all triggered read and write requests have completed, an exception event is generated, and the processing of the currently processed valid data segment is confirmed to be complete. Triggering read and write requests for unprocessed valid data segments will then cease. If all checks are normal, and the preceding checks are successful, write requests and the next untriggered read requests for untriggered tasks will be triggered.

[0174] For ease of understanding, see [link to relevant documentation]. Figure 3 In the read / write diagram shown, as Figure 3As shown, assuming the first storage area includes three valid data segments (valid data segment 0, valid data segment 1, and valid data segment 2), the three valid data segments are divided into multiple data segments according to a predetermined size, generating multiple task lists corresponding to the data segments. The task lists include the request tasks corresponding to each data segment. For example, valid data segment 0 can be divided into four data segments, which correspond to request tasks IO0, IO1, IO2, and IO3, respectively. Valid data segment 1 can be divided into three data segments, which correspond to request tasks IO4, IO5, and IO6, respectively. Valid data segment 2 can be divided into three data segments, which correspond to request tasks IO7, IO8, and IO9, respectively. For example, the number of parallel requests can be set to 5. Based on the number of parallel requests and the task execution order, the initial number of parallel read requests can be triggered in parallel. This could involve simultaneously triggering read requests corresponding to IO0-IO3 for valid data segment 0 and IO4 for valid data segment 1. Before the initial parallel triggering of IO0-IO4, a pre-detection check can be performed. If the pre-detection is normal, the read requests corresponding to IO0-IO4 will be triggered in parallel. The system then checks if any read request response from IO0-IO3 for valid data segment 0 has resulted in a read error. If a read error exists, the triggering of write requests corresponding to IO0-IO3 for valid data segment 0, write requests corresponding to IO4 for valid data segment 1, and read and write requests corresponding to IO5 and IO6 will be stopped. After the read requests corresponding to IO0-IO4 have finished responding, an exception event is generated, and it is determined that processing of valid data segment 0 and valid data segment 1 has ended. The triggering of read and write requests corresponding to the unprocessed task for valid data segment 2 will then cease. If no read error exists, a pre-check is performed after all read requests corresponding to IO0-IO3 have finished responding. If the pre-check is normal, the write request corresponding to IO0 can be triggered. After the write request corresponding to IO0 finishes responding, a check is performed to see if a write error exists. If a write error exists, the triggering of write requests corresponding to IO1-IO3 in valid data segment 0, write requests corresponding to IO4 in valid data segment 1, and read and write requests corresponding to IO5 and IO6 is stopped. At this point, among the already triggered read requests corresponding to IO0-IO4 and write requests corresponding to IO0, the read requests corresponding to IO0-IO3 and the write requests corresponding to IO0 have finished responding. After confirming that the read request corresponding to IO4 has finished responding, an exception event can be generated, and it is confirmed that the processing of valid data segment 0 and valid data segment 1 has ended. The triggering of read and write requests corresponding to the unprocessed request task in valid data segment 2 is stopped.If all goes well, then if the pre-detection is normal, the write request corresponding to the next triggered request task IO1 and the read request corresponding to the next untriggered request task IO5 can be triggered simultaneously.

[0175] As an alternative implementation, the method may further include: triggering a read request for the next untriggered request task when the read request response of any triggered request task has ended.

[0176] In this optional implementation, once the read request response of any triggered request task ends, the read request corresponding to the next untriggered request task can be triggered, so that the triggering of the read request is not affected by the write request, further ensuring the parallel processing of read requests.

[0177] In this optional implementation, if a write request from any triggered request task is responded to, a read request corresponding to the next untriggered request task can also be triggered to ensure that read and write requests are processed in parallel.

[0178] For example, if a read request response for any given task has ended, the execution result of the read request can be used to determine if a read error exists. If a read error exists, triggering any untriggered read or write requests corresponding to the currently processed valid data segment is stopped. After the responses to triggered write and read requests corresponding to the currently processed valid data segment have ended, an exception event is generated, and the processing of the currently processed valid data segment is confirmed to have ended. Furthermore, the existence of an exception event can be detected, and triggering read and write requests corresponding to unprocessed valid data segments can be stopped. If no read error exists, a pre-detection process can be performed. If the pre-detection result is normal, the read request corresponding to the next untriggered task can be triggered.

[0179] Optionally, if the read request response of any triggered request task ends, triggering the read request corresponding to the next untriggered request task can be done as follows: if the read request response of any triggered request task ends, and the number of request tasks with triggered read requests and untriggered write requests is less than a preset value, then trigger the read request corresponding to the next untriggered request task.

[0180] For example, if the read requests corresponding to IO1-IO9 have been triggered, but the write requests corresponding to IO1-IO9 have not been triggered, then the number of requests with triggered read requests and those with untriggered write requests is 9. If the preset value is 6, then the condition of being less than the preset value is not met, and the next untriggered request task cannot be triggered.

[0181] Since processing read requests requires memory resources to store the data segments read from the first storage area, setting a preset value can prevent the number of tasks that have triggered read requests and those that have not triggered write requests from becoming too large, thus avoiding excessive memory consumption and performance impact.

[0182] According to the number of parallel requests and the order of task execution, this embodiment can trigger read requests corresponding to at least two request tasks without waiting for the previous request task to finish, enabling multiple read requests to be processed in parallel. After the read request responses corresponding to multiple request tasks have finished, write requests corresponding to multiple request tasks are triggered sequentially to ensure sequential writing. This embodiment, through the parallel processing of multiple read requests, since at least two request tasks can correspond to one or more valid data segments, enables parallel processing of request tasks across data segments, improving throughput, making full use of processing resources, thereby improving data processing efficiency, ensuring garbage collection efficiency, and ensuring the performance of the storage device.

[0183] As another optional implementation, parallel processing can be achieved by counting the number of request tasks. Therefore, such as Figure 4 In the illustrated embodiment, another data processing method provided in this application may include the following steps:

[0184] 401: Determine the first storage region and the second storage region in the storage device.

[0185] 402: Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments.

[0186] Step 401 can be found in [reference]. Figure 1 Steps 101 and 402 of the illustrated embodiment can be found in [reference needed]. Figure 1 Step 102 of the illustrated embodiment will not be described in detail here.

[0187] 403: Based on the number of parallel requests and the order of task execution, trigger the read requests corresponding to at least two of the untriggered request tasks.

[0188] 404: When multiple request tasks belonging to the same valid data segment have completed their respective read request responses, write requests corresponding to the multiple request tasks are triggered sequentially.

[0189] 405: Counts the number of currently triggered read request tasks.

[0190] 406: Update the number of read request tasks if the read request response for any triggered request task has ended.

[0191] 407: Check if the number of read request tasks has reached the number of concurrent requests. If not, proceed to step 408; if yes, continue checking.

[0192] 408: Following the task execution order, trigger the read request corresponding to the next untriggered request task, and return to step 405 to continue execution.

[0193] 409: Detect the presence of abnormal events.

[0194] 410: In the event of an abnormal event, stop triggering any untriggered read and write requests.

[0195] As described above, detecting the existence of abnormal events can include: detecting the existence of abnormal events before triggering any read request, before triggering any write request, after the response to any read request has ended, or after the response to any write request has ended.

[0196] Pre-detection can be performed before triggering any read request or any write request. For example, it can include at least one of the following detection operations: whether the first storage area has an abnormal state, whether the second storage area has an abnormal state, whether the remaining space of the second storage area meets the write conditions, and whether any valid data segment has an abnormal state.

[0197] If a read request response ends, the system can determine if a read error exists based on the execution result of the read request; if a write request response ends, the system can determine if a write error exists based on the execution result of the write request. If either a read or write error exists, triggering any untriggered read or write requests corresponding to the currently processed valid data segment is stopped. After the triggered write and read requests corresponding to the currently processed valid data segment have finished responding, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Furthermore, if an exception event is detected, triggering any read or write requests corresponding to the unprocessed valid data segment is stopped.

[0198] Therefore, in this implementation, a pre-detection can be performed before triggering the read requests corresponding to at least two untriggered request tasks according to the number of parallel requests and the task execution order. If an exception exists, triggering the read requests corresponding to the at least two untriggered request tasks is stopped; otherwise, the read requests corresponding to the at least two untriggered request tasks are triggered. When the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, it can be checked whether a read error occurred at the end of any one of the read requests. If a read error exists, triggering the waiting write requests is stopped. After all triggered read and write requests have finished responding, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Triggering the read and write requests corresponding to the unprocessed valid data segments is then stopped. If no read error occurs, a pre-detection is performed when all read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding. If an exception exists, triggering the waiting write requests is stopped, and it is determined that the processing of the currently processed valid data segment has ended. Triggering the unprocessed valid data segments is then stopped. If everything is normal, the untriggered write requests can be triggered.

[0199] Specifically, upon completion of a read request response for any given request task, a check can be performed to determine if a read error occurred. If a read error is found, triggering the next waiting read request task's corresponding read request is stopped. After all triggered read and write requests have completed, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Triggering read and write requests corresponding to unprocessed valid data segments is then stopped. If no read error occurs, and the preceding checks are successful, the next request task's corresponding read request is triggered.

[0200] For example, still assuming the first storage area includes three valid data segments (valid data segment 0, valid data segment 1, and valid data segment 2), the three valid data segments are divided into multiple data segments according to a predetermined size, generating multiple task lists corresponding to the data segments. The task lists include the request tasks corresponding to each data segment. For example, valid data segment 0 can be divided into four data segments, which correspond to request tasks IO0, IO1, IO2, and IO3, respectively; valid data segment 1 can be divided into three data segments, which correspond to request tasks IO4, IO5, and IO6, respectively; and valid data segment 2 can be divided into three data segments, which correspond to request tasks IO7, IO8, and IO9, respectively. For example, the number of parallel requests can be set to 5. Initially, 5 read requests can be triggered in parallel: IO0-IO3 corresponding to valid data segment 0 and IO4 corresponding to valid data segment 1. The number of currently triggered read request tasks is counted as the number of parallel requests, which is 5. The system continues to monitor. When the read request corresponding to IO0 finishes responding, the number of currently triggered read request tasks is decremented by 1, and the number of request tasks is updated to 4. If it is detected that the number of read request tasks 4 cannot reach the number of parallel requests 5, then the next untriggered request task corresponding to read request IO5 is triggered. The number of currently triggered read request tasks will be increased from 4 to 5. The number of currently triggered read request tasks is counted as 5. If it is detected that the number of read request tasks has reached the number of parallel requests 5, the system continues to monitor until it is detected that the number of read request tasks cannot reach the number of parallel requests, at which point the next untriggered request task corresponding to read request is triggered.

[0201] Before the initial parallel triggering of read requests corresponding to IO0-IO4, a pre-detection check can be performed. If the pre-detection check is normal, the read requests corresponding to IO0-IO4 are triggered in parallel. Then, it checks whether any read request corresponding to IO0-IO3 within valid data segment 0 has a read error upon completion. If a read error exists, the triggering of write requests corresponding to IO0-IO3 within valid data segment 0, the write request corresponding to IO4 within valid data segment 1, and the read and write requests corresponding to IO5 and IO6 are stopped. After the read requests corresponding to IO0-IO4 complete their responses, an exception event is generated, and it is determined that processing of valid data segment 0 and valid data segment 1 is complete. The triggering of read and write requests corresponding to the unprocessed request task in valid data segment 2 is stopped. If no read error exists, after all read requests corresponding to IO0-IO3 have completed their responses, a pre-detection check is performed. If the pre-detection check is normal, the write request corresponding to IO0 can be triggered. Specifically, it can detect whether a read error exists at the end of the read request response corresponding to IO0. If a read error exists, it stops triggering any untriggered read and write requests corresponding to valid data segment 0 and valid data segment 1. When the read request responses corresponding to IO0-IO4 end, and the write request response corresponding to IO0 (which may have already been triggered) ends, it generates exception events and determines that the processing of valid data segment 0 and valid data segment 1 is complete. It then stops triggering read and write requests corresponding to the unprocessed request task corresponding to valid data segment 2. If the read request response corresponding to IO0 ends without a read error, and the preceding checks are normal, it triggers the read request corresponding to the next request task, IO5.

[0202] This embodiment, based on the number of parallel requests and the task execution order, can trigger read requests corresponding to at least two request tasks without waiting for the completion of the previous request task. This allows for the parallel processing of multiple read requests. Furthermore, after the read requests corresponding to multiple request tasks belonging to the same valid data segment have finished responding, write requests corresponding to multiple request tasks are triggered sequentially to ensure sequential writing. This achieves parallel processing of multiple read requests, which, compared to serial processing, improves throughput, fully utilizes processing resources, thereby improving garbage collection efficiency, storage device utilization, and the performance and availability of the storage device. In addition, based on the statistics and comparison of the number of read request tasks and the number of parallel requests, after the read request corresponding to any triggered request task has finished responding, the next untriggered request task's read request can be triggered, ensuring that the number of read request tasks reaches the number of parallel requests, thus fully utilizing processing resources and improving throughput.

[0203] In some embodiments, the method may further include: counting the number of currently triggered write request tasks; updating the number of write request tasks when the write request response corresponding to any triggered request task has ended; detecting whether the number of write request tasks is equal to zero; and, if the number of write request tasks is equal to zero, triggering the write request corresponding to the next triggered request task according to the task execution order.

[0204] For example, by counting the number of currently triggered write request tasks, the number of write request tasks can be limited to no more than 1.

[0205] Specifically, if a write request response for any given task has ended, the system can determine whether a write error exists based on the execution result of the write request. If a write error exists, triggering the next write request is stopped. After the responses to all triggered write and read requests corresponding to the currently processed valid data segment have ended, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Furthermore, the existence of the exception event can be detected, and triggering the corresponding read and write requests for the unprocessed valid data segments can be stopped. If no write error exists, a pre-detection process can be performed. If the pre-detection result is normal, the next write request can be triggered.

[0206] By counting the number of write request tasks, the system can trigger the next write request task after any given task completes. Read request triggering is independent of write request triggering, allowing for parallel processing of read and write requests. This avoids the problems of read / write sequential processing that can cause significant bandwidth fluctuations and latency due to mutual waiting, improving data processing efficiency, ensuring efficient garbage collection, and ultimately increasing storage device utilization, reducing write latency, and enhancing storage device performance and availability.

[0207] In some embodiments, as described above, the storage engine runs in user mode, and the read or write requests triggered by the storage engine need to be sent to the storage device for the storage device to respond to. Therefore, the read requests corresponding to the at least two triggered and untriggered request tasks may include: sending the read requests corresponding to the at least two untriggered request tasks to the storage device so that the storage device can read the data segments corresponding to the read requests and write them into memory.

[0208] The storage device can read the data segment corresponding to the read request and write it into memory based on the read request information. The read request information may include, for example, storage address information such as a second storage area identifier, offset, and length. The storage device can read the corresponding data segment based on the storage address information and write it into memory. The storage device can then feed back the memory address of the data segment in memory to the storage engine.

[0209] The above-mentioned sequential triggering of at least two write requests corresponding to the request tasks may include: sequentially sending the write requests corresponding to at least two request tasks to the storage device, so that the storage device can read the data segments corresponding to the write requests from memory and write them into the second storage area.

[0210] The storage device can read the data segment corresponding to the write request and write it to the second storage area based on the write request information. The write request information may include, for example, the second storage area identifier and memory address. The storage device can determine the data segment to be retrieved from memory based on the memory address, and determine the corresponding second storage area to write the data segment based on the second storage area identifier.

[0211] In this embodiment, the storage device can be managed by a kernel-mode device driver, while the technical solution can be executed by a storage engine running in user mode. When the user-mode storage engine needs to read or write data to the storage device, it can send read / write requests to the kernel. After receiving these read / write requests, the kernel can interact with the storage device through the corresponding device driver to enable the storage device to perform actual read / write operations.

[0212] In some embodiments, the method may further include: if the request tasks corresponding to the multiple data segments are successfully processed, switching the state of the first storage area to a garbage state in order to perform an erasure operation.

[0213] In this scenario, once the request tasks corresponding to multiple data segments have been processed, all valid data in the first storage area is migrated to the second storage area. The first storage area contains only invalid data. In the garbage state, all invalid data in the first storage area can be erased, and the first storage area can be reset to the Free state, becoming a writable storage space. By reclaiming the space that stores invalid data, more available space is provided for new data writing, which can improve the utilization of the storage device, reduce write latency, and improve the performance and availability of the storage device.

[0214] For ease of understanding, Figure 5 This diagram illustrates a scenario interaction of an embodiment of this application in a practical application. The technical solution of this embodiment can be applied to garbage collection of storage devices involved in a distributed storage system. The distributed storage system can consist of multiple storage servers, and each storage server can deploy at least one storage device, and a storage engine can be deployed to manage the at least one storage device. Figure 5 As shown, storage server 500 can deploy storage engine 501 and multiple storage devices. The storage engine can run in user space, while the storage devices run in kernel space. In practical applications, the storage server can be implemented as a physical machine.

[0215] For ease of description, let's take storage device 502, one of multiple storage devices, as an example. See [link to documentation]. Figure 4 The processing flowchart shown illustrates that storage engine 501 can determine a first storage area in storage device 502 where the proportion of invalid data reaches a predetermined percentage, and select a second storage area that is in an idle state. It then divides at least one valid data segment in the first storage area into multiple data segments and generates request tasks corresponding to each data segment, thereby obtaining a task list consisting of multiple request tasks. Storage engine 501 can trigger read requests from at least two request tasks according to the number of parallel requests and the task execution order. For example, storage engine 501 can initially trigger a number of parallel read requests. Specifically, the read requests are sent to storage device 502, which can simultaneously respond to multiple read requests and read the corresponding data segments and write them into memory.

[0216] Storage engine 501 can sequentially send write requests corresponding to multiple request tasks to storage device 502 after the read request responses for multiple request tasks belonging to the same valid data segment have ended. Storage device 502 can read the corresponding data segment from memory and write it into the second storage area according to the request information of the write request.

[0217] Storage engine 501 can initially trigger a number of concurrent read requests. It can detect when the number of read request tasks reaches the number of concurrent requests and continue detecting. If the response to a read request corresponding to any triggered request task ends, the number of read request tasks can be decremented by 1. If the number of read request tasks cannot reach the number of concurrent requests, the next untriggered request task's corresponding read request can be triggered, and the number of read request tasks can be incremented by 1. If the number of read request tasks reaches the number of concurrent requests, the detection can continue.

[0218] Storage engine 501 can trigger a write request corresponding to the first triggered request task among multiple triggered request tasks when the read request responses for multiple request tasks belonging to the same valid data segment have ended. Storage engine 501 can count the number of currently triggered write request tasks as 1. When the write request response for any triggered request task ends, it updates the number of write request tasks to 0. If the number of write request tasks is 0, it will trigger the write request corresponding to the next triggered request task according to the task execution order.

[0219] Specifically, a pre-detection process can be performed before triggering read requests corresponding to at least two untriggered request tasks, based on the number of parallel requests and the task execution order. If an anomaly exists, triggering read requests corresponding to the at least two untriggered request tasks is stopped; otherwise, triggering read requests corresponding to the at least two untriggered request tasks is performed. When multiple read requests belonging to the same valid data segment have finished responding, a check can be performed to determine if a read error occurred at the end of any one of the read requests. If a read error exists, triggering pending write requests is stopped. After all triggered read and write requests have finished responding, an exception event is generated, and it is determined that the currently processed valid data segment has finished processing. Triggering read and write requests corresponding to unprocessed valid data segments is then stopped. If no read error occurs, a pre-detection process is performed after all read requests belonging to the same valid data segment have finished responding. If an anomaly exists, triggering pending write requests is stopped, and it is determined that the currently processed valid data segment has finished processing. Triggering unprocessed valid data segments is then stopped. If everything is normal, untriggered write requests can be triggered.

[0220] Specifically, upon completion of a read request response for any given request task, a check can be performed to determine if a read error occurred. If a read error is found, triggering the next waiting read request task's corresponding read request is stopped. After all triggered read and write requests have completed, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Triggering read and write requests corresponding to unprocessed valid data segments is then stopped. If everything is normal, and the preceding checks are successful, the next request task's corresponding read request is triggered.

[0221] Specifically, if a write request response for any given task has ended, the execution result of the write request can be used to determine if a write error exists. If a write error exists, triggering the next write request is stopped. After the responses to all triggered write and read requests corresponding to the currently processed valid data segment have ended, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. Furthermore, the existence of the exception event can be detected, and triggering read and write requests corresponding to unprocessed valid data segments is stopped. If no write error exists, pre-detection can be performed. If the pre-detection result is normal, the next write request can be triggered.

[0222] Still assuming the first storage area includes three valid data segments (valid data segment 0, valid data segment 1, and valid data segment 2), the three valid data segments are divided into multiple data segments according to a predetermined size, generating a task list corresponding to each data segment. The task list includes the request tasks corresponding to each data segment. For example, valid data segment 0 can be divided into four data segments, which correspond to request tasks IO0, IO1, IO2, and IO3, respectively. Valid data segment 1 and valid data segment 2 can be divided into multiple data segments, which correspond to request tasks IO4, IO5, etc. For example, the number of parallel requests can be set to 5. Initially, 5 read requests can be triggered in parallel: IO0-IO3 corresponding to valid data segment 0 and IO4 corresponding to valid data segment 1. The number of currently triggered read request tasks is counted as the number of parallel requests, which is 5. The system continues to monitor. When the read request corresponding to IO0 finishes responding, the number of currently triggered read request tasks is decremented by 1, and the number of request tasks is updated to 4. If it is detected that the number of read request tasks 4 cannot reach the number of parallel requests 5, then the next untriggered request task corresponding to read request IO5 is triggered. The number of currently triggered read request tasks will be increased from 4 to 5. The number of currently triggered read request tasks is counted as 5. If it is detected that the number of read request tasks has reached the number of parallel requests 5, the system continues to monitor until it is detected that the number of read request tasks cannot reach the number of parallel requests, at which point the next untriggered request task corresponding to read request is triggered.

[0223] For example, before the initial parallel triggering of read requests corresponding to IO0-IO4, a pre-detection can be performed. If the pre-detection is normal, the read requests corresponding to IO0-IO4 are triggered in parallel. The system checks whether a read error exists after the response of any read request corresponding to IO0-IO3 within valid data segment 0. If a read error exists, the triggering of write requests corresponding to IO0-IO3 within valid data segment 0, the write request corresponding to IO4 within valid data segment 1, and the read and write requests corresponding to IO5 and IO6 are stopped. After the read requests corresponding to IO0-IO4 have finished responding, an exception event is generated, and it is determined that the processing of valid data segment 0 and valid data segment 1 has ended. The triggering of read and write requests corresponding to the unprocessed request tasks in valid data segment 2 is stopped. If no read error exists, after all read requests corresponding to IO0-IO3 have finished responding, a pre-detection is performed. If the pre-detection is normal, the write request corresponding to IO0 can be triggered. Specifically, it can detect whether a read error exists at the end of the read request response corresponding to IO0. If a read error exists, it stops triggering any untriggered read and write requests corresponding to valid data segment 0 and valid data segment 1. When the read request responses corresponding to IO0-IO4 end, and the write request response corresponding to IO0 (which may have already been triggered) ends, it generates exception events and determines that the processing of valid data segment 0 and valid data segment 1 is complete. It then stops triggering read and write requests corresponding to the unprocessed request task corresponding to valid data segment 2. If the read request response corresponding to IO0 ends without a read error, and the preceding checks are normal, it triggers the read request corresponding to the next request task, IO5.

[0224] Specifically, it can detect whether a write error occurred upon the completion of the write request response corresponding to IO0. If a write error exists, it stops triggering any untriggered read and write requests within the currently processed valid data segment. After the responses of the triggered read and write requests corresponding to the currently processed valid data segment have ended, an exception event can be generated to confirm the completion of processing for the valid data segments corresponding to the triggered read and write requests, and to stop triggering read and write requests for the request tasks corresponding to the unprocessed valid data segments. If the write request response corresponding to IO0 does not have a write error, and the preceding checks are normal, it triggers the write request corresponding to the next request task, IO1.

[0225] This embodiment performs pre-checks before triggering any read request or write request, enabling the early detection of potential problems and avoiding unnecessary read and write operations, thereby saving processing resources. If any read request or write request encounters a read error or write error, triggering of untriggered read and write requests can be stopped immediately, without waiting for all triggered read and write requests to respond and for an abnormal event to be confirmed before stopping untriggered read and write requests. This allows for timely response to errors, reduces latency, prevents error accumulation, avoids further resource waste, and improves resource utilization.

[0226] By counting the number of read request tasks, the number of read request tasks can reach the number of parallel requests, allowing multiple read requests to be processed in parallel to fully utilize processing resources and improve throughput. By counting the number of write request tasks, the next triggered write request task can be triggered after the completion of the write request corresponding to any triggered request task. The triggering of read requests and write requests is independent, enabling parallel processing of read and write requests. This avoids the problems of read / write serial processing that can cause drastic fluctuations in throughput bandwidth and latency due to mutual waiting between read and write operations, thus improving data processing efficiency, ensuring garbage collection efficiency, improving storage device utilization, reducing write latency, and ultimately improving the performance and availability of the storage device.

[0227] In this scenario, once the request tasks corresponding to multiple data segments in the first storage area are successfully processed (meaning all valid data in the first storage area has been read into the second storage area), the state of the first storage area can be switched to a garbage state to perform an erase operation. After erasing all invalid data in the first storage area, it can be reset to a free state, becoming writable storage space. By reclaiming the space used to store invalid data, more available space is provided for new data writing, improving storage device utilization, reducing write latency, and enhancing the performance and availability of the storage device.

[0228] This embodiment implements parallel processing of multiple read requests and parallel processing of read and write requests. Compared with serial processing, it can improve throughput, make full use of processing resources, thereby improving garbage collection efficiency, improving storage device utilization, and improving storage device performance and availability.

[0229] Figure 6 This application provides a schematic diagram of the structure of a data processing apparatus according to one embodiment. The apparatus includes:

[0230] The determining module 601 is used to determine the first storage area and the second storage area in the storage device;

[0231] The generation module 602 is used to divide at least one valid data segment in the first storage area into multiple data segments, and to generate request tasks corresponding to the multiple data segments respectively.

[0232] The first triggering module 603 is used to trigger the read requests corresponding to at least two untriggered request tasks according to the task execution order.

[0233] The second triggering module 604 is used to sequentially trigger the write requests corresponding to the multiple request tasks when the read request responses corresponding to the multiple request tasks belonging to the same valid data segment have ended.

[0234] Detection module 605 is used to detect whether there are abnormal events;

[0235] The stop triggering module 606 is used to stop triggering untriggered read and write requests in the event of an abnormal event.

[0236] In some embodiments, the first triggering module may, in the absence of any abnormal events, trigger the read requests corresponding to at least two untriggered request tasks according to the task execution order.

[0237] In some embodiments, the device may further include:

[0238] The first exception generation module is used to determine whether a read error exists based on the execution result of the read request when any read request response ends; to determine whether a write error exists based on the execution result of the write request when any write request response ends; and to generate an exception event if a read error or a write error exists.

[0239] In some embodiments, the device may further include:

[0240] The second exception generation module is used to perform at least one of the following detection operations before triggering any read request or any write request: whether there is a status exception in the first storage area, whether there is a status exception in the second storage area, whether the remaining space in the second storage area meets the write conditions, and whether there is a status exception in any valid data segment; and to generate an exception event if there is a status exception in the first storage area, a status exception in the second storage area, the remaining space in the second storage area does not meet the write conditions, or any valid data segment has a status exception.

[0241] In some embodiments, the detection module may detect the presence of an abnormal event by detecting whether an abnormal event exists before triggering any read request, before triggering any write request, after the response to any read request has ended, or after the response to any write request has ended.

[0242] In some embodiments, the first exception generation module generates an exception event in the event of a read error or write error, including: stopping the triggering of untriggered read requests and write requests corresponding to the currently processed valid data segment in the event of a read error or write error; generating an exception event after the response of the triggered write request and triggered read request corresponding to the currently processed valid data segment has ended, and determining that the processing of the currently processed valid data segment has ended.

[0243] Accordingly, the stop triggering module can stop triggering untriggered read and write requests in the event of an abnormal event. This can include stopping the triggering of read and write requests corresponding to unprocessed valid data segments in the event of an abnormal event.

[0244] In some embodiments, the device may further include:

[0245] The instruction generation module is used to respond to abnormal events and generate a recycling end instruction after the processing of the currently processed valid data segment is completed; the recycling end instruction is used to instruct the resource release operation.

[0246] In some embodiments, the device may further include:

[0247] The exception execution module is used to respond to exception events and perform exception handling operations after the processing of the currently processed valid data segment has ended.

[0248] The exception handling operation can include one or more of the following implementation methods:

[0249] As one implementation method, if the exception type of the abnormal event is that there is a state abnormality in the second storage area, an exception flag is set for the second storage area, and the second storage area corresponding to the first storage area is reselected, and the step of splitting at least one valid data segment in the first storage area into multiple data segments is returned to continue execution;

[0250] As another implementation method, if the exception type of the abnormal event is that there is a state abnormality in the first storage area, an exception flag can be set for the first storage area.

[0251] As another implementation method, when the exception type of the abnormal event is that there is an abnormal state of the valid data segment, an exception flag can be set for the valid data segment;

[0252] As another implementation method, the state of the second storage area can be switched to the closed state when the exception type of the abnormal event is that the remaining space of the second storage area does not meet the write condition.

[0253] Accordingly, the determination module can determine the first storage area and the second storage area in the storage device by: determining the first storage area in the storage device that has no abnormality flag set, and determining the second storage area in the storage device that has no abnormality flag set and is in the open state.

[0254] In some embodiments, the generation module may divide at least one valid data segment in the first storage area into multiple data segments and generate request tasks corresponding to each of the multiple data segments, including: determining at least one valid data segment in the first storage area; dividing the at least one valid data segment into multiple data segments according to a predetermined size; generating a task list corresponding to the multiple data segments; the task list includes request tasks corresponding to each of the multiple data segments.

[0255] In some embodiments, the device may further include:

[0256] The resource release module is used to release the resources occupied by a request task when the write request response for any triggered request task has ended.

[0257] In some embodiments, the first triggering module triggering the read requests corresponding to at least two untriggered request tasks may include: simultaneously sending the read requests corresponding to at least two untriggered request tasks to the storage device, so that the storage device can read the data segments corresponding to the read requests and write them into memory.

[0258] The second triggering module sequentially triggers write requests corresponding to multiple request tasks, including: sequentially sending the write requests corresponding to the multiple request tasks to the storage device, so that the storage device can read the data segments corresponding to the write requests from the memory and write them into the second storage area.

[0259] In some embodiments, the device may further include:

[0260] The state switching module is used to switch the state of the first storage area to the garbage state when the request tasks corresponding to multiple data segments are successfully processed, so as to perform the erase operation.

[0261] In some embodiments, the second triggering module sequentially triggering the write requests corresponding to the at least two request tasks includes: triggering the write request corresponding to the next request task when the write request response of the previous request task among the multiple request tasks has ended.

[0262] In some embodiments, the first triggering module may trigger the read requests corresponding to the untriggered request tasks according to the task execution order. This may involve triggering the read requests corresponding to at least two untriggered request tasks respectively, according to the number of parallel requests and the task execution order.

[0263] The first triggering module can also be used to trigger a read request for the next untriggered request task when the write request response for any triggered request task has ended.

[0264] In some embodiments, the device may further include:

[0265] The third triggering module is used to trigger the read request corresponding to the next untriggered request task when the read request response of any triggered request task has ended.

[0266] If the read request response of any triggered request task ends, the read request corresponding to the next untriggered request task can be triggered as follows: if the read request response of any triggered request task ends, and the number of request tasks with triggered read requests and untriggered write requests is less than a preset value, then the read request corresponding to the next untriggered request task can be triggered.

[0267] In some embodiments, the first triggering module triggers the read requests corresponding to at least two untriggered request tasks according to the task execution order, including: triggering the read requests corresponding to at least two untriggered request tasks according to the number of parallel requests and the task execution order;

[0268] The device may also include:

[0269] The data statistics module is used to count the number of currently triggered read request tasks; and update the number of request tasks when the write request response corresponding to any triggered request task ends.

[0270] The data detection module is used to detect whether the number of read request tasks has reached the number of parallel requests; if the number of read request tasks has not reached the number of parallel requests, the next read request corresponding to the untriggered request task is triggered according to the task execution order.

[0271] Optionally, the data statistics module can also count the number of currently triggered write request tasks; and update the number of write request tasks when the write request response for any triggered request task has ended.

[0272] The data detection module can also detect whether the number of write request tasks is equal to zero; if the number of write request tasks is equal to zero, the next write request corresponding to the triggered request task will be triggered according to the task execution order.

[0273] Figure 6 The data processing device can perform Figure 1 , Figure 2 ,or Figure 4The implementation principle and technical effects of the data processing method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the data processing device in the above embodiments performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0274] This application also provides a computing device, such as... Figure 7 As shown, the device may include a storage component 701 and a processing component 702;

[0275] The storage component 701 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component to achieve, for example... Figure 1 or Figure 2 or Figure 4 The data processing method described in the illustrated embodiment.

[0276] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0277] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0278] The processing component 702 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0279] Storage component 701 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0280] It should be noted that the aforementioned computing devices can be physical devices or elastic computing hosts provided by cloud computing platforms. They can be implemented as a distributed cluster of multiple servers or terminal devices, or as a single server or a single terminal device.

[0281] It should be noted that the above-mentioned computing devices implement Figure 1 or Figure 2 or Figure 4 In the case of the data processing method described in the illustrated embodiment, it can be a physical device or an elastic computing host provided by a cloud computing platform. It can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device.

[0282] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 or Figure 2 or Figure 4 The data processing method described in the illustrated embodiment. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0283] This application also provides a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program, when executed by a computer, can perform the above-described functions. Figure 1 or Figure 2 or Figure 4 The data processing method described in the illustrated embodiment. In such an embodiment, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the various functions defined in the system of this application.

[0284] In the foregoing embodiments, the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0285] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0286] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0287] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0288] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Determine the first storage region and the second storage region in the storage device; Divide at least one valid data segment in the first storage area into multiple data segments, and generate request tasks corresponding to each of the multiple data segments; According to the task execution order, trigger the read requests corresponding to at least two untriggered request tasks respectively; The read request is used to read the corresponding data segment and write it into memory; the at least two request tasks correspond to one or more valid data segments; When the read request responses for multiple request tasks belonging to the same valid data segment have ended, the write requests corresponding to the multiple request tasks are triggered sequentially. The write request is used to read the corresponding data segment from the memory and write the data segment into the second storage area; Detect for any abnormal events; In the event of an abnormal event, stop triggering any read and write requests that have not yet been triggered.

2. The method according to claim 1, characterized in that, The step of triggering read requests corresponding to at least two non-triggered request tasks according to the task execution order includes: In the absence of any abnormal events, the read requests corresponding to at least two untriggered request tasks will be triggered according to the task execution order.

3. The method according to claim 1, characterized in that, Also includes: If any read request response ends, determine whether a read error exists based on the execution result of the read request; If any write request response ends, determine whether a write error exists based on the execution result of the write request; The exception event is generated in the event of a read error or write error.

4. The method according to claim 1, characterized in that, Also includes: Before triggering any read request or any write request, perform at least one of the following detection operations: whether the first storage area has an abnormal state, whether the second storage area has an abnormal state, whether the remaining space in the second storage area meets the write conditions, and whether any valid data segment has an abnormal state. The abnormal event is generated when there is an abnormal state in the first storage area, an abnormal state in the second storage area, the remaining space in the second storage area does not meet the write conditions, or any valid data segment has an abnormal state.

5. The method according to any one of claims 1 to 4, characterized in that, The detection of whether there are abnormal events includes: Before triggering any read request, before triggering any write request, after the response to any read request has ended, or after the response to any write request has ended, check for any abnormal events.

6. The method according to claim 3, characterized in that, The generation of the exception event in the event of a read error or write error includes: In the event of a read error or write error, stop triggering any untriggered read or write requests corresponding to the currently processed valid data segment; After the triggered write request and triggered read request corresponding to the currently processed valid data segment have finished responding, an exception event is generated, and it is determined that the processing of the currently processed valid data segment has ended. The step of stopping the triggering of untriggered read and write requests in the event of an abnormal event includes: In the event of an abnormal event, stop triggering read and write requests corresponding to unprocessed valid data segments.

7. The method according to claim 1, characterized in that, Also includes: In response to the abnormal event, and after the processing of the currently processed valid data segment has ended, a recycling end command is generated; the recycling end command is used to instruct the resource release operation.

8. The method according to claim 1, characterized in that, Also includes: In response to the abnormal event, and after the processing of the currently processed valid data segment has ended, an abnormal handling operation is performed.

9. The method according to claim 8, characterized in that, The exception handling operation includes one or more of the following implementation methods: If the anomaly type of the abnormal event is that the second storage area has an abnormal state, an anomaly flag is set for the second storage area, and the second storage area corresponding to the first storage area is reselected. Then, the step of dividing at least one valid data segment in the first storage area into multiple data segments is returned to continue execution. If the anomaly type of the abnormal event is that the first storage area has a state anomaly, an anomaly flag is set for the first storage area; If the anomaly type of the abnormal event is that there is an abnormal state in the valid data segment, an anomaly flag is set for the valid data segment; If the exception type of the abnormal event is that the remaining space in the second storage area does not meet the write condition, the state of the second storage area will be switched to the closed state.

10. The method according to claim 1, characterized in that, The determination of the first storage region and the second storage region in the storage device includes: A first storage area in the storage device that is not marked with an anomaly flag is identified, and a second storage area in the storage device that is not marked with an anomaly flag and is in an open state is identified.

11. The method according to claim 1, characterized in that, The sequential triggering of the write requests corresponding to the plurality of request tasks includes: If the write request response for any triggered request task has ended, then the write request for the next triggered request task will be triggered.

12. The method according to claim 11, characterized in that, The step of triggering read requests corresponding to at least two non-triggered request tasks according to the task execution order includes: Based on the number of parallel requests and the order of task execution, trigger the read requests corresponding to at least two untriggered request tasks respectively; The method further includes: If the write request response for any triggered request task has ended, then the read request for the next untriggered request task will be triggered.

13. The method according to claim 1, characterized in that, Also includes: If the read request response for any triggered request task ends, then the read request for the next untriggered request task will be triggered.

14. The method according to claim 13, characterized in that, The step of triggering a read request for the next untriggered request task after the read request response for any triggered request task has ended includes: If the read request response for any triggered request task ends, and the number of request tasks that have triggered read requests and have not triggered write requests is less than a preset value, then the read request for the next untriggered request task will be triggered.

15. The method according to claim 1, characterized in that, The step of triggering read requests corresponding to at least two non-triggered request tasks according to the task execution order includes: Based on the number of parallel requests and the order of task execution, trigger the read requests corresponding to at least two untriggered request tasks respectively; The method further includes: Count the number of currently triggered read request tasks; If the read request response for any triggered request task ends, update the number of read request tasks. Detect whether the number of read request tasks has reached the number of parallel requests; If the number of read request tasks has not reached the number of parallel requests, the next read request corresponding to the untriggered request task will be triggered according to the task execution order.

16. The method according to claim 1, characterized in that, Also includes: Count the number of write request tasks that have been triggered. If the write request response for any triggered request task ends, update the number of write request tasks. Check if the number of write request tasks is equal to zero; If the number of write request tasks is zero, the next write request corresponding to the already triggered request task will be triggered according to the task execution order.

17. The method according to claim 1, characterized in that, Also includes: When the write request response for any triggered request task ends, the resources occupied by the request task are released.

18. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the data processing method as described in any one of claims 1 to 17.

19. A computer storage medium, characterized in that, The device contains a computer program that, when executed by a computer, implements the data processing method as described in any one of claims 1 to 17.

20. A computer program product, characterized in that, It includes a computer program / instruction, which, when executed by a computer, implements the data processing method as described in any one of claims 1 to 17.

Images