Data processing method and electronic equipment

Abstract

The invention provides a data processing method and electronic equipment, and the method comprises the steps: setting a redundant array of independent disks, and configuring the size of an actual strip and a grouping adjustment index of the actual strip, the grouping adjustment index being a non-negative integer; based on the grouping adjustment index, the grouping rule of the actual strips is determined, the actual strips belonging to the same group form a logic strip, and the addresses of the actual strips in the same logic strip are continuous; acquiring a data volume corresponding to the target task, and updating the grouping adjustment index based on the data volume; based on the updated grouping adjustment index and the grouping rule, the size of a logic strip is determined, and the size of the logic strip is determined through operation of the product of the size of an actual strip and the second power of the grouping adjustment index of 2; and in response to the data reading request, reading all the logic strips from the corresponding single hard disk, and in response to the data writing request, modifying the corresponding strips of the corresponding hard disk.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method and electronic device. Background Technology

[0002] In a Redundant Array of Independent Disks (RAID) that supports striping, the stripe size has a significant impact on data read and write operations. The stripe size is fixed when the volume is created, but a fixed stripe size cannot adapt to different workload requirements. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method and an electronic device.

[0004] The embodiments of this application adopt the following technical solution: a data processing method, comprising: Configure a redundant array of independent disks, and configure the actual stripe size and the grouping adjustment index of the actual stripe, wherein the grouping adjustment index is a non-negative integer; Based on the grouping adjustment index, the grouping rules of the actual stripes are determined, wherein actual stripes belonging to the same group form a logical stripe, and the addresses of actual stripes within the same logical stripe are contiguous; Obtain the amount of data corresponding to the target task, and update the grouping adjustment index based on the amount of data; The size of the logical stripe is determined based on the updated grouping adjustment index and the grouping rules, wherein the size of the logical stripe is determined by the product of the actual stripe size and 2 raised to the power of the grouping adjustment index; In response to a read data request, all logical stripes are read from the corresponding single hard disk; in response to a write data request, the corresponding stripe on the corresponding hard disk is modified.

[0005] In some embodiments, configuring the actual strip size and grouping adjustment index includes: Configure the actual strip size based on the target task scenario; The grouping adjustment index is configured as n, where the initial value of n is a non-negative integer adapted to the actual strip size.

[0006] In some embodiments, obtaining the data volume corresponding to the target task and updating the grouping adjustment index based on the data volume includes: The amount of data corresponding to the target task is obtained periodically, and the growth rate of the data amount is determined. When the growth rate of the data volume reaches a first set threshold, the group adjustment index is updated to a first group adjustment index, wherein the first group adjustment index is larger than the current group adjustment index.

[0007] In some embodiments, the method further includes: If the growth rate of the data volume does not reach the first set threshold, determine whether the current period is the first statistical period; If the current period is the first statistical period, the group adjustment index will be set to 0; If the current period is not the first statistical period, the current grouping adjustment index shall be maintained.

[0008] In some embodiments, the method further includes: If the growth rate of the data volume is negative and the negative growth rate reaches a second set threshold, the group adjustment index will be updated to a second group adjustment index, wherein the second group adjustment index is smaller than the current group adjustment index.

[0009] In some embodiments, determining the grouping rules for the actual stripes based on the grouping adjustment index includes: The grouping rule is to group the actual stripes according to the power of 2.

[0010] In some embodiments, reading all the logical stripes from the corresponding single hard disk in response to a read data request includes: In response to a read data request, determine the stripe index corresponding to the read data request; Based on the stripe index, read the data of the logical stripe corresponding to the stripe index.

[0011] In some embodiments, modifying the corresponding stripe on the corresponding hard disk in response to a write data request includes: In response to a write data request, determine whether the data of the logical stripe corresponding to the write data request is the data corresponding to the configured actual stripe; If the data of the logical stripe is not the actual configured stripe, the data is validated. The verified data are merged, and the data corresponding to the logical stripe is updated.

[0012] In some embodiments, the method further includes: After updating the data corresponding to the logical stripe, determine the stripe index affected by the update of the data corresponding to the logical stripe; Update the affected stripe index.

[0013] This application also provides an electronic device, including a memory and a processor, wherein the memory stores an executable program, and the processor is used to execute: Configure a redundant array of independent disks, and configure the actual stripe size and the grouping adjustment index of the actual stripe, wherein the grouping adjustment index is a non-negative integer; Based on the grouping adjustment index, the grouping rules of the actual stripes are determined, wherein actual stripes belonging to the same group form a logical stripe, and the addresses of actual stripes within the same logical stripe are contiguous; Obtain the amount of data corresponding to the target task, and update the grouping adjustment index based on the amount of data; The size of the logical stripe is determined based on the updated grouping adjustment index and the grouping rules, wherein the size of the logical stripe is determined by the product of the actual stripe size and 2 raised to the power of the grouping adjustment index. In response to a read data request, all logical stripes are read from the corresponding single hard disk; in response to a write data request, the corresponding stripe on the corresponding hard disk is modified. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A diagram of a RAID5 array including four hard drives; Figure 2 This is a flowchart illustrating the RAID creation process for this application; Figure 3 This is a flowchart of the data processing method in this application; Figure 4 A flowchart illustrating the grouping adjustment index update mechanism for this application; Figure 5 This diagram illustrates the logic striping setup when the grouping adjustment index is 1 in this application. Figure 6 This diagram illustrates the logic striping setup when the grouping adjustment index for this application is 2. Detailed Implementation

[0016] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0017] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0018] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0019] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0020] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0021] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0022] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0023] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0024] First, let's introduce the technical terms and application scenarios involved in this application: Redundant RAID (Redundant Array of Independent Disks) is a technology that combines multiple independent physical hard drives into a virtual storage device through hardware (such as a RAID controller) or software logic. Multiple independent physical hard drives can process input / output requests in parallel, improving storage performance (such as read / write speed) while ensuring data reliability through redundancy mechanisms, preventing data loss due to the failure of a single hard drive. There are different types of RAID, such as RAID 0 (no redundancy), RAID 1 (mirrored backup, reliability priority), and RAID 5 (distributed parity, a balance between performance and reliability). Figure 1This shows a RAID 5 array. RAID 5, RAID 6 (dual parity, higher reliability), etc.

[0025] A stripe is the basic data unit used in RAID technology to split and distribute data. It can split continuous data to be stored into multiple data blocks of a fixed size (i.e., stripe size, such as 64KB or 128KB), and then distribute these data blocks sequentially to different physical hard drives in the RAID.

[0026] A stripe group is composed of stripes at corresponding locations on all hard drives in a redundant array of independent disks (e.g., Figure 1 As shown, a RAID 5 array has 4 hard drives. The first stripe of each hard drive together forms a stripe group. A RAID 5 array contains 3 data stripes and 1 parity stripe. The 3 data stripes correspond to 3 data blocks, and the 1 parity stripe corresponds to 1 parity block.

[0027] In a redundant array of independent disks that supports striping, the stripe size is fixed when the volume (array) is created. For example... Figure 2 As shown, volume creation is a fundamental operation in programming, and an array is a data structure used to store multiple elements of the same type. For reading and writing smaller files, a fixed stripe size leads to redundant metadata. During the write process, even if only part of the data changes, the entire stripe needs to be rewritten, resulting in redundant write amplification. For reading and writing larger files, a single input / output operation can only process a small amount of data, impacting data processing efficiency. Some solutions involve pre-assessing business load requirements and using an appropriate stripe size, but this approach still suffers from the problem of the stripe size being difficult to change once set, lacking flexibility.

[0028] Therefore, this application proposes a data processing method that, unlike the solutions mentioned above, can adjust the size of the logical stripes according to the load requirements, thereby improving data read and write performance and achieving the effect of flexibly meeting load requirements.

[0029] Combination Figure 3 This application provides a data processing method, which may include the following steps: S10, Set up an independent disk redundancy array, configure the actual stripe size and the grouping adjustment index of the actual stripe, wherein the grouping adjustment index is a non-negative integer.

[0030] For example, here, the actual stripe size refers to the size of a single data block split. When setting up a redundant array of independent disks, the actual stripe size can be configured beforehand. For instance, when creating a RAID array, the actual stripe size can be configured through the RAID controller or software. If the business requirement to be processed mainly involves smaller files, the actual stripe size can be set smaller. For example, if the business requirement is office documents, the actual stripe size can be configured to 16KB or 64KB. If the business requirement to be processed mainly involves larger files, the actual stripe size can be set larger. For example, if the business requirement is video, the actual stripe size can be configured to 256KB or 1MB. Of course, the actual stripe size can also be other numerical values; this is merely an example and does not constitute a limitation of the claims.

[0031] Furthermore, this embodiment of the application also configures the grouping adjustment index of the actual stripes when setting up a redundant array of independent disks. The grouping adjustment index is used to determine the grouping rules of the actual stripes, which will be described below, and to adjust the grouping of the actual stripes. Compared to the fixed stripe size of traditional RAID, which cannot adapt to scenarios with changing task data volume, this embodiment of the application dynamically adjusts the size of the logical stripes, which will be described below, by setting the grouping adjustment index of the actual stripes, so that the RAID can better adapt to the data volume of the current task and improve the flexibility of processing the target task. It should be noted that the grouping adjustment index can be 0 or a positive integer greater than 0.

[0032] S20, based on the grouping adjustment index, determine the grouping rules of the actual stripes, wherein actual stripes belonging to the same group form a logical stripe, and the addresses of actual stripes within the same logical stripe are consecutive.

[0033] For example, based on the grouping adjustment index configured in step S10, the grouping rules for the actual stripes are determined, and the actual stripes are grouped according to the determined grouping rules. The grouping rules may be to group the actual stripes according to 2... n The groups are grouped according to the grouping rules, where n is the grouping adjustment index.

[0034] Here, actual stripes grouped into the same group according to the grouping rules form a logical stripe; that is, a logical stripe can include at least one stripe group. The number of stripe groups included in a logical stripe is related to the data volume of the target task, which will be discussed below. When the data volume of the target task is large, the logical stripe includes more stripe groups to enable the data corresponding to the target task to be read / written to multiple hard drives simultaneously, improving the processing efficiency of the target task. Conversely, when the data volume of the target task is small, the logical stripe includes fewer stripe groups to avoid generating redundant metadata when processing the target task.

[0035] The addresses of actual stripes belonging to the same logical stripe are kept continuous. This allows data corresponding to the target task to be split across different physical hard drives according to the stripes corresponding to consecutive addresses, avoiding frequent track switching and improving the speed of data processing for the target task. Furthermore, in RAID 5, RAID 6, and other classes with parity mechanisms, the same logical stripe contains data stripes and parity stripes. Consecutive stripe addresses allow parity data to form a fixed logical association with the corresponding data stripes; for example, a parity stripe can correspond to several adjacent consecutive data stripes. When a hard drive fails, the controller can quickly locate the parity information corresponding to the damaged data stripe and other normal data stripes based on the continuous addresses, and quickly recover the data using the parity algorithm. If the addresses are not continuous, the correspondence between parity data and data stripes will be chaotic, making it difficult to quickly match associated data in case of failure. This not only prolongs recovery time but may also increase the risk of data recovery failure.

[0036] S30, combined Figure 4 Obtain the amount of data corresponding to the target task, and update the grouping adjustment index based on the amount of data.

[0037] For example, the data volume corresponding to a single task in the target application system can be counted, and the grouping and adjustment index can be updated based on the data volume. For instance, methods for obtaining the data volume of the target task include, but are not limited to, counting bytes during read / write operations. The specific method for obtaining the data volume depends on the application scenario of the target task, and no limitation is imposed here.

[0038] After obtaining the data volume for the target task, the adjustment index can be increased or decreased based on historical data changes. For example, if historical data changes indicate that the data volume has increased by 100% within a certain statistical period, the adjustment index can be increased; if historical data changes indicate that the data volume has decreased by 50% within a certain statistical period, the adjustment index can be decreased.

[0039] If the updated grouping adjustment index still cannot meet the data volume processing requirements of the current target task, it can be updated again. The update method can be a one-time adjustment based on changes in the data volume corresponding to the target task within a statistical period. For example, if the historical data change representing the data volume corresponding to the target task increases by 100% within a statistical period, the grouping adjustment index can be increased by 1; conversely, if the historical data change representing the data volume increases by 200% within a statistical period, the grouping adjustment index can be increased by 2. A similar adjustment method is used to decrease and update the grouping adjustment index when the data volume corresponding to the target task decreases.

[0040] Of course, the magnitude of the grouping adjustment index can also correspond to other historical changes in the amount of data. This is only used as an example and does not constitute a limitation on the scope of protection of the claims.

[0041] By continuously and dynamically adjusting the grouping adjustment index based on the changes in the amount of data in the target task, the grouping of the actual stripes is adjusted so that the logical stripes composed of the actual stripes can meet the current data processing requirements of the target task.

[0042] S40, based on the updated grouping adjustment index and the grouping rules, determine the size of the logical stripe, wherein the size of the logical stripe is determined by the product of the actual stripe size and 2 raised to the power of the grouping adjustment index.

[0043] For example, after updating the grouping adjustment index, the current grouping adjustment index can be adapted to the current workload of the target task, so that the size of the logical stripe determined based on the grouping adjustment index can meet the operational requirements of the current data volume of the target task. Based on the grouping rules introduced in step S20, the size of the logical stripe can be determined. The size of the logical stripe is determined by multiplying the actual stripe size by 2 raised to the power of the grouping adjustment index, that is, the size of the logical stripe is the actual stripe size multiplied by 2. n The product of the two. Taking a RAID 0 array consisting of 4 hard drives as an example, the actual stripe size is 4KB. With the group adjustment exponent set to 0, the logical stripe size is 4 × 4KB × 2. 0 That is, the logical stripe size is 16KB; with a grouping adjustment index of 1, the logical stripe size is 4×4KB×2. 1 That is, the logical stripe size is 32KB. Combined with... Figure 1 , Figure 5 and Figure 6 Taking a RAID5 array with 4 hard drives as an example, the actual stripe size is 4KB. With the group adjustment index set to 0, the logical stripe consists of one stripe group, and the logical stripe size is 3 × 4KB × 2. 0 That is, the logical stripe size is 12KB; with a grouping adjustment index of 1, the logical stripe consists of two stripe groups, and the logical stripe size is 3×4KB×2. 1 This means the logical stripe size is 24KB. With a grouping adjustment index of 2, the logical stripe consists of 4 stripe groups, and the logical stripe size is 3 × 4KB × 2. 2 That is, the size of the logical stripe is 48KB.

[0044] S50, in response to a read data request, reads all the logical stripes from the corresponding single hard disk; in response to a write data request, modifies the corresponding stripe of the corresponding hard disk.

[0045] For example, when a read data request is generated, in response to the request, such as a request to read data corresponding to one of the logical stripes, data from all physical hard drives belonging to the same logical stripe can be accessed simultaneously. Taking RAID 0 with three physical hard drives as an example, the actual stripe size is 64KB, and a stripe group consisting of three physical hard drives can store 192KB of data. With a grouping adjustment index of 1, the logical stripe consists of two stripe groups, and the logical stripe size is 384KB. In response to a read data request, the size of the single disk data block read each time is twice the actual stripe size, enabling the simultaneous reading of 384KB of data.

[0046] When a write request is generated, a differential write mechanism can be used in response, writing only the changed portion of the data instead of overwriting the entire data. That is, when it is necessary to modify data already stored in a striped disk array, the original data and the new data can be compared first to determine the differences (differential data), and then the differential data can be written to the corresponding hard drive of the logical stripe.

[0047] Here, when the grouping adjustment index is greater than 0, the presence of logical stripes can improve the speed of writing data. For example, when the grouping adjustment index is 1, the speed of writing data can be twice as fast as before, and when the grouping adjustment index is 2, the speed of writing data can be four times as fast as before.

[0048] By grouping actual stripes according to grouping rules determined by a grouping adjustment index, logical stripes that can be dynamically resized are formed. This enables striped disk arrays with logical stripes to adapt to different workload requirements, reduce the generation of redundant metadata, and improve data processing speed.

[0049] In some embodiments, configuring the actual strip size and grouping adjustment index includes: S101, based on the scenario of the target task, configure the actual strip size.

[0050] For example, in different application scenarios, the actual stripe size configuration will affect the processing effect on the data corresponding to the target task, such as the amount of data processed, the processing speed, and the balance between performance and overhead in the data processing process.

[0051] In high-concurrency scenarios with small data volumes, such as web server scenarios, the primary task is to read web page files, images, and static resources. While there are numerous concurrent requests, the data size of a single request is relatively small (e.g., a web page HTML file is only a few KB in size, and a small image is tens of KB). Random reads are dominant, with fewer write operations, occurring only when updating static resources. In this scenario, the actual stripe size can be configured from 4KB to 16KB, and can be paired with RAID10 or RAID5. For example, a news website's web server uses four hard drives in a RAID5 configuration, and the actual stripe size can be set to 8KB. The website's news pages are composed of multiple small files, and an 8KB stripe can quickly respond to concurrent requests from multiple users for different small files. Small stripes distribute requests from different users across different hard drives, avoiding overloading a single drive, while also matching the size of smaller static resources, reducing invalid data loading during data reading, and improving concurrent access performance.

[0052] In scenarios involving continuous reading of large files, such as audio and video editing, which may involve high-definition video editing, special effects rendering, and audio mixing, the files processed are often tens or even hundreds of gigabytes in size. The read / write operation is primarily sequential, requiring high data transfer rates. Inappropriate stripe configuration can lead to rendering stuttering and other issues. In such scenarios, the actual stripe size can be configured to 128KB to 256KB, and can be paired with RAID 0 or RAID 10. For example, a video editing workstation in a film studio uses six hard drives in a RAID 0 configuration, with an actual stripe size of 128KB. When editing 4K high-definition video, the video data is split into 128KB stripes and distributed across multiple hard drives, allowing simultaneous read and write operations. Too small a stripe size will cause the input / output of a single transaction to be split across too many hard drives, increasing hard drive seek times and controller overhead. Too large a stripe size will reduce parallelism, failing to leverage the advantages of parallel read / write operations across multiple hard drives and leading to increased latency. Large striping can split large files into reasonably sized data blocks, which can then be distributed across multiple hard drives for parallel reading and writing. This reduces the number of data splits, maximizes the utilization of the parallel reading and writing capabilities of multiple disks, reduces data transmission interruptions, and ensures smooth playback and editing of high-definition videos. Furthermore, RAID 0 has no redundancy overhead, which can further improve read and write speeds.

[0053] In different scenarios of the above-mentioned target tasks, the actual strip configuration size can also be other strip sizes. Using this as a reference does not constitute a limitation on the scope of protection of the claims.

[0054] S102, configure the grouping adjustment index as n, where the initial value of n is a non-negative integer adapted to the actual strip size.

[0055] For example, in conjunction with the embodiment of step S101, in a high-concurrency small-file scenario, the initial value of the grouping adjustment exponent n can be set to 0 to increase the number of input / output operations per second. In a scenario of continuous reading of large files, the initial value of the grouping adjustment exponent n can be set to 1 or 2 to increase the throughput of data processing. Of course, the specific value of the initial value of n can also be other values; this is merely an example and does not constitute a limitation on the scope of the claims.

[0056] The group adjustment index can be set at the operating system software level or adjusted by the user according to the needs of the target task. An initial setup interface and dynamic setting of the group adjustment index during operation can be provided in the RAID controller. The RAID controller can enable the setting of virtual logical stripe sizes when configuring RAID on hard drives and the actual stripe size.

[0057] In some embodiments, obtaining the data volume corresponding to the target task and updating the grouping adjustment index based on the data volume includes: S301, periodically obtain the data volume corresponding to the target task, and determine the growth rate of the data volume.

[0058] For example, the data volume corresponding to the target task within a specified time interval can be statistically analyzed, and the data volume of the target task in the current statistical period can be compared with the data volume of the target task in the previous statistical period to determine the changes in the data volume of the target task at different times during execution. The data volume corresponding to the target task may change during execution; the data volume in the current statistical period may increase or decrease compared to the data volume in the previous statistical period. The rates of increase and decrease may also differ. That is, the growth rate of the data volume in this step can correspond to a positive growth rate or a negative growth rate.

[0059] S302, when the growth rate of the data volume reaches a first set threshold, the group adjustment index is updated to a first group adjustment index, wherein the first group adjustment index is larger than the current group adjustment index.

[0060] For example, if the data volume growth rate is positive and reaches a first set threshold, it indicates that the data volume corresponding to the target task has significantly increased by the current statistical period. A grouping adjustment index that is too small cannot adapt to the current data volume of the target task; for example, insufficient data processing speed may cause lag in the target task's execution. In this case, the current grouping adjustment index can be adjusted to become a first grouping adjustment index, which can be larger than the current grouping adjustment index. The first grouping adjustment index can be 1, 2, or other values ​​larger than the current grouping index to ensure that the first grouping adjustment index is suitable for the data volume corresponding to the target task.

[0061] For example, RAID 0 includes four hard drives, with an actual stripe size of 4KB. The current group adjustment index is 0, and the current logical stripe size is 16KB. It can simultaneously read data from four stripes belonging to the same logical stripe on the four hard drives. If the target task's data volume in the previous statistical period is the first data volume, then 16KB of data can be read simultaneously from the four hard drives (4KB from each drive). If the target task's data volume in the current statistical period is the second data volume, and the growth rate of the second data volume compared to the first data volume is 100% (a first set threshold), then the first group adjustment index can be updated to 1. Due to the update of the group adjustment index, the logical stripe size also updates accordingly. The updated logical stripe size is twice the size of the original logical stripe (the logical stripe size is updated to 32KB), allowing simultaneous reading of data from eight stripes on the four hard drives, i.e., simultaneously reading 32KB of data from the four hard drives, resulting in faster data reading speeds. Before and after the grouping adjustment index update, the data read capability (4KB) of each stripe remains the same. However, the increased size of the logical stripes after the update significantly reduces the time required for data reads. This also improves the speed of write operations, which will not be elaborated further here. By updating the grouping adjustment index based on changes in data volume, it is possible to process the current data volume corresponding to the target task more quickly, reduce the number of input / output requests, and decrease data processing latency.

[0062] It should be noted that the length of the statistical period can vary depending on the target task. The duration of the statistical period can be set according to the target task, such as the data type (continuous large files / discrete small data). For continuous large files, the statistical period can be longer than that for discrete small data. For continuous large files, the single input / output volume is large, and the data volume changes infrequently, so there is no need to frequently and dynamically adjust the grouping and adjustment index in a short period of time. For discrete small data, the data volume changes relatively frequently, so the statistical period can be set shorter.

[0063] In some embodiments, the method further includes: S303, if the growth rate of the data volume does not reach the first set threshold, determine whether the current period is the first statistical period.

[0064] For example, after determining the growth rate of the data volume corresponding to the target task in step S301, if the growth rate does not reach the first set threshold, the following situations may exist: the current statistical period is the first statistical period, or the change in data volume is small. It should be noted that there is no requirement for the order of steps S303 and S302.

[0065] Whether a statistical period is the first statistical period can be determined based on period identifiers, time anchors, historical records, etc. For example, some target tasks have a set period start anchor, meaning the start time for the data volume corresponding to the target task can be recorded as `start_time`. The statistical period can be set in minutes, hours, days, etc., and each statistical period can have a corresponding period identifier; for example, period number = 1 indicates it is the first statistical period, period number = 2 indicates it is the first statistical period, and so on. If the target task does not have a set start anchor, it can be determined by querying whether historical period data exists. If the query result is empty, it means the current period is the first statistical period; if historical records exist, it means the current period is not the first statistical period.

[0066] Of course, it is understandable that other methods can be used to determine whether the current statistical period is the first statistical period.

[0067] S304, when the current period is the first statistical period, set the group adjustment index to 0.

[0068] For example, when the current period is the first statistical period, the grouping adjustment index can be set to 0. This means that a logical stripe group is formed, and the size of the logical stripe is the sum of the sizes of all stripes within the same group. As the target task is executed, the adjustment of the grouping adjustment index is determined based on changes in the amount of data corresponding to the target task.

[0069] S305, if the current period is not the first statistical period, maintain the current grouping adjustment index.

[0070] For example, if the current period is not the first statistical period, it indicates that the amount of data corresponding to the target task has not changed significantly as the target task is executed. Therefore, the grouping adjustment index does not need to be adjusted; the current grouping adjustment index can be maintained. For instance, if the current period is the second statistical period and the current grouping adjustment index n is 1, then n can remain at 1 without any corresponding modification to the grouping adjustment index. In the third statistical period, the grouping adjustment index n will still be 1, and the logical stripe size will remain the same as in the second statistical period. This reduces the modification operations on the grouping adjustment indication at the system software layer, thus reducing the consumption of system software layer resources.

[0071] In some embodiments, the method further includes: If the growth rate of the data volume is negative and the negative growth rate reaches a second set threshold, the group adjustment index will be updated to a second group adjustment index, wherein the second group adjustment index is smaller than the current group adjustment index.

[0072] For example, if the growth rate of the data volume is negative and the negative growth rate reaches the second set threshold, it indicates that as the target task is executed, the data volume corresponding to the target task is significantly reduced. If the current grouping adjustment index and logical strip size are maintained at this time, it may cause the generation of redundant data and the problem of redundant write amplification.

[0073] For example, RAID 0 includes 4 hard drives, with an actual stripe size of 4KB. The current group adjustment index is 2, and the current logical stripe size is 64KB. It can simultaneously read data from 16 stripes belonging to the same logical stripe on the 4 hard drives. If the target task's data volume in the previous statistical period was the third data volume, then 64KB of data can be read simultaneously from the 4 hard drives (4KB per hard drive). If the target task's data volume in the current statistical period is the fourth data volume, and the growth rate of the fourth data volume compared to the first data volume is -50% (a second set threshold), then the first group adjustment index can be updated to 1. This allows simultaneous reading of data from 8 stripes on the 4 hard drives, i.e., 32KB of data can be read simultaneously from the 4 hard drives. Before and after the group adjustment index update, the data read capability (4KB) of each stripe remains the same. However, after the update, the logical stripe size becomes smaller, eliminating the need to read data from other stripes to meet the current data volume requirement, thus reducing the addressing time for other stripes. For write operations, redundant write amplification can be reduced, which will not be elaborated here.

[0074] In some embodiments, determining the grouping rules for the actual stripes based on the grouping adjustment index includes: The grouping rule is to group the actual stripes according to the power of 2.

[0075] For example, the actual stripe grouping rules are related to the stripe size, which is the data block size of a single stripe on each hard drive in the RAID array. Common values ​​are powers of 2. For example, stripes can only be 4k, 8k, 16k, 64k, etc. The actual stripes are grouped according to powers of 2 so that the logical stripe size conforms to the above stripe size rules.

[0076] In some embodiments, reading all the logical stripes from the corresponding single hard disk in response to a read data request includes: S501, in response to a read data request, determine the stripe index corresponding to the read data request.

[0077] For example, consider a RAID 0 array consisting of 4 hard drives. It has no redundancy, and the actual number of stripes within the same stripe group is the same as the number of hard drives (4 in total). Each actual stripe is 64KB in size, and each stripe group is 256KB (64KB × 4). The logical address (contiguous logical addresses exposed to the outside world) ranges from 0 to N × 256KB (N is the number of stripe groups). The actual stripes are sequentially numbered L0, L1, L2, L3 (corresponding to the first stripe group); L4, L5, L6, L7 (corresponding to the second stripe group). Each actual stripe is mapped to one hard drive: L0 to hard drive 1, L1 to hard drive 2, L2 to hard drive 3, L3 to hard drive 4; L4 to hard drive 1, L5 to hard drive 2, and so on. For example: L0: 0KB~63KB; L1: 64KB~127KB; L2: 128KB~191KB; L3: 192KB~255KB (end of the first stripe group); L4: 256KB~319KB... When an application initiates a read request to read data in the range of 100KB to 200KB, the RAID controller and system software layer can determine the stripe index corresponding to the read data through the mapping relationship between logical addresses and stripe indexes, that is, by comparing the stripe intervals one by one: 100KB~127KB, falling within the L1 address range; 128KB~191KB, falling within the L2 address range; 192KB~200KB, falling within the L3 address range.

[0078] The stripe indices corresponding to this data read request are determined to be L1, L2, and L3.

[0079] S502, based on the stripe index, read the data of the logical stripe corresponding to the stripe index.

[0080] For example, based on the embodiment combined with step 501, after determining the stripe index corresponding to the current read data request, the RAID controller can perform read operations in parallel according to the physical hard disk mapping relationship of the stripe index.

[0081] You can first locate the physical hard drive: Stripe L1 corresponds to hard disk 2; Stripe L2 corresponds to hard disk 3; Stripe L3 corresponds to hard drive 4.

[0082] Parallel reading of data from the physical hard drives located above: The controller simultaneously sends read commands to hard disks 2, 3, and 4. It reads 100KB~127KB of data (28KB total) from L1 on hard disk 2; 128KB~191KB of complete data (64KB total) from L2 on hard disk 3; and 192KB~200KB of data (9KB total) from L3 on hard disk 4. When the group adjustment index n is 0, L0, L1, L2, and L3 belong to the same logical stripe, and corresponding data on hard disks 2 and 3 can be read simultaneously. When the group adjustment index n is 1 or greater than 1, L0, L1, L2, L3, and L4 belong to the same logical stripe, and corresponding data on hard disks 2, 3, and 4 can be read simultaneously (e.g., ...). Figure 5 The diagram shown is a graph showing the relationship between actual and logical stripes when the grouping adjustment index is 2.

[0083] The data read from the three hard drives (hard drive 2, hard drive 3, and hard drive 4) can be concatenated in address order (28KB + 64KB + 9KB = 101KB) to form a complete data block of 100KB to 200KB, which is then returned to the application.

[0084] Of course, the way stripe indexes are set and the way data is read, spliced, and returned may differ for different types of RAID arrays. For example, in a RAID 5 array, a stripe group contains one parity stripe. When reading data, if a hard drive fails, the controller will calculate the data of the failed stripe using the remaining data stripes and the parity stripe, and then splice it back. This involves one more parity calculation than a RAID 0 array, but the logic for determining the stripe index is the same, which will not be elaborated here.

[0085] In some embodiments, modifying the corresponding stripe on the corresponding hard disk in response to a write data request includes: S503, in response to a write data request, determine whether the data of the logical stripe corresponding to the write data request is the data corresponding to the configured actual stripe.

[0086] For example, combined again Figure 1Taking a RAID 5 array consisting of 4 hard drives as an example, a stripe group includes 4 actual stripes, of which 3 are data stripes and 1 is a parity stripe. These 4 actual stripes are located on the 4 hard drives respectively. Each actual stripe is 64KB in size, and the size of a stripe group is 192KB (3 × 64KB). Figure 3 The grouping adjustment index is 1. The mapping relationship between stripes and hard drives is as follows: Stripe group 1 includes: stripe L0 corresponding to hard drive 1, stripe L1 corresponding to hard drive 2, stripe L2 corresponding to hard drive 3, and parity stripe L3 corresponding to hard drive 4. Stripe group 2 includes: parity stripe L4 corresponding to hard drive 4, stripe L5 corresponding to hard drive 1, stripe L6 corresponding to hard drive 2, stripe L7 corresponding to hard drive 3, and so on. Logical stripe 1 includes stripe group 1 and stripe group 2. Stripe group 1's L0-L2 stores data blocks A1-A3 (0-191KB), and L3 stores the corresponding parity value P1.

[0087] The application initiates a write data request, requesting modification of data within the logical address range of 80KB to 120KB (original data is A2, new data is A2'). The write data request address range is split according to the actual stripe size (64KB), with 80KB to 120KB covering L1 (64KB to 127KB, corresponding to hard drive 2). Only one actual stripe, L1, needs modification. The RAID controller's stripe mapping table is queried; the configured actual stripe L1 should map to the 64KB to 127KB physical address range of hard drive 2. The mapping between the write data request's L1 and the configured actual stripe is checked to determine if they match, and thus whether the data in the logical stripe corresponding to the write data request is the same as the data in the configured actual stripe.

[0088] S504, if the data of the logical stripe is not the actual configured stripe, the data is validated.

[0089] For example, continuing with the embodiment of step S503, if the verification finds that the L1 of the write data request does not match the actual stripe mapping configured, for example, the mapping table of the controller cache is abnormal, showing that L1 is mapped to hard disk 3, then the data verification process can be triggered to verify the data.

[0090] If this verification finds that the L1 of the write data request matches the actual stripe mapping configured, then no data verification is required.

[0091] For example, taking the XOR algorithm-based data consistency verification of a RAID5 array as an example, the controller can read all data stripes and parity stripes of stripe group 1: Read L0 (disk 1): 0KB~63KB, data A1; Reading L1 (disk 2): 64KB~127KB, raw data A2 (actual physical address is correct); Reading L2 (disk 3): 128KB~191KB, data A3; Read L3 (disk 4): check stripe, check value P1 (P1 = A1 XOR A2 XOR A3).

[0092] To verify the consistency between the data and the checksum, the checksum can be recalculated: P1' = A1 XOR A2 XOR A3. Compare P1' with the read P1. If they match, the data is undamaged; if they don't match, the erroneous data can be repaired using the checksum (e.g., P1' = P0, the data is undamaged, only the mapping table is abnormal). This allows for the correction of the logical stripe mapping, adjusting the L1 mapping relationship of the controller cache to correspond to 64KB~127KB on disk 2.

[0093] Calculate the checksum of the corrected data. Based on the corrected new data A2', recalculate the new checksum P1'. The old checksum logic was P1 = A1 XOR A2 XOR A3, which leads to A2 = A1 XOR A3 XOR P1. The new checksum logic is P1' = A1 XOR A2' XOR A3, only replacing A2 with A2', without needing to recalculate everything.

[0094] S505, merge the verified data and update the data corresponding to the logical strip.

[0095] For example, the validated data is merged to complete the write update. The data and checksum are merged, and the valid data to be written is organized. The new data is A2' (80KB~120KB, covering part of the data in L1); the new checksum is P1'.

[0096] The update operation writes new data, writing A2' to the 80KB~120KB physical address of hard disk 2, overwriting only the changed part of L1, using differential write optimization. A new parity value is then written, i.e., P1' is written to the L3 parity stripe of hard disk 4.

[0097] Return "Write successful" to the application and update the metadata of stripe group 1, such as record data version, checksum update time, etc.

[0098] In some embodiments, the method further includes: After updating the data corresponding to the logical stripe, determine the stripe index affected by the update of the data corresponding to the logical stripe; Update the affected stripe index.

[0099] For example, in conjunction with the embodiments of steps S503 to S505, in a RAID array with parity, updates to data stripes affect parity stripes within the same stripe group. Based on the logical address of L1 (64KB~127KB), it can be determined that it belongs to stripe group 1. The directly affected stripe is L1, whose data is updated, and its index metadata needs to be marked as a version change. Additionally, the parity stripe L3 is also affected. The data for L0 and L2 remains unchanged, and their indexes do not need to be updated. The affected stripe indices are determined to be L1 and L3.

[0100] The RAID controller maintains a stripe index metadata table, which records the data version, parity version, physical mapping address, and update time for each stripe. The metadata for L1 and L3 needs to be updated synchronously. For example, the following can be updated: the data version number, update time, and "data updated" status for data stripe L1; and the parity version number, parity value, and associated data stripe for the parity stripe.

[0101] It can also synchronize stripe index updates to the controller cache. For example, new metadata for L1 and L3 can be written to the RAID controller cache to ensure that subsequent input / output requests read the latest index first.

[0102] This application also provides an electronic device, including a memory and a processor, wherein the memory stores an executable program, and the processor is used to execute: Configure a redundant array of independent disks, and configure the actual stripe size and the grouping adjustment index of the actual stripe, wherein the grouping adjustment index is a non-negative integer; Based on the grouping adjustment index, the grouping rules of the actual stripes are determined, wherein actual stripes belonging to the same group form a logical stripe, and the addresses of actual stripes within the same logical stripe are contiguous; Obtain the amount of data corresponding to the target task, and update the grouping adjustment index based on the amount of data; The size of the logical stripe is determined based on the updated grouping adjustment index and the grouping rules, wherein the size of the logical stripe is determined by the product of the actual stripe size and 2 raised to the power of the grouping adjustment index. In response to a read data request, all logical stripes are read from the corresponding single hard disk; in response to a write data request, the corresponding stripe on the corresponding hard disk is modified.

[0103] The foregoing has described in detail several embodiments of this application, but this application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications based on the concept of this application, and all such variations and modifications should fall within the scope of protection claimed in this application.

Claims

1. A data processing method, comprising: Configure a redundant array of independent disks, and configure the actual stripe size and the grouping adjustment index of the actual stripe, wherein the grouping adjustment index is a non-negative integer; Based on the grouping adjustment index, the grouping rules of the actual stripes are determined, wherein actual stripes belonging to the same group form a logical stripe, and the addresses of actual stripes within the same logical stripe are contiguous; Obtain the amount of data corresponding to the target task, and update the grouping adjustment index based on the amount of data; The size of the logical stripe is determined based on the updated grouping adjustment index and the grouping rules, wherein the size of the logical stripe is determined by the product of the actual stripe size and 2 raised to the power of the grouping adjustment index; In response to a read data request, all logical stripes are read from the corresponding single hard disk; in response to a write data request, the corresponding stripe on the corresponding hard disk is modified.

2. The method according to claim 1, wherein configuring the actual strip size and grouping adjustment index includes: Configure the actual strip size based on the target task scenario; The grouping adjustment index is configured as n, where the initial value of n is a non-negative integer adapted to the actual strip size.

3. The method according to claim 1, wherein obtaining the data volume corresponding to the target task and updating the grouping adjustment index based on the data volume includes: The amount of data corresponding to the target task is obtained periodically, and the growth rate of the data amount is determined. When the growth rate of the data volume reaches a first set threshold, the group adjustment index is updated to a first group adjustment index, wherein the first group adjustment index is larger than the current group adjustment index.

4. The method according to claim 3, wherein, The method further includes: If the growth rate of the data volume does not reach the first set threshold, determine whether the current period is the first statistical period; If the current period is the first statistical period, the group adjustment index will be set to 0; If the current period is not the first statistical period, the current grouping adjustment index shall be maintained.

5. The data processing method according to claim 3, wherein, The method further includes: If the growth rate of the data volume is negative and the negative growth rate reaches a second set threshold, the group adjustment index will be updated to a second group adjustment index, wherein the second group adjustment index is smaller than the current group adjustment index.

6. The data processing method according to claim 1, wherein determining the grouping rule of the actual stripes based on the grouping adjustment index includes: The grouping rule is to group the actual stripes according to the power of 2.

7. The data processing method according to claim 1, wherein reading all the logical stripes from the corresponding single hard disk in response to a read data request comprises: In response to a read data request, determine the stripe index corresponding to the read data request; Based on the stripe index, read the data of the logical stripe corresponding to the stripe index.

8. The data processing method according to claim 1, wherein modifying the corresponding stripe of the corresponding hard disk in response to a write data request includes: In response to a write data request, determine whether the data of the logical stripe corresponding to the write data request is the data corresponding to the configured actual stripe; If the data of the logical stripe is not the actual configured stripe, the data is validated. The verified data are merged, and the data corresponding to the logical stripe is updated.

9. The data processing method according to claim 7, wherein, The method further includes: After updating the data corresponding to the logical stripe, determine the stripe index affected by the update of the data corresponding to the logical stripe; Update the affected stripe index.

10. An electronic device comprising a memory and a processor, wherein the memory stores an executable program, and the processor is configured to execute: Configure a redundant array of independent disks, and configure the actual stripe size and the grouping adjustment index of the actual stripe, wherein, The grouping adjustment index is a non-negative integer; Based on the grouping adjustment index, the grouping rules of the actual stripes are determined, wherein actual stripes belonging to the same group form a logical stripe, and the addresses of actual stripes within the same logical stripe are contiguous; Obtain the amount of data corresponding to the target task, and update the grouping adjustment index based on the amount of data; The size of the logical stripe is determined based on the updated grouping adjustment index and the grouping rules, wherein the size of the logical stripe is determined by the product of the actual stripe size and 2 raised to the power of the grouping adjustment index. In response to a read data request, all logical stripes are read from the corresponding single hard disk; in response to a write data request, the corresponding stripe on the corresponding hard disk is modified.

Images