A data processing method, computer program product, device and computer medium
By obtaining and processing the calculation parameter list and start address of the disk array in hardware, and extracting the calculation parameter group of the target task and subtask, the problem of poor performance of the disk array during multitasking is solved, and more efficient data processing is achieved.
Patent Information
- Application Number
- CN202410882268.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-03
AI Technical Summary
When existing disk arrays are multitasked, software frequently interacts with hardware, resulting in reduced system performance and difficult to improve system performance for data processing.
By obtaining the calculation parameter list and starting address transmitted by the software, the hardware can determine the calculation parameter group of all disks and extract the calculation parameter group required by the target task and sub-task from it, reducing the number of interactions between the software and the hardware and improving data processing efficiency.
Reduces the frequency of software and hardware interaction, reduces performance overhead, and improves the system performance of data processing by applying disk arrays.
Smart Images

Figure CN118409718B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of storage technology, and more specifically, to a data processing method, a computer program product, an electronic device, and a computer-readable storage medium. Background Art
[0002] Redundant Arrays of Independent Disks (RAID) use data striping, mirroring, and data verification technology to achieve high performance, reliability, fault tolerance, and scalability. It distributes data to each disk in blocks. When writing disk data, it is necessary to obtain the corresponding calculation parameters. For example, the calculation parameters required for each task are analyzed by software and then sent to the hardware for processing. However, when there are multiple tasks, the software needs to interact with the hardware frequently, resulting in reduced system performance.
[0003] In summary, how to improve the system performance of using disk arrays for data processing is a problem that needs to be solved urgently by those skilled in the art. Summary of the invention
[0004] The purpose of the present invention is to provide a data processing method, which can solve the technical problem of how to improve the system performance of data processing using a disk array to a certain extent. The present invention also provides a computer program product, an electronic device and a computer readable storage medium.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] In a first aspect, a data processing method is provided, which is applied to hardware and includes:
[0007] Obtaining a calculation parameter list transmitted by the software, wherein the calculation parameter list is used to record the calculation parameters stored by each disk according to the target disk array;
[0008] Obtaining the starting address of the calculation parameters transmitted by the software;
[0009] Determine a first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameter;
[0010] Extracting a second computing parameter group required for the target task from the first computing parameter group;
[0011] Extracting a target computing parameter group required for each target subtask from the second computing parameter group, wherein the target subtask includes a task obtained after the target task is split;
[0012] The processing result of the corresponding target subtask is determined according to the target calculation parameter group.
[0013] On the other hand, according to the calculation parameter list and the starting address of the calculation parameter, determining the first calculation parameter group corresponding to all disks includes:
[0014] Determine the location of the calibration disk;
[0015] Determining a unit size value of the calculation parameter;
[0016] Determining a calculation parameter address according to the position of the check disk, the starting address of the calculation parameter and the unit size value;
[0017] The first calculation parameter group corresponding to all disks is read from the calculation parameter list according to the calculation parameter address.
[0018] On the other hand, determining the calculation parameter address according to the position of the check disk, the starting address of the calculation parameter and the unit size value includes:
[0019] Subtract one from the position of the calibration disk to obtain a target value;
[0020] Generate a product value of the target value and the unit size value;
[0021] The starting address of the calculation parameter and the product value are added to obtain the calculation parameter address.
[0022] On the other hand, extracting a second computing parameter group required for the target task from the first computing parameter group includes:
[0023] Generate a disk flag for each disk under the target task, wherein the disk flag is used to reflect whether data is written to the disk;
[0024] Determine the disk mark reflecting writing data to the disk as a target mark;
[0025] In the first computing parameter group, the computing parameters of the disk corresponding to the target mark are extracted to obtain the second computing parameter group required for the target task.
[0026] On the other hand, extracting a target computing parameter group required for each target subtask from the second computing parameter group includes:
[0027] Acquire a data pointer of the target subtask, where the data pointer is used to mark the disk to be processed by the target subtask;
[0028] Generate a target identification bit of the disk according to the data pointer, wherein the target identification bit is used to reflect whether the disk is the disk processed by the target subtask;
[0029] The target computing parameter group required for the target subtask is extracted from the second computing parameter group according to the target identification bit.
[0030] On the other hand, generating disk flags of each disk under the target task includes:
[0031] Determining a target writing mode for the target task;
[0032] The disk flag of each disk under the target task is generated according to the target write mode.
[0033] On the other hand, generating the disk flag of each disk under the target task according to the target write mode includes:
[0034] In response to the target write mode being full stripe write, the disk flag of the generated check disk is generated as not writing data, and the disk flag of the generated data disk is generated as writing data.
[0035] On the other hand, generating a target identification bit of the corresponding disk according to the data pointer includes:
[0036] According to the rule that all the data pointers are valid, the target identification bits generated for all disks are all first identification bits, and the first identification bit is used to reflect that the disk is the disk processed by the target subtask.
[0037] On the other hand, generating the disk flag of each disk under the target task according to the target write mode includes:
[0038] In response to the target write mode being read-modify-write, the disk flag for generating the check disk is not to write data, the disk flag for generating the data disk to which data needs to be written is to write data, and the disk flag for generating the data disk to which data does not need to be written is not to write data.
[0039] On the other hand, generating a target identification bit of the corresponding disk according to the data pointer includes:
[0040] Generate the target identification bits of all disks according to the rule that the target identification bits between the modified disk and the disk to which data is written are the same, the target identification bit of the disk to which data is to be written is the first identification bit, and the target identification bits of the remaining disks are the second identification bits;
[0041] The first identification bit is used to reflect that the disk is a disk processed by the target subtask, and the second identification bit is used to reflect that the disk is not a disk processed by the target subtask.
[0042] On the other hand, generating the disk flag of each disk under the target task according to the target write mode includes:
[0043] In response to the target write mode being reconstruction write, the disk flag of the generated check disk is not to write data, and the disk flag of the generated data disk is to write data.
[0044] On the other hand, generating a target identification bit of the corresponding disk according to the data pointer includes:
[0045] Generate the target identification bits of all disks according to the rule that the target identification bits between the modified disk and the disk to which data is written are mutually exclusive, the target identification bit of the disk to be modified and which does not need data to be written is the first identification bit, and the target identification bit of the disk to be modified and which needs data to be written is the second identification bit;
[0046] The first identification bit is used to reflect that the disk is a disk processed by the target subtask, and the second identification bit is used to reflect that the disk is not a disk processed by the target subtask.
[0047] In a second aspect, a computer program product is provided, comprising a computer program / instruction, which implements the steps of any of the above data processing methods when executed by a processor.
[0048] According to a third aspect, an electronic device is provided, including:
[0049] Memory for storing computer programs;
[0050] A processor is used to implement the steps of any of the above data processing methods when executing the computer program.
[0051] According to a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above data processing methods are implemented.
[0052] The present invention provides a data processing method, which is applied to hardware, obtains a calculation parameter list transmitted by software, and the calculation parameter list is used to record the calculation parameters stored by each disk according to the target disk array; obtains the starting address of the calculation parameter transmitted by the software; determines the first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameter; extracts the second calculation parameter group required for the target task from the first calculation parameter group; extracts the target calculation parameter group required for each target subtask from the second calculation parameter group, and the target subtask includes the task obtained after the target task is split; and determines the processing result of the corresponding target subtask according to the target calculation parameter group. The beneficial effect of the present invention is that the software only transmits the calculation parameter list and the starting address of the calculation parameter to the hardware once, so that the hardware can know the calculation parameters stored by each disk according to the target disk array, so that the hardware can use the starting address of the calculation parameter to extract the target calculation parameter group required by the target subtask from the calculation parameter list for data processing. In the whole process, the hardware only needs to interact with the software twice, the interaction frequency between the software and the hardware is low, and the performance overhead is reduced, so that the system performance of using the disk array for data processing can be improved. The computer program product, electronic device and computer-readable storage medium provided by the present invention also solve the corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0054] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention;
[0055] Figure 2 A schematic diagram of the calculation parameter list;
[0056] Figure 3 It is a schematic diagram of the hardware structure;
[0057] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;
[0058] Figure 5 Another structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0060] See also Figure 1 , Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention.
[0061] A data processing method provided by an embodiment of the present invention, applied to hardware, may include the following steps:
[0062] Step S101: obtaining a calculation parameter list transmitted by the software, where the calculation parameter list is used to record the calculation parameters stored in each disk according to the target disk array.
[0063] In actual applications, the software organizes the calculation parameters stored in each disk according to the target disk array (Redundant Arrays of Independent Disks, RAID) into a calculation parameter list, and sends the calculation parameter list to the hardware, so that the hardware can know the corresponding relationship between the disk and the calculation parameter, which is convenient for subsequent application of the calculation parameter list for data processing. Among them, the type of the target disk array, the content of the calculation parameter list, etc. can be flexibly determined according to the application scenario.
[0064] For ease of understanding, it is assumed that the target disk array is RAID5, and the RAID5 stripe distribution is as shown in Table 1, where D1, D2, and Dn are data on different disks, and P is a check disk, where the position of the P disk is not fixed.
[0065] Table 1 RAID5 stripe distribution
[0066]
[0067] Then it satisfies the following operational relationship:
[0068] K1*D1⊕K2*D2⊕K3*D3⊕…⊕Kn*Dn⊕P=0; where * represents Galois Field multiplication, K1, K2, K3…Kn are calculation parameters of Galois multiplication, and have the same meanings in the following text.
[0069] The coding formula is: P=K1*D1⊕K2*D2⊕K3*D3⊕…⊕Kn*Dn;
[0070] The decoding formula is: Dx=Kx^(-1)*(K1*D1⊕K2*D2⊕K3*D3⊕…⊕Kn*Dn⊕P) (excluding the Dx term);
[0071] When any data (disk) of RAID5 is lost, it can be recovered through other data.
[0072] RAID6 adds a check disk Q on the basis of RAID5. Disk P and disk Q are two consecutive disks. There is a special case, that is, when disk P is the last disk, disk Q is the first disk. The stripe distribution of RAID6 is shown in Table 2:
[0073] Table 2 RAID6 stripe distribution
[0074]
[0075] D1, D2, and Dn are data, P and Q are checks, and they satisfy the following calculation relationship:
[0076] P=K11*D1⊕K12*D2⊕K13*D3⊕…⊕K1n*Dn;
[0077] Q=K21*D1⊕K22*D2⊕K23*D3⊕…⊕K2n*Dn;
[0078] According to the principle of solving a set of two-dimensional linear equations, RAID6 can recover the missing data of two disks at most. Assuming that a stripe contains multiple sub-stripes, the actual calculation and verification process uses the sub-strip as the unit, that is, one check is calculated for each sub-strip, as shown in Table 3.
[0079] Table 3 Disk distribution diagram
[0080]
[0081] In Table 3, one stripe is divided into four sub-stripes, and a total of four checks need to be calculated. During the processing, the hardware can set one stripe as a task and one sub-strip as a subtask for processing.
[0082] It should be noted that the hardware in the present invention is used to process data on the disk in a disk array manner, and can be hardware based on digital circuits, chips, etc., and the software can be hardware front-end software, etc. In addition, when the total number of disks is determined to be n, there are n situations for the position of the check disk P. Different positions of the check disks correspond to a set of different calculation parameter distributions, which are n groups in total, as shown in Table 4; n groups of parameters can form a calculation parameter list, such as Figure 2 As shown, the software writes all n groups of calculation parameters into the hardware at one time, such as the continuous space of the on-chip memory of the hardware, and the size of each group of parameters is the same.
[0083] Table 4 Disk and computing parameter distribution table
[0084]
[0085] Step S102: Obtain the starting address of the calculation parameters transmitted by the software.
[0086] In actual applications, after the software transmits the calculation parameter list to the hardware, it also needs to transmit the starting address of the calculation parameter to the hardware. For example, the software can write the starting address of the calculation parameter in the on-chip memory to the configuration register of the hardware, so that the hardware can subsequently apply the starting address of the calculation parameter to extract the calculation parameters required for the task. After that, the software will no longer send the calculation parameters to the hardware, and the hardware will extract the required calculation parameters from the calculation parameter list when executing the task.
[0087] Step S103: Determine the first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameter.
[0088] In actual applications, when the hardware receives a task to be processed, it can first determine the first calculation parameter group corresponding to all disks based on the calculation parameter list and the starting address of the calculation parameters. The content of the task can be flexibly determined according to the application scenario. For example, the task can include the following information: the location number of the check disk, that is, which disk is the check disk; the number of subtasks, the number of data written to each disk; the starting number of the data disk to which data is written, that is, from which data disk to start writing data; the number of disks for written data; the starting address of the current task data pointer; the mode of the current task, etc.
[0089] In a specific application scenario, when the hardware determines the first computing parameter group corresponding to all disks according to the computing parameter list and the starting address of the computing parameter, it can determine the location of the check disk; determine the unit size value of the computing parameter; determine the computing parameter address according to the location of the check disk, the starting address of the computing parameter and the unit size value; and read the first computing parameter group corresponding to all disks from the computing parameter list according to the computing parameter address. In this way, the task only needs to specify the location of the check disk, and the hardware can accurately calculate the computing parameter address according to the location of the check disk, the starting address of the computing parameter and the unit size value, so that all the first computing parameter groups corresponding to the task can be accurately read from the computing parameter list according to the computing parameter address, and the influence of other computing parameters on the task can be eliminated, so that the subsequent hardware can quickly determine the computing parameters required for the task from the first computing parameter group for data processing.
[0090] In specific application scenarios, when the hardware determines the calculation parameter address based on the position of the check disk, the starting address of the calculation parameter and the unit size value, the calculation parameter address can be quickly determined through operations based on the relationship between the check disk and the calculation parameter address. For example, the position of the check disk can be subtracted by one to obtain the target value; the product value of the target value and the unit size value can be generated; the starting address of the calculation parameter and the product value are added to obtain the calculation parameter address.
[0091] Step S104: extracting a second computing parameter group required for the target task from the first computing parameter group.
[0092] Step S105: extracting a target computing parameter group required for each target subtask from the second computing parameter group, where the target subtask includes tasks obtained after the target task is split.
[0093] Step S106: Determine the processing result of the corresponding target subtask according to the target calculation parameter group.
[0094] In practical applications, considering that when the hardware processes tasks, data processing will be performed in the form of subtasks, and the subtasks are the tasks obtained after splitting the tasks. Therefore, after obtaining the first computing parameter group, it is necessary to extract the second computing parameter group required for the target task from the first computing parameter group, and extract the target computing parameter group required for each target subtask from the second computing parameter group. Finally, the processing result of the corresponding target subtask is determined according to the target computing parameter group, such as determining the data required to be restored for the target subtask according to the target computing parameter group.
[0095] In actual applications, in the process of hardware extracting the second computing parameter group required for the target task from the first computing parameter group, a disk flag for each disk under the target task can be generated, and the disk flag is used to reflect whether data is written to the disk; the disk flag reflecting writing data to the disk is determined as the target flag; in the first computing parameter group, the computing parameters of the disk corresponding to the target flag are extracted to obtain the second computing parameter group required for the target task, that is, the computing parameters of the current task are screened by the target flag, and the computing parameters of the disk corresponding to the target flag for writing data to the disk are extracted, and a set of all valid computing parameter sets are generated in disk order as the second computing parameter group, and the second computing parameter group only contains the computing parameters of the disk corresponding to the target flag for writing data to the disk.
[0096] In other words, the hardware can obtain the disk number where data is written according to the total number of disks, the starting number of the data disk and the position of the check disk. For ease of understanding, assume that the check disk position number is 1, the starting number of the data disk is 2, and the total number of disks where data is written is 5. Then the disks where data is written are numbered 2-6. In special cases, for example, if the check disk is numbered 4, the disks where data is written are numbered 2, 3, 5, 6, and 7, and each disk generates a flag bit. The target flag of the disk that needs to write data can be 1, and the target flag of the disk that does not need to write data can be 0. The target flag corresponding to the check disk is 0, and so on.
[0097] It should be noted that the determination process of the second calculation parameter group is affected by the disk write data mode. For example, in FSW mode, the target flag corresponding to the check disk is 0, and the target flag corresponding to the data disk is 1. The second calculation parameter group of FSW mode is the calculation parameters corresponding to all data disks, that is, {K1, K2, K3, K4...Kn-1}, and the extraction process is shown in Table 5; in RMW mode, the target flag of the disk that needs to write data is 1, and the target flag of the disk that does not need to write data is 0, and the target flag corresponding to the check disk is 0. The second calculation parameter group of RMW mode is the calculation parameters corresponding to the updated data disk. For example, if data needs to be written to three disks D2, D3, and D4, the generated valid second calculation parameter group is {K2, K3, K4}, as shown in Table 6; in RCW mode, the target flag corresponding to the check disk is 0, and the target flag corresponding to the data disk is 1. The second calculation parameter group of RCW mode is the calculation parameters corresponding to all data disks, which is the same as the FSW mode.
[0098] Table 5 Schematic diagram of extracting the second calculation parameter group of FSW mode
[0099]
[0100] Table 6 Schematic diagram of extracting the second calculation parameter group of RMW mode
[0101]
[0102] It should also be noted that the three modes of writing disk data in the present invention are interpreted as follows: FSW mode, that is, full stripe writing, in other words, the data on all disks will be modified, and all new data will be used to calculate the checksum, P´=K1*D1´⊕K2*D2´⊕K3*D3´⊕…⊕Kn*Dn´, and the calculation parameters used in the FSW mode are the calculation parameters corresponding to all data disks; RMW mode, that is, read-modify-write, in other words, the old data (D) to be modified and the original check data are read out, and then the checksum value is calculated together with the newly written data (D´). The RMW mode is used to modify the scenario where the disk data is relatively small (such as less than half of the stripe), for example, among n magnetic blocks, D2 and D3 will be modified, etc., P´=P⊕K2*(D2⊕D2´)⊕K3(D3⊕D3´), and the calculation parameters used in the RMW mode are the calculation parameters corresponding to the modified data disk. RCW mode, namely reconstruction writing, in other words, reads out the data (D) on the stripe that does not need to be modified, and calculates the checksum together with the data to be newly written (D´). RCW mode is suitable for scenarios where a lot of disk data is modified (for example, more than half of the stripe). For example, among n magnetic blocks, D2 and D3 will be modified, then the new checksum P´ calculation formula is P´=K1*D1⊕K4*D4…⊕Kn*Dn⊕(K2*D2´⊕K3*D3´). The calculation parameters used in RCW mode are the parameters corresponding to all data disks. In the actual calculation process, the parameters corresponding to the unmodified data disks need to be used first, and then the parameters corresponding to the modified data disks need to be used.
[0103] In actual applications, after the target hard disk obtains the second computing parameter group, in the process of extracting the target computing parameter group required for each target subtask from the second computing parameter group, the data pointer of the target subtask can be obtained. The data pointer is used to mark the disk that the target subtask needs to process. For example, the task information received by the hardware contains the starting address of the data pointer, and the hardware reads all the data pointers of the current task according to the starting address; considering that each task contains several subtasks, the number of data written to the disk by each subtask may also be different. For example, the disks that need to be written to the entire task are D2, D3, and D4, and the disk written to subtask 1 may be D2, D3, the disks written by subtask 2 are D3 and D4. Each subtask may not use all the calculation parameters in the second calculation parameter group. It is necessary to extract the calculation parameters required by each subtask from the second calculation parameter group, that is, generate the target identification bit of the disk according to the data pointer, and the target identification bit is used to reflect whether the disk is the disk processed by the target subtask; according to the target identification bit, extract the target calculation parameter group required by the target subtask from the second calculation parameter group. For example, in the second calculation parameter group, extract the calculation parameters of the disk corresponding to the target identification bit reflecting that the disk is the disk processed by the target subtask as the target calculation parameter group, etc.
[0104] In this way, the disk that needs to be processed by the target subtask can be marked with the help of the data pointer, and the data pointer can be converted into a target identification bit to reflect whether the disk is the disk that needs to be processed by the target subtask. In this way, the target calculation parameter group required for the target subtask can be quickly extracted from the second calculation parameter group by simply defining the disk according to the target identification bit, thereby improving the efficiency of determining the target calculation parameter group; and the target calculation parameter group only contains the calculation parameters required for the target subtask, and does not contain the calculation parameters required for other subtasks, which can eliminate the influence of the calculation parameters between subtasks, improve the matching degree between the target subtask and the target calculation parameter group, and then improve the operation efficiency of the method.
[0105] In a specific application scenario, the hardware can determine the target write mode of the target task in the process of generating the disk flags of each disk under the target task; and generate the disk flags of each disk under the target task according to the target write mode. For example, in response to the target write mode being full stripe write, the disk flag of the check disk is generated as not writing data, and the disk flag of the data disk is generated as writing data. Accordingly, according to the rule that all data pointers are valid, the target identification bits generated for all disks are all the first identification bits, and the first identification bit is used to reflect that the disk is the disk processed by the target subtask. For another example, in response to the target write mode being read-modify-write, the disk flag of the check disk is generated as not writing data, the disk flag of the data disk that needs to write data is generated as writing data, and the disk flag of the data disk that does not need to write data is generated as not writing data. Accordingly, according to the rule that the target identification bits between the modified disk and the disk that writes data are the same, and the target identification bit of the disk that needs to write data is the first identification bit, and the target identification bits of the remaining disks are the second identification bits, the target identification bits of all disks are generated. For another example, in response to the target write mode being reconstruction write, the disk flag of the generated check disk is not to write data, and the disk flag of the generated data disk is to write data. Accordingly, according to the rule that the target identification bits between the modified disk and the disk for writing data are mutually exclusive, the target identification bit of the modified disk that does not need to write data is the first identification bit, and the target identification bit of the modified disk that needs to write data is the second identification bit, the target identification bits of all disks are generated. Among them, the first identification bit is used to reflect that the disk is a disk processed by the target subtask, and the second identification bit is used to reflect that the disk is not a disk processed by the target subtask. In this way, the disk flag and target identification bit under each write mode can be quickly determined according to the setting process of the corresponding mode, and the present invention covers the determination methods of all write modes, so that various subtasks can be processed comprehensively. In other words, no matter what kind of subtask, the hardware of the present invention can process it, the method is highly compatible and has good applicability. In addition, only the valid identification bit of the data pointer is needed to extract the calculation parameters needed for the subtask, which reduces the complexity of extracting parameters and saves hardware resources.
[0106] That is, in the present invention, the results of the data pointer and the target identification bit can be determined according to the mode of writing disk data. For ease of understanding, it is assumed that the data pointer of each subtask is arranged in the following manner: the FSW mode updates all data disks, and the data pointer arrangement is D1´D2´D3´...; the RMW mode updates part of the data disks, for example, D2, D3, and D4 data disks need to be updated, then the data pointer arrangement is D2D3D4D2´D3´D4´, wherein D2D3D4 is the original data that needs to be modified, and D2´D3´D4´ is the newly written data; the RCW mode updates part of the data disks, for example, D1-D5 is the data disk, and D2, D3, and D4 data disks need to be updated, then the data pointer arrangement is D1D2D3D4D5D1´D2´D3´D4´D5´; a task can only be in one of the modes, and the data pointer arrangement of subtasks in the same mode is the same. Assume that the data pointer of each subtask has a 1-bit flag to indicate whether the current data is valid. A data pointer flag of 0 indicates that the current data is invalid, and a data pointer flag of 1 indicates that the current data is valid. Invalid data does not need to be written to the disk, and the calculation parameters corresponding to the disk do not need to be used. The type of subtask valid data distribution can be shown in Table 7.
[0107] Table 7 Subtask effective data distribution table
[0108]
[0109] Assume that as shown in Table 7, the disks to be written by the entire task include D2, D3, and D4; the disks written by subtask 1 and subtask 2 are D3 and D4; the disks written by subtask 3 are D2, D3, D4, and; the disks written by subtask 4 are D1 and D2. After the hardware obtains each data pointer of the subtask, it combines the valid identification bits of the data pointer into a group of target identification bits, and the generation methods of the target identification bits in different modes are as follows: in FSW mode, all data pointers are valid, and the target identification bits of all subtasks are {1,1,1...}; in RMW mode, the data valid identification bits of Dx and Dx' are exactly the same, the valid identification bit of the data disk to be written is 1, and the target identification bit is the valid identification bit of Dx, which is the valid identification bit of all Dx. For example, the pointer arrangement of subtask 1 is D2D3D4D2'D3'D4', and the valid identification bits are {0,1,1,0,1,1}, among which the data valid identification bits corresponding to D2 and D2' are 0, and the target identification bits of subtask 1 are marked as {0,1,1}. Similarly, the target identification bits of subtask 2 are marked as { 0,1,1}, the target identification bit of subtask 3 is marked as {1,1,1}, and the target identification bit of subtask 4 is marked as {1,1,0}; in RCW mode, the data valid identification bits of Dx and Dx' are mutually exclusive. The valid identification bit of the data disk that does not need to be written in Dx is 1, and the valid identification bit of the data disk that is written is 0. The valid identification bit of the data disk that does not need to be written in Dx' is 0, and the valid identification bit of the data disk that is written is 1. Each subtask generates two groups of target identification bits, one for Dx and one for Dx'. For example, the pointer arrangement of subtask 1 is D1D2D3D4D1'D2'D3'D4', and the valid identification bits are {1,1,0,0,0,0,1,1}, the valid identification bits corresponding to Dx are {1,1,0,0}, and the valid identification bits corresponding to Dx' are {0,0,1,1}.
[0110] It should be noted that the number of target identification bit marks of each subtask is equal to the number of calculation parameters in the second calculation parameter group, and corresponds one-to-one to the disk order. Therefore, the calculation parameters at the corresponding positions in the second calculation parameter group can be extracted according to the position of 1 in the target identification bit mark, and the extracted calculation parameters are written into the parameter cache. The extracted calculation parameters are the target calculation parameter group ultimately needed by the current subtask. For example, the extraction process of the subtask corresponding to Table 7 can be shown in Table 8.
[0111] Table 8 Schematic diagram of target calculation parameter group extraction
[0112]
[0113] It should be noted that the association between hardware and software in the present invention can be flexibly determined according to the application scenario, and the structure of the hardware can be determined according to actual needs. For example, the structure of the hardware can be as follows: Figure 3As shown, the configuration register is used to store the starting address of the calculation parameters stored by the software by writing the configuration register, and the on-chip storage is used to store the calculation parameter list transmitted by the software. The expanded description of the remaining modules can refer to the above embodiments.
[0114] It should also be noted that in specific application scenarios, the hardware can execute in parallel the steps of determining the first calculation parameter group corresponding to all disks based on the calculation parameter list and the starting address of the calculation parameters, generating the disk flag of each disk under the target task, and obtaining the data pointer of the target subtask, so as to utilize the synchronous processing advantages of the hardware to improve the efficiency of extracting calculation parameters.
[0115] The invention provides a data processing method, which is applied to hardware, obtains a calculation parameter list transmitted by software, and the calculation parameter list is used to record the calculation parameters stored by each disk according to the target disk array; obtains the starting address of the calculation parameter transmitted by the software; determines the first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameter; extracts the second calculation parameter group required for the target task from the first calculation parameter group; extracts the target calculation parameter group required for each target subtask from the second calculation parameter group, and the target subtask includes the task obtained after the target task is split; and determines the processing result of the corresponding target subtask according to the target calculation parameter group. The beneficial effect of the invention is that the software only transmits the calculation parameter list and the starting address of the calculation parameter to the hardware once, so that the hardware can know the calculation parameters stored by each disk according to the target disk array, so that the hardware can use the starting address of the calculation parameter to extract the target calculation parameter group required for the target subtask from the calculation parameter list for data processing. In the whole process, the hardware only needs to interact with the software twice, and the interaction frequency between the software and the hardware is low, which reduces the performance overhead, thereby improving the system performance of using the disk array for data processing.
[0116] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor:
[0117] Obtaining a calculation parameter list transmitted by the software, the calculation parameter list is used to record the calculation parameters stored in each disk according to the target disk array;
[0118] Get the starting address of the calculation parameters transmitted by the software;
[0119] Determine the first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameters;
[0120] Extracting a second computing parameter group required for the target task from the first computing parameter group;
[0121] Extracting a target computing parameter group required for each target subtask from the second computing parameter group, the target subtask including tasks obtained after the target task is split;
[0122] The processing result of the corresponding target subtask is determined according to the target calculation parameter group.
[0123] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: determining the position of the check disk; determining the unit size value of the calculation parameter; determining the calculation parameter address according to the position of the check disk, the starting address of the calculation parameter and the unit size value; and reading the first calculation parameter group corresponding to all disks from the calculation parameter list according to the calculation parameter address.
[0124] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: subtracting one from the position of the check disk to obtain a target value; generating a product value of the target value and a unit size value; adding the starting address of the calculation parameter and the product value to obtain the calculation parameter address.
[0125] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: generating a disk flag for each disk under a target task, the disk flag being used to reflect whether data is written to the disk; determining the disk flag reflecting data being written to the disk as a target flag; and extracting the calculation parameters of the disk corresponding to the target flag from a first calculation parameter group to obtain a second calculation parameter group required for the target task.
[0126] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: obtaining a data pointer of a target subtask, the data pointer is used to mark the disk to be processed by the target subtask; generating a target identification bit of the disk according to the data pointer, the target identification bit is used to reflect whether the disk is a disk to be processed by the target subtask; and extracting a target computing parameter group required for the target subtask from a second computing parameter group according to the target identification bit.
[0127] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: determining a target write mode for a target task; and generating disk flags for each disk under the target task according to the target write mode.
[0128] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: in response to the target write mode being a full stripe write, a disk flag for a check disk is generated as not writing data, and a disk flag for a data disk is generated as writing data.
[0129] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: according to the rule that all data pointers are valid, the target identification bits of all disks are generated as the first identification bit, and the first identification bit is used to reflect that the disk is a disk processed by the target subtask.
[0130] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: in response to the target write mode being read-modify-write, a disk flag for a check disk is generated as not writing data, a disk flag for a data disk to which data needs to be written is generated as writing data, and a disk flag for a data disk to which data does not need to be written is generated as not writing data.
[0131] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: generating target identification bits for all disks according to the rule that the target identification bits between the modified disk and the disk to which data is written are the same, the target identification bit of the disk to which data needs to be written is the first identification bit, and the target identification bits of the remaining disks are the second identification bits; wherein the first identification bit is used to reflect that the disk is a disk processed by the target subtask, and the second identification bit is used to reflect that the disk is not a disk processed by the target subtask.
[0132] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: in response to the target write mode being a reconstruction write, a disk flag of a check disk is generated as not writing data, and a disk flag of a data disk is generated as writing data.
[0133] Based on the above embodiments, the present invention provides a computer program product, including a computer program / instruction, which implements the following steps when executed by a processor: generating target identification bits of all disks according to the rule that the target identification bits of the modified disk and the disk to which data is written are mutually exclusive, the target identification bit of the modified disk that does not need to write data is the first identification bit, and the target identification bit of the modified disk that needs to write data is the second identification bit;
[0134] The first identification bit is used to reflect that the disk is a disk processed by the target subtask, and the second identification bit is used to reflect that the disk is not a disk processed by the target subtask.
[0135] The present invention also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the data distribution method provided in the embodiment of the present invention. Figure 4 , Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.
[0136] An electronic device provided by an embodiment of the present invention includes a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, the following steps are implemented:
[0137] Obtaining a calculation parameter list transmitted by the software, the calculation parameter list is used to record the calculation parameters stored in each disk according to the target disk array;
[0138] Get the starting address of the calculation parameters transmitted by the software;
[0139] Determine the first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameters;
[0140] Extracting a second computing parameter group required for the target task from the first computing parameter group;
[0141] Extracting a target computing parameter group required for each target subtask from the second computing parameter group, the target subtask including tasks obtained after the target task is split;
[0142] The processing result of the corresponding target subtask is determined according to the target calculation parameter group.
[0143] See also Figure 5Another electronic device provided in an embodiment of the present invention may also include: an input port 203 connected to the processor 202, used to transmit commands input from the outside to the processor 202; a display unit 204 connected to the processor 202, used to display the processing results of the processor 202 to the outside; a communication module 205 connected to the processor 202, used to realize communication between the electronic device and the outside. The display unit 204 can be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes but is not limited to mobile high-definition link technology (Mobile High-Definition Link, MHL), Universal Serial Bus (Universal Serial Bus, USB), High-Definition Multimedia Interface (High-Definition Multimedia Interface, HDMI), wireless connection: wireless fidelity technology (WIreless Fidelity, WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, and communication technology based on IEEE802.11s.
[0144] An embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:
[0145] Obtaining a calculation parameter list transmitted by the software, the calculation parameter list is used to record the calculation parameters stored in each disk according to the target disk array;
[0146] Get the starting address of the calculation parameters transmitted by the software;
[0147] Determine the first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameters;
[0148] Extracting a second computing parameter group required for the target task from the first computing parameter group;
[0149] Extracting a target computing parameter group required for each target subtask from the second computing parameter group, the target subtask including tasks obtained after the target task is split;
[0150] The processing result of the corresponding target subtask is determined according to the target calculation parameter group.
[0151] The computer-readable storage medium involved in the present invention includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM (Compact Disc Read-Only Memory), or any other form of storage medium known in the technical field.
[0152] For the description of the relevant parts of a computer program product, an electronic device, and a computer-readable storage medium provided by an embodiment of the present invention, please refer to the detailed description of the corresponding parts in a data distribution method provided by an embodiment of the present invention, which will not be repeated here. In addition, the parts of the above technical solutions provided by the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.
[0153] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0154] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data processing method, It is characterized in that Applied to hardware, including: Obtaining a calculation parameter list transmitted by the software, wherein the calculation parameter list is used to record the calculation parameters stored by each disk according to the target disk array; Obtaining the starting address of the calculation parameters transmitted by the software; Determine a first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameter; Extracting a second computing parameter group required for the target task from the first computing parameter group; Extracting a target computing parameter group required for each target subtask from the second computing parameter group, wherein the target subtask includes a task obtained after the target task is split; The processing result of the corresponding target subtask is determined according to the target calculation parameter group.
2. The data processing method according to claim 1, It is characterized in that Determining a first calculation parameter group corresponding to all disks according to the calculation parameter list and the starting address of the calculation parameter includes: Determine the location of the calibration disk; Determining a unit size value of the calculation parameter; Determining a calculation parameter address according to the position of the check disk, the starting address of the calculation parameter and the unit size value; The first calculation parameter group corresponding to all disks is read from the calculation parameter list according to the calculation parameter address.
3. The data processing method according to claim 2, It is characterized in that Determining the calculation parameter address according to the position of the check disk, the starting address of the calculation parameter and the unit size value includes: Subtract one from the position of the calibration disk to obtain a target value; Generate a product value of the target value and the unit size value; The starting address of the calculation parameter and the product value are added to obtain the calculation parameter address.
4. The data processing method according to claim 1, It is characterized in that Extracting a second computing parameter group required for the target task from the first computing parameter group includes: Generate a disk flag for each disk under the target task, wherein the disk flag is used to reflect whether data is written to the disk; Determine the disk mark reflecting writing data to the disk as a target mark; In the first computing parameter group, the computing parameters of the disk corresponding to the target mark are extracted to obtain the second computing parameter group required for the target task.
5. The data processing method according to claim 4, It is characterized in that Extracting a target computing parameter group required for each target subtask from the second computing parameter group includes: Acquire a data pointer of the target subtask, where the data pointer is used to mark the disk to be processed by the target subtask; Generate a target identification bit of the disk according to the data pointer, wherein the target identification bit is used to reflect whether the disk is the disk processed by the target subtask; The target computing parameter group required for the target subtask is extracted from the second computing parameter group according to the target identification bit.
6. The data processing method according to claim 5, It is characterized in that Generate disk flags for each disk under the target task, including: Determining a target writing mode for the target task; The disk flag of each disk under the target task is generated according to the target write mode.
7. The data processing method according to claim 6, It is characterized in that Generating the disk flag of each disk under the target task according to the target write mode includes: In response to the target write mode being full stripe write, the disk flag of the generated check disk is generated as not writing data, and the disk flag of the generated data disk is generated as writing data.
8. The data processing method according to claim 7, It is characterized in that Generating a target identification bit of a corresponding disk according to the data pointer includes: According to the rule that all the data pointers are valid, the target identification bits generated for all disks are all first identification bits, and the first identification bit is used to reflect that the disk is the disk processed by the target subtask.
9. The data processing method according to claim 6, It is characterized in that Generating the disk flag of each disk under the target task according to the target write mode includes: In response to the target write mode being read-modify-write, the disk flag for generating the check disk is not to write data, the disk flag for generating the data disk to which data needs to be written is to write data, and the disk flag for generating the data disk to which data does not need to be written is not to write data.
10. The data processing method according to claim 9, It is characterized in that Generating a target identification bit of a corresponding disk according to the data pointer includes: Generate the target identification bits of all disks according to the rule that the target identification bits between the modified disk and the disk to which data is written are the same, the target identification bit of the disk to which data is written is the first identification bit, and the target identification bits of the remaining disks are the second identification bits; The first identification bit is used to reflect that the disk is a disk processed by the target subtask, and the second identification bit is used to reflect that the disk is not a disk processed by the target subtask.
11. The data processing method according to claim 6, It is characterized in that Generating the disk flag of each disk under the target task according to the target write mode includes: In response to the target write mode being reconstruction write, the disk flag of the generated check disk is not to write data, and the disk flag of the generated data disk is to write data.
12. The data processing method according to claim 11, It is characterized in that Generating a target identification bit of a corresponding disk according to the data pointer includes: Generate the target identification bits of all disks according to the rule that the target identification bits between the modified disk and the disk to which data is written are mutually exclusive, the target identification bit of the disk to be modified and which does not need data to be written is the first identification bit, and the target identification bit of the disk to be modified and which needs data to be written is the second identification bit; The first identification bit is used to reflect that the disk is a disk processed by the target subtask, and the second identification bit is used to reflect that the disk is not a disk processed by the target subtask.
13. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 12 are implemented.
14. An electronic device, It is characterized in that include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 12 when executing the computer program.
15. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 12 are implemented.
Citation Information
Patent Citations
Method for configuring disk array of electronic equipment and electronic equipment
CN107357525A
Disk array controller, and IO data processing method and apparatus
CN108121600A
Control method and device for group disk array, electronic equipment and readable storage medium
CN114442930A
Method, device and equipment for realizing memory address mapping
CN116627856A
Solid state disk configuration management method and device, computer equipment and storage medium
CN117311646A
