Data processing method and electronic device

Abstract

The application discloses a data processing method and an electronic device. The method comprises the following steps: in response to a write / read data request, determining a storage address corresponding to to-be-written / read data; determining a bitmap mark corresponding to the storage address, the bitmap mark being used for representing whether data calculated by an erasure code exists in the storage address; and writing the to-be-written data in the storage address or reading the to-be-read data from the storage address according to the bitmap mark. In this way, when data is saved by using a large-scale strip erasure code, each time data is written in or read from the storage address, only the bitmap mark corresponding to the storage address needs to be confirmed, and the data is written in or read from the storage address based on the confirmation result of the bitmap mark, that is, the data can be directly written in the strip by using a local updating mode without reconstructing the data, so that time spent on reconstructing the data in the strip is saved, and user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method and an electronic device. BACKGROUND

[0002] In the field of high-performance computing (HPC), large-scale strip erasure code is often used to save data. Therefore, when facing the problem of how to write data in the strip of the storage unit, updating the entire strip often cannot be used as a feasible solution.

[0003] In the existing solution, data can be written in the strip by means of local update, but this method needs to ensure that the data in the strip is calculated by erasure code, and usually there is old dirty data in the strip, which does not meet this condition. Therefore, the data in the strip needs to be reconstructed before writing data, and the reconstruction method includes clearing all the data in the strip, or reading out the valid data in the strip, then calculating the erasure code, and writing the calculated erasure code to the check bit. However, whenever the data in the strip is reconstructed, the full amount of data space needs to be accessed, which will take a lot of time and reduce user experience in large storage systems (such as PB (petabyte) level). SUMMARY

[0004] Therefore, the embodiments of the present application provide a data processing method and an electronic device to at least solve the above technical problems in the prior art.

[0005] According to a first aspect of the present application, the embodiments of the present application provide a data processing method, comprising: in response to a write / read data request, determining a storage address corresponding to to-be-written / read data; determining a bitmap mark corresponding to the storage address, the bitmap mark being used to represent whether there is data calculated by erasure code in the storage address; and writing the to-be-written data in the storage address or reading the to-be-read data from the storage address according to the bitmap mark.

[0006] Optionally, the bitmap mark corresponding to the storage address is determined by: determining a target data block corresponding to the storage address, the target data block being one data block in the strip; determining the bitmap mark of the target data block; and taking the bitmap mark of the target data block as the bitmap mark corresponding to the storage address.

[0007] Optionally, the to-be-written data is written in the storage address according to the bitmap mark, comprising: obtaining storage data corresponding to the storage address and first check data corresponding to the storage address according to the bitmap mark, to obtain return data; performing erasure code calculation on the return data and the to-be-written data to obtain second check data; and writing the to-be-written data into the storage address and replacing the first check data corresponding to the storage address with the second check data.

[0008] Optionally, the return data and the to-be-written data are subjected to erasure code calculation to obtain second check data, including: performing incremental calculation on the storage data corresponding to the storage address in the return data and the to-be-written data to obtain incremental data; and performing erasure code calculation on the incremental data and the first check data corresponding to the storage address in the return data to obtain the second check data.

[0009] Optionally, before the to-be-written data is written into the storage address and the second check data is used to replace the first check data corresponding to the storage address, the data processing method further includes: setting a dirty bitmap flag corresponding to the storage address as a dirty flag; and the dirty bitmap flag is used to indicate whether the storage data in the storage address needs to be recovered.

[0010] After the to-be-written data is written into the storage address and the second check data is used to replace the first check data corresponding to the storage address, the data processing method further includes: setting a dirty bitmap flag corresponding to the storage address as a non-dirty flag.

[0011] Optionally, the to-be-read data is read from the storage address according to the bitmap flag, including: obtaining the storage data corresponding to the storage address and the first check data corresponding to the storage address according to the bitmap flag to obtain return data.

[0012] Optionally, the storage data corresponding to the storage address and the first check data corresponding to the storage address are obtained according to the bitmap flag to obtain return data, including: in a case where it is determined that the bitmap flag is a write flag, reading the storage data in the storage address and the first check data corresponding to the storage address to obtain the return data; and in a case where it is determined that the bitmap flag is a non-write flag, determining a first constant as the storage data corresponding to the storage address and a second constant as the first check data corresponding to the storage address, and obtaining the return data based on the first constant and the second constant, wherein the first constant and the second constant satisfy: an erasure code obtained by performing erasure code calculation on the constant is equal to the constant; and wherein the write flag is used to indicate that there is data subjected to erasure code calculation in the storage address, and the non-write flag is used to indicate that there is no data subjected to erasure code calculation in the storage address.

[0013] Optionally, before the storage address corresponding to the to-be-written / read data is determined in response to the write / read data request, the data processing method further includes: allocating a plurality of data blocks of a plurality of stripes from the storage unit as a data storage space according to a data storage capacity requested by a user; and initializing bitmap flags corresponding to the plurality of data blocks of the plurality of stripes.

[0014] Optionally, the data processing method further comprises: in response to system startup, checking the dirty bitmap marks corresponding to the data blocks of each stripe in the storage unit; for any stripe, in the case where it is confirmed that there is a data block with a dirty bitmap mark, reading the data in the data block with a non-dirty mark in the stripe; performing erasure code calculation on the data to obtain correction data; writing the correction data into the data block with a dirty mark; and flipping the dirty mark to a non-dirty mark.

[0015] According to the second aspect of the present application, the embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to the first aspect or any of the embodiments of the first aspect.

[0016] The data processing method and the electronic device provided by the embodiments of the present application, by responding to the write / read data request, determining the storage address corresponding to the to-be-written / read data; determining the bitmap mark corresponding to the storage address, the bitmap mark being used to represent whether there is data calculated by the erasure code in the storage address; and writing the to-be-written data in the storage address according to the bitmap mark, or reading the to-be-read data from the storage address; in this way, when the data is saved by using the large-scale stripe erasure code, each time the data is written in the storage address or read from the storage address, only the bitmap mark corresponding to the storage address needs to be confirmed, and the data is written in the storage address or read from the storage address based on the confirmation result of the bitmap mark, and it is not necessary to ensure that the data in the stripe is calculated by the erasure code, and the data in the stripe is reconstructed, that is, the data can be directly written in the stripe by the local updating mode without reconstructing the data, thereby saving the time spent on reconstructing the data in the stripe and improving the user experience.

[0017] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the specific embodiments of the present application can be implemented according to the content of the description, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of a data processing method in an embodiment of the present application is shown;

[0019] Figure 2 A structural diagram of a storage unit in an embodiment of the present application is shown;

[0020] Figure 3 Another flowchart of a data processing method in an embodiment of the present application is shown;

[0021] Figure 4 Fig. 1 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0023] First, several terms related to the embodiments of the present application are explained:

[0024] Erasure coding (EC): a data protection method that divides data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographic locations.

[0025] Stripe: a method of dividing continuous data into data blocks of the same size and writing each piece of data to different disks in the array. Simply put, stripe is a method of combining multiple disk drives into a volume. In many cases, this is done through a hardware controller.

[0026] Dirty read: refers to data in the source system that is not within a given range or is meaningless for actual business, or data format is illegal, and there are non-standard encoding and ambiguous business logic in the source system.

[0027] Logical block: the smallest unit of memory that ensures atomicity in the present application, such as when the system loses power, the write in this area will not be partially successful or partially failed. It is usually the size of a disk sector or a physical block, such as a hard disk can guarantee integrity above 512, and an NVMe storage system (Non-Volatile Memory express) disk can guarantee integrity above 4096. The data blocks in the stripe correspond one-to-one to the logical blocks.

[0028] Storage bitmap: the area to which the data blocks of multiple stripes in the storage unit belong. The storage bitmap corresponds one-to-one to the logical blocks by bit.

[0029] Bitmap flag: used to represent whether there is data calculated by erasure coding in the storage address; the bitmap flag corresponds one-to-one to the logical blocks, so the data blocks correspond one-to-one to the bitmap flags.

[0030] Dirty bit flag: used to mark whether there is dirty data in the data block, and the dirty bit flag corresponds to the data block one by one.

[0031] The embodiment of the application provides a data processing method, which can be applied to a scheme of saving data by using a strip-based erasure code. Figure 1 As shown in the figure, the data processing method comprises:

[0032] S101, in response to a write / read data request, determining a storage address corresponding to to-be-written / read data.

[0033] In the embodiment, the write data request can be a request of writing data to a storage unit in which dirty data is stored, or a request of writing data to a new storage unit. The read data request can be a request of reading data from a storage unit in which dirty data is stored, or a request of reading data calculated by the erasure code from the storage unit. The storage unit includes but is not limited to a persistent memory device, a disk. The persistent memory device includes but is not limited to a non-volatile dual in-line memory module (NVDIMM), Optane, a persistent memory region (PMR) of an NVMe storage system disk. The disk includes but is not limited to a mechanical disk, an NVMe storage system solid state disk, and an NVMe storage system disk. The storage address corresponding to the to-be-written data is a position of the to-be-written data in the storage unit, and the storage address corresponding to the to-be-read data is a position of the to-be-read data in the storage unit.

[0034] S102, determining a bit flag corresponding to the storage address, the bit flag being used to represent whether there is data calculated by the erasure code in the storage address.

[0035] In the embodiment, when data is written into the storage unit, the to-be-written data needs to be calculated by the erasure code, and before that, the data in the storage unit has not been reconstructed, so it cannot be determined whether there is data calculated by the erasure code in the storage address. If there is data calculated by the erasure code in the storage address corresponding to the to-be-written data, the to-be-written data can be directly written into the storage address by a local update manner, so that subsequent data repair can be ensured; if there is dirty data in the storage address corresponding to the to-be-written data, the dirty data has not been calculated by the erasure code, so the to-be-written data cannot be directly written into the storage address by the local update manner, otherwise the effect of the erasure code cannot be achieved.

[0036] The logical block corresponding to the storage address has a bitmap mark. The bitmap mark can represent whether the data in the storage address is calculated by the erasure code. Therefore, the bitmap mark corresponding to the storage address needs to be determined to determine whether the data stored in the storage address is calculated by the erasure code.

[0037] In S103, the data to be written is written in the storage address according to the bitmap mark, or the data to be read is read from the storage address.

[0038] In this embodiment, if the bitmap mark indicates that the data in the storage address is calculated by the erasure code, the data to be written can be directly written in the storage address in the form of overwrite by the local update, or the data to be read can be directly read from the storage address. If the bitmap mark indicates that the data in the storage address is not calculated by the erasure code, the data in the storage address can be regarded as a constant, for example, 0, and the data to be written can be directly written in the storage address in the form of overwrite by the local update, or the constant is returned to obtain the data to be read read from the storage address.

[0039] The data processing method provided by the embodiment of the application can determine the storage address corresponding to the data to be written / read in response to the data writing / reading request, determine the bitmap mark corresponding to the storage address, and write the data to be written in the storage address or read the data to be read from the storage address according to the bitmap mark. In this way, when the data is written in the storage address or read from the storage address, only the bitmap mark corresponding to the storage address needs to be confirmed, and the data is written in the storage address or read from the storage address based on the confirmation result of the bitmap mark, without ensuring that all the data in the stripe is calculated by the erasure code and reconstructing the data in the stripe. That is, the data can be directly written in the stripe by the local update without reconstructing the data, thereby saving the time spent on reconstructing the data in the stripe and improving the user experience.

[0040] In an optional embodiment, for the old storage unit, if a large amount of dirty data is stored in the storage unit, the dirty data is irrelevant to the data to be written at present, and the bitmap marks of the data blocks in the storage unit are initialized when the storage system is established. The bitmap mark includes two states of a written mark and an unwritten mark. When the bitmap mark is initialized, the written mark in the bitmap mark is cleared and set as the unwritten mark. The ways of finding the bitmap mark of the data block as the written mark include direct traversal and optimized search by the Bloom filter.

[0041] Thus, by initializing the bitmap markers of the data blocks in the storage unit and then writing data, the data in the data blocks can be completely overwritten, just like using a new storage unit, and the original dirty data cannot be read from the storage unit. Since only the bitmap markers of the data blocks in the storage unit are initialized, the time required is very short, and the user has no perception. Compared with the large amount of time spent on data reconstruction of the storage unit in the prior art, the user can quickly use the old storage unit, improving the user experience. For example, the amount of data to be written is 1 / blk-size / 8, which is much smaller than the total capacity of the storage unit. According to the total capacity 2P and the logical block 8k, the amount of data to be written is only 32G. By using the data processing method of the embodiment of the present application, writing 32G data in the storage unit can be completed in a few seconds. Compared with the data reconstruction of the entire storage space of the storage unit in the prior art, a large amount of data reconstruction time is saved.

[0042] In an optional embodiment, the user usually needs to allocate data storage space for data before using the storage system, and the allocation of data storage space includes creating a volume, creating a logical unit number (LUN), etc. Therefore, in the embodiment of the present application, before determining the storage address corresponding to the data to be written / read in step S101 in response to the data write / read request, the data processing method further comprises: allocating a plurality of data blocks of a plurality of stripes as data storage space from the storage unit according to the data storage capacity requested by the user; and initializing the bitmap markers corresponding to the plurality of data blocks of the plurality of stripes.

[0043] Specifically, the data storage capacity requested by the user is less than or equal to the total capacity of the storage unit. The structure of the storage unit is shown in Figure 2 If the bitmap markers corresponding to the plurality of data blocks of the plurality of stripes as data storage space are not initialized, the bitmap markers corresponding to the plurality of data blocks of the plurality of stripes in the data storage space also need to be initialized.

[0044] Thus, the data storage space can be reasonably allocated for data according to the user's demand for data storage capacity, improving the user experience.

[0045] In an optional embodiment, step S102 of determining the bitmap marker corresponding to the storage address comprises: determining a target data block corresponding to the storage address, the target data block being one data block in the stripe; determining the bitmap marker of the target data block; and taking the bitmap marker of the target data block as the bitmap marker corresponding to the storage address.

[0046] In a specific implementation, the target data block of the stripe in which the storage address is located can be calculated by using the storage address, the stripe capacity and the data block capacity. For example, a first ratio of the storage address to the stripe capacity is calculated first to obtain the stripe in which the storage address is located, and then a second ratio of the storage address to the data block capacity is calculated to obtain the block in which the storage address is located. According to the stripe in which the storage address is located and the block in which the storage address is located, the target data block of the stripe in which the storage address is located is determined.

[0047] The bitmap marker of the target data block can be found in the mapping relationship between the data block and the bitmap marker by using the mapping lookup method. Then it is determined that the bitmap marker of the target data block is the bitmap marker corresponding to the storage address.

[0048] In the embodiment, since each data block of the stripe in the storage unit corresponds to a bitmap marker, the bitmap marker corresponding to the storage address can be accurately determined by determining the target data block corresponding to the storage address and finding the bitmap marker of the target data block.

[0049] In an optional embodiment, in step S103, the data to be written in the storage address according to the bitmap marker includes: obtaining the storage data corresponding to the storage address and the first check data corresponding to the storage address according to the bitmap marker to obtain return data; performing erasure code calculation on the return data and the data to be written to obtain second check data; writing the data to be written in the storage address and replacing the first check data corresponding to the storage address with the second check data.

[0050] In some embodiments, obtaining the storage data corresponding to the storage address and the first check data corresponding to the storage address according to the bitmap marker to obtain return data includes: in the case where it is determined that the bitmap marker is a write marker, reading the storage data in the storage address and the first check data corresponding to the storage address to obtain the return data; in the case where it is determined that the bitmap marker is an un-written marker, determining a first constant as the storage data corresponding to the storage address and determining a second constant as the first check data corresponding to the storage address, and obtaining the return data based on the first constant and the second constant, wherein the first constant and the second constant both satisfy that the erasure code obtained by performing erasure code calculation on the constant is equal to the constant; wherein the write marker is used to represent that there is data calculated by erasure code in the storage address, and the un-written marker is used to represent that there is no data calculated by erasure code in the storage address.

[0051] In a specific implementation, the write marker and the un-written marker can be replaced by symbols or data, for example, the write marker is 1 and the un-written marker is 0. The constant equation satisfied by the first constant and the second constant is EC(a+b+c+d)=e+f, where a=b=c=d=e=f. In the embodiment, when the first constant and the second constant are 0, the above constant equation is satisfied, and therefore, in the embodiment, the first constant and the second constant can be set to 0.

[0052] In the embodiment, when the bitmap mark is the write mark, it is indicated that the data calculated by the erasure code exists in the storage address, and the storage data in the storage address and the first check data corresponding to the storage address are directly read to obtain the return data; when the bitmap mark is the non-write mark, the first constant is directly determined as the storage data corresponding to the storage address, and the second constant is determined as the first check data corresponding to the storage address, so that it can be guaranteed that whether the data calculated by the erasure code exists in the storage address or not, the return data can meet the requirement of the erasure code, and the dirty data in the storage address can be avoided to be read, so that the local updating manner can be directly used to write the data in the stripe.

[0053] In some embodiments, the return data and the to-be-written data are subjected to erasure code calculation to obtain second check data, including: incrementally calculating the storage data corresponding to the storage address in the return data and the to-be-written data to obtain incremental data; and performing erasure code calculation on the incremental data and the first check data corresponding to the storage address in the return data to obtain the second check data.

[0054] In specific implementation, the incremental calculation can be an exclusive OR operation. The first check data can be stored in a check block of the storage unit, and each stripe can correspond to one or more check blocks. The first check data is used to recover the data in the target data block. After the second check data is calculated, the second check data needs to replace the first check data in the check block.

[0055] In the embodiment, the storage data corresponding to the storage address and the first check data corresponding to the storage address are obtained according to the bitmap mark to obtain the return data; the return data and the to-be-written data are subjected to erasure code calculation to obtain the second check data; and the to-be-written data is written into the storage address, and the second check data replaces the first check data corresponding to the storage address, so that the to-be-written data can be subjected to erasure code calculation when being written into the target data block, and the data stored in the target data block can be protected.

[0056] In an optional embodiment, before the to-be-written data is written into the storage address and the second check data replaces the first check data corresponding to the storage address, the data processing method further includes: setting the dirty bitmap mark corresponding to the storage address to the dirty mark; and the dirty bitmap mark is used to indicate whether the storage data in the storage address needs to be recovered.

[0057] After the to-be-written data is written into the storage address and the second check data replaces the first check data corresponding to the storage address, the data processing method further includes: setting the dirty bitmap mark corresponding to the storage address to the non-dirty mark.

[0058] In the embodiment, through the flip operation control of the dirty bit flag, when data is written into the system and a whole power failure occurs, the power failure repair work can be completed by traversing the dirty bit flag of the storage unit, so that the data loss caused by the power failure can be avoided, the strip alignment condition does not need to be met when data is written, no additional read-write is needed, the proportion of read and write data blocks is fixed, the delay is stable, and the data writing speed and reading speed can be improved. When the number of members (including data blocks and check blocks) of a strip is large, the advantage is more obvious. For example, for a certain strip (4 data blocks + 2 check blocks), in the way of writing data in the whole strip, the data of 4 data blocks needs to be read, after data updating and operation, 4 data blocks and 2 check blocks are written again, a total of 10 times of input / output (IO) operations are needed; and in the embodiment of the present application, the data of 1 data block and 2 check blocks is read, after data updating and operation, 1 data block and 2 check blocks are written again, a total of 6 times of IO operations + 2 times of bit flag marking are needed, but the delay of 2 times of bit flag marking is far lower than the writing delay of a disk (mechanical disk, solid state disk, NVMe storage system disk, etc.).

[0059] In an optional embodiment, the step S103 of reading the to-be-read data from the storage address according to the bit flag includes: obtaining the storage data corresponding to the storage address and the first check data corresponding to the storage address according to the bit flag to obtain the return data.

[0060] In a specific implementation, the step of obtaining the storage data corresponding to the storage address and the first check data corresponding to the storage address according to the bit flag to obtain the return data includes: in a case where it is determined that the bit flag is a write flag, reading the storage data in the storage address and the first check data corresponding to the storage address to obtain the return data; in a case where it is determined that the bit flag is an unwritten flag, determining a first constant as the storage data corresponding to the storage address and determining a second constant as the first check data corresponding to the storage address, and obtaining the return data based on the first constant and the second constant, wherein the first constant and the second constant satisfy: a first constant and a second constant satisfy: an erasure code obtained by performing erasure code calculation on the constant is equal to the constant; and the write flag is used to represent that there is data calculated by the erasure code in the storage address, and the unwritten flag is used to represent that there is no data calculated by the erasure code in the storage address.

[0061] More specifically, the write flag and the unwritten flag can be replaced by symbols or data, for example, the write flag is 1 and the unwritten flag is 0. The first constant and the second constant satisfy the constant equation: EC(a+b+c+d)=e+f, where a=b=c=d=e=f. In the embodiment, when the first constant and the second constant are 0, the above constant equation is satisfied, and therefore, in the embodiment, the first constant and the second constant can be set to 0.

[0062] In the embodiment, when the bitmap is marked as a write mark, it is indicated that there is data calculated by the erasure code in the storage address, and thus the storage data in the storage address and the first check data corresponding to the storage address are directly read to obtain the return data; when the bitmap is marked as a non-write mark, a first constant is directly determined as the storage data corresponding to the storage address, and a second constant is directly determined as the first check data corresponding to the storage address, so that it can be guaranteed that whether there is data calculated by the erasure code in the storage address, the return data can meet the requirement of the erasure code, and dirty data in the storage address can be avoided.

[0063] In an optional embodiment, the data processing method further includes: in response to system startup, checking the dirty bitmap marks corresponding to the data blocks of each stripe in the storage unit; for any stripe, in a case where it is confirmed that there is a data block with a dirty bitmap mark, reading data in a data block with a non-dirty mark in the stripe; performing erasure code calculation on the data to obtain correction data; writing the correction data into the data block with the dirty mark; and flipping the dirty mark to a non-dirty mark.

[0064] Specifically, when the system starts, it is necessary to check the dirty marks in the storage bitmap of the storage unit, and when a dirty mark is found, stripe recovery needs to be performed, indicating that the storage system has experienced an unexpected stop, including power failure and the like. The method of stripe recovery is to read out the data with a non-dirty mark in the stripe, calculate the correction data through EC operation, write the correction data into the data block corresponding to the dirty mark through concurrent or non-concurrent mode, and clear the dirty mark corresponding to the data block. The searching method includes direct traversal searching and optimized searching through a Bloom filter.

[0065] In the embodiment, at each startup of the storage system, the dirty bitmap marks corresponding to the data blocks of each stripe in the storage unit are traversed to find the data block with dirty data in the stripe in time and repair it, so that data loss caused by power failure of the storage system can be avoided.

[0066] The following describes the specific steps of writing data in the stripe of the persistent memory through local updating, which further introduces the data processing method of the application.

[0067] When data is written, the stripe and the target data block of the stripe corresponding to the written data are first calculated according to the writing position. Then the following process is performed, as shown in Figure 3 .

[0068] a. When the data in the target data block is read, the bitmap mark of the target data block is first checked, and if no write mark is found, a constant of the identity equation is returned, including 0, otherwise the existing data in the target data block and the check data in the check block corresponding to the stripe are read, and the data is returned to obtain the return data.

[0069] b. Incrementally calculate the existing data in the return data or the constant of the constant equation with the data to be written, and the incrementally calculated data can be: XOR operation.

[0070] c. Calculate the new check data by inputting the incrementally calculated data and the check data calculated in b into the EC update operation.

[0071] d. Set the dirty bit map mark of the position of the persistent memory to the dirty mark.

[0072] e. Write the data to be written into the position of the data.

[0073] f. Write the new check data into the original check position, wherein step f and step e can be concurrently executed, but the concurrent number is less than or equal to the number of check blocks.

[0074] g. If the mark corresponding to the target data block in the persistent memory bit map mark is the unwritten mark, flip the mark corresponding to the target data block in the persistent memory bit map mark to the written mark.

[0075] h. Set the dirty bit map mark of the position of the persistent memory to the non-dirty mark.

[0076] The data processing method provided by the embodiments of the present application, when large-scale strip erasure code is used to save data, only needs to confirm the bit map mark corresponding to the storage address each time data is written in the storage address, and write data in the storage address based on the confirmation result of the bit map mark, and does not need to ensure that the data in the strip is all calculated by erasure code, and reconstruct the data in the strip, that is, the data can be directly written in the strip by the local update mode without reconstructing the data, thereby saving the time spent on reconstructing the data in the strip, and improving the user experience.

[0077] According to the embodiments of the present application, the present application also provides an electronic device and a readable storage medium.

[0078] Figure 4 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0079] AsFigure 4 As shown, the device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0080] Various components in the device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc., an output unit 807, such as various types of displays, speakers, etc., a storage unit 808, such as a magnetic disk, an optical disk, etc., and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0081] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the data processing method by any other appropriate means, such as by means of firmware.

[0082] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0083] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0084] In the context of the present application, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0085] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0086] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0087] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0088] It should be understood that various forms of flow shown above can be used, re-ordered, added to, or deleted from without departing from the technology disclosed in this application. For example, the steps recited in this application can be executed in parallel, in series, or in different orders, as long as the desired results of the technology disclosed in this application are achieved, and this application is not limited herein.

[0089] In addition, the terms "first", "second", etc., are used herein only to describe different instances, and do not imply or suggest relative importance or an implied number of the indicated technical features. Thus, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "a plurality" is two or more, unless otherwise specifically limited.

[0090] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method comprising: determining a storage address corresponding to data to be written / read in response to a write / read data request; determining a bitmap flag corresponding to the storage address, the bitmap flag being used to represent whether data calculated by an erasure code exists in the storage address; writing data to be written in the storage address according to the bitmap flag, or reading data to be read from the storage address according to the bitmap flag; writing data to be written in the storage address according to the bitmap flag comprises: obtaining storage data corresponding to the storage address and first check data corresponding to the storage address according to the bitmap flag to obtain return data; performing erasure code calculation on the return data and the data to be written to obtain second check data; setting a dirty bitmap flag corresponding to the storage address to a dirty flag; the dirty bitmap flag is used to represent whether storage data in the storage address needs to be recovered; writing the data to be written in the storage address, replacing the first check data corresponding to the storage address with the second check data; and setting the dirty bitmap flag corresponding to the storage address to a non-dirty flag.

2. The data processing method of claim 1, wherein the determining the bitmap flag corresponding to the storage address comprises: determining a target data block corresponding to the storage address, the target data block being one data block in a stripe; determining a bitmap flag of the target data block; and taking the bitmap flag of the target data block as the bitmap flag corresponding to the storage address.

3. The data processing method of claim 1, wherein the performing erasure code calculation on the return data and the data to be written to obtain second check data comprises: performing incremental calculation on the storage data corresponding to the storage address in the return data and the data to be written to obtain incremental data; and performing erasure code calculation on the incremental data and the first check data corresponding to the storage address in the return data to obtain second check data.

4. The data processing method of claim 1, wherein the reading data to be read from the storage address according to the bitmap flag comprises: obtaining storage data corresponding to the storage address and first check data corresponding to the storage address according to the bitmap flag to obtain return data.

5. The data processing method of claim 1 or 4, wherein the obtaining storage data corresponding to the storage address and first check data corresponding to the storage address according to the bitmap flag to obtain return data comprises: in a case where the bitmap flag is determined to be a write flag, reading storage data in the storage address and first check data corresponding to the storage address to obtain return data; and in a case where the bitmap flag is determined to be a non-write flag, determining a first constant as the storage data corresponding to the storage address and a second constant as the first check data corresponding to the storage address, and obtaining return data based on the first constant and the second constant, wherein the first constant and the second constant both satisfy that an erasure code calculated by the constant is equal to the constant. The write mark is used to represent that the data calculated by the erasure code exists in the storage address, and the non-write mark is used to represent that the data calculated by the erasure code does not exist in the storage address.

6. The data processing method of claim 1, before the determining the storage address corresponding to the data to be written / read in response to the write / read data request, further comprising: allocating a plurality of data blocks of a plurality of stripes from the storage unit as a data storage space according to a data storage capacity requested by a user; initializing the bitmap marks corresponding to the plurality of data blocks of the plurality of stripes.

7. The data processing method of claim 6, further comprising: checking the dirty bitmap marks corresponding to the data blocks of each stripe in the storage unit in response to a system startup; for any stripe, in a case where it is confirmed that there is a data block with a dirty bitmap mark, reading data in a data block with a non-dirty mark in the stripe; performing erasure code calculation on the data to obtain corrected data; writing the corrected data into the data block with the dirty bitmap mark; flipping the dirty mark to a non-dirty mark.

8. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform the data processing method of any one of claims 1-7.

Images