Data processing method and apparatus, and electronic device

Abstract

The application discloses a data processing method and device and electronic equipment. When writing to-be-written data, it is determined whether there is first data in the to-be-written data. If the first data exists, a first data identifier of the first data and a corresponding first storage area are determined. A second data identifier of to-be-written second data is generated, and a second data area corresponding to the second data identifier is written. In this way, when a data rollback instruction is received, a corresponding target storage area can be determined based on a target data identifier, and rollback data can be obtained. Related data can be written based on a data identifier, and a low-efficiency backup mode of reading and writing is not required to support rollback. The data can be rolled back based on the data identifier, and the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and more specifically to a data processing method, apparatus, and electronic device. Background Technology

[0002] With the widespread adoption of software-defined storage, rich interfaces are provided, and business applications are widely supported. The amount of data stored in these systems is also increasing rapidly. Based on common replication mechanisms, cost pressures rise exponentially with the data volume. Therefore, more and more users are adopting error-correcting coding (EC) as a data redundancy method. EEC technology requires backups of the original data to enable data to be rewritten and redirected to the appropriate state when needed.

[0003] For systems that support erasure coding based on the file system, the only way to modify and write data backup is to copy the data out and then write the new data. This will cause severe write amplification due to erasure coding, resulting in low modification and write performance. Summary of the Invention

[0004] In view of the above, this application provides the following technical solution:

[0005] A data processing method, comprising:

[0006] If there is first data corresponding to the received data to be written, determine the first data identifier of the first data, so that the first storage area corresponding to the first data can be obtained through the first data identifier;

[0007] A second data identifier is generated for the data to be written. The second data identifier is different from the first data identifier and corresponds to a second storage area that is different from the first storage area.

[0008] Write the data to be written into the second storage area;

[0009] In response to receiving a data rollback command, obtain the target data identifier corresponding to the data rollback command;

[0010] The target storage area is determined based on the target data identifier, so that rollback data can be obtained through the target storage area.

[0011] Optionally, the method further includes:

[0012] Generate an index for the stored data in the storage area, wherein the index represents the mapping relationship between the location information and data identifiers of the stored data in the storage area;

[0013] The index is stored in the index storage area;

[0014] The step of determining the target storage area based on the target data identifier includes:

[0015] Invoke the index of the index storage area, and obtain the target storage area corresponding to the target data identifier based on the index;

[0016] Obtain the rollback data of the target storage area.

[0017] Optionally, the method further includes:

[0018] If no first data corresponding to the data to be written exists, the data to be written is written to a free storage area.

[0019] Optionally, writing the data to be written to the second storage area includes:

[0020] Based on the index, obtain the address where the data to be written to the second storage area;

[0021] Based on the address to be written, determine the writable length of the data in the second storage area;

[0022] If the writable length of the data is greater than or equal to the length of the data to be written, the data to be written is written to the second storage area;

[0023] If the writable length of the data is less than the length of the data to be written, a data extension area is determined, and the data to be written is written into the data extension area.

[0024] Optionally, the method further includes:

[0025] The index is updated based on the write address of the data extension region corresponding to the data to be written and the second data identifier of the data to be written.

[0026] Optionally, obtaining rollback data through the target storage area includes:

[0027] Based on the location information of the stored data corresponding to the target data identifier, determine whether the data extension area is included;

[0028] If not, retrieve the target data from the target storage area;

[0029] If so, acquire the target data in the data extension area.

[0030] Optionally, the data rollback instruction includes an erasure coding rollback request, and the method further includes:

[0031] In response to the completion of data rollback of the rollback data, rollback status information is added to the data identifier corresponding to the rollback data.

[0032] Optionally, the method further includes:

[0033] If the target storage area stores data with multiple different data identifiers, check if any data meets the deletion conditions. If so, delete the data that meets the deletion conditions in the target storage area.

[0034] Update the index of the index storage area based on the deleted data.

[0035] A data processing apparatus, comprising:

[0036] The first determining unit is configured to determine a first data identifier of the first data if there is first data corresponding to the received data to be written, so that the first storage area corresponding to the first data can be obtained through the first data identifier.

[0037] A generation unit is used to generate a second data identifier for the data to be written, wherein the second data identifier is different from the first data identifier and corresponds to a second storage area that is different from the first storage area.

[0038] A writing unit is used to write the data to be written into the second storage area;

[0039] The acquisition unit is configured to acquire the target data identifier corresponding to the data rollback instruction in response to receiving the data rollback instruction;

[0040] The second determining unit is used to determine the target storage area based on the target data identifier, so that rollback data can be obtained through the target storage area.

[0041] An electronic device, comprising:

[0042] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the data processing method as described in any of the preceding claims.

[0043] As can be seen from the above technical solutions, this application discloses a data processing method, apparatus, and electronic device. When writing data to be written, it determines whether first data exists in the data to be written; if it exists, it determines a first data identifier and a corresponding first storage area; it generates a second data identifier to be written and writes it to the corresponding second data area. Thus, upon receiving a data rollback command, it can determine the corresponding target storage area based on the target data identifier and obtain the rollback data. This enables relevant data to be written based on data identifiers, eliminating the need for inefficient read-write backup methods to support rollback, and allowing rollback to the corresponding data based on the data identifier, thereby improving data processing efficiency. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0046] Figure 2 A flowchart illustrating a method for writing data to be written, provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram illustrating the overall structure and principle of metadata provided in this application embodiment;

[0048] Figure 4 A schematic diagram of a data writing process provided in an embodiment of this application;

[0049] Figure 5 A schematic diagram of a data rollback process provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a data processing method applied to the field of erasure coding. Faced with ever-increasing data volumes, more and more users are adopting erasure coding as a data redundancy method. To ensure that modified writes can be rolled back to the previous consistent state when needed, it is necessary to back up the original data. For systems supporting file-based erasure coding, the only method for backing up modified write data is to copy the data and then write new data, resulting in severe write amplification during erasure coding modifications and writes, leading to reduced performance. To solve the above problems, this application extends the kernel-mode file system metadata storage structure to support multiple data identifiers for data at the same location. This allows overwritten data to be distinguished by data identifiers, eliminating the need for inefficient read-write backup methods to support rollback. It enables rollback to any version of the data without additional read-write operations. Erasure coding (EC) is a data protection method that divides data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations.

[0053] See Figure 1 The diagram illustrates a data processing method provided in an embodiment of this application, which may include the following steps:

[0054] S101. If there is first data corresponding to the received data to be written, determine the first data identifier of the first data.

[0055] The data to be written is data that needs to be written to a storage area. However, there are different partitions within the storage area, corresponding to different storage regions. Each storage region may already contain data or may be a free storage region. The data stored in different storage regions are related. For example, the data from two adjacent storage regions, after being combined, could represent data generated in adjacent time periods. Alternatively, data of the same type or with the same attributes can be allocated to specific storage regions.

[0056] The first data corresponding to the data to be written is data with the same attributes as the data to be written, such as data collected from the target device in the target state, even if they are collected at different times or with different versions. Each data identifier corresponds to data stored in its corresponding storage area, but data with the same attributes can belong to the same set of storage areas. Therefore, the first storage area corresponding to the first data can be obtained through the first data identifier. The first data identifier may include globally unique information such as the data version number.

[0057] S102, Generate a second data identifier for the data to be written.

[0058] S103. Write the data to be written to the second storage area.

[0059] After determining the first data identifier of the first data, a second data identifier of the data to be written is generated based on the first data identifier. That is, the second data identifier is different from the first data identifier. Since the second data identifier also has a corresponding second storage area, the second storage area is different from the first storage area.

[0060] Therefore, a second storage area is allocated corresponding to the second data identifier, and the data to be written is written to the second storage area.

[0061] In this embodiment, data is distinguished by adding data identifiers. Different data identifiers correspond to different storage areas, and the storage areas of related data can belong to the same data storage set. When the data identifier is the version information of the data, this enables multiple versions of data corresponding to the same data storage set, thus achieving version management for overwritten data.

[0062] S104. In response to receiving a data rollback command, obtain the target data identifier corresponding to the data rollback command.

[0063] S105. Determine the target storage area based on the target data identifier, so that rollback data can be obtained through the target storage area.

[0064] Data rollback refers to restoring data to its previous correct state when a data processing error occurs. In the embodiments of this application, a data rollback instruction refers to a data recovery instruction. Since data in the same location (i.e., the same data storage set) has identification information, upon receiving a data rollback instruction, the data can be restored to the corresponding identified data state, and the resulting rollback data is the data in that state. Specifically, the corresponding target data identifier is obtained based on the data rollback instruction. Then, the target storage area can be determined based on this data identifier, and the rollback data, i.e., the data to be restored to the corresponding version, can be obtained through this target storage area.

[0065] Therefore, in this embodiment, there is no longer a need for the inefficient backup method of reading and writing to roll back, and it is possible to roll back to the data corresponding to any data identifier without adding extra reading and writing.

[0066] To facilitate the use of data identifiers for stored data, in one possible implementation of this application, an index can be generated for the stored data in a storage area. This index represents the mapping relationship between the location information of the stored data in the storage area and the data identifiers. That is, a one-to-one correspondence is established between the location information of the stored data and the data identifier; for example, a first storage area corresponds to the first data identifier of first data, and a second storage area corresponds to the second data identifier of second data. This allows the corresponding data storage area to be found based on the data identifier. To facilitate index retrieval, the index is stored in an index storage area. This means that storing the index in a specific area facilitates its use and updating without affecting the retrieval and processing of other data.

[0067] The process of determining the target storage region based on the target identifier includes: calling the index of the index storage region and obtaining the target storage region corresponding to the target data identifier based on the index; and obtaining the rollback data of the target storage region. This enables the lookup of the storage region corresponding to the data identifier using an index.

[0068] In the above embodiments of this application, writing the data to be written refers to overwriting the data, such as when there is a previous version of the data to be written. If the data to be written does not have a corresponding previous version of the data, then the writing process of the data to be written is equivalent to writing ordinary data, that is, writing to its corresponding storage area, i.e., the free storage area.

[0069] See Figure 2 The illustration shows a flowchart of a data writing method provided in the application embodiment, the method comprising:

[0070] S201. Based on the index, obtain the address of the data to be written in the second storage area;

[0071] S202. Determine the writable length of data in the second storage area based on the address to be written;

[0072] S203. If the writable length of the data is greater than or equal to the length of the data to be written, the data to be written is written to the second storage area.

[0073] S204. If the writable length of the data is less than the length of the data to be written, determine the data extension area and write the data to be written into the data extension area.

[0074] Because the index records the mapping relationship between the location information of the stored data and the data identifier, the actual area to be written can be determined by the index and the length of the data to be written. Specifically, this can be achieved based on the current write offset, length, and data length, which will be explained in detail in subsequent embodiments and will not be elaborated here.

[0075] Since the index records the location of the stored data, when the data to be written is written to the corresponding data extension area, the index will be updated based on the write address of the data extension area corresponding to the data to be written and the second data identifier of the data to be written, so that the accurate location information of the data can be obtained when using the index to find the data later.

[0076] Based on the above embodiments, obtaining rollback data through the target storage area includes:

[0077] Based on the location information of the stored data corresponding to the target data identifier, determine whether to include the data extension area;

[0078] If not, retrieve the target data from the target storage area;

[0079] If so, retrieve the target data in the data extension area.

[0080] In this implementation, target data can be obtained based on the accurate location information of the stored data, allowing the data to be rolled back to the accurate state.

[0081] When the data rollback instruction includes an erasure coding rollback request, the method further includes:

[0082] In response to the completion of the data rollback, rollback status information is added to the data identifier corresponding to the rollback data. This rollback status information may include rollback time, rollback version information, etc., facilitating the retrieval of detailed data information.

[0083] In one possible implementation of the embodiments of this application, if the target storage area stores data with multiple different data identifiers, it is detected whether there is data that meets the deletion conditions. If so, the data that meets the deletion conditions is deleted from the target storage area; the index of the index storage area is updated according to the deleted data.

[0084] The deletion criteria here can be determined based on the actual application scenario, such as data status information or data write time. For example, after the data has been completely written to disk, the data is deleted from its corresponding storage area (target storage area) to free up storage capacity. The index is then updated based on the deleted data, ensuring that the index reflects the actual state of the data in real time.

[0085] The above data processing method is explained below using data identifiers as data versions. Erasure coding write operations can be differentiated using the following method.

[0086] One possible implementation is to add POSIX (Portable Operating System Interface) to distinguish erasure coding write operations. The POSIX standard defines the interface standard that the operating system should provide to applications. Its execution function could be "ssize_t ecwrite(int fd, uint64_t tid, const void *buf, size_t count)" which can define the following information:

[0087] fd: File handle;

[0088] tid: The transaction number of the current write operation, for example, it can be a version number, which is globally unique;

[0089] buf: The content of the currently written data;

[0090] count: Current length of data being written.

[0091] Another possible implementation is to extend the `writev` function provided by Posix (which represents an interface for storing multiple data together and writing data residing in two or more unconnected buffers at once) to distinguish erasure coding write operations. The function could be `Ssize_t writev(int fd, iovec iov, size_t count)`. The following information can be defined:

[0092] fd: file handle;

[0093] iov: buffer array, user data;

[0094] count: The number of data items.

[0095] By reserving the last element of the iov array for type differentiation, this space is removed during data writing to achieve a special meaning, such as array identifier information.

[0096] For example:

[0097] Struct ecinfo{

[0098] Uint64_t tid; / / Current write transaction

[0099] int magic; / / Magic number, checksum

[0100] }

[0101] This embodiment of the invention also includes metadata structure extension, where metadata is data describing data. An additional 4 bytes are added to the metadata inode to store the corresponding identification information (NAT page id), and the data storage area (page) stores the EC node page (erasure coding node storage area) information in the form of a doubly linked list. The page records the modified content as follows:

[0102] Struct tidinfo{

[0103] Uint64 tid; / / Modify write transaction

[0104] int offset; / / Modifies the file offset to be written.

[0105] int length; / / Modifies the length of the corresponding data block to be written.

[0106] int nid; / / nid extended block address, storing pages exceeding 32 block addresses.

[0107] struct f2fs_blkaddr addr

[32] / / Address of the inbound block

[0108] }

[0109] See Figure 3 This illustrates a schematic diagram of the overall structure and principle of metadata provided in an embodiment of this application. Figure 3 The metadata shown in the embodiment describes a NAT structure, but it can also be used for other structures. The overall structure and principle of the metadata are similar to those in this embodiment, and will not be described in detail here. The NodeAddress Table, or NAT for short, is a structure used by F2FS to manage nodes. It records that each node is assigned a nodeID (nid for short, node identifier). The system can use the nid to find the physical address of the node on the flash device from the NAT, and then read it from the flash device.

[0110] Typically, F2FS divides the entire volume into the following areas: SB (super block) contains basic partition information and F2FS default parameters, generated during formatting and cannot be changed; CB (checkpoint) stores the file system state; SIT (segment information table) contains segment information, such as the number of valid blocks and a bitmap of all valid blocks in the main area; NAT (node ​​address table) is a block address table that finds all "node blocks" stored in the main area; SSA (segment summary area) stores summary entries representing the owner information of all blocks in the main area. It should be noted that, because the embodiments of this application facilitate erasure coding overwrite, a node block entry area (NAT entry) is also added, which includes inode (inode) through which relevant information in the index information table can be obtained, and also includes block address (block address) representing the location information of the data block.

[0111] The index list in this application includes inodes (index nodes) which are used to store index information and ec nid (erasure code block address identifier). The version information and write position information corresponding to erasure code data can be obtained through this index list.

[0112] exist Figure 3 The middle section shows the structure of an erasure coding node (EC node), which includes a header, a footer, and an erasure coding entry area (EC entry). The EC entry includes valid and invalid positions (or unused areas). Valid positions record the write operation transaction number (TID), which can be a version number and is a globally unique identifier. Invalid areas record invalid information, and unused areas can be areas where no content has been written. The valid positions can record information such as address, extent nid, write operation transaction number (TID), block count, and offset. Correspondingly, the header and footer content can be customized according to actual application requirements. For example, the header can include: padding, last nid, next nid, previous nid, version, write operation transaction count (TID count), and start position information (start pos). Specifically, the next ec node can be accessed through the next block address identifier, and the previous ec node can be accessed through the previous block address identifier.

[0113] During erasure coding write operations, if the writable data length of the current data region is less than the length of the data to be written, the corresponding extended erasure coding node (extent ec node) can be obtained through the extended block address identifier, thus obtaining the extended area where the data can be written. Then, the data is written to its corresponding data page through the block address of the valid area, and the data is written to the corresponding data page through the extended erasure coding write address.

[0114] This application embodiment is passed through Figure 3 The metadata structure extension shown enables multiple versions of data corresponding to the same location, i.e. Figure 3 Different valid regions in the code can correspond to different tids, which can be different versions. Erasure coding can be used to overwrite data blocks, which can be managed through versions. This allows for rollback to any version without adding extra reads or writes.

[0115] See Figure 4 It illustrates a data writing process provided in this application, including the following steps:

[0116] S301. Determine if the erasure code block address identifier exists. If it does, proceed to step S302; otherwise, proceed to step S303.

[0117] This step is used to determine whether the erasure code is full.

[0118] S302. Check if the current erasure coding data block is full. If it is, proceed to S303; otherwise, proceed to step S304.

[0119] S303, Request a new erasure code block address identifier.

[0120] S304. Is the current write length greater than 128K? If yes, proceed to S305; otherwise, proceed to step S306.

[0121] S305, Request an extended block address.

[0122] S306. Read the block address corresponding to the current write offset and length and record it in the inline block address.

[0123] S307. Read the block address corresponding to the current write offset and length and record it in the large block address.

[0124] S308. Set the original block address to null and update the data block address reverse description information.

[0125] See Figure 5 The diagram illustrates a data rollback process provided in an embodiment of this application.

[0126] S401. Determine if the erasure code block address exists in the current file inode. If it exists, proceed to step S402; otherwise, end.

[0127] S402. Check if the rollback version number is in the list. If it exists, execute S403; otherwise, end.

[0128] S403. Match the block address information covered in the storage area based on the version information.

[0129] S404. Release the block address corresponding to the original location.

[0130] S405. Determine whether an extended data block has been requested. If yes, proceed to step S406; otherwise, proceed to step S407.

[0131] S406. Obtain the extended data block and proceed to step S408.

[0132] S407. Obtain the block address recorded in the inline erasure coding data block and restore it to the original file.

[0133] S408. Read the data block address recorded in the extended block address and restore it to the corresponding position in the original file.

[0134] S409. Update the data status corresponding to the current version to the rolled-back state, update the reverse information, and associate the data file and location information.

[0135] This application provides a data processing method. When writing data to be written, it determines whether first data exists in the data to be written. If it exists, it determines a first data identifier and a corresponding first storage area for the first data. A second data identifier to be written is generated and written to the corresponding second data area. Thus, upon receiving a data rollback command, the corresponding target storage area can be determined based on the target data identifier to obtain the rollback data. This allows related data to be written based on data identifiers, eliminating the need for inefficient read-write backup methods to support rollback. Rollback can be performed based on data identifiers to the corresponding data, improving data processing efficiency.

[0136] See Figure 6 In this application embodiment, a data processing apparatus is also provided, including:

[0137] The first determining unit 10 is configured to determine a first data identifier of the first data if there is first data corresponding to the received data to be written, so that the first storage area corresponding to the first data can be obtained through the first data identifier.

[0138] The generation unit 20 is used to generate a second data identifier for the data to be written, wherein the second data identifier is different from the first data identifier and corresponds to a second storage area that is different from the first storage area;

[0139] The writing unit 30 is used to write the data to be written into the second storage area;

[0140] Acquisition unit 40 is used to acquire the target data identifier corresponding to the data rollback instruction in response to receiving the data rollback instruction;

[0141] The second determining unit 50 is used to determine the target storage area based on the target data identifier, so that rollback data can be obtained through the target storage area.

[0142] In the above embodiments, the device further includes:

[0143] An index generation unit is used to generate an index for the stored data in a storage area, wherein the index represents the mapping relationship between the location information and data identifiers of the stored data in the storage area;

[0144] An index storage unit is used to store the index in an index storage area;

[0145] Specifically, the second determined unit is used for:

[0146] Invoke the index of the index storage area, and obtain the target storage area corresponding to the target data identifier based on the index;

[0147] Obtain the rollback data of the target storage area.

[0148] Based on the above embodiments, the writing unit is further configured to:

[0149] If no first data corresponding to the data to be written exists, the data to be written is written to a free storage area.

[0150] Based on the above embodiments, the writing unit is specifically used for:

[0151] Based on the index, obtain the address where the data to be written to the second storage area;

[0152] Based on the address to be written, determine the writable length of the data in the second storage area;

[0153] If the writable length of the data is greater than or equal to the length of the data to be written, the data to be written is written to the second storage area;

[0154] If the writable length of the data is less than the length of the data to be written, a data extension area is determined, and the data to be written is written into the data extension area.

[0155] Based on the above embodiments, the device further includes:

[0156] The index update unit is used to update the index based on the write address of the data extension region corresponding to the data to be written and the second data identifier of the data to be written.

[0157] Based on the above embodiments, the second determining unit is further configured to:

[0158] Based on the location information of the stored data corresponding to the target data identifier, determine whether the data extension area is included;

[0159] If not, retrieve the target data from the target storage area;

[0160] If so, acquire the target data in the data extension area.

[0161] Based on the above embodiments, the data rollback instruction includes an erasure coding rollback request, and the apparatus further includes:

[0162] An information adding unit is used to add rollback status information to the data identifier corresponding to the rollback data in response to the completion of data rollback of the rollback data.

[0163] Based on the above embodiments, the device further includes:

[0164] The deletion unit is used to detect whether there is data that meets the deletion conditions if the target storage area stores data with multiple different data identifiers; if so, delete the data that meets the deletion conditions in the target storage area.

[0165] The index update unit is also used to update the index of the index storage area based on the deleted data.

[0166] Based on the foregoing embodiments, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the data processing methods described above, as well as sub-steps or associated steps related to the data processing method.

[0167] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the data processing method as described above.

[0168] It should be noted that the aforementioned processor or CPU can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. Understandably, the electronic device implementing the above processor function can also be other types, and this application does not specifically limit its capabilities.

[0169] It should be noted that the aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; it can also be various terminals that include one or any combination of the above-mentioned memory, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0171] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, in the various embodiments of this application, all functional units can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0174] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0175] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0176] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method applied to the field of erasure coding, wherein the erasure coding segments data into fragments and expands and encodes redundant data blocks, the method comprising: If there is first data corresponding to the received data to be written, determine the first data identifier of the first data, so that the first storage area corresponding to the first data can be obtained through the first data identifier, and the write operation of the data to be written is erasure coding data overwrite; A second data identifier is generated for the data to be written. The second data identifier is different from the first data identifier. The second data identifier corresponds to a second storage area that is different from the first storage area. The second storage area and the first storage area belong to the same data storage set. The kernel-mode file system metadata storage structure is extended to enable multiple data identifiers for data at the same location. Write the data to be written into the second storage area; In response to receiving a data rollback command, obtain the target data identifier corresponding to the data rollback command; The target storage area is determined based on the target data identifier, so that rollback data can be obtained through the target storage area.

2. The method according to claim 1, further comprising: Generate an index for the stored data in the storage area, wherein the index represents the mapping relationship between the location information and data identifiers of the stored data in the storage area; The index is stored in the index storage area; The step of determining the target storage area based on the target data identifier includes: Invoke the index of the index storage area, and obtain the target storage area corresponding to the target data identifier based on the index; Obtain the rollback data of the target storage area.

3. The method according to claim 1, further comprising: If no first data corresponding to the data to be written exists, the data to be written is written to a free storage area.

4. The method according to claim 2, wherein writing the data to be written to the second storage area comprises: Based on the index, obtain the address where the data to be written to the second storage area; Based on the address to be written, determine the writable length of the data in the second storage area; If the writable length of the data is greater than or equal to the length of the data to be written, the data to be written is written to the second storage area; If the writable length of the data is less than the length of the data to be written, a data extension area is determined, and the data to be written is written into the data extension area.

5. The method according to claim 4, further comprising: The index is updated based on the write address of the data extension region corresponding to the data to be written and the second data identifier of the data to be written.

6. The method according to claim 4, wherein obtaining rollback data through the target storage area comprises: Based on the location information of the stored data corresponding to the target data identifier, determine whether the data extension area is included; If not, retrieve the target data from the target storage area; If so, acquire the target data in the data extension area.

7. The method according to claim 1, wherein the data rollback instruction includes an erasure coding rollback request, and the method further includes: In response to the completion of data rollback of the rollback data, rollback status information is added to the data identifier corresponding to the rollback data.

8. The method according to claim 2, further comprising: If the target storage area stores data with multiple different data identifiers, check if any data meets the deletion conditions. If so, delete the data that meets the deletion conditions in the target storage area. Update the index of the index storage area based on the deleted data.

9. A data processing apparatus, applied in the field of erasure coding, wherein the erasure coding segments data into fragments and expands and encodes redundant data blocks, the apparatus comprising: The first determining unit is configured to determine a first data identifier of the first data if there is first data corresponding to the received data to be written, so that the first storage area corresponding to the first data can be obtained through the first data identifier, and the writing operation of the data to be written is erasure coding data overwrite. A generation unit is used to generate a second data identifier for the data to be written. The second data identifier is different from the first data identifier. The second data identifier corresponds to a second storage area that is different from the first storage area. The second storage area and the first storage area belong to the same data storage set. The kernel-mode file system metadata storage structure is extended to enable multiple data identifiers for data at the same location. A writing unit is used to write the data to be written into the second storage area; The acquisition unit is configured to acquire the target data identifier corresponding to the data rollback instruction in response to receiving the data rollback instruction; The second determining unit is used to determine the target storage area based on the target data identifier, so that rollback data can be obtained through the target storage area.

10. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the data processing method of any one of claims 1-8.

Images