Erasure code data updating method and apparatus

Abstract

Disclosed in the present application are an erasure code data updating method and apparatus. The method comprises: in response to a write operation for a target shard of a target object in a main object storage device, acquiring a state identifier of the target shard; if the state identifier is a first state identifier, storing, in a reserved object in a parity object storage device, initial data in the target shard and initial parity data for the target object, using updated data to overwrite the initial data, and modifying the state identifier to a second state identifier; if the state identifier is the second state identifier, using the updated data to overwrite current data in the target shard and updated data that is stored in the reserved object; and when a distributed object storage system is idle, determining target parity data on the basis of the initial parity data and the initial data and the updated data that are currently stored in the reserved object, and modifying the state identifier to the first state identifier.

Description

Erasure code data updating method and device

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023117237174, filed on December 14, 2023, entitled “Error Code Data Update Method and Device,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of distributed storage technology, and more specifically, to a method and device for updating erasure code data. Background Art

[0004] When updating data, traditional erasure codes need to read all data blocks within the stripe containing the updated data, calculate the code, and then update all data blocks and check blocks. This results in low cluster performance and additional read and write overhead, which cannot meet users' high-performance requirements.

[0005] The existing solutions are as follows: One is to read the original data block of the unupdated part each time the data is updated, merge it with the updated data, calculate the erasure code, and then write it into the entire erasure code group. However, this solution requires multiple reads and writes and erasure code calculations each time the data is updated, which consumes the cluster disk performance as well as the cluster CPU (Central Processing Unit). Unit, central processing unit); the second is to update the data of the erasure code based on the tree structure, that is, k optimal update trees are generated according to the number of network hops from each data node to all verification nodes. After receiving the information, the data node sends the updated data block and other information along the structure of the optimal update tree to all verification nodes. However, this scheme only considers the shortest network distance and does not consider the problem of fault domain. When the cluster network is abnormal, it is easy to cause data loss, which in turn affects data security; the third is to update the erasure code based on the replica data log, that is, each updated data is saved to the data log. When the data log is full, it starts to be recycled, reads the corresponding data block and data log record to generate a check difference, and forwards it to the verification end. When the check log is full, the check data is generated by the check difference and the check block is updated. This scheme frequently performs check difference calculation and network transmission, which will greatly affect the cluster performance. At the same time, the log space appends multiple data changes at the same location, which consumes a lot of system resources and causes waste of system resources.

[0006] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0007] Summary of the Invention

[0008] In a first aspect, an embodiment of the present application provides an erasure code data update method, comprising: in response to a write operation to write updated data to a target shard of a target stripe of a target object in a primary object storage device, obtaining a status identifier of the target shard; if the status identifier is a first status identifier indicating that the data in the target shard has not been overwritten, reading the initial data in the target shard, obtaining initial verification data corresponding to the target object, overwriting the initial data with the updated data, and modifying the first status identifier to a second status identifier indicating that the data in the target shard has been overwritten, storing the initial data and the updated data in a reserved object in the verification object storage device; if the status identifier is the second status identifier, overwriting the current data in the target shard with the updated data, and overwriting the updated data currently stored in the reserved object with the updated data; when the distributed object storage system is idle, determining the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, and modifying the status identifier of the target shard to the first status identifier.

[0009] In some embodiments, in response to a write operation to write updated data to a target shard of a target stripe of a target object in a primary object storage device, obtaining a status identifier of the target shard includes: in response to a write operation in a client to write updated data to a target object in the primary object storage device, distributing an update task to the target shard of the target stripe of the target object through the primary object storage device; obtaining attribute information of the target shard based on a map structure, and determining a status identifier in the attribute information, wherein, in the attribute information, the identifier of the target stripe is used as a key, and the identifier and status identifier of the target shard are used as values.

[0010] In some embodiments, before obtaining the status identifier of the target shard, the method also includes: comparing the data length of the updated data and the shard length of the target shard; if the data length is not greater than the shard length, continuing to obtain the status identifier of the target shard; if the data length is greater than the shard length, stopping the update operation and generating a prompt message, wherein the prompt message is used to prompt that the updated data is too long and cannot be updated to the target shard.

[0011] In some embodiments, after overwriting the updated data currently stored in the reserved object with the updated data, the method further includes: returning the updated data to the primary object storage device, and returning the updated data to the client via the primary object storage device.

[0012] In some embodiments, when the distributed object storage system is idle, the target verification data corresponding to the target object is determined based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, including: periodically using a performance analysis tool to detect the load operation status of the distributed object storage system, and when the load operation status meets the preset idle standard, triggering the correction and erasure code verification data merging operation, and determining the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data.

[0013] In some embodiments, target verification data corresponding to the target object is determined based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, including: determining first difference data between the initial data currently stored in the reserved object and the updated data; determining the product of the first difference data and a preset matrix coding coefficient to obtain second difference data; and merging the second difference data with the initial verification data to obtain target verification data.

[0014] In some embodiments, in response to a read operation on a target shard in a client, a status identifier of the target shard is obtained; if the status identifier is a first status identifier, the initial data in the target shard is read, and the initial data is fed back to the client; if the status identifier is a second status identifier, the updated data currently stored in the reserved object is read, and the updated data currently stored in the reserved object is fed back to the client.

[0015] In a second aspect, an embodiment of the present application further provides an erasure code data update device, comprising: an acquisition module for acquiring a status identifier of a target shard in response to a write operation of writing updated data to a target shard of a target stripe of a target object in a primary object storage device; a first update module for reading the initial data in the target shard and acquiring initial verification data corresponding to the target object when the status identifier is a first status identifier indicating that the data in the target shard has not been overwritten, overwriting the initial data with the updated data, and modifying the first status identifier to a second status identifier indicating that the data in the target shard has been overwritten, and storing the initial data and the updated data in a reserved object in the verification object storage device; a second update module for overwriting the current data in the target shard with the updated data and overwriting the updated data currently stored in the reserved object with the updated data when the status identifier is the second status identifier; a determination module for determining, when the distributed object storage system is idle, the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, and modifying the status identifier of the target shard to the first status identifier.

[0016] In a third aspect, an embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned erasure code data update method by running the computer program.

[0017] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned erasure code data update method through the computer program.

[0018] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] FIG1 is a schematic diagram of the structure of a distributed object storage system according to an embodiment of the present application;

[0021] FIG2 is a schematic structural diagram of a computer terminal according to an embodiment of the present application;

[0022] FIG3 is a schematic flow chart of a method for updating erasure code data according to an embodiment of the present application;

[0023] FIG4 is a flowchart of a write operation in a method for updating erasure code data according to an embodiment of the present application;

[0024] FIG5 is a flowchart of a read operation in a method for updating erasure code data according to an embodiment of the present application;

[0025] FIG6 is a schematic structural diagram of an erasure code data updating device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] To help those skilled in the art better understand the present invention, the following will provide a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. It should be understood that the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0028] In order to better understand the embodiments of the present application, some nouns or terms that appear in the description of the embodiments of the present application are first translated and explained as follows:

[0029] Erasure Coding (EC): is a data protection method that divides data into fragments, expands and encodes redundant data blocks, and stores them in different locations.

[0030] Distributed Storage System: A system that stores data in a distributed manner on multiple independent devices.

[0031] Object Storage Device (OSD): is a storage device management process in a distributed storage system, mainly used for storing data, replicating data, balancing data, and recovering data.

[0032] Ceph: is a unified, distributed file system designed for excellent performance, reliability, and scalability.

[0033] Object: It is the lowest-level storage unit of Ceph. Each object contains metadata and original data. When users want to store data in a Ceph cluster, the stored data will be divided into multiple objects.

[0034] Example 1

[0035] When updating data, traditional erasure codes require multiple reads and writes and erasure code calculations, and multiple data changes at the same location must be recorded in the log space. This approach greatly affects cluster performance and consumes a large amount of system resources, thereby wasting system resources. In order to solve the above problems, a distributed object storage system is first proposed in an embodiment of the present application. As shown in Figure 1, the system includes: a primary object storage device 11 for storing original data and a verification object storage device 12 for storing verification data.

[0036] Based on the above-mentioned distributed object storage system, an embodiment of the present application further provides a method for updating erasure coded data implemented by the distributed object storage system. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] The distributed object storage system provided in the embodiments of the present application can be a mobile terminal, a computer terminal, or a similar computing device. Figure 2 shows a hardware block diagram of a computer terminal (or mobile device) for implementing an erasure code data update method. As shown in Figure 2, the computer terminal 20 (or mobile device 20) can include one or more processors 202 (illustrated as 202a, 202b, ..., 202n in the figure) (the processor 202 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 204 for storing data, and a transmission device 206 for communication functions. In addition, it can also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that the structure shown in Figure 2 is merely illustrative and does not limit the structure of the electronic device described above. For example, the computer terminal 20 can also include more or fewer components than shown in Figure 2, or have a configuration different from that shown in Figure 2.

[0038] It should be noted that the one or more processors 202 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 20 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0039] The memory 204 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the erasure code data update method in the embodiment of the present application. The processor 202 executes various functional applications and data processing by running the software programs and modules stored in the memory 204, that is, implementing the vulnerability detection method of the above-mentioned application. The memory 204 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 204 may further include a memory remotely located relative to the processor 202, and these remote memories may be connected to the computer terminal 20 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0040] Transmission device 206 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of computer terminal 20. In one embodiment, transmission device 206 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 206 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0041] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 20 (or mobile device).

[0042] In the above operating environment, an embodiment of the present application provides a method for updating erasure code data, as shown in FIG3 , which includes the following steps:

[0043] Step S302 : in response to a write operation of writing updated data to a target slice of a target stripe of a target object in a primary object storage device, obtaining a status identifier of the target slice.

[0044] As an optional implementation, before obtaining the status identifier of the target shard, it is first necessary to compare the data length of the updated data and the shard length of the target shard; if the data length is not greater than the shard length, continue to obtain the status identifier of the target shard; if the data length is greater than the shard length, stop the update operation and generate a prompt message, wherein the prompt message is used to prompt that the updated data is too long and cannot be updated to the target shard.

[0045] When the data length is not greater than the shard length, the status identifier of the target shard can be obtained through the following method: in response to a write operation in the client to write updated data to the target object in the primary object storage device, distribute the update task to the target shard of the target stripe of the target object through the primary object storage device; obtain the attribute information of the target shard based on the map structure, and determine the status identifier in the attribute information, where, in the attribute information, the identifier of the target stripe is used as the key, and the identifier and status identifier of the target shard are used as the values.

[0046] In this embodiment, taking the Ceph distributed storage system as an example, the corresponding object storage device can be an OSD, the target object can be an object, and correspondingly, the attribute information of the target shard based on the map structure can record whether the nth shard of the mth stripe in an object is repeatedly hit for writing. The specific attribute information can be: std::map<stripe id, std::vector<std::pair<shard id, flag(overwrite / clean)>>> hitter.

[0047] Step S304, if the status identifier is the first status identifier clean indicating that the data in the target shard has not been overwritten, read the initial data d(0) in the target shard, and obtain the initial check data p(0) corresponding to the target object. Use the updated data d(1) to overwrite the initial data, and modify the first status identifier clean to the second status identifier overwrite indicating that the data in the target shard has been overwritten. Store the initial data d(0) and the updated data d(1) in the reserved object in the check object storage device.

[0048] Step S306, if the status identifier is the second status identifier overwrite, use the updated data d(n), n > 1 to overwrite the current data d(n - 1) in the target shard, and use the updated data to overwrite the updated data d(n - 1) currently stored in the reserved object.

[0049] After using the updated data to overwrite the updated data currently stored in the reserved object, the updated data needs to be returned to the primary object storage device, and the updated data is returned to the client through the primary object storage device to ensure strong data consistency.

[0050] Step S308, when the distributed object storage system is idle, determine the target check data corresponding to the target object based on the initial check data, the initial data and the updated data currently stored in the reserved object, and modify the status identifier of the target shard to the first status identifier.

[0051] In some embodiments, the target verification data corresponding to the target object can be determined in the following manner: periodically using a performance analysis tool to detect the load operation status of the distributed object storage system, and when the load operation status meets the preset idle standard, triggering the correction and erasure code verification data merging operation, and determining the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data.

[0052] Among them, the performance analysis tool can use the perf tool, which can usually perform: CPU performance analysis, that is, analyzing the CPU performance bottleneck by monitoring indicators such as CPU utilization, cache hit rate, and number of instruction executions; memory performance analysis, that is, analyzing the memory performance bottleneck by monitoring indicators such as memory usage, memory allocation and release times; disk performance analysis, that is, analyzing the disk performance bottleneck by monitoring indicators such as disk read and write speed and number of I / O operations; network performance analysis, that is, analyzing the network performance bottleneck by monitoring indicators such as network bandwidth, transmission rate, and network latency.

[0053] Specifically, the target verification data corresponding to the target object can be determined based on the initial verification data, the initial data currently stored in the reserved object, and the updated data in the following manner: determining the first difference data between the initial data currently stored in the reserved object and the updated data; determining the product of the first difference data and the preset matrix coding coefficient to obtain the second difference data; merging the second difference data with the initial verification data to obtain the target verification data.

[0054] For example, suppose the initial data currently stored in the reserved object is Update data to The preset matrix coding coefficient is a ij , the initial calibration data is The specific formula for the target verification data is:

[0055] FIG4 shows a schematic flow chart of a write operation in an erasure code data update method. The specific flow of the write operation is as follows:

[0056] S1, in response to a write operation of writing updated data to a target shard of a target object in a primary object storage device;

[0057] S2: Check whether the length of the updated data is greater than the target shard length. If so, execute S3; otherwise, execute S4 and then S5.

[0058] S3, stop the update operation and generate a prompt message;

[0059] S4, obtain the status identifier of the target shard;

[0060] S5: Is the status identifier of the target shard the first status identifier or the second status identifier? If it is the first status identifier, execute S6 and then S7; if it is the second status identifier, execute S8;

[0061] S6, overwriting the initial data in the target shard with the updated data, and changing the first state identifier to the second state identifier;

[0062] S7, storing the initial data and the updated data in a reserved object in the verification object storage device;

[0063] S8, using the updated data to overwrite the current data in the target shard and update the updated data currently stored in the reserved object.

[0064] After completing the above-mentioned update data write operation, in response to the read operation on the target shard in the client, the status identifier of the target shard is obtained; if the status identifier is the first status identifier clean, the initial data d(0) in the target shard is read, and the initial data d(0) is fed back to the client; if the status identifier is the second status identifier overwrite, the updated data d(n) currently stored in the reserved object is read, n>1, and the updated data d(n) currently stored in the reserved object, n>1 is fed back to the client.

[0065] FIG5 shows a schematic flow chart of a read operation in an erasure code data update method. The specific flow of the read operation is as follows:

[0066] S1, responds to the read operation of the target shard in the client;

[0067] S2, obtain the status identifier of the target shard;

[0068] S3: Is the status identifier of the target shard the first status identifier or the second status identifier? If it is the first status identifier, execute step S4; if it is the second status identifier, execute step S5;

[0069] S4, reads the initial data in the target shard and feeds it back to the client;

[0070] S5, read the updated data currently stored in the reserved object and feed it back to the client.

[0071] In an embodiment of the present application, in response to a write operation to write updated data to a target slice of a target stripe of a target object in a primary object storage device, a status identifier of the target slice is obtained; if the status identifier is a first status identifier indicating that the data in the target slice has not been overwritten, the initial data in the target slice is read, and initial verification data corresponding to the target object is obtained, the initial data is overwritten with the updated data, and the first status identifier is modified to a second status identifier indicating that the data in the target slice has been overwritten, and the initial data and the updated data are stored in a reserved object in the verification object storage device; if the status identifier is the second status identifier, the current data in the target slice is overwritten with the updated data, and the updated data currently stored in the reserved object is overwritten with the updated data; when the distributed object storage system is idle, the target verification data corresponding to the target object is determined based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, and the status identifier of the target slice is modified to the first status identifier. Among them, using direct overwrite to perform write update operations can prevent the amplification of write operations and reduce storage space waste; updating data distribution operations are controlled through the main object storage device, and the update efficiency and update security of the correction and correction code are comprehensively considered, ensuring data security and reliability while improving performance; periodically updating the verification data only when the distributed object storage system is idle can reduce CPU consumption and effectively solve the technical problem of [keywords].

[0072] Example 2

[0073] According to an embodiment of the present application, an erasure code data updating device for implementing the erasure code data updating method in Example 1 is also provided. As shown in FIG6 , the erasure code data updating device includes at least: an acquisition module 61, a first update module 62, a second update module 63, and a determination module 64, wherein:

[0074] The acquisition module 61 may acquire a status identifier of a target slice in a target stripe of a target object in a primary object storage device in response to a write operation to write updated data to the target slice.

[0075] As an optional implementation, before obtaining the status identifier of the target shard, it is first necessary to compare the data length of the updated data and the shard length of the target shard; if the data length is not greater than the shard length, continue to obtain the status identifier of the target shard; if the data length is greater than the shard length, stop the update operation and generate a prompt message, wherein the prompt message is used to prompt that the updated data is too long and cannot be updated to the target shard.

[0076] When the data length is not greater than the shard length, the status identifier of the target shard can be obtained in the following manner: In response to a write operation in the client to write updated data to a target object in the primary object storage device, the primary object storage device distributes an update task to the target shard of the target stripe of the target object; obtain the attribute information of the target shard based on the map structure, and determine the status identifier in the attribute information, where, in the attribute information, the identifier of the target stripe serves as the key, and the identifier and status identifier of the target shard serve as the values.

[0077] In this embodiment, taking the Ceph distributed storage system as an example, the corresponding object storage device can be an OSD, the target object can be an object, and correspondingly, the attribute information of the target shard based on the map structure can record whether the nth shard of the mth stripe in an object is repeatedly hit for writing. The specific attribute information can be: std::map<stripe id, std::vector<std::pair<shard id, flag(overwrite / clean)>>hitter.

[0078] The first update module 62 can, when the status identifier is the first status identifier indicating that the data in the target shard has not been overwritten, read the initial data in the target shard, obtain the initial check data corresponding to the target object, overwrite the initial data with the updated data, modify the first status identifier to the second status identifier indicating that the data in the target shard has been overwritten, and store the initial data and the updated data in the reserved object in the check object storage device.

[0079] The second update module 63 can, when the status identifier is the second status identifier, overwrite the current data in the target shard with the updated data, and overwrite the updated data currently stored in the reserved object with the updated data.

[0080] After the second update module overwrites the updated data currently stored in the reserved object with the updated data, it is necessary to return the updated data to the primary object storage device, and the primary object storage device returns the updated data to the client, so as to ensure strong consistency of the data.

[0081] The determination module 64 can, when the distributed object storage system is idle, determine the target check data corresponding to the target object based on the initial check data, the initial data currently stored in the reserved object, and the updated data, and modify the status identifier of the target shard to the first status identifier.

[0082] In some embodiments, the determination module can determine the target verification data corresponding to the target object in the following manner: periodically use a performance analysis tool to detect the load operation status of the distributed object storage system, and when the load operation status meets the preset idle standard, trigger the correction and erasure code verification data merging operation, and determine the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data.

[0083] Among them, the performance analysis tool can use the perf tool, which can usually perform: CPU performance analysis, that is, analyzing the CPU performance bottleneck by monitoring indicators such as CPU utilization, cache hit rate, and number of instruction executions; memory performance analysis, that is, analyzing the memory performance bottleneck by monitoring indicators such as memory usage, memory allocation and release times; disk performance analysis, that is, analyzing the disk performance bottleneck by monitoring indicators such as disk read and write speed and number of I / O operations; network performance analysis, that is, analyzing the network performance bottleneck by monitoring indicators such as network bandwidth, transmission rate, and network latency.

[0084] Specifically, the determination module can determine the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data in the following manner: determine the first difference data between the initial data currently stored in the reserved object and the updated data; determine the product of the first difference data and the preset matrix coding coefficient to obtain the second difference data; merge the second difference data with the initial verification data to obtain the target verification data.

[0085] FIG4 shows a flowchart of a write operation in an optional erasure code data update method. The specific process of the write operation is as follows:

[0086] S1, in response to a write operation of writing updated data to a target shard of a target object in a primary object storage device;

[0087] S2: Check whether the length of the updated data is greater than the target shard length. If so, execute S3; otherwise, execute S4 and then S5.

[0088] S3, stop the update operation and generate a prompt message;

[0089] S4, obtain the status identifier of the target shard;

[0090] S5: Is the status identifier of the target shard the first status identifier or the second status identifier? If it is the first status identifier, execute S6 and then S7; if it is the second status identifier, execute S8;

[0091] S6, overwriting the initial data in the target shard with the updated data, and changing the first state identifier to the second state identifier;

[0092] S7, storing the initial data and the updated data in a reserved object in the verification object storage device;

[0093] S8, using the updated data to overwrite the current data in the target shard and update the updated data currently stored in the reserved object.

[0094] After completing the above-mentioned update data write operation, in response to the read operation on the target shard in the client, the status identifier of the target shard is obtained; if the status identifier is the first status identifier clean, the initial data d(0) in the target shard is read, and the initial data d(0) is fed back to the client; if the status identifier is the second status identifier overwrite, the updated data d(n) currently stored in the reserved object is read, n>1, and the updated data d(n) currently stored in the reserved object, n>1 is fed back to the client.

[0095] FIG5 shows a flowchart of a read operation in an optional erasure code data update method. The specific flow of the read operation is as follows:

[0096] S1, responds to the read operation of the target shard in the client;

[0097] S2, obtain the status identifier of the target shard;

[0098] S3: Is the status identifier of the target shard the first status identifier or the second status identifier? If it is the first status identifier, execute step S4; if it is the second status identifier, execute step S5;

[0099] S4, reads the initial data in the target shard and feeds it back to the client;

[0100] S5, read the updated data currently stored in the reserved object and feed it back to the client.

[0101] It should be noted that the modules in the erasure code data updating device in the embodiment of the present application correspond one-to-one to the implementation steps of the erasure code data updating method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0102] Example 3

[0103] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the erasure code data update method in Example 1 by running the computer program.

[0104] Specifically, the device where the non-volatile storage medium is located implements the following steps by running the computer program: in response to a write operation to write updated data to a target slice of a target stripe of a target object in a primary object storage device, obtaining a status identifier of the target slice; if the status identifier is a first status identifier indicating that the data in the target slice has not been overwritten, reading initial data in the target slice, obtaining initial verification data corresponding to the target object, overwriting the initial data with the updated data, and modifying the first status identifier to a second status identifier indicating that the data in the target slice has been overwritten, storing the initial data and the updated data in a reserved object in the verification object storage device; if the status identifier is the second status identifier, overwriting the current data in the target slice with the updated data, and overwriting the updated data currently stored in the reserved object with the updated data; when the distributed object storage system is idle, determining the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, and modifying the status identifier of the target slice to the first status identifier.

[0105] According to an embodiment of the present application, a processor is further provided, which is used to run a computer program, wherein the erasure code data updating method in Example 1 is executed when the computer program is running.

[0106] Specifically, the computer program executes the following steps when it is run: in response to a write operation to write updated data to a target slice of a target stripe of a target object in a primary object storage device, obtain a status identifier of the target slice; if the status identifier is a first status identifier indicating that data in the target slice has not been overwritten, read initial data in the target slice, obtain initial verification data corresponding to the target object, overwrite the initial data with the updated data, modify the first status identifier to a second status identifier indicating that data in the target slice has been overwritten, and store the initial data and the updated data in a reserved object in a verification object storage device; if the status identifier is the second status identifier, overwrite the current data in the target slice with the updated data, and overwrite the updated data currently stored in the reserved object with the updated data; when the distributed object storage system is idle, determine the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, and modify the status identifier of the target slice to the first status identifier.

[0107] According to an embodiment of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the erasure code data update method in Example 1 through the computer program.

[0108] Specifically, the computer program executes the following steps when it is run: in response to a write operation to write updated data to a target slice of a target stripe of a target object in a primary object storage device, obtain a status identifier of the target slice; if the status identifier is a first status identifier indicating that data in the target slice has not been overwritten, read initial data in the target slice, obtain initial verification data corresponding to the target object, overwrite the initial data with the updated data, modify the first status identifier to a second status identifier indicating that data in the target slice has been overwritten, and store the initial data and the updated data in a reserved object in a verification object storage device; if the status identifier is the second status identifier, overwrite the current data in the target slice with the updated data, and overwrite the updated data currently stored in the reserved object with the updated data; when the distributed object storage system is idle, determine the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object, and the updated data, and modify the status identifier of the target slice to the first status identifier.

[0109] The serial numbers of the above embodiments are for description only and do not represent the advantages or disadvantages of the embodiments.

[0110] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0112] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0113] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code.

[0115] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for updating erasure code data, applied to a distributed object storage system, wherein the distributed object storage system comprises: A primary object storage device for storing original data and a verification object storage device for storing verification data, the method comprising: In response to a write operation of writing updated data to a target slice of a target stripe of a target object in the primary object storage device, obtaining a status identifier of the target slice; If the state identifier is a first state identifier indicating that the data in the target shard has not been overwritten, read the initial data in the target shard, obtain the initial verification data corresponding to the target object, overwrite the initial data with the updated data, modify the first state identifier to a second state identifier indicating that the data in the target shard has been overwritten, and store the initial data and the updated data in a reserved object in the verification object storage device; If the state identifier is the second state identifier, overwriting the current data in the target shard with the updated data, and overwriting the updated data currently stored in the reserved object with the updated data; When the distributed object storage system is idle, the target verification data corresponding to the target object is determined according to the initial verification data, the initial data currently stored in the reserved object, and the update data, and the state identifier of the target shard is modified to the first state identifier.

2. The method according to claim 1, wherein: In response to a write operation of writing updated data to a target slice of a target stripe of a target object in the primary object storage device, obtaining a status identifier of the target slice includes: In response to a write operation in a client to write updated data to the target object in the primary object storage device, distributing an update task to the target slice of the target stripe of the target object through the primary object storage device; Acquire map-structured attribute information of the target shard, and determine the state identifier in the attribute information, wherein in the attribute information, the identifier of the target stripe is used as a key, and the identifier of the target shard and the state identifier are used as values.

3. The method according to claim 1, wherein: Before obtaining the status identifier of the target slice, the method further includes: Comparing the data length of the update data with the slice length of the target slice; If the data length is not greater than the fragment length, continue to obtain the status identifier of the target fragment; If the data length is greater than the segment length, the update operation is stopped and a prompt message is generated, wherein the prompt message is used to prompt that the update data is too long and cannot be updated to the target segment.

4. The method according to claim 2, wherein: After overwriting the updated data currently stored in the reserved object with the updated data, the method further includes: The updated data is returned to the primary object storage device, and the updated data is returned to the client through the primary object storage device.

5. The method according to claim 1, wherein: When the distributed object storage system is idle, determining the target verification data corresponding to the target object according to the initial verification data, the initial data currently stored in the reserved object, and the updated data, including: A performance analysis tool is periodically used to detect the load operation status of the distributed object storage system. When the load operation status meets a preset idle standard, an erasure code verification data merging operation is triggered, and the target verification data corresponding to the target object is determined based on the initial verification data, the initial data currently stored in the reserved object, and the updated data.

6. The method according to claim 5, wherein: Determining target verification data corresponding to the target object according to the initial verification data, the initial data currently stored in the reserved object, and the updated data includes: Determining first difference data between the initial data and the updated data currently stored in the reserved object; Determine the product of the first difference data and a preset matrix coding coefficient to obtain second difference data; The second difference data is combined with the initial verification data to obtain the target verification data.

7. The method according to claim 2, wherein: The method further comprises: In response to a read operation on the target shard in the client, acquiring the status identifier of the target shard; If the state identifier is the first state identifier, read the initial data in the target shard, and feed the initial data back to the client; If the state identifier is the second state identifier, the updated data currently stored in the reserved object is read, and the updated data currently stored in the reserved object is fed back to the client.

8. A device for updating erasure code data, comprising: An acquisition module, configured to acquire a status identifier of a target slice in a target stripe of a target object in a primary object storage device in response to a write operation of writing updated data to the target slice; A first update module is used to read the initial data in the target shard and obtain the initial verification data corresponding to the target object when the state identifier is a first state identifier indicating that the data in the target shard has not been overwritten, overwrite the initial data with the updated data, and modify the first state identifier to a second state identifier indicating that the data in the target shard has been overwritten, and store the initial data and the updated data in a reserved object in the verification object storage device; A second update module, configured to, when the state identifier is the second state identifier, overwrite the current data in the target slice with the update data, and overwrite the update data currently stored in the reserved object with the update data; A determination module is used to determine the target verification data corresponding to the target object based on the initial verification data, the initial data currently stored in the reserved object and the updated data when the distributed object storage system is idle, and to modify the state identifier of the target shard to the first state identifier.

9. A non-volatile storage medium comprising a stored computer program, wherein: The device where the non-volatile storage medium is located executes the erasure code data updating method described in any one of claims 1 to 7 by running the computer program.

10. An electronic device, comprising: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the erasure code data updating method according to any one of claims 1 to 7 through the computer program.

Images