Data processing method, device, and storage medium

Abstract

The present application discloses a data processing method, device, and storage medium, which splits a first object in a cache area to obtain a first number of first sub-blocks, and stores the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe; a second object in the cache area is allocated to the first number of first data blocks, wherein the data volume of the second object is less than the first data volume; if the first number of first data blocks meet a verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.

Description

Technical Field

[0001] The present application relates to wireless communication technology, and in particular to a data processing method and device, and a storage medium. Background Art

[0002] Erasure coding (EC) is a data protection method that divides data into fragments, performs checksum calculations on the data blocks to generate redundant blocks, and then distributes all the blocks across different and independent storage nodes to ensure fault tolerance. Erasure coding modes include the following: self-coding (SC) and cross-coding (CC). SC splits a single object into different sub-blocks, then generates stripes by performing coding calculations between the sub-blocks to obtain checksum data. In the SC coding mode, each stripe corresponds to a unique object. CC aggregates and splices multiple objects into a single data block, then performs coding calculations on the multiple spliced ​​data blocks, and then organizes them into stripes. In the CC coding mode, each stripe corresponds to multiple different objects.

[0003] In the related art, an erasure code system only supports a single encoding mode. It can be seen that the erasure code system in the related art is aimed at the scenario with a single load and cannot meet the requirements of the scenario with a mixed object with complex and variable loads. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method and device, and a storage medium that are compatible with the encoding of objects of different sizes and achieve flexible and efficient hybrid EC deployment.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] This embodiment of the present application provides a data processing method, the method comprising:

[0007] Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as a first number of first data blocks; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe;

[0008] Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;

[0009] If the first number of first data blocks meets the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks. The first number of first data blocks and the second number of first verification blocks constitute a first stripe.

[0010] An embodiment of the present application provides an electronic device, including:

[0011] a splitting unit configured to split a first object in a cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as a first number of first data blocks; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe;

[0012] An allocating unit configured to allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;

[0013] The verification unit is configured to verify the first number of first data blocks to obtain a second number of first verification blocks if the first number of first data blocks meet a verification condition, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.

[0014] An embodiment of the present application provides an electronic device, including a processor, wherein the processor is configured to:

[0015] Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as a first number of first data blocks; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe;

[0016] Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;

[0017] If the first number of first data blocks meets the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks. The first number of first data blocks and the second number of first verification blocks constitute a first stripe.

[0018] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned data processing method are implemented.

[0019] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0020] The chip provided in an embodiment of the present application is used to implement the above-mentioned data processing method. The chip includes: a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the above-mentioned data processing method.

[0021] The data processing method, device, and storage medium provided in the embodiments of the present application split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe; the second object in the cache area is allocated to the first number of first data blocks, wherein the data volume of the second object is less than the first data volume; when the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe; thereby, in a mixed load scenario where the cache area includes a first object and a second object, by splitting the first object and aggregating the second object, the data of the first object and the data of the second object can be simultaneously present in one stripe, which can effectively cope with mixed loads of large and small objects, be compatible with encoding of objects of different sizes, and achieve flexible and efficient hybrid EC deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an optional structural diagram of the strip provided in the embodiment of the present application.

[0023] Figure 2 This is an optional flowchart of the data processing method provided in the embodiment of the present application;

[0024] Figure 3 This is an optional position diagram of a boundary provided in an embodiment of the present application;

[0025] Figure 4 This is an optional location diagram of the first area and the second area divided based on the boundary provided in an embodiment of the present application;

[0026] Figure 5 This is an optional flowchart of the SC mode provided in the embodiment of the present application;

[0027] Figure 6 This is an optional flowchart of the CC mode provided in an embodiment of the present application;

[0028] Figure 7 This is a schematic diagram of an optional storage mode provided in an embodiment of the present application;

[0029] Figure 8 This is an optional schematic diagram of the strip organization provided in the embodiment of the present application;

[0030] Figure 9 is an optional schematic diagram of the data processing method provided in an embodiment of the present application;

[0031] Figure 10 is an optional schematic diagram of the data processing method provided in an embodiment of the present application;

[0032] Figure 11 This is an optional schematic diagram of metadata provided by an embodiment of the present application;

[0033] Figure 12 This is an optional schematic diagram of a repair scenario provided in an embodiment of the present application;

[0034] Figure 13 This is an optional structural diagram of an electronic device provided in an embodiment of the present application;

[0035] Figure 14 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0036] Figure 15 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0037] Figure 16 This is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] The embodiments of the present application may provide a data processing method, device, and storage medium. In practical applications, the data processing method may be implemented by an electronic device, and the functional entities in the electronic device may be collaboratively implemented by hardware resources of a computer device (such as a terminal device, a server, etc.), such as computing resources such as a processor, and communication resources (such as those used to support various communication methods such as optical cables and cellular communications).

[0040] Of course, the embodiments of the present application are not limited to being provided as methods and hardware, and can also be implemented in various ways, such as being provided as a storage medium (storing instructions for executing the data processing method provided by the embodiments of the present application).

[0041] Before describing the data processing method provided in the embodiments of the present application, the following terms are explained.

[0042] An object is a data to be encoded. Different objects can have different sizes, for example, one object can be 1KB in size and another can be 10MB in size. Different objects can also contain different types of content, such as audio data, text data, image data, etc. In this embodiment of the application, there is no limitation on the type of content contained in an object.

[0043] A chunk includes a data chunk and a check chunk. A data chunk is a chunk whose content is the data of the object, and a check chunk is a chunk whose content is check data obtained by checking the data chunk.

[0044] A stripe is a set of blocks consisting of a group of data blocks to be verified and the verification blocks obtained by verifying the group of data blocks.

[0045] The encoding parameters include a first number k and a second number n, where the first number is the number of data blocks in a stripe, wherein the first number of data blocks in a stripe constitutes a group of data blocks, and the number of check blocks in a stripe is nk, where n is greater than k.

[0046] In one example, in the EC encoding of (5, 3) where n is 5 and k is 3, Figure 1 As shown, stripe 100 includes five blocks: block 101, block 102, block 103, block 104, and block 105, wherein blocks 101, 102, and 103 are data blocks, and blocks 104 and 105 are check blocks.

[0047] It is understandable that different blocks in a stripe may be distributed in different storage nodes, and different storage nodes may be independent physical devices.

[0048] Below, various embodiments of the data processing method, device, and storage medium provided in the embodiments of the present application are described.

[0049] The data processing method provided in the embodiment of the present application is applied to an electronic device, which can be used as a data encoding server, such as Figure 2 Shown, including:

[0050] S201. The electronic device splits the first object in the cache area to obtain a first number of first sub-blocks, and stores the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe.

[0051] When the amount of data of objects received in the cache area reaches a first data amount threshold, the electronic device may distribute the objects in the cache area across different data blocks for verification and striping. In the embodiment of the present application, the process of distributing objects in the cache area across different data blocks may also be referred to as striping.

[0052] For an object in the cache area, the electronic device determines whether the object belongs to a first object or a second object, wherein the data volume of the first object is larger than the first data volume and the data volume of the second object is smaller than the first data volume. The first object and the second object can be respectively called a large object and a small object.

[0053] The size of the first data volume can be set according to the size of the load of the current buffer area.

[0054] In some embodiments, the first data amount is greater than or less than a first data amount upper limit, and the first data amount upper limit is a data amount upper limit of one data block.

[0055] The cache area may include at least one or more first objects. For each first object, the electronic device splits the first object to obtain a first number of first sub-blocks, where the data amount, i.e., size, of different first sub-blocks may be the same or different. The electronic device stores the first number of split first sub-blocks in a first number of first data blocks, such that a first stripe containing the first data block includes one first object, and the first object is distributed across each first data block in the first stripe.

[0056] It is understandable that after the electronic device obtains the first number of first sub-blocks, one first sub-block can be understood as a first data block. In this case, the first data block only includes the data of the large object.

[0057] The stripe organization of this application is explained by taking the following objects as an example in the cache area for stripe organization: 1.8MB object 1, 50KB object 2, 12KB object 3, 32KB object 4, 1KB object 5, where the first data volume is 1M, n is 4, and k is 2, then object 1 belongs to the first object, and the electronic device splits object 1 into two sub-blocks: sub-block 1 and sub-block 2, and sub-block 1 and sub-block 2 are stored as data block 1 and data block 2, respectively.

[0058] The electronic device storing the first sub-block of the first object as the first data block can be understood as storing the first sub-block in another storage space, and using the first sub-block as the first data block including only the first sub-block.

[0059] In practical applications, when performing stripe organization, the electronic device may select objects whose retention time in the cache area exceeds the first time period for stripe organization.

[0060] In an embodiment of the present application, when a first object is stored as a first data block, object metadata for the first object can be determined. The object metadata includes at least one of the following: a stripe identifier, a block index, a data amount, and an offset. The stripe identifier is used to identify the stripe to which the corresponding object belongs, the block index is used to index the data block to which the corresponding object belongs, the data amount is the data amount, i.e., the size, of the corresponding object, and the offset is used to identify the offset of the storage position of the corresponding object in the data block to which it belongs relative to the initial position of the block. The block index can be the sequence number of the corresponding object within the stripe to which it belongs.

[0061] In this embodiment of the present application, a sub-block of a first object is stored as a data block in only one first stripe. Different first objects may be stored as data blocks in different first stripes, thereby ensuring that only one large object is included in one first stripe. If the number of small objects in the cache exceeds the number of large objects, and a first stripe only includes one large object, then the first stripe corresponds to only one large object. This simplifies the management of large objects based on the encoding requirements of the cache.

[0062] S202: The electronic device allocates a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume.

[0063] For the second object, the electronic device aggregates the second objects in the cache area and allocates the aggregated second objects to a first number of data blocks, thereby adding each striped second object to the first number of first data, wherein one second object is allocated to only one first data block and is spliced ​​with a sub-block in the first data block, thereby increasing the data volume of the first data block.

[0064] The electronic device allocating the second object to the first data block can be understood as the electronic device storing the second object in a storage area where the first data block is located, and splicing the second object with the existing data in the first data block, so that the first object is added to the first data block and the number of the first data block increases.

[0065] Similarly, the allocation of the second object in the stripe organization process is explained by taking the following objects as an example, in which the objects for stripe organization in the cache area include: 1.8MB object 1, 50KB object 2, 12KB object 3, 32KB object 4, and 1KB object 5, where the first data volume is 1M, n is 4, and k is 2, then object 1 belongs to the first object, and object 2, object 3, object 4, and object 5 belong to the second object. After the electronic device stores the sub-blocks 1 and sub-blocks 2 split from object 1 as data blocks 1 and data blocks 2 respectively, it allocates object 2 to data block 1 to splice object 2 and sub-block 1 to obtain a new data block 1, and allocates objects 3, object 4, and object 5 to data block 2 to splice object 3, object 4, object 5 and sub-block 2 to obtain a new data block 2.

[0066] In the embodiment of the present application, when the electronic device stores the second object as the first data block, the object metadata of the second object can be determined.

[0067] In the embodiment of the present application, the second objects organized in stripes in the cache area may be allocated using a binning algorithm so that the sizes of the first data blocks are balanced.

[0068] S203. If the first number of first data blocks meet a verification condition, the electronic device verifies the first number of first data blocks to obtain a second number of first verification blocks. The first number of first data blocks and the second number of first verification blocks constitute a first stripe.

[0069] During the distribution of the second object, the electronic device monitors the first number of first data blocks obtained. If the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain nk first verification blocks. The block set consisting of the first number k first data blocks and nk first verification blocks is called a first stripe.

[0070] After determining the first stripe, the electronic device distributes the n first data blocks in the first stripe to the n nodes.

[0071] In the embodiment of the present application, the verification conditions include but are not limited to at least one of the following conditions:

[0072] The total amount of data in the data blocks exceeds the reference amount;

[0073] No object is received within the second time period to fill the gap in the data block.

[0074] In some embodiments, the data processing method provided in the embodiments of the present application may be implemented as an erasure code, and the data processing method provided in the embodiments of the present application may be applied to electronic devices implemented as an erasure code system.

[0075] In the data processing method provided in the embodiment of the present application, large objects and small objects are mixedly encoded, so that the generated first data block includes both the data of large objects and the data of small objects, which is suitable for scenarios with mixed loads of large and small objects. Considering that the first data block includes both the data of large objects and the data of small objects, the first data block can be called a mixed data block, and the first stripe obtained by checking the first data block can be called a hybrid coding (Hybrid-Coding, HC) stripe. The process of performing hybrid encoding on large objects and small objects in the buffer can be called HC mode.

[0076] The data processing method provided in the embodiment of the present application is to split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe; the second object in the cache area is allocated to the first number of first data blocks, wherein the data volume of the second object is less than the first data volume; when the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe; thereby, in a mixed load scenario where the cache area includes the first object and the second object, by splitting the first object and aggregating the second object, the data of the first object and the data of the second object can be simultaneously present in one stripe, which can effectively cope with mixed loads of large and small objects, be compatible with encoding of objects of different sizes, and achieve flexible and efficient hybrid EC deployment.

[0077] In the data processing method provided in an embodiment of the present application, a first data block in a first stripe includes both data of a first object belonging to a large object and data of a second object belonging to a small object, and the first stripe can be considered as a hybrid coding stripe.

[0078] In some embodiments, the electronic device further performs the following processing:

[0079] For an object in the cache area, the size of the object is obtained to obtain the second data volume, and the object type of the object is determined based on the relationship between the first data volume, the third data volume and the second data volume. The object with the first type of object is the first object, and the object with the second type of object is the second object.

[0080] For an object organized in stripes, the electronic device determines the object type of the object based on the relationship between the second data volume, the first data volume, and the third data volume of the object, thereby determining whether the object belongs to the first object or the second object. If the object type of the object is the first type, the object belongs to the first object; if the object type of the object is the second type, the object belongs to the second object.

[0081] In this embodiment of the present application, the first data volume is used to determine whether an object can be completely contained in a data block. If an object is a large object, it is considered that the object's data cannot be completely contained in a data block. If an object is a small object, it is considered that the object can be contained in a data block. Here, being completely contained in a data block means that there is insufficient space to accommodate the data block when the data of other objects are spliced ​​together.

[0082] In the embodiment of the present application, a large object cannot be carried in a data block, but can be carried in a stripe.

[0083] The third data amount is used to determine whether an object can be carried in a stripe. In some embodiments, the third data amount is the second data amount upper limit, where the second data amount upper limit is the product of the first data amount upper limit and the first number, representing the data amount upper limit of a stripe.

[0084] If the first data amount is the first data amount upper limit c, then the second data amount upper limit is c*k.

[0085] In some embodiments, determining the object type of the object based on the relationship between the first data volume, the third data volume, and the second data volume includes:

[0086] If the second data amount is greater than the first data amount and less than the third data amount, the object type of the object is determined to be the first type; if the second data amount is less than the first data amount, the object type of the object is determined to be the second type.

[0087] For an object, if the object's data size is less than the first data size, the object is considered to be fully configurable within a data block. If the object's data size is greater than the first data size but less than the third data size, the object is considered to be unable to be fully configurable within a data block but can be fully configurable within a stripe. An object's full configurability within a stripe means that the data blocks within a stripe can accommodate all of the object's data.

[0088] Taking the first data volume as c and the third data volume as k*c as an example, if the s of an object is greater than c and less than k*c, then the object is a first object and needs to be split, and after the split, it is located in multiple data blocks. If the s of an object is less than c, then the object is a second object.

[0089] In some embodiments, determining the object type of the object based on the relationship between the first data volume, the third data volume, and the second data volume also includes: if the second data volume is larger than the third data volume, splitting the object according to the third data volume to obtain at least two split objects; if the data volume of the split object is larger than the first data volume and smaller than the third data volume, treating the split object as a first object of the first type; if the data volume of the split object is smaller than the first data volume, treating the split object as a second object of the second type.

[0090] Here, for an object, if its data volume is greater than the third data volume, it is considered that one stripe cannot fully accommodate all the data of the object, and the object is an ultra-large object. For an ultra-large object, its data volume needs to be distributed in multiple stripes. At this time, the electronic device splits the object based on the third data volume to obtain at least two split objects, and the data volume of the split objects is less than or equal to the third data volume. For the at least two split objects obtained by the split, the at least two split objects are respectively regarded as an independent object, and different split objects are organized in different stripes. It should be noted that for ultra-large objects, they are distributed in different stripes, so one ultra-large object corresponds to multiple groups of object metadata, and different groups of object metadata correspond to different stripes.

[0091] If the number of at least two split objects is h, then the data volume of h-1 split objects or h split objects is the third number. If the data volume of h-1 split objects is the third number, then the capacity of the remaining split object is less than the third number.

[0092] In the case where there are h-1 split objects with the third data volume, the data volume of the remaining split object is determined, and the type of the split object is determined based on the relationship between the data volume of the split object and the first data volume and the third data volume, and it is determined whether the split object belongs to the first object or the second object.

[0093] In some embodiments, the electronic device also performs the following processing: if the data amount of the split object is the third data amount, the split object is divided into the first number of second sub-blocks, and the size of the second sub-blocks is the first data amount; the first number of second sub-blocks are checked to obtain a second number of second check blocks, and the first number of second sub-blocks and the second number of second check blocks constitute a second stripe.

[0094] For the split object whose data amount is equal to the third data amount among the h objects, it can be understood as a large object after splitting. The electronic device splits the split object to obtain k second sub-blocks, and the k second sub-blocks are verified as k second data blocks to obtain a second stripe.

[0095] If the third data amount is k*c, the data amount of each sub-block in the k second sub-blocks is c, that is, the data amount of each second data block reaches the upper limit of the first data amount.

[0096] In an embodiment of the present application, for a split object in an ultra-large object whose data volume is equal to the third data, the split object is individually distributed in different data blocks of a second stripe, and the corresponding second stripe only includes the data of the split object, and does not include the data of other small objects, so that one second stripe corresponds to only one object.

[0097] In the embodiment of the present application, a stripe corresponding to only one object is called an SC stripe. For an extremely large object, its data is distributed in at least two or more SC stripes, or at least one SC stripe and one HC stripe, thereby realizing the encoding of the extremely large object.

[0098] In an embodiment of the present application, in a data block, the area where the large object, i.e., the sub-block of the first object, is located can be called the first area, and the area where the small object, i.e., the second object, is located can be called the second area, and there is a boundary between the first area and the second area. In a data block, the size of the first area can change with the size of the first object, thereby causing the position of the boundary to change, and therefore, the boundary can be called a variable boundary. Among them, the first area is used to splice multiple small objects and can be called a cross-zone (CZ), and the second area is used to store a fixed sub-block of a large object and can be called a self-zone (SZ).

[0099] In one example, if Figure 3 As shown, the middle boundary of data block 301 is located at position 3011 , the middle boundary of data block 302 is located at position 3021 , the middle boundary of data block 303 is located at position 3031 , and the middle boundary of data block 304 is located at position 3041 .

[0100] In an embodiment of the present application, the first area may be located at the starting position of the data block, and the second area may be spliced ​​after the first area.

[0101] Based on the change in the position of the boundary between the first area and the second area in the data block, the data processing method provided in the embodiment of the present application can be applied to the following three scenarios:

[0102] Scenario 1: Large object load scenario;

[0103] Scenario 2: Small object load scenario;

[0104] Scenario 3: Mixed load of large and small objects.

[0105] For scenario 1, the cache area contains only large objects. The electronic device encodes the large objects in the cache area using the SC mode, resulting in an SC stripe corresponding to a single large object. For scenario 2, the cache area contains only small objects. The electronic device encodes the small objects in the cache area using the CC mode, resulting in CC stripes corresponding to multiple small objects.

[0106] For scenario 3, the cache area includes both large objects and small objects. The electronic device encodes the large objects and small objects in the cache area through the HC mode to obtain HC stripes corresponding to the large objects and small objects, and also obtains an SC stripe corresponding to only one large object.

[0107] for Figure 3 The data blocks shown are as follows: Figure 4 As shown, area 3012 of data block 301 is the first area and area 3013 is the second area, area 3022 of data block 302 is the first area and area 3023 is the second area, and for data block 301 or data block 302, it is a data block in the HC stripe obtained under scenario 3. Figure 4 All areas in the data block 303 shown are first areas, excluding the second area, and are data blocks in the CC strip obtained under scenario 2. Figure 4 All areas in the data block 304 shown are second areas, excluding the first area, and are data blocks in the SC strip obtained under scenario 1.

[0108] In this embodiment of the present application, the electronic device may first determine whether the striped object in the cache area belongs to the first object or the second object, and then determine the scene to which the cache area belongs based on the determination results of the first and second objects, and then execute the corresponding encoding mode in the corresponding scene. For scene 1, the SC mode is executed; for scene 2, the CC mode is executed; and for scene 3, the HC mode is executed.

[0109] It should be noted that the objects in the cache area in the embodiment of the present application may change with the implementation of the electronic device, and the scenes in the cache area may also change with the changes in the objects. The electronic device executes the encoding of the corresponding mode according to the changes in the scenes.

[0110] In actual applications, when the scene to which the cache area belongs is scene 3 and encoding in the HC mode is performed, the scene to which the cache area belongs is changed to scene 2, and encoding in the CC mode is continued in the scene.

[0111] Taking the first data volume as c and the third data volume as k*c as an example, the process of determining whether an object belongs to the first object or the second object in the embodiments of the present application is described. The process of determining whether an object belongs to the first object or the second object includes, but is not limited to, the following steps 1 to step 3:

[0112] Step 1: For an object whose size is greater than the strip data block capacity k*c, that is, an object with s greater than k*c, split it into stripes. At this time, the part that is less than one stripe is regarded as another object (which still needs to be recorded as the same object in terms of metadata) and is placed in steps 2 and 3 for further consideration.

[0113] Step 2: For an object whose size is less than the strip data block capacity k*c but greater than the single data block capacity c, that is, an object with c < s < k*c, split it into k sub-blocks with a size of s / k according to the encoding parameters and distribute them in all blocks of the same stripe. At this time, s / k serves as the boundary between the first area and the second area of this stripe.

[0114] Step 3: For all the remaining objects, they can be considered as small objects and an aggregation operation is performed on them. This operation is a typical binning, and the capacity of the "bins" is c or multiple c - s / k. At this time, the binning approximation algorithm can be used for processing.

[0115] It should be noted that the value of s for each object is independent.

[0116] In some embodiments, S202 splits the first object to obtain a first number of first sub-blocks, including: splitting the first object based on the first number to obtain a first number of first sub-blocks, and the data volumes of different first sub-blocks in the first number of sub-blocks are the same.

[0117] In the embodiments of the present application, for a large object encoded by HC, the data volume of this large object in each first data block in the first stripe is the same.

[0118] When the sizes of the first sub-blocks of the first object are the same, the positions of the boundaries of different first data blocks in the first stripe are the same. For different first stripes, the positions of the boundaries in the first data blocks may be the same or different.

[0119] In an embodiment of the present application, if the amount of data of the large objects included in each data block in the first stripe is the same, then the large objects in the first stripe have the same offset in each data block, thereby enabling the size attribute of the current object to be determined based on the object size, object offset, and block size in the object metadata. For example, in a 3+1 stripe (n is 4 and k is 3), the block size is 1MB, and the object metadata of an object is <size=0.9MB, offset=0.7MB>. Then, it can be determined that this object is composed of the last 0.3MB of each data block in the stripe, and the last 0.3MB is the second area in the data block, and the object is a large object.

[0120] In some embodiments, the electronic device further performs the following processing:

[0121] Determining target object metadata based on the keyword of the third object; the target object metadata is object metadata of the third object, and the object metadata includes: stripe index, block index, object size and offset;

[0122] Determine target stripe metadata based on a stripe index in the target object metadata, where the target stripe metadata is stripe metadata of the target stripe where the third object is located;

[0123] The data of the third object is obtained based on the block index, object size and offset in the target object metadata and the target stripe metadata.

[0124] The electronic device determines object metadata for the third object in the object index table based on the keyword of the third object; the object metadata includes but is not limited to: a stripe identifier, a block index, an object size, and an offset, wherein the stripe identifier refers to the sequence number of the stripe, the block index indicates the sequence number of the block within the stripe (0 to n-1), the object size is the amount of data in the object, and the offset is the offset of the object within the block. The third object is any object that needs to be accessed.

[0125] In an embodiment of the present application, an electronic device is provided with an object index table. The object index table includes a mapping relationship between keywords of different objects and object metadata. The object metadata may be located in a 64-bit bucket. The electronic device determines the object metadata corresponding to the keyword of the third object in the object index table as the target object metadata.

[0126] After determining the target object metadata, the electronic device determines target stripe metadata, ie, stripe metadata of the target stripe, in the stripe index table according to the stripe index in the target object metadata. The target stripe is the stripe where the third object is located.

[0127] In an embodiment of the present application, a stripe index table is provided in the electronic device, and the stripe index table includes a mapping relationship between a stripe index and stripe metadata. The stripe metadata includes: a mapping relationship between a block index and a block identifier of a block in a corresponding stripe. The electronic device determines the stripe metadata corresponding to the stripe identifier in the target object metadata in the stripe index table as the target stripe metadata. The identifier of the target object metadata is used to identify the target stripe, and the stripe metadata corresponding to the stripe identifier in the target object metadata in the stripe index table is the target stripe metadata, i.e., the stripe metadata of the target stripe. The stripe metadata includes a mapping relationship between the block index and the block identifier of different blocks in the corresponding stripe. The target stripe metadata includes a mapping relationship between the index and the block identifier of each block in the target stripe. For a large object, the data of a large object is stored in k data blocks, and the index of the large object corresponds to k block identifiers. For a small object, the data of a small object is stored in 1 data block, and the index of the small object corresponds to 1 block identifier.

[0128] After determining the target stripe metadata, the electronic device obtains data of the third object based on the target stripe metadata, depending on the accessibility of the third object. The accessibility of the third object includes accessible and inaccessible, and the reasons for the inaccessibility of the third object include, but are not limited to, at least one of the following: failure, node overheating, and network congestion.

[0129] In the embodiment of the present application, the electronic device accesses the third object in at least one of the following ways:

[0130] Access method 1: when the third object is accessible, directly read the data of the third object from the data block storing the third object, i.e., the target data block;

[0131] Access method two: when the third object is inaccessible, data of the third object is obtained by repairing objects in other data blocks in the target stripe.

[0132] Access method 1

[0133] The electronic device obtains the data of the third object based on the block index, object size and offset in the target object metadata and the target stripe metadata, including:

[0134] When the third object is accessible, the target data block metadata is determined based on the block index in the target object metadata and the target stripe metadata, where the target data block metadata is the block metadata of the target data block where the third object is located; the address of the target data block is determined in the target data block metadata; and the data of the third object is read based on the address of the target data block and the object size and offset in the target object metadata.

[0135] The electronic device determines the block data of the target data block storing the third object, namely the target data block metadata, based on the target stripe data in the block index table. The target data block metadata includes the address of the target data block, namely the initial physical address of the target data block.

[0136] In an embodiment of the present application, an electronic device is provided with a block index table that includes a mapping between block identifiers and block metadata. The block metadata includes the address of the corresponding data block, i.e., the starting physical address of the corresponding data block. The electronic device determines the address corresponding to the block identifier corresponding to the third object in the target stripe metadata in the block index table as the address of the target data block.

[0137] The electronic device determines the target address corresponding to the third object based on the address of the target data block and the offset in the target object data, and reads data of the object size starting from the target address to obtain data of the third object, wherein the target address is the address of the target data block offset by the offset in the target object data.

[0138] In one example, assume that the sizes of objects in the cache are 1.9MB, 50KB, 12KB, 32KB, and 1KB, respectively. The 1.9MB object is split into two 0.95MB sub-blocks. The 50KB, 12KB, 32KB, and 1KB objects are aggregated and distributed across two data blocks, with the 50KB object allocated to one data block and the 12KB, 32KB, and 1KB objects allocated to another data block. A parity block is encoded from the two data blocks. Assume that the storage blocks in the erasure coding system are a, b, and c from top to bottom, and the keys of the cache objects are 0, 1, 2, 3, and 4 from top to bottom and left to right, respectively. The stripe ID they constitute is 0, and the capacity limit of a single data block in the erasure coding system is 1MB. For a 1KB object, Through the object's Key: 3, we get index A: <0,1,1KB,44KB> in the object index. Based on the stripe number 0 in index A, we locate the index of Stripe0, and by querying the element 1 of the mapping array in the index of Stripe0, we locate block a. Thus, based on block ID a, we locate the physical block address where the object is stored, and obtain the data of the object with a size of 1KB. For an object with a size of 1.9MB, similarly, through the object's Key4, we get index A: <0,0,1.9MB,50KB>. Therefore, we locate the stripe index through stripe number 0, and determine all the storage blocks containing the large object, namely ab, based on the mapping array in the stripe index item. Then, we access all the remaining data from the starting address + offset of blocks a and b, respectively, to obtain the data of the object with a size of 1.9MB.

[0139] It can be seen that the access method for accessible objects provided in the embodiment of the present application is applicable to large objects and small objects, thereby adapting to the reading of accessible objects in scenarios with mixed loads of large and small objects.

[0140] Access method 2

[0141] The electronic device obtains the data of the third object based on the block index, object size and offset in the target object metadata and the target stripe metadata, including:

[0142] In the case that the third object is inaccessible, a fourth number of reference block metadata is determined based on the block index in the target object metadata and the target stripe metadata; the reference block metadata is block metadata of a reference block, the reference block is a block in the target stripe that does not include the third object, and the fourth number is the number of reference blocks included in the target stripe; for each reference block metadata in the fourth number of reference block metadata, reference data at a target position range in the corresponding reference block is determined based on the reference block metadata; the reference data included in the fourth number of reference blocks is repaired to obtain data of the third object.

[0143] In the case where the third object is inaccessible, the electronic device determines the block index of the target data block through the target object data, and uses the block index in the target stripe other than the block index of the target data block as the block index of the reference block, and determines the block identifier corresponding to the reference block based on the block index of the reference block. The reference block is a block in the target stripe that does not include the third object, and the number of reference blocks in the target stripe is the fourth number (n-1). In one example, n is 5 and k is 3, and the target stripe includes the following blocks: chunk a, chunk b, chunk c, chunk d, and chunk e, and the index of each block is 0, 1 to 4; if the third object is located in chunk 1, the block index in the target object metadata is 1, and includes the following reference block block indexes: 0, 2, 3, 4, and the following block identifiers are determined based on 0, 2, 3, 4: a, c, d, e, and the following reference blocks are determined: chunk a, chunk c, chunk d, and chunk e.

[0144] After the electronic device determines the block identifier of the reference block, it determines the block metadata of each reference block, i.e., the reference block metadata, from the block index based on the block identifier of each reference block. For a reference block metadata, if the reference block is a data block, it includes the address of each reference block and the position information of each object, i.e., the candidate object, included in the reference block. The electronic device determines the data of the object, i.e., the reference object, located within the target position range from the candidate objects included in the reference block based on the reference block metadata. For a reference block metadata, if the reference block is a check block, the check information it includes is a whole. Therefore, all the data of the reference block can be considered as the reference data of the reference block. Among them, the target position range is the position of the third object in the target data block. The electronic device can determine the target position range based on the object size and offset in the target object metadata.

[0145] After determining the reference data in each reference block, the electronic device decodes all the reference data to obtain the repaired data of the third object, thereby repairing the inaccessible data of the third object.

[0146] In an embodiment of the present application, for a large object, its offset and position in different data blocks are the same. Therefore, the electronic device can obtain data of other sub-blocks that can form a large object with the third object based on the position of the same offset of the third object in the reference block belonging to the data block, and obtain the data of the modified third object by decoding the data of these sub-blocks and the check block.

[0147] In an embodiment of the present application, in the data acquisition process of the inaccessible object, that is, the repair process of the inaccessible object, the third object is repaired based on the data in other blocks other than the target data block in the target stripe that are at the same position as the third data block. There is no need to read all the data in other blocks, thereby reducing unnecessary data reading overhead and realizing partial decoding of the stripe; especially for the repair of small data, targeted reading of other object data aligned with the read object to be degraded can reduce the read amplification ratio.

[0148] In some embodiments, based on the reference block metadata, reference data at a target position range in a corresponding reference block is determined, including: determining a pointer to a candidate object included in the reference block metadata, and determining corresponding candidate object metadata, wherein the candidate object metadata is object metadata of the candidate object, and the candidate object is an object included in the reference block corresponding to the reference block metadata; determining the position of the corresponding candidate based on the candidate object metadata of each candidate object in the reference block; selecting the reference object from the candidate objects included in the reference block based on the target position of the third object in the target data block and the position of each candidate object in the reference block, wherein the reference data includes data of the reference object in the reference block.

[0149] In embodiments of the present application, block metadata, in addition to including the physical address of the corresponding block, may also include a pointer corresponding to an object in the corresponding block, pointing to the object metadata of the corresponding object. This allows for a link between not only the object metadata and the block metadata, but also the block metadata and the object metadata. In some embodiments, the code metadata may include a linked list connecting the objects in the corresponding block. Within the linked list, for each object, the following two components may be included: a pointer to the object metadata of the corresponding object, and a pointer "Next" pointing to the next object.

[0150] The electronic device obtains object metadata of each candidate object in the corresponding reference block based on the pointer of each object in the reference block metadata, wherein the stripe index and block index in the object metadata of the candidate objects in the same reference block are the same.

[0151] For a candidate object in a reference block, the electronic device determines whether the candidate object is a reference object based on the position of the candidate object, and if so, reads the data of the reference object. The reference data read by the electronic device includes the data of all reference objects in the reference block.

[0152] Among them, for a reference object, the electronic device determines the position range of the corresponding candidate object based on the object size and offset in the object metadata of a candidate object. When the position range overlaps with the target position range, the candidate object can be considered as a reference object, and the data of the reference object is read based on the address in the reference block and the position range of the reference object.

[0153] The method provided in the embodiment of the present application realizes unified indexing of large and small objects by adopting a multi-level, closed index structure. The multi-level is reflected in three different indexes: the object index, i.e., the mapping relationship in the object index table; the stripe index, i.e., the mapping relationship in the stripe index table; and the block index, i.e., the mapping relationship in the block index table. The closed structure is reflected in that any index item can be used to locate other index items associated with it, thereby supporting mixed loads of large and small objects. In addition, the unified index provided in the embodiment of the present application is compatible with large objects in the Self Coding mode, and at the same time, additional information is added to allow the multi-level indexes to be related to each other, thereby realizing the closure of the index, and further providing comprehensive index information for various system operations, such as downgraded reading, repair, etc.

[0154] The data processing method provided in the embodiments of the present application is further described below.

[0155] The erasure coding schemes for object storage systems include the following two: Figure 5 The SC and Figure 6 As shown in CC. Figure 5 As shown in , SC splits a single object into different sub-blocks (splits), and then obtains the check data through coding calculation between the sub-blocks to generate strips. In the SC coding mode, each strip corresponds to a unique object. Figure 6 As shown, CC aggregates or splices multiple objects including object 0, object 1, ..., object n into multiple data blocks, then performs encoding calculations on the multiple spliced ​​data blocks, and then organizes them into strips. In the CC encoding mode, each strip corresponds to multiple different objects.

[0156] Due to the existence of parallel input / output (I / O), SC-based erasure coding systems have superior read and write performance for object access, as well as better load balancing and tail latency. Furthermore, due to its stripe-as-object nature, SC has very low repair bandwidth when repairing / downgrading single objects. However, SC also suffers from the problem of unstable data blocks and multiple metadata queries for a single I / O. Therefore, SC is more suitable for large objects where queries are not time-sensitive and there is significant room for optimization of I / O performance.

[0157] In contrast to SC-based erasure coding systems, CC-based erasure coding systems feature fixed data block sizes and single I / O and single metadata queries. However, due to a lack of parallelism, they exhibit poor performance for single-object access and suffer from severe read amplification issues during single-object repair. Therefore, CC is more suitable for small objects where queries are time-sensitive and there is limited room for I / O performance optimization.

[0158] It should be noted that repair / downgrade reads are reads of data that is temporarily inaccessible (due to failure, node overheating, network congestion, etc.) through erasure code repair. For inaccessible data, cross-node access is required to read other data in the same stripe to calculate and decode the unavailable data. The network bandwidth generated by cross-node access is the repair bandwidth. Due to the organization of CC, repairing a small object requires repairing the entire stripe, which will increase the repair bandwidth. For example, to repair 1KB of data, the entire 4MB stripe needs to be restored, and after restoration, it is only to obtain the required 1KB of data. However, since the object of SC is the stripe, 4MB of bandwidth is required to repair and obtain the required 4MB of data. Therefore, SC has a very low repair bandwidth when repairing / downgrading a single object, but SC has a serious read amplification problem during the repair process of a single object.

[0159] CC is aggregated, so the stripe block size is fixed. Metadata queries refer to the mapping from objects to physical addresses. In CC mode, the data for a single object is generally concentrated in a single block, so only a single node needs to be accessed. In contrast, in SC mode, accessing a single piece of data requires initiating multiple I / O requests, accessing multiple nodes in parallel.

[0160] In one example, the erasure coding system under different object sizes may be as shown in Table 1.

[0161] Table 1. Example of an erasure code system

[0162]

[0163] In related technologies, as shown in Table 1, various erasure coding systems only consider scenarios where large objects or small objects are the sole focus. In the design of these erasure coding systems, the data load is single, such as streaming data dominated by GB-level large objects or key-value pairs dominated by byte-level small objects. While a single SC / CC model can effectively leverage its advantages in specific scenarios, ensuring efficient and stable operation of the entire EC system, with the rapid development of cloud computing, the data load faced by cloud systems is often extremely uneven.

[0164] Figure 7 The workload of the real object storage pool is displayed. Storage model 71 is the capacity ratio of storage pool 1, storage model 72 is the quantity ratio of storage pool 1, storage model 73 is the capacity ratio of storage pool 2, and storage model 74 is the quantity ratio of storage pool 2.

[0165] For storage model 71, the capacity of data [0, 64KB) accounts for 0.2%, the capacity of data [64KB, 4MB) accounts for 30.0%, the capacity of data [4MB, 16KB) accounts for 20.5%, the capacity of data [16MB, 64MB) accounts for 17.4%, the capacity of data [64MB, 1GB) accounts for 26.9%, and the capacity of data larger than 1GB accounts for 4.9%.

[0166] For storage model 72, the amount of data in [0, 64KB) accounts for 27.7%, the amount of data in [64KB, 4MB) accounts for 62.4%, the amount of data in [4MB, 16KB) accounts for 8.0%, the amount of data in [16MB, 64MB) accounts for 1.4%, the amount of data in [64MB, 1GB) accounts for 0.2%, and the capacity of data larger than 1GB accounts for 0%.

[0167] For storage model 73, the capacity of data [0, 64KB) accounts for 4.3%, the capacity of data [64KB, 4MB) accounts for 1.8%, the capacity of data [4MB, 16KB) accounts for 3.0%, the capacity of data [16MB, 64MB) accounts for 6.9%, the capacity of data [64MB, 1GB) accounts for 30.0%, and the capacity of data larger than 1GB accounts for 53.9%.

[0168] For storage model 74, the amount of data in [0, 64KB) accounts for 98.7%, the amount of data in [64KB, 4MB) accounts for 1.2%, the amount of data in [4MB, 16KB) accounts for 0.1%, the amount of data in [16MB, 64MB) accounts for 0%, the amount of data in [64MB, 1GB) accounts for 0%, and the capacity of data larger than 1GB accounts for 0%.

[0169] based on Figure 7 It can be determined that the object size distribution is extremely uneven: in some business scenarios, small objects may account for 98% of the storage pool, while in other scenarios, small objects may only account for 27%. Clearly, adopting different EC frameworks for different storage pools is unreasonable, as the complexity and high operational costs would place an excessive burden on the enterprise. However, adopting a single EC encoding scheme across different storage pools cannot effectively address the diverse and complex mixed-object scenarios. For example, if storage pools 1 and 2 use a CC-mode EC system, the CC mode will deparallelize all object accesses in storage pool 1, limiting the I / O performance of 82% of large objects to single-channel bandwidth. If storage pools 1 and 2 use a SC-mode EC system, the SC mode will degrade the read and write performance of 98% of small objects in storage pool 2 due to the startup overhead of concurrent I / O access, as their small data size does not benefit from parallel I / O.

[0170] Therefore, the erasure code system in the related art adopts a single encoding mode solution and cannot flexibly handle mixed object load scenarios: a single SC / CC mode deployment will always optimize the read and write performance of some objects at the expense of other objects.

[0171] For large objects, SC is used for encoding, thereby fully utilizing the performance advantages of parallel I / O. Since large objects take a long time to access, the cost of multiple metadata query overheads brought by SC is acceptable. For small objects, CC is used for encoding, thereby avoiding I / O amplification caused by object splitting and metadata query overheads caused by frequent access to small objects. However, when processing mixed load scenarios with objects of different sizes, a single SC / CC mode is inflexible and has problems with performance degradation in some businesses. Therefore, the embodiment of the present application will propose a hybrid erasure code encoding design to address this dilemma, making it compatible with the encoding of large and small objects, and realizing flexible and efficient hybrid EC deployment and management.

[0172] The data processing method provided in the embodiments of this application integrates the SC and CC coding modes into the same stripe, enabling flexible EC stripe deployment for objects of varying sizes. This is referred to as the Hybrid-Coding (HC) mode. In HC mode, each block in a stripe is split into two distinct zones: the CZ and the SZ. Small objects are clustered in the CZ, while large objects are separated in the SZ. When the amount of data stored in a stripe reaches a threshold, encoding is triggered, encoding a mix of large and small objects into the same stripe.

[0173] Due to the huge disparity in the total storage volume for large and small objects, a single large object in each HC stripe can meet the requirements of most EC deployments with mixed workloads. Therefore, to simplify the design, the number of large objects in an HC stripe is at most 1. Under this premise, different blocks in the same stripe can have the same boundaries. When an HC stripe includes multiple large objects, a block may contain data for different large objects. In this case, the boundaries between large objects need to be distinguished. For example, the first KB to the third KB of each block in the stripe are large object A, and the block after 3KB is large object B. This design complicates the management and design of metadata.

[0174] It is worth noting that the boundary between SZ and CZ in different stripes can be different. The location of the boundary depends on the distribution of the sizes of the objects participating in the stripe. This means that HC can flexibly degenerate into SC or CC according to the needs of the current workload.

[0175] Figure 8Various possibilities of HC stripes are shown in Figure 1: For case 0, when the data sizes in the cache are different, the system will select a suitable boundary and organize a single large object and multiple small objects of appropriate sizes into the same stripe (stripe 0); for case 1, when the object in the cache is a large file, the system will push the boundary to the head of the stripe (stripe 1), causing the stripe to degenerate into SC mode, and directly split and encode the single large object; for case 2, when the object in the cache is a small file, the system will push the boundary to the end of the stripe (stripe 2), causing the stripe to degenerate into CC mode, and aggregately encode multiple small objects. It can be understood that the boundary of a stripe can be understood as the boundary of each data block included in the stripe, and the position of the boundary of each data block is the same. A data block is divided into two areas, CZ and SZ, based on the boundary.

[0176] In an embodiment of the present application, the encoding parameters of a stripe may include information such as the number of data blocks and the number of check blocks in the stripe. For example: (n, k)-EC encoding means encoding k data blocks into n blocks, where the number of check blocks is nk. Figure 8 The encoding in can be identified as a (3, 2) stripe, where a stripe includes 3 blocks: 2 data blocks and 1 check block.

[0177] Next, the stripe organization and metadata management, and degraded read operations included in the data processing method provided in the embodiment of the present application are described respectively.

[0178] Strip organization

[0179] To address mixed loads of large and small objects, a hybrid coding stripe organization method is proposed. By adaptively dividing the SZ and CZ, the two coding modes, Self Coding and Cross Coding, are integrated into a single stripe to achieve mixed coding of large and small objects. The boundaries of the Self Zone / Cross Zone within the stripe are determined in real time based on the system load, and in extreme cases, they can degenerate to a single Self Coding / Cross Coding mode. Compared to a single Self Coding / Cross Coding mode, hybrid coding can effectively handle mixed loads of large and small objects, improving the read and write performance of large and small objects while ensuring system fault tolerance.

[0180] Here, with Figure 9 As an example, the (3,2)-HC strip is shown to be effective in the face of mixed loads (i.e. Figure 8The organization process for Case 0 is shown in the figure. When the cache receives objects with sufficient data, the erasure coding system will begin organizing HC stripes. First, the erasure coding system selects a large object in the cache and splits it according to the encoding parameters (n, k). After the split, the erasure coding system divides the boundary between the SZ and CZ based on the sub-block size of the large object. The erasure coding system then gathers small objects from the cache and distributes them to CZs of different blocks without crossing the boundary. When the aggregation of small objects reaches the storage upper bound of the data block, the erasure coding system encodes and calculates the data blocks belonging to an HC stripe to generate a parity block including parity data. At this point, an HC stripe containing objects of different sizes is generated.

[0181] like Figure 9 As shown, the sizes of objects in the cache are: 1.9MB, 50KB, 12KB, 32KB, 1KB, and the object with a size of 1.9MB is split into two sub-blocks with a size of 0.95MB, and the two sub-blocks are SZs of different data blocks. Objects with sizes of 50KB, 12KB, 32KB, and 1KB are aggregated and distributed to two data blocks, among which the object of 50KB is allocated to the CZ of one data block, and the objects of 12KB, 32KB, and 1KB are distributed in the CZ of another data block. The two data blocks are encoded to obtain a check block.

[0182] To ensure high availability and low latency, the striping algorithm should be an online, polynomial-time algorithm. This "online" requirement means that the size of the next object requested by the user cannot be predicted, and the algorithm can only consider the sequence of objects currently in the cache. Furthermore, to stripe the cache data as quickly as possible, the algorithm should be a polynomial-time algorithm to ensure efficient system operation.

[0183] To address these requirements, the striping algorithm in this application's data processing method is based on the following premise: When the amount of data in the cache reaches a preset threshold, the erasure coding system initiates the striping algorithm and stripes objects that have been in the cache for longer than a certain amount of time. Therefore, the striping algorithm is abstracted as follows:

[0184] Input: N object entities x, entity x has size attribute s, and collection capacity upper limit c;

[0185] Output: m sets P, the i-th set P i There are k sets, each of which is an entity x;

[0186] Constraint: For the jth set in the i-th set group satisfy:

[0187] Objective: While minimizing m, minimize it, where m is the number of stripes.

[0188] In the above stripe organization algorithm, a set can be understood as a data block, and a set group can be understood as a data block set composed of data blocks in a stripe.

[0189] In the embodiments of the present application, stripe organization can be understood as a binning problem or equivalently as a set partitioning problem. In computational complexity theory, such problems belong to NPC problems, so it is unreasonable to seek an optimal solution. To meet the requirement that objects do not overflow the data block, the data entity x in the buffer must face the following three considerations:

[0190] (1) For an object with a size larger than the stripe data block capacity k*c, that is, an object with s > k*c, split it into stripes. At this time, the part that does not fill a stripe is regarded as another object (which still needs to be recorded as the same object in terms of metadata) and is placed in (2) and (3) for further consideration.

[0191] (2) For an object with a size smaller than the stripe data block capacity k*c but larger than the single data block capacity c, that is, an object with c < s < k*c, split it into k sub-blocks with a size of s / k according to the encoding parameter and distribute them in all blocks of the same stripe. At this time, s / k serves as the boundary of SZ / CZ of this stripe.

[0192] (3) For all the remaining objects, they can be considered as small objects and aggregated. This operation is a typical binning, and the capacity of the "bin" is c or multiple c - s / k. At this time, a binning approximation algorithm can be used for processing.

[0193] In the embodiments of the present application, the logic of the stripe organization algorithm is: split large objects -> allocate sub-blocks of large objects -> perform an approximate binning algorithm on small objects.

[0194] In the data processing algorithm provided by the embodiments of the present application, a simple replica redundancy strategy will be adopted to solve the fault tolerance problem in the generation process of HC stripes. During the generation process of HC stripes, such as [[ID=

[29] ]] Figure 10As shown, parity block computation is delegated to the parity nodes. Subsequently, during data transmission, data replicas are distributed to nk data nodes, forming an (n-k+1)-way replica. This means that nk replicas exist for the original data, ensuring fault tolerance during the striping process. After data distribution, the data on each parity block can perform its own parity calculations, switching the replication mode to EC mode. If data on a node is lost, since it has copies on other nodes, the data is not permanently lost and can be directly replicated for continued use.

[0195] Metadata management

[0196] For mixed encoding of large and small objects, the data processing method provided in the embodiment of the present application adopts a multi-level, closed index structure to achieve unified indexing of large and small objects. The multi-level is reflected in three different indexes: object, stripe, and block index, and the closedness is reflected in the ability to locate other index items associated with any index item. Existing indexing technologies are all based on a single Self Coding / Cross Coding and cannot support mixed loads of large and small objects. Compared with existing indexing technologies, the unified index proposed in the embodiment of the present application is to expand the index items on the basis of the Cross Coding mode index, so that it can be compatible with large objects in the Self Coding mode, and at the same time add additional information to make the multi-level indexes interrelated, realize the closure of the index, and then provide comprehensive index information for various system operations, such as downgraded reading, repair, etc.

[0197] In the data processing method provided in the embodiment of the present application, a multi-level index structure will be designed for HC stripes to uniformly manage the metadata of large and small objects. The index of the HC stripe will consist of three hash tables: object index, stripe index and storage block index. Figure 11 As shown in the figure, the three indexes are linked to each other. The erasure code system can query any object / strip / storage block information related to any index, thereby meeting the query requirements of operations such as read / write / repair / degraded read.

[0198] like Figure 11 As shown in Figure 2, the object index (table) is a hash table that maps object keys to 64-bit buckets. Each bucket contains the values ​​of four parameters: stripe ID, chunk index (Chunk Idx), size (Size), and offset (Offset). The meanings of these four parameters are shown in Table 2. These four parameters can be used to locate the specific physical storage location of the object. By analyzing the relationship between size and offset, the size of the current object can be determined without the need for additional tags.

[0199] Table 2. Definition of object parameters

[0200] Bit value describe 0-31 Stripe ID A serial number used to refer to the stripe 32-35 Chunk Idx Sequence number of the block within the stripe (0 to n-1) 36-51 Size The size of the object 52-63 Offset The offset of the object in the block

[0201] like Figure 11 As shown, the stripe index (table) is a hash table that maps Stripe IDs to stripe metadata. The stripe metadata stores an array of mappings from Chunk IDs within the stripe to chunk IDs, thereby recording the block organization of the stripe. This allows the target storage chunk to be located using the Stripe ID and Chunk ID.

[0202] like Figure 11 As shown in the figure, the block index (table) is a hash table that maps Chunk ID to block metadata. In addition to storing the physical address of the block, the block metadata also stores a pointer list to record the organization of objects in the block. The structure of the linked list represents the object structure of the block, and as shown in the figure, Figure 11 As shown, the contents of the linked list are pointers (Ptr) pointing to the corresponding object metadata. This allows for quick understanding of the organization of storage blocks and implements indexing from the physical storage layer to object information (this design also provides an indexing basis for optimizing degraded read performance for small objects).

[0203] Below, Figure 9 This article will use HC encoding as an example to explain the object data access process in detail. Assume that the storage blocks in an erasure code system are a, b, and c from top to bottom, and the keys of the cache objects are 0, 1, 2, 3, and 4 from top to bottom and left to right. The stripe ID is 0, and the capacity of a single data block in the erasure code system is limited to 1MB.

[0204] For an object with a size of 1KB in CZ, we obtain index A: <0,1,1KB,44KB> in the object index through the object's Key: 3. We locate the index of Stripe0 based on the stripe number 0 in index A, and locate block a by querying the element 1 of the mapping array in the index of Stripe0. Thus, we locate the physical block address where the object is stored based on block ID a. At this point, since the object's Offset + Size is less than 1MB, we can determine that the object is a small object and is within CZ. Therefore, we only need to access 1KB of data from the starting address + offset of block a.

[0205] For a 1.9MB object in SZ, the object's Key4 yields index A: <0,0,1.9MB,50KB>. Since the object's Offset + Size > 1MB, it can be determined to be a large object and located within SZ. Therefore, the stripe index is located using stripe number 0, and the mapping array in the stripe index entry identifies all storage blocks containing the large object, namely ab. The remaining data can then be accessed from the starting address + offset of blocks a and b, respectively.

[0206] The above-mentioned process access demonstrates the advantages of the metadata design provided by the embodiment of the present application: the same metadata structure can manage the data in SZ and CZ at the same time, and also realizes the unification of the access process.

[0207] When deploying EC for small objects, neither the existing CC mode nor the HC mode proposed in the embodiment of this application can avoid the serious read amplification problem caused by downgraded reads of small objects. This is because there are often multiple small objects in a stripe in the CC / HC mode. At this time, when a downgraded read operation is performed on a single small object, the entire stripe will be decoded and calculated. At this time, the read amplification ratio will reach (k*block size) / object size. If Figure 9 Taking the degraded read of 1KB data as an example, the read amplification ratio will reach an astonishing 2MB / 1KB=2000.

[0208] Degraded read operations

[0209] In order to optimize this read amplification ratio, an embodiment of the present application proposes an extended technical solution: using block indexes to specifically read other object data aligned with the read object to be degraded, thereby realizing partial decoding of the stripe, thereby reducing unnecessary data reading overhead and lowering the read amplification ratio.

[0210] by Figure 12 The (4, 3) EC of the stripe is used as an example to illustrate how to achieve local decoding of the stripe through selective reading. Here, it is assumed that object 1 is currently inaccessible and a degraded read operation is required. In the traditional degraded read process, the erasure code system will arbitrarily select k (3) storage blocks that do not contain object 1 from the stripe, namely Chunk 1, Chunk 2, Chunk 3, and then perform decoding calculations, restore Chunk 0, and then truncate object 1. In the embodiment of the present application, the index items of each block can be analyzed to select objects whose offsets coincide with object 1, namely objects 4, 5, 7 and the check block, and then perform local decoding operations to restore object 1. In this process, access to useless object data 3, 6, and 8 can be avoided, thereby optimizing the degraded read ratio of small objects.

[0211] It should be noted that although the above example only shows the degraded read optimization solution in CC mode, its concept is also applicable in CZ of HC.

[0212] The data processing method provided in the embodiment of the present application has the following technical effects:

[0213] As for the benefits of large objects, since all large objects are stored in SZ, the access mode of large objects is consistent with the SC mode, and its read and write performance can reach the performance level of a single SC mode. When accessing large objects through the index process provided in the embodiment of the present application, the erasure code system will also perform parallel I / O, which will improve the read and write throughput of the object while reducing the tail latency of the object access. The characteristics and advantages of this access mode have been verified in practice in EC-Cache. At the same time, since the SZ-CZ boundaries in the HC stripe are aligned, the degraded read of large objects is the same as in the SC mode, and there is no problem of read amplification.

[0214] As for the benefits of small objects, since all small objects are stored in CZ, the access mode of small objects is consistent with the CC mode, and its read and write performance can reach the performance level of a single CC mode. By accessing small objects through the index process provided in the embodiment of the present application, the erasure code system will only start a single I / O, thereby avoiding the bottleneck caused by the startup overhead of multiple I / Os and reducing the access performance of small objects. In addition, due to the additional design of block indexes, there can also be additional designs to optimize the problem of degraded read I / O amplification of small objects.

[0215] Combined with the above-mentioned benefits for large objects and small objects, the data processing method provided in the embodiment of the present application has the following performance advantages: HC strips can be flexibly organized adaptively for different object sizes, so that all types of objects can adopt the optimal encoding mode, thereby obtaining performance benefits and avoiding performance losses in some business scenarios.

[0216] The unified framework design for data storage in this application has good flexibility and is compatible with most object load scenarios. At the same time, there is no need to maintain two different frameworks to manage large / small objects respectively. This gives the EC framework great advantages in development, expansion, and operation and maintenance.

[0217] An electronic device according to an embodiment of the present application, such as Figure 13 As shown, the electronic device 1300 includes:

[0218] A first splitting unit 1301 is configured to split a first object in a cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as a first number of first data blocks, respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe;

[0219] An allocating unit 1302 is configured to allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;

[0220] The verification unit 1303 is configured to verify the first number of first data blocks to obtain a second number of first verification blocks when the first number of first data blocks meet the verification condition. The first number of first data blocks and the second number of first verification blocks constitute a first stripe.

[0221] In some embodiments, the electronic device 1300 further includes a classification unit configured to:

[0222] For an object in the cache area, the size of the object is obtained to obtain the second data volume, and the object type of the object is determined based on the relationship between the first data volume, the third data volume and the second data volume. The object with the first type of object is the first object, and the object with the second type of object is the second object.

[0223] In some embodiments, the classification unit is further configured to:

[0224] If the second data amount is greater than the first data amount and less than the third data amount, determining that the object type of the object is the first type;

[0225] If the second data amount is smaller than the first data amount, it is determined that the object type of the object is a second type.

[0226] In some embodiments, the classification unit is further configured to:

[0227] If the second data amount is greater than the third data amount, splitting the object according to the third data amount to obtain at least two split objects;

[0228] If the data volume of the split object is greater than the first data volume and less than the third data volume, the split object is treated as a first object of the first type;

[0229] If the data volume of the split object is smaller than the first data volume, the split object is treated as a second object of a second type.

[0230] In some embodiments, the electronic device 1300 further includes a second splitting unit configured to, if the data amount of the split object is the third data amount, divide the split object into the first number of second sub-blocks, and the size of the second sub-blocks is the first data amount;

[0231] The verification unit 1303 is further configured to verify the first number of second sub-blocks to obtain a second number of second verification blocks, where the first number of second sub-blocks and the second number of second verification blocks constitute a second stripe.

[0232] In some embodiments, the first splitting unit 1301 is further configured to:

[0233] The first object is split based on the first number to obtain a first number of first sub-blocks, and the data amounts of different first sub-blocks in the first number of sub-blocks are the same.

[0234] In some embodiments, the electronic device 1300 further includes an access unit configured to:

[0235] Determining target object metadata based on the keyword of the third object; the target object metadata is object metadata of the third object, and the object metadata includes: stripe index, block index, object size and offset;

[0236] Determine target stripe metadata based on a stripe index in the target object metadata, where the target stripe metadata is stripe metadata of the target stripe where the third object is located;

[0237] The data of the third object is obtained based on the block index, object size and offset in the target object metadata and the target stripe metadata.

[0238] In some embodiments, the access unit is further configured to:

[0239] In a case where the third object is accessible, determining target data block metadata based on the block index in the target object metadata and the target stripe metadata, the target data block metadata being block metadata of the target data block where the third object is located;

[0240] Determining an address of the target data block in the target data block metadata;

[0241] The data of the third object is read based on the address of the target data block and the object size and offset in the target object metadata.

[0242] In some embodiments, the access unit is further configured to:

[0243] In a case where the third object is inaccessible, determining a fourth number of reference block metadata based on the block index in the target object metadata and the target stripe metadata; the reference block metadata is block metadata of a reference block, the reference block is a block in the target stripe that does not include the third object, and the fourth number is the number of reference blocks included in the target stripe;

[0244] For each reference block metadata in the fourth number of reference block metadata, determining reference data at a target position range in the corresponding reference block based on the reference block metadata;

[0245] The data of the reference object included in the fourth number of reference blocks is repaired to obtain the data of the third object.

[0246] In some embodiments, the acquisition unit is further configured to:

[0247] Determine a pointer to a candidate object included in the reference block metadata, and determine corresponding candidate object metadata, wherein the candidate object metadata is object metadata of a candidate object, and the candidate object is an object included in the reference block corresponding to the reference block metadata;

[0248] determining a position of the corresponding candidate based on the candidate object metadata of each candidate object in the reference block;

[0249] Based on the target position of the third object in the target data block and the positions of the candidate objects in the reference block, the reference object is selected from the candidate objects included in the reference block, and the reference data includes data of the reference objects in the reference block.

[0250] In practical applications, the above-mentioned first splitting unit, allocation determination unit, verification unit, classification unit, second splitting unit, access unit and other modules can be implemented by a processor located on the electronic device, specifically a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).

[0251] Those skilled in the art should understand that the relevant description of the above-mentioned electronic device in the embodiment of the present application can be understood with reference to the relevant description of the data processing method in the embodiment of the present application.

[0252] Figure 14 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application is shown in FIG. Figure 14 As shown, an embodiment of the present application provides an electronic device 1400, including an electronic chip 1401, which can implement the data processing algorithm described in one or more of the above embodiments.

[0253] An embodiment of the present application provides an electronic device, Figure 15A schematic diagram of the structure of another optional electronic device provided in an embodiment of the present application, such as Figure 15 As shown, an embodiment of the present application provides an electronic device 1500, including:

[0254] A processor 1501 and a storage medium 1502 storing executable instructions of the processor 1501. The storage medium 1502 relies on the processor 1501 to perform operations through a communication bus 1503. When the instructions are executed by the processor 1501, the data processing method performed in one or more of the above embodiments is executed.

[0255] It should be noted that in actual application, the various components in the terminal are coupled together through the communication bus 1503. It is understandable that the communication bus 1503 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 1503 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 15 Various buses are labeled as communication buses 1503.

[0256] An embodiment of the present application provides a computer storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the data processing method described in one or more of the above embodiments.

[0257] An embodiment of the present application provides a schematic structural diagram of an electronic device 1600. Figure 16 The electronic device 1600 shown includes a processor 1610. The processor 1610 is configured to:

[0258] Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as a first number of first data blocks; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe;

[0259] Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;

[0260] If the first number of first data blocks meets the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks. The first number of first data blocks and the second number of first verification blocks constitute a first stripe.

[0261] In the embodiment of the present application, the processor 1610 can call and run a computer program from the memory to implement the data processing method in the embodiment of the present application.

[0262] Alternatively, as Figure 16 As shown, the electronic device 1600 may further include a memory 1620. The processor 1610 may call and run a computer program from the memory 1620 to implement the data processing method in the embodiment of the present application.

[0263] The memory 1620 may be a separate device independent of the processor 1610 , or may be integrated into the processor 1610 .

[0264] Alternatively, as Figure 16 As shown, the electronic device 1600 may further include a transceiver 1630. The processor 1610 may control the transceiver 1630 to communicate with other devices, specifically, to receive signals sent by other devices. Here, the transceiver may include at least two antennas.

[0265] Optionally, the electronic device 1600 can implement the corresponding processes implemented by the electronic device in each method of the embodiment of the present application. For the sake of brevity, they are not described here in detail. It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method embodiments can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0266] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0267] It should be understood that the above-mentioned memories are exemplary but not restrictive. For example, the memories in the embodiments of the present application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM RAM (DR RAM), etc. In other words, the memories in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0268] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0269] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0270] An embodiment of the present application also provides a computer program product, including computer program instructions.

[0271] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0272] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0273] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0274] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0275] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0276] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0277] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0278] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as a first number of first data blocks; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe; Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume; If the first number of first data blocks meets the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks. The first number of first data blocks and the second number of first verification blocks constitute a first stripe.

2. The method according to claim 1, characterized in that The method further comprises: For an object in the cache area, the size of the object is obtained to obtain the second data volume, and the object type of the object is determined based on the relationship between the first data volume, the third data volume and the second data volume. The object with the first type of object is the first object, and the object with the second type of object is the second object.

3. The method according to claim 2, characterized in that The determining the object type of the object according to the relationship between the first data volume, the third data volume, and the second data volume includes: If the second data amount is greater than the first data amount and less than the third data amount, determining that the object type of the object is the first type; If the second data amount is smaller than the first data amount, it is determined that the object type of the object is a second type.

4. The method according to claim 3, characterized in that The determining the object type of the object according to the relationship between the first data volume, the third data volume, and the second data volume further includes: If the second data amount is greater than the third data amount, splitting the object according to the third data amount to obtain at least two split objects; If the data volume of the split object is greater than the first data volume and less than the third data volume, the split object is treated as a first object of the first type; If the data volume of the split object is smaller than the first data volume, the split object is treated as a second object of a second type.

5. The method according to claim 4, characterized in that The method further comprises: If the data amount of the split object is the third data amount, dividing the split object into the first number of second sub-blocks, and the size of the second sub-blocks is the first data amount; The first number of second sub-blocks is verified to obtain a second number of second verification blocks, and the first number of second sub-blocks and the second number of second verification blocks constitute a second stripe.

6. The method according to claim 1, characterized in that The splitting of the first object to obtain a first number of first sub-blocks includes: The first object is split based on the first number to obtain a first number of first sub-blocks, and the data amounts of different first sub-blocks in the first number of sub-blocks are the same.

7. The method according to claim 1, characterized in that The method further comprises: Determine target object metadata based on the keyword of the third object; the target object metadata is object metadata of the third object, the object metadata includes: stripe index, block index, object size and offset; the third object is any object that needs to be accessed; Determine target stripe metadata based on a stripe index in the target object metadata, where the target stripe metadata is stripe metadata of the target stripe where the third object is located; The data of the third object is obtained based on the block index, object size and offset in the target object metadata and the target stripe metadata.

8. The method according to claim 7, characterized in that The obtaining of the data of the third object based on the block index, the object size and the offset in the target object metadata and the target stripe metadata includes: In a case where the third object is accessible, determining target data block metadata based on the block index in the target object metadata and the target stripe metadata, the target data block metadata being block metadata of the target data block where the third object is located; Determining an address of the target data block in the target data block metadata; The data of the third object is read based on the address of the target data block and the object size and offset in the target object metadata.

9. The method according to claim 7, characterized in that The obtaining of the data of the third object based on the block index, the object size and the offset in the target object metadata and the target stripe metadata includes: In a case where the third object is inaccessible, determining a fourth number of reference block metadata based on the block index in the target object metadata and the target stripe metadata; the reference block metadata is block metadata of a reference block, the reference block is a block in the target stripe that does not include the third object, and the fourth number is the number of reference blocks included in the target stripe; For each reference block metadata in the fourth number of reference block metadata, determining reference data at a target position range in the corresponding reference block based on the reference block metadata; The data of the reference object included in the fourth number of reference blocks is repaired to obtain the data of the third object.

10. The method according to claim 9, characterized in that The determining, based on the reference block metadata, reference data at a target position range in the corresponding reference block includes: Determine a pointer to a candidate object included in the reference block metadata, and determine corresponding candidate object metadata, wherein the candidate object metadata is object metadata of a candidate object, and the candidate object is an object included in the reference block corresponding to the reference block metadata; determining a position of the corresponding candidate based on the candidate object metadata of each candidate object in the reference block; Based on the target position of the third object in the target data block and the positions of the candidate objects in the reference block, the reference object is selected from the candidate objects included in the reference block, and the reference data includes data of the reference objects in the reference block.

11. An electronic device, characterized in that: The electronic device comprises: a first splitting unit configured to split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as a first number of first data blocks; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe; An allocating unit configured to allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume; The verification unit is configured to verify the first number of first data blocks to obtain a second number of first verification blocks if the first number of first data blocks meet a verification condition, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.

12. An electronic device comprising a processor, characterized in that: The processor is configured to: Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as a first number of first data blocks; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe; Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume; If the first number of first data blocks meets the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks. The first number of first data blocks and the second number of first verification blocks constitute a first stripe.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the data processing method according to any one of claims 1 to 10 are implemented.

14. A storage medium storing an executable program, characterized in that: When the executable program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

15. A chip comprising a processor, characterized in that: The processor is configured to execute the data processing method according to any one of claims 1 to 10.

Images