Data processing method and device and storage medium
By splitting and aggregating objects in the cache area and generating hybrid encoding strips, the problem that the existing technology cannot be compatible with the encoding of objects of different sizes is solved, hybrid EC deployment is realized, and the system's fault tolerance and read and write performance is improved.
Patent Information
- Application Number
- CN202310627504.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-05-30
AI Technical Summary
The existing erasure coding system only supports a single encoding mode, and cannot effectively deal with mixed object scenarios with variable loads, and cannot be compatible with object encoding of different sizes.
By splitting and aggregating objects in the cache area, hybrid encoding strips are generated, hybrid EC deployment is realized, compatible with object encoding of different sizes, and meeting the needs of hybrid load scenarios.
In hybrid load scenarios, flexible and efficient data processing is achieved, compatible with object coding of different sizes, and improves the system's fault tolerance and read and write performance.
Smart Images

Figure CN119065584A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to wireless communication technology, and in particular to a data processing method and device, and a storage medium. Background Art
[0002] Erasure coding (EC) is a data protection method that divides data into fragments, performs verification calculations on data blocks to generate redundant blocks, and then distributes all blocks on different and independent storage nodes to ensure fault tolerance. Erasure coding modes include the following two: Self-Coding (SC) and Cross-Coding (CC). SC is to split a single object into different sub-blocks, and then obtain verification data through coding calculations between sub-blocks to generate stripes. In the SC coding mode, each stripe corresponds to a unique object. CC is to aggregate and splice multiple objects into a single data block, and then perform coding calculations on the multiple spliced data blocks to organize them into stripes. In the CC coding mode, each stripe will correspond to multiple different objects.
[0003] In the related art, an erasure code system only supports a single encoding mode. It can be seen that the erasure code system in the related art is aimed at the scenario with a single load and cannot meet the scenario with mixed objects with variable and complex loads. Summary of the invention
[0004] The embodiments of the present application provide a data processing method and device, and a storage medium, which are compatible with the encoding of objects of different sizes and realize flexible and efficient hybrid EC deployment.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The present application provides a data processing method, the method comprising:
[0007] Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe;
[0008] Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;
[0009] When the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
[0010] An embodiment of the present application provides an electronic device, including:
[0011] a splitting unit configured to split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe;
[0012] an allocating unit, configured to allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;
[0013] The verification unit is configured to verify the first number of first data blocks to obtain a second number of first verification blocks if the first number of first data blocks meet the verification condition, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
[0014] An embodiment of the present application provides an electronic device, including a processor, wherein the processor is configured to:
[0015] Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe;
[0016] Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;
[0017] When the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
[0018] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the above-mentioned data processing method are implemented.
[0019] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.
[0020] The chip provided in the embodiment of the present application is used to implement the above-mentioned data processing method, and the chip includes: a processor, which is used to call and run a computer program from a memory so that a device equipped with the chip executes the above-mentioned data processing method.
[0021] The data processing method, device, and storage medium provided in the embodiments of the present application split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe; the second object in the cache area is allocated to the first number of first data blocks, wherein the data volume of the second object is less than the first data volume; when the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe; thereby, in a mixed load scenario where the cache area includes a first object and a second object, by splitting the first object and aggregating the second object, the data of the first object and the data of the second object can be simultaneously present in one stripe, which can effectively cope with mixed loads of large and small objects, be compatible with encoding of objects of different sizes, and achieve flexible and efficient hybrid EC deployment. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is an optional structural diagram of the strip provided in the embodiment of the present application.
[0023] Figure 2 It is an optional flow chart of the data processing method provided in the embodiment of the present application;
[0024] Figure 3 is an optional position diagram of a boundary provided in an embodiment of the present application;
[0025] Figure 4 This is an optional location diagram of a first area and a second area divided based on a boundary provided in an embodiment of the present application;
[0026] Figure 5 It is an optional flow chart of the SC mode provided in the embodiment of the present application;
[0027] Figure 6 This is an optional flow chart of the CC mode provided in the embodiment of the present application;
[0028] Figure 7 It is an optional schematic diagram of a storage mode provided in an embodiment of the present application;
[0029] Figure 8 is an optional schematic diagram of the strip organization provided in the embodiment of the present application;
[0030] Fig. 9 is an optional schematic diagram of a data processing method provided in an embodiment of the present application;
[0031] Fig.10 is an optional schematic diagram of a data processing method provided in an embodiment of the present application;
[0032] Fig.11 is an optional schematic diagram of metadata provided by an embodiment of the present application;
[0033] Fig.12 is an optional schematic diagram of a repair scenario provided in an embodiment of the present application;
[0034] Fig.13 is an optional structural diagram of an electronic device provided in an embodiment of the present application;
[0035] Fig.14 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;
[0036] Fig.15 is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application;
[0037] Fig.16 It is an optional schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0039] The embodiments of the present application can be provided as a data processing method, device and storage medium. In practical applications, the data processing method can be implemented by an electronic device, and each functional entity in the electronic device can be collaboratively implemented by hardware resources of a computer device (such as a terminal device, a server and other devices), such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular).
[0040] Of course, the embodiments of the present application are not limited to being provided as methods and hardware, and there may be a variety of implementation methods, such as being provided as a storage medium (storing instructions for executing the data processing method provided by the embodiments of the present application).
[0041] Before describing the data processing method provided in the embodiments of the present application, the following terms are explained.
[0042] Object, the content is the data to be encoded, different objects may have different sizes, for example, the size of one object is 1KB, and the size of another object is 10MB. Different objects may also contain different types of content: audio data, text data, image data, etc. In the embodiment of the present application, there is no limitation on the type of content contained in the object.
[0043] A chunk includes a data chunk and a check chunk. A data chunk is a chunk whose content is object data, and a check chunk is a chunk whose content is check data obtained by checking the data chunk.
[0044] A stripe is a set of blocks consisting of a group of data blocks to be verified and the verification blocks obtained by verifying the group of data blocks.
[0045] The coding parameter includes a first number k and a second number n, wherein the first number is the number of data blocks included in a stripe, wherein the first number of data blocks included in a stripe constitutes a group of data blocks, and the number of check blocks included in a stripe is nk, wherein n is greater than k.
[0046] In one example, in performing (5, 3) EC encoding, where n is 5 and k is 3, Figure 1 As shown, stripe 100 includes five blocks: block 101, block 102, block 103, block 104, and block 105, wherein blocks 101, 102, and 103 are data blocks, and blocks 104 and 105 are check blocks.
[0047] It is understandable that different blocks in a stripe may be distributed in different storage nodes, and different storage nodes may be independent physical devices.
[0048] The following describes various embodiments of the data processing method, device, and storage medium provided in the embodiments of the present application.
[0049] The data processing method provided in the embodiment of the present application is applied to an electronic device, and the electronic device can be used as a data encoding server, such as Figure 2 As shown, including:
[0050] S201. The electronic device splits the first object in the cache area to obtain a first number of first sub-blocks, and stores the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe.
[0051] When the amount of data of the object received in the cache area reaches a first data amount threshold, the electronic device can distribute the object in the cache area in different data blocks to generate stripes for verification. In the embodiment of the present application, the process of distributing the objects in the cache area in different data blocks can also be called stripe organization.
[0052] For an object in the cache area, the electronic device determines that the object belongs to a first object or a second object, wherein the data volume of the first object is greater than the first data volume, and the data volume of the second object is less than the first data volume. The first object and the second object can be respectively called a large object and a small object.
[0053] The size of the first data volume can be set according to the size of the load of the current buffer area.
[0054] In some embodiments, the first data amount is greater than or less than a first data amount upper limit, and the first data amount upper limit is an upper limit of the data amount of a data block.
[0055] The cache area may include at least one or more first objects. For a first object, the electronic device splits the first object to obtain a first number of first sub-blocks, wherein the data amount, i.e., size, of different first sub-blocks may be the same or different. The electronic device stores the first number of split first sub-blocks in a first number of first data blocks, so that the first stripe where the first data block is located includes a first object, and the first object is distributed in each first data block of the first stripe.
[0056] It is understandable that after the electronic device obtains the first number of first sub-blocks, one first sub-block can be understood as a first data block. At this time, the first data block only includes the data of the large object.
[0057] The stripe organization of the present application is explained by taking the following objects as an example in which the objects to be striped in the cache area include: 1.8MB object 1, 50KB object 2, 12KB object 3, 32KB object 4, 1KB object 5, where the first data volume is 1M, n is 4, k is 2, then object 1 belongs to the first object, and the electronic device splits object 1 into two sub-blocks: sub-block 1 and sub-block 2, and sub-block 1 and sub-block 2 are stored as data block 1 and data block 2, respectively.
[0058] The electronic device storing the first sub-block of the first object as the first data block can be understood as storing the first sub-block in another storage space, and using the first sub-block as the first data block including only the first sub-block.
[0059] In actual applications, when performing stripe organization, the electronic device may select objects whose retention time in the cache area exceeds the first time length for stripe organization.
[0060] In an embodiment of the present application, when the first object is stored as a first data block, the object metadata of the first object can be determined. The object metadata includes at least one of the following: a stripe identifier, a block index, a data amount, and an offset, wherein the stripe identifier is used to identify the stripe to which the corresponding object belongs, the block index is used to index the data block to which the corresponding object belongs, the data amount is the data amount, i.e., the size, of the corresponding object, and the offset is used to identify the offset of the storage position of the corresponding object in the data block to which it belongs relative to the initial position of the block. The block index can be the serial number of the corresponding object in the stripe to which it belongs.
[0061] In the embodiment of the present application, a sub-block of a first object is stored as a data block of only one first stripe, and different first objects can be stored as data blocks in different first stripes, so that only one large object is included in one first stripe. When the number of small objects in the cache is greater than the number of large objects, and a first stripe only includes one large object, one first stripe corresponds to only one large object, which can simplify the management of large objects based on the encoding requirements of the cache.
[0062] S202: The electronic device allocates a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume.
[0063] For the second object, the electronic device aggregates the second objects in the cache area and allocates the aggregated second objects to a first number of data blocks, thereby adding each striped second object to the first number of first data, wherein one second object is allocated to only one first data block and is spliced with a sub-block in the first data block, thereby increasing the data volume of the first data block.
[0064] The electronic device allocating the second object to the first data block can be understood as the electronic device storing the second object in a storage area where the first data block is located, and splicing the second object with the existing data in the first data block, so that the first object is added to the first data block and the number of the first data block increases.
[0065] Similarly, the allocation of the second object in the stripe organization process is explained by taking the following objects as an example, in which the objects for stripe organization in the cache area include: object 1 of 1.8MB, object 2 of 50KB, object 3 of 12KB, object 4 of 32KB, and object 5 of 1KB, wherein the first data volume is 1M, n is 4, and k is 2, then object 1 belongs to the first object, and object 2, object 3, object 4, and object 5 belong to the second object. After the electronic device stores the sub-blocks 1 and sub-blocks 2 split from object 1 as data blocks 1 and data blocks 2 respectively, it allocates object 2 to data block 1 to splice object 2 and sub-block 1 to obtain a new data block 1, and allocates objects 3, object 4, and object 5 to data block 2 to splice object 3, object 4, object 5 and sub-block 2 to obtain a new data block 2.
[0066] In the embodiment of the present application, when the electronic device stores the second object as the first data block, the object metadata of the second object can be determined.
[0067] In the embodiment of the present application, the second objects organized in stripes in the cache area may be allocated by a binning algorithm so that the sizes of the first data blocks are balanced.
[0068] S203: When the first number of first data blocks meet a verification condition, the electronic device verifies the first number of first data blocks to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
[0069] During the process of allocating the second object, the electronic device monitors the first number of first data blocks obtained. If the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain nk first verification blocks. The block set consisting of the first number k first data blocks and nk first verification blocks is called a first stripe.
[0070] After determining the first stripe, the electronic device distributes the n first data blocks in the first stripe to the n nodes.
[0071] In the embodiment of the present application, the verification conditions include but are not limited to at least one of the following conditions:
[0072] The total amount of data in the data blocks exceeds the reference amount;
[0073] No object is received to fill the empty space in the data block within the second time period.
[0074] In some embodiments, the data processing method provided in the embodiments of the present application may be implemented as an erasure code, and the data processing method provided in the embodiments of the present application may be applied to an electronic device implemented as an erasure code system.
[0075] In the data processing method provided in the embodiment of the present application, large objects and small objects are mixedly encoded, so that the generated first data block includes both the data of the large object and the data of the small object, which is suitable for scenarios with mixed loads of large and small objects. Considering that the first data block includes both the data of the large object and the data of the small object, the first data block can be called a mixed data block, and the first stripe obtained by checking the first data block can be called a hybrid coding (Hybrid-Coding, HC) stripe, and the process of performing hybrid encoding on large objects and small objects in the buffer can be called HC mode.
[0076] The data processing method provided in the embodiment of the present application is to split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in a stripe; for the second object in the cache area, the first number of first data blocks are allocated, wherein the data volume of the second object is less than the first data volume; when the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe; thereby, in a mixed load scenario where the cache area includes the first object and the second object, by splitting the first object and aggregating the second object, the data of the first object and the data of the second object can be simultaneously present in one stripe, which can effectively cope with the mixed load of large and small objects, be compatible with the encoding of objects of different sizes, and realize flexible and efficient hybrid EC deployment.
[0077] In the data processing method provided in the embodiment of the present application, a first data block in a first stripe includes both data of a first object belonging to a large object and data of a second object belonging to a small object, and the first stripe can be considered as a hybrid coding stripe.
[0078] In some embodiments, the electronic device further performs the following processing:
[0079] For an object in the cache area, the size of the object is obtained to obtain a second data volume, and based on the relationship between the first data volume, the third data volume and the second data volume, the object type of the object is determined. An object with the first type of object is the first object, and an object with the second type of object is the second object.
[0080] For an object that is organized in stripes, the electronic device determines the object type of the object according to the relationship between the second data volume of the object and the first data volume and the third data volume, thereby determining whether the object belongs to the first object or the second object. If the object type of the object is the first type, the object belongs to the first object, and if the object type of the object is the second type, the object belongs to the second object.
[0081] In the embodiment of the present application, the first data volume is used to determine whether an object can be completely carried in a data block. When an object is a large object, it is considered that the data of the object cannot be completely carried in a data block, and when an object is a small object, it is considered that the object can be carried in a data block. Here, being completely carried in a data block can be considered that when the data of other objects are spliced, there is not enough space to accommodate the data block.
[0082] In the embodiment of the present application, a large object cannot be carried in a data block, but can be carried in a stripe.
[0083] The third data volume is used to determine whether an object can be carried in a stripe. In some embodiments, the third data volume is the second data volume upper limit, wherein the second data volume upper limit is the product of the first data volume upper limit and the first number, representing the data volume upper limit of a stripe.
[0084] If the first data amount is the first data amount upper limit c, then the second data amount upper limit is c*k.
[0085] In some embodiments, determining the object type of the object according to the relationship between the first data volume, the third data volume, and the second data volume includes:
[0086] If the second data amount is greater than the first data amount and less than the third data amount, the object type of the object is determined to be the first type; if the second data amount is less than the first data amount, the object type of the object is determined to be the second type.
[0087] For an object, if the data volume of the object is less than the first data volume, it is considered that the object can be completely carried in one data block; if the data volume of the object is greater than the first data volume and less than the third data volume, it is considered that the object cannot be completely carried in one data block, but can be completely carried in one stripe. An object can be completely carried in one stripe, which can be understood as the data blocks included in one stripe can accommodate all the data of the object.
[0088] Taking the first data volume as c and the third data volume as k*c as an example, if the s of an object is greater than c and less than k*c, then the object is a first object and needs to be split, and after the split, it is located in multiple data blocks. If the s of an object is less than c, then the object is a second object.
[0089] In some embodiments, determining the object type of the object based on the relationship between the first data amount and the third data amount and the second data amount also includes: if the second data amount is larger than the third data amount, splitting the object according to the third data amount to obtain at least two split objects; if the data amount of the split object is larger than the first data amount and smaller than the third data amount, treating the split object as a first object with a first type of object type; if the data amount of the split object is smaller than the first data amount, treating the split object as a second object with a second type of object type.
[0090] Here, for an object, if its data volume is greater than the third data volume, it is considered that one stripe cannot fully accommodate all the data of the object, and the object is an extra-large object. For an extra-large object, its data volume needs to be distributed in multiple stripes. At this time, the electronic device splits the object based on the third data volume to obtain at least two split objects, and the data volume of the split objects is less than or equal to the third data volume. For the at least two split objects obtained by the split, the at least two split objects are respectively regarded as an independent object, and different split objects are organized in different stripes. It should be noted that for extra-large objects, which are distributed in different stripes, one extra-large object corresponds to multiple groups of object metadata, and different groups of object metadata correspond to different stripes.
[0091] If the number of at least two split objects is h, the data volume of h-1 split objects or h split objects is the third number. If the data volume of h-1 split objects is the third number, the capacity of the remaining 1 split object is less than the third number.
[0092] For the case where there are h-1 split objects with the third data volume, the data volume of the remaining split object is determined, and the type of the split object is determined based on the relationship between the data volume of the split object and the first data volume and the third data volume, and it is determined whether the split object belongs to the first object or the second object.
[0093] In some embodiments, the electronic device also performs the following processing: if the data amount of the split object is the third data amount, the split object is divided into the first number of second sub-blocks, and the size of the second sub-blocks is the first data amount; the first number of second sub-blocks are checked to obtain a second number of second check blocks, and the first number of second sub-blocks and the second number of second check blocks constitute a second stripe.
[0094] For the split object whose data amount is equal to the third data amount in h objects, it can be understood as a large object after splitting. The electronic device splits the split object to obtain k second sub-blocks, and the k second sub-blocks are verified as k second data blocks to obtain a second stripe.
[0095] If the third data amount is k*c, the data amount of each sub-block in the k second sub-blocks is c, that is, the data amount of each second data block reaches the upper limit of the first data amount.
[0096] In an embodiment of the present application, for a split object in an ultra-large object whose data volume is equal to the third data, the split object is individually distributed in different data blocks of a second stripe, and the corresponding second stripe only includes the data of the split object, and does not include the data of other small objects, so that one second stripe corresponds to only one object.
[0097] In the embodiment of the present application, a stripe corresponding to only one object is called an SC stripe. For an extremely large object, its data is distributed in at least two or more SC stripes, or at least one SC stripe and one HC stripe, thereby realizing the encoding of the extremely large object.
[0098] In an embodiment of the present application, in a data block, the area where the large object, i.e., the sub-block of the first object, is located can be called the first area, and the area where the small object, i.e., the second object, is located can be called the second area, and there is a boundary between the first area and the second area. In a data block, the size of the first area can change with the size of the first object, so that the position of the boundary changes, and therefore, the boundary can be called a variable boundary. Among them, the first area is used to splice multiple small objects, which can be called a cross-zone (Cross-Zone, CZ), and the second area is used to store a fixed sub-block of a large object, which can be called a self-zone (Self-Zone, SZ).
[0099] In one example, if Figure 3 As shown, the middle boundary of data block 301 is located at position 3011 , the middle boundary of data block 302 is located at position 3021 , the middle boundary of data block 303 is located at position 3031 , and the middle boundary of data block 304 is located at position 3041 .
[0100] In the embodiment of the present application, the first area may be located at the starting position of the data block, and the second area may be spliced after the first area.
[0101] Based on the change in the position of the boundary between the first area and the second area in the data block, the data processing method provided in the embodiment of the present application can be applied to the following three scenarios:
[0102] Scenario 1: Large object load scenario;
[0103] Scenario 2: Small object load scenario;
[0104] Scenario 3: Mixed load of large and small objects.
[0105] For scenario 1, the cache area includes only large objects, and the electronic device encodes the large objects in the cache area through the SC mode to obtain an SC strip corresponding to only one large object. For scenario 2, the cache area includes only small objects, and the electronic device encodes the small objects in the cache area through the CC mode to obtain CC strips corresponding to multiple small objects.
[0106] For scenario 3, the cache area includes both large objects and small objects. The electronic device encodes the large objects and small objects in the cache area through the HC mode to obtain HC stripes corresponding to the large objects and small objects, and also obtains SC stripes corresponding to only one large object.
[0107] for Figure 3 The data blocks shown are as follows Figure 4 As shown, area 3012 of data block 301 is the first area and area 3013 is the second area, area 3022 of data block 302 is the first area and area 3023 is the second area, and for data block 301 or data block 302, it is a data block in the HC stripe obtained under scenario 3. Figure 4 All regions in the data block 303 shown are first regions, excluding the second region, and are data blocks in the CC strip obtained under scenario 2. Figure 4 All regions in the data block 304 shown are second regions, excluding the first region, and are data blocks in the SC strip obtained under scenario 1.
[0108] In the embodiment of the present application, the electronic device may first determine whether the object in the cache area to be striped belongs to the first object or the second object, determine the scene to which the cache area belongs according to the determination result of the first object and the second object, and execute the corresponding encoding mode in the corresponding scene. For scene 1, the SC mode is executed; for scene 2, the CC mode is executed; and for scene 3, the HC mode is executed.
[0109] It should be noted that in the embodiment of the present application, the object of the cache area may change with the implementation of the electronic device, and the scene of the cache area may also change with the change of the object. The electronic device executes the encoding of the corresponding mode according to the change of the scene.
[0110] In actual applications, when the scene to which the cache area belongs is scene 3, after encoding in the HC mode is performed, the scene to which the cache area belongs is changed to scene 2, and encoding in the CC mode is continued to be performed in the scene.
[0111] Taking the first data volume as c and the third data volume as k*c as an example, the process of determining whether an object belongs to the first object or the second object in the embodiments of the present application will be described. The process of determining whether an object belongs to the first object or the second object includes, but is not limited to, the following steps 1 to 3:
[0112] Step 1: For an object whose size is greater than the stripe data block capacity k*c, that is, an object with s greater than k*c, split it into strips. At this time, the part that is less than one strip is regarded as another object (which still needs to be recorded as the same object in terms of metadata) and is placed in Steps 2 and 3 for further consideration.
[0113] Step 2: For an object whose size is less than the stripe data block capacity k*c but greater than the single data block capacity c, that is, an object with c < s < k*c, split it into k sub-blocks with a size of s / k according to the encoding parameters and distribute them in all blocks of the same stripe. At this time, s / k serves as the boundary between the first region and the second region of the stripe.
[0114] Step 3: For all the remaining objects, which can be considered as small objects, perform an aggregation operation on them. This operation is a typical binning, and the capacity of the "bins" is c or multiple c - s / k. At this time, the binning approximation algorithm can be used for processing.
[0115] It should be noted that the value of s for each object is independent.
[0116] In some embodiments, S202 splits the first object to obtain a first number of first sub-blocks, including: splitting the first object based on the first number to obtain a first number of first sub-blocks, and the data volumes of different first sub-blocks in the first number of sub-blocks are the same.
[0117] In the embodiments of the present application, for a large object encoded by HC, the data volume of the large object in each first data block in the first stripe is the same.
[0118] When the sizes of the first sub-blocks of the first object are the same, the positions of the boundaries of different first data blocks in the first stripe are the same. For different first stripes, the positions of the boundaries in the first data blocks may be the same or different.
[0119] In the embodiment of the present application, if the amount of data of the large object included in each data block in the first stripe is the same, then the large object in the first stripe has the same offset in each data block, so that the size attribute of the current object can be determined by the object size, object offset and block size in the object metadata. For example, in a 3+1 (n is 4 and k is 3) stripe, the block size is 1MB, and the object metadata of an object is <size = 0.9MB, offset is 0.7MB>, then it can be determined that this object is composed of the last 0.3MB of each data block in the stripe, and the last 0.3MB is the second area in the data block, and the object belongs to a large object.
[0120] In some embodiments, the electronic device further performs the following processing:
[0121] Determine target object metadata based on the keyword of the third object; the target object metadata is object metadata of the third object, and the object metadata includes: stripe index, block index, object size and offset;
[0122] Determine target stripe metadata based on the stripe index in the target object metadata, where the target stripe metadata is the stripe metadata of the target stripe where the third object is located;
[0123] Based on the block index, object size and offset in the target object metadata, and the target stripe metadata, data of the third object is obtained.
[0124] The electronic device determines the object metadata of the third object in the object index table based on the keyword of the third object; the object metadata includes but is not limited to: stripe identification, block index, object size and offset, wherein the stripe identification is used to refer to the serial number of the stripe, the block index is used to indicate the serial number of the block in the stripe (0 to n-1), the object size is the data amount of the object, and the offset is the offset of the object in the block. The third object is any object that needs to be accessed.
[0125] In an embodiment of the present application, the electronic device is provided with an object index table. The object index table includes a mapping relationship between keywords of different objects and object metadata. The object metadata may be located in a 64-bit bucket. The electronic device determines the object metadata corresponding to the keyword of the third object in the object index table as the target object metadata.
[0126] After determining the target object metadata, the electronic device determines the target stripe metadata, ie, the stripe metadata of the target stripe, in the stripe index table according to the stripe index in the target object metadata. The target stripe is the stripe where the third object is located.
[0127] In an embodiment of the present application, a stripe index table is provided in the electronic device, and the stripe index table includes a mapping relationship between a stripe index and stripe metadata. The stripe metadata includes: a mapping relationship between a block index and a block identifier of a block in a corresponding stripe. The electronic device determines the stripe metadata corresponding to the stripe identifier in the target object metadata in the stripe index table as the target stripe metadata. The identifier of the target object metadata is used to identify the target stripe, and the stripe metadata corresponding to the stripe identifier in the target object metadata in the stripe index table is the target stripe metadata, i.e., the stripe metadata of the target stripe. The stripe metadata includes a mapping relationship between the block index and the block identifier of different blocks in the corresponding stripe. The target stripe metadata includes a mapping relationship between the index and the block identifier of each block in the target stripe. For a large object, the data of a large object is stored in k data blocks, and the index of the large object corresponds to k block identifiers. For a small object, the data of a small object is stored in 1 data block, and the index of the small object corresponds to 1 block identifier.
[0128] After the electronic device determines the target strip metadata, it obtains the data of the third object based on the target strip metadata according to the accessibility of the third object. The accessibility of the third object includes accessible and inaccessible, and the reason for the inaccessibility of the third object includes but is not limited to at least one of the following: failure, node overheating, and network congestion.
[0129] In the embodiment of the present application, the electronic device accesses the third object in at least one of the following ways:
[0130] Access method 1: when the third object is accessible, directly read the data of the third object in the data block storing the third object, that is, the target data block;
[0131] Access method two: when the third object is inaccessible, the data of the third object is obtained by repairing objects in other data blocks in the target stripe.
[0132] Access method 1
[0133] The electronic device obtains the data of the third object based on the block index, the object size and the offset in the target object metadata, and the target stripe metadata, including:
[0134] In the case that the third object is accessible, the target data block metadata is determined based on the block index in the target object metadata and the target stripe metadata, the target data block metadata being the block metadata of the target data block where the third object is located; the address of the target data block is determined in the target data block metadata; and the data of the third object is read based on the address of the target data block and the object size and offset in the target object metadata.
[0135] The electronic device determines the block data of the target data block storing the third object, namely, the target data block metadata, based on the target stripe data in the block index table. The target data block metadata includes the address of the target data block, namely, the initial physical address of the target data block.
[0136] In an embodiment of the present application, a block index table is provided in the electronic device, and the block index table includes a mapping relationship between a block identifier and block metadata. The block metadata includes: the address of the corresponding data block, that is, the starting physical address of the corresponding data block. The electronic device determines the address corresponding to the block identifier corresponding to the third object in the metadata of the corresponding target stripe in the block index table as the address of the target data block.
[0137] The electronic device determines the target address corresponding to the third object based on the address of the target data block and the offset in the target object data, and reads data of the object size starting from the target address to obtain the data of the third object, wherein the target address is the address of the target data block offset by the offset in the target object data.
[0138] In one example, assume that the sizes of objects in the cache are: 1.9MB, 50KB, 12KB, 32KB, 1KB, and the object of size 1.9MB is split into two sub-blocks of size 0.95MB, and the objects of size 50KB, 12KB, 32KB, and 1KB are aggregated and distributed to two data blocks, where the object of 50KB is allocated to one data block, and the objects of 12KB, 32KB, and 1KB are distributed in another data block, and the two data blocks are encoded to obtain a check block. Assume that the storage blocks in the erasure code system are a, b, and c from top to bottom, and the keys of the cache objects are 0, 1, 2, 3, and 4 from top to bottom and from left to right, respectively, and the stripe ID they constitute is 0. The upper limit of the capacity of a single data block in the erasure code system is 1MB. For an object of size 1KB. Through the object's Key: 3, we get index A: <0,1,1KB,44KB> in the object index. We locate the index of Stripe0 according to the stripe number 0 in index A, and locate block a by querying the element 1 of the mapping array in the index of Stripe0. Then, we locate the physical block address where the object is stored according to block ID a, and get the data of the object with a size of 1KB. For an object with a size of 1.9MB. Similarly, through the object's Key4, we can get index A: <0,0,1.9MB,50KB>. Therefore, we locate the stripe index through stripe number 0, and determine all the storage blocks containing the large object, i.e. ab, according to the mapping array in the stripe index item. Then, we access all the remaining data from the starting address + offset of blocks a and b respectively, and get the data of the object with a size of 1.9MB.
[0139] It can be seen that the access method for accessible objects provided in the embodiment of the present application is applicable to large objects and small objects, thereby adapting to the reading of accessible objects in a scenario with mixed loads of large and small objects.
[0140] Access method 2
[0141] The electronic device obtains the data of the third object based on the block index, the object size and the offset in the target object metadata, and the target stripe metadata, including:
[0142] In the case that the third object is inaccessible, a fourth number of reference block metadata is determined based on the block index in the target object metadata and the target stripe metadata; the reference block metadata is block metadata of a reference block, the reference block is a block in the target stripe that does not include the third object, and the fourth number is the number of reference blocks included in the target stripe; for each reference block metadata in the fourth number of reference block metadata, based on the reference block metadata, reference data at a target position range in a corresponding reference block is determined; the reference data included in the fourth number of reference blocks is repaired to obtain data of the third object.
[0143] In the case where the third object is inaccessible, the electronic device determines the block index of the target data block through the target object data, and uses the block index in the target stripe except the block index of the target data block as the block index of the reference block, and determines the block identifier corresponding to the reference block based on the block index of the reference block. Wherein, the reference block is a block in the target stripe that does not include the third object, and the number of reference blocks in the target stripe is the fourth number (n-1). In one example, n is 5 and k is 3, and the target stripe includes the following blocks: chunk a, chunk b, chunk c, chunk d, and chunk e, and the index of each block is 0, 1 to 4; if the third object is located in chunk 1, the block index in the target object metadata is 1, and includes the following block indexes of the reference blocks: 0, 2, 3, 4, and the following block identifiers are determined based on 0, 2, 3, 4: a, c, d, e, and the following reference blocks are determined: (chunk) a, chunk c, chunk d, and chunk e.
[0144] After the electronic device determines the block identifier of the reference block, it determines the block metadata of each reference block, i.e., the reference block metadata, from the block index based on the block identifier of each reference block. For a reference block metadata, if the reference block is a data block, it includes the address of each reference block and the position information of each object, i.e., the candidate object, included in the reference block. The electronic device determines the data of the object, i.e., the reference object, located within the target position range from the candidate objects included in the reference block based on the reference block metadata to obtain the reference block. For a reference block metadata, if the reference block is a check block, the check information it includes is a whole, and therefore, all the data of the reference block can be considered as the reference data of the reference block. Among them, the target position range is the position of the third object in the target data block. The electronic device can determine the target position range based on the object size and offset in the target object metadata.
[0145] After the electronic device determines the reference data in each reference block, it decodes all the reference data to obtain the repaired data of the third object, thereby repairing the inaccessible data of the third object.
[0146] In an embodiment of the present application, for a large object, its offset and position in different data blocks are the same. Therefore, the electronic device can obtain data of other sub-blocks that can form a large object with the third object based on the position of the same offset of the offset of the third object in the reference block belonging to the data block, and obtain the data of the modified third object by decoding the data of these sub-blocks and the check block.
[0147] In an embodiment of the present application, in the data acquisition process of the inaccessible object, that is, the repair process of the inaccessible object, the third object is repaired based on the data in other blocks other than the target data block in the target stripe that have the same position as the third data block, and there is no need to read all the data in other blocks, thereby reducing unnecessary data reading overhead and realizing partial decoding of the stripe; especially for the repair of small data, targeted reading of other object data aligned with the read object to be degraded can reduce the read amplification ratio.
[0148] In some embodiments, based on the reference block metadata, reference data at a target position range in a corresponding reference block is determined, including: determining a pointer to a candidate object included in the reference block metadata, and determining corresponding candidate object metadata, wherein the candidate object metadata is object metadata of the candidate object, and the candidate object is an object included in the reference block corresponding to the reference block metadata; determining the position of the corresponding candidate based on the candidate object metadata of each candidate object in the reference block; selecting the reference object from the candidate objects included in the reference block based on the target position of the third object in the target data block where it is located, and the position of each candidate object in the reference block, wherein the reference data includes data of the reference object in the reference block.
[0149] In the embodiment of the present application, in addition to the physical address of the corresponding block, the block metadata may also correspond to the pointer set for the object in the corresponding block to point to the object metadata of the corresponding object, thereby realizing not only the link of the object metadata to the block metadata, but also the link of the block metadata to the object metadata. In some embodiments, the code metadata may include a linked list connecting the objects in the corresponding block, and in the linked list, for one object, the following two parts may be included: a pointer to the object metadata of the corresponding object and a pointer Next pointing to the next object.
[0150] The electronic device obtains object metadata of each candidate object in the corresponding reference block based on the pointer of each object in the reference block metadata, wherein the stripe index and block index in the object metadata of the candidate objects in the same reference block are the same.
[0151] For a candidate object in a reference block, the electronic device determines whether the candidate object is a reference object based on the position of the candidate object, and if so, reads the data of the reference object. The reference data read by the electronic device includes the data of all reference objects in the reference block.
[0152] Among them, for a reference object, the electronic device determines the location range of the corresponding candidate object based on the object size and offset in the object metadata of a candidate object. When the location range overlaps with the target location range, the candidate object can be considered as a reference object, and the data of the reference object is read based on the address in the reference block and the location range of the reference object.
[0153] The method provided in the embodiment of the present application realizes unified indexing of large and small objects by adopting a multi-level, closed index structure. The multi-level is reflected in three different indexes: object index, i.e., the mapping relationship in the object index table, stripe index, i.e., the mapping relationship in the stripe index table, and block index, i.e., the mapping relationship in the block index table. The closedness is reflected in that any index item can be used to locate other index items associated with it, thereby supporting mixed loads of large and small objects. In addition, the unified index provided in the embodiment of the present application is compatible with large objects in the Self Coding mode, and at the same time, additional information is added to make the multi-level indexes interrelated, thereby realizing the closure of the index, and further providing comprehensive index information for various system operations, such as downgraded reading, repair, etc.
[0154] The following is a further description of the data processing method provided in the embodiments of the present application.
[0155] The erasure coding modes of object storage systems include the following two: Figure 5 The SC and Figure 6 As shown in CC. Figure 5 As shown in , SC splits a single object into different sub-blocks (splits), and then obtains the check data through coding calculation between the sub-blocks, and then generates stripes. In the coding mode of SC, each stripe corresponds to a unique object. Figure 6 As shown, CC aggregates or splices multiple objects including object 0, object 1, ..., object n into multiple data blocks, and then performs encoding calculations on the multiple spliced data blocks, and then organizes them into strips. In the encoding mode of CC, each strip corresponds to multiple different objects.
[0156] Due to the existence of parallel input / output (I / O), the erasure code system based on SC has better read and write performance in object access, as well as better load balancing and tail latency performance. At the same time, due to its stripe-as-object feature, SC has a very low repair bandwidth when repairing / downgrading a single object. However, SC also has the problem of unstable data blocks and multiple metadata queries in a single I / O. Therefore, SC is more suitable for large objects that are not sensitive to query time and have a large room for optimization of I / O performance.
[0157] The CC-based erasure code system is the opposite of the SC-based erasure code system. It has the characteristics of fixed data block size and single I / O single metadata query. However, due to the lack of parallelism, its performance in single object access is poor, and there is a serious read amplification problem in the repair process of a single object. Therefore, CC is more suitable for small objects with time-sensitive queries and limited I / O performance optimization space.
[0158] It should be noted that repair / downgrade reading is to read data that is temporarily inaccessible (such as failure, node overheating, network congestion, etc.) through erasure code repair. For inaccessible data, it is necessary to read other data in the same stripe through cross-node access to calculate and decode the unavailable data. The network bandwidth generated by cross-node access is the repair bandwidth. Because of the organization of CC, repairing a small object requires repairing the entire stripe, and the repair bandwidth will be more. For example, in order to repair 1KB of data, the 4MB stripe needs to be restored in full, and the restoration is only to obtain the required 1KB of data. Since the SC object is a stripe, 4MB of bandwidth is required to repair and obtain the required 4MB of data. Therefore, SC has a very low repair bandwidth when repairing / downgrading a single object, but SC has a serious read amplification problem in the repair process of a single object.
[0159] CC is aggregated, so the block size of the stripe is fixed. The number of metadata queries refers to the mapping query from the object to the physical address. In CC mode, the data of a single object is generally concentrated in one block, so it is enough to access only one node. In contrast, in SC mode, accessing a single data requires initiating multiple I / O requests to access multiple nodes in parallel.
[0160] In one example, erasure coding systems under different object sizes may be as shown in Table 1.
[0161] Table 1. Example of erasure code system
[0162]
[0163] In the related art, as shown in Table 1, different erasure code systems only consider the scenario of large object / small object exclusiveness. In the design of these erasure code systems, the data load is single, such as streaming data dominated by GB-level large objects, or key-value pairs dominated by Byte-level small objects. Although in specific occasions, a single SC / CC mode can play its advantages well and make the entire EC system run efficiently and stably. However, with the growing development of cloud computing today, the data load faced by cloud systems is often extremely unbalanced.
[0164] Figure 7 The workload of the real object storage pool is displayed, where storage model 71 is the capacity ratio of storage pool 1, storage model 72 is the quantity ratio of storage pool 1, storage model 73 is the capacity ratio of storage pool 2, and storage model 74 is the quantity ratio of storage pool 2.
[0165] For storage model 71, the capacity of data [0, 64KB) accounts for 0.2%, the capacity of data [64KB, 4MB) accounts for 30.0%, the capacity of data [4MB, 16KB) accounts for 20.5%, the capacity of data [16MB, 64MB) accounts for 17.4%, the capacity of data [64MB, 1GB) accounts for 26.9%, and the capacity of data greater than 1GB accounts for 4.9%.
[0166] For storage model 72, the amount of data [0, 64KB) accounts for 27.7%, the amount of data [64KB, 4MB) accounts for 62.4%, the amount of data [4MB, 16KB) accounts for 8.0%, the amount of data [16MB, 64MB) accounts for 1.4%, the amount of data [64MB, 1GB) accounts for 0.2%, and the capacity of data greater than 1GB accounts for 0%.
[0167] For storage model 73, the capacity of data [0, 64KB) accounts for 4.3%, the capacity of data [64KB, 4MB) accounts for 1.8%, the capacity of data [4MB, 16KB) accounts for 3.0%, the capacity of data [16MB, 64MB) accounts for 6.9%, the capacity of data [64MB, 1GB) accounts for 30.0%, and the capacity of data greater than 1GB accounts for 53.9%.
[0168] For storage model 74, the amount of data in [0, 64KB) accounts for 98.7%, the amount of data in [64KB, 4MB) accounts for 1.2%, the amount of data in [4MB, 16KB) accounts for 0.1%, the amount of data in [16MB, 64MB) accounts for 0%, the amount of data in [64MB, 1GB) accounts for 0%, and the capacity of data greater than 1GB accounts for 0%.
[0169] based on Figure 7 It can be determined that the size distribution of objects is extremely uneven: in some business scenarios, the number of small objects will account for 98% of the storage pool, while in other business scenarios, the number of small objects may only account for 27% of the storage pool. Obviously, it is unreasonable to use different EC frameworks for different storage pools, and its complex and high operation and maintenance costs will bring too much burden to the enterprise. However, using a single EC encoding mode for different storage pools cannot well meet the changing and complex mixed object scenarios. For example, if storage pool 1 and storage pool 2 use the CC mode EC system, for storage pool 1, using the CC mode will deparallelize all object accesses, among which 82% of the large object I / O performance will be limited by the bandwidth of single channel access. If storage pool 1 and storage pool 2 use the SC mode EC system, for storage pool 2, using the SC mode will reduce the read and write performance of 98% of small objects due to the startup overhead of concurrent I / O access, because their small data volume is not enough to benefit from parallel I / O.
[0170] Therefore, the erasure code system in the related art adopts a single encoding mode solution and cannot flexibly handle mixed object load scenarios: a single SC / CC mode deployment will always optimize the read and write performance of some objects at the expense of other objects.
[0171] For large objects, the SC method is used for encoding, so as to fully utilize the performance advantages of parallel I / O. Since large objects take a long time to access, the cost of multiple metadata query overhead brought by SC is acceptable. For small objects, the CC method is used for encoding, thereby avoiding the I / O amplification caused by object splitting and the metadata query overhead caused by frequent access to small objects. However, when dealing with mixed load scenarios with objects of different sizes, the single SC / CC mode is inflexible and has problems with performance degradation of some services. Therefore, the embodiment of the present application will propose a hybrid erasure code encoding design to address this dilemma, so that it is compatible with the encoding of large and small objects, and realizes flexible and efficient hybrid EC deployment and management.
[0172] In the data processing method provided in the embodiment of the present application, the coding modes of SC and CC are integrated into the same stripe, so that EC stripe deployment can be flexibly performed when facing objects of different sizes. This is called the Hybrid-Coding (HC) mode. In the HC mode, each block in the stripe is split into two different areas: CZ and SZ. In the CZ, small objects are gathered; in the SZ, large objects are split. When the amount of data stored in the stripe reaches a threshold, the encoding is triggered, and the data mixed with large and small objects are encoded together in the same stripe.
[0173] Due to the huge gap in the total storage volume of large and small objects, a single large object in each HC stripe can meet the EC deployment requirements of most mixed loads. Therefore, for the purpose of simplifying the design, the number of large objects in an HC stripe is at most 1. Under this premise, different blocks in the same stripe can have the same boundaries. When an HC stripe includes multiple large objects, there may be data of different large objects in a block, and the boundaries between large objects need to be distinguished. For example, the 1KB to 3KB of each block in the stripe are large object A, and the large object B is after 3KB. This design makes the management and design of metadata complicated.
[0174] It is worth noting that the division of the boundary between SZ and CZ in different stripes can be different, and the location of the boundary depends on the distribution of the object sizes participating in the stripe, which means that HC can flexibly degenerate into SC or CC according to the needs of the current workload.
[0175] Figure 8Various possibilities of HC stripes are shown in the figure: for case 0, when the data size of the cache area is different, the system will select a suitable boundary to organize a single large object and multiple small objects of appropriate size into the same stripe (stripe 0); for Case 1, when the object in the cache area is a large file, the system will push the boundary to the head of the stripe (stripe 1), so that the stripe degenerates to the SC mode, and directly splits and encodes a single large object; for Case 2, when the object in the cache area is a small file, the system will push the boundary to the end of the stripe (stripe 2), so that the stripe degenerates to the CC mode, and multiple small objects are aggregated and encoded. It can be understood that the boundary of a stripe can be understood as the boundary of each data block included in the stripe, and the position of the boundary of each data block is the same. A data block is divided into two areas, CZ and SZ, based on the boundary.
[0176] In an embodiment of the present application, the encoding parameters of a stripe may include information such as the number of data blocks and the number of check blocks in the stripe, for example: (n, k)-EC encoding means encoding k data blocks into n blocks, where the number of check blocks is nk. Figure 8 The encoding in can be identified as a (3, 2) stripe, where a stripe includes 3 blocks: 2 data blocks and 1 check block.
[0177] Next, the stripe organization and metadata management, and the degraded read operation included in the data processing method provided in the embodiment of the present application are described respectively.
[0178] Strip organization
[0179] In response to the mixed load of large and small objects, a hybrid coding stripe organization method is proposed. By adaptively dividing SZ and CZ, the two coding modes of Self Coding and Cross Coding are integrated into a single stripe to achieve mixed coding of large and small objects. Among them, the boundary of the Self Zone / Cross Zone in the stripe is determined in real time according to the system load, and in extreme cases it can be degraded to a single Self Coding / Cross Coding mode. Compared with a single Self Coding / Cross Coding, hybrid coding can effectively cope with mixed loads of large and small objects, while ensuring the system's fault tolerance, it improves the read and write performance of large and small objects.
[0180] Here, Fig. 9 As an example, the (3,2)-HC strip is shown to be effective in the face of mixed loads (i.e. Figure 8The organization process for Case 0) is shown in the figure. When the cache accepts objects with sufficient data, the erasure code system will start to organize HC stripes. First, the erasure code system will select a large object in the cache and split the large object according to the encoding parameters (n, k). After the split, the erasure code system divides the boundary between SZ and CZ based on the sub-block size of the large object. The erasure code system then gathers small objects from the cache and distributes them to CZs of different blocks without crossing the boundary. When the aggregation of small objects reaches the storage upper limit of the data block, the erasure code system encodes and calculates the data blocks belonging to an HC stripe to generate a check block including check data. At this point, an HC stripe mixed with objects of different sizes is generated.
[0181] like Fig. 9 As shown, the sizes of objects in the cache are: 1.9MB, 50KB, 12KB, 32KB, 1KB, and the object with a size of 1.9MB is split into two sub-blocks with a size of 0.95MB, and the two sub-blocks are SZs of different data blocks. Objects with sizes of 50KB, 12KB, 32KB, and 1KB are aggregated and distributed to two data blocks, wherein the object of 50KB is allocated to the CZ of one data block, and the objects of 12KB, 32KB, and 1KB are distributed in the CZ of another data block, and the two data blocks are encoded to obtain a check block.
[0182] In order to meet the characteristics of high availability and low latency, the stripe organization algorithm should be an online, polynomial time algorithm. Online (online) means that the size of the next object requested by the user cannot be predicted, and the algorithm can only consider the object sequence already in the current cache when executing; at the same time, in order to stripe the data in the cache as soon as possible, the algorithm should be a polynomial time algorithm to ensure the efficient operation of the system.
[0183] Considering the above requirements, the stripe organization algorithm in the data processing method of this application will be based on the following premise: when the amount of data in the cache accumulates to a preset threshold, the erasure code system will start the stripe organization algorithm and perform stripe organization on objects whose retention time in the cache exceeds a certain value. Therefore, the stripe organization algorithm abstraction will be as follows:
[0184] Input: N object entities x, entity x has size attribute s, and collection capacity upper limit c;
[0185] Output: m sets P, the i-th set P i There are k sets, each of which is an entity x;
[0186] Constraint: For the jth set in the i-th set group satisfy:
[0187] Objective: While satisfying the minimum of m, is minimized, where m is the number of stripes.
[0188] In the above stripe organization algorithm, a set can be understood as a data block, and a set group can be understood as a data block set composed of data blocks in a stripe.
[0189] In the embodiments of the present application, stripe organization can be understood as a binning problem or equivalently a set partitioning problem. In computational complexity theory, such problems belong to NPC problems, so it is unreasonable to seek an optimal solution. In order to meet the requirement that objects do not overflow the data block, the data entity x in the buffer area will necessarily face the following three considerations:
[0190] (1) For objects whose size is greater than the stripe data block capacity k*c, that is, objects with s greater than k*c, split them into stripes. At this time, the part that is less than one stripe is regarded as another object (which still needs to be recorded as the same object in terms of metadata) and placed in (2) and (3) for further consideration.
[0191] (2) For objects whose size is less than the stripe data block capacity k*c but greater than the single data block capacity c, that is, objects with c < s < k*c, split them into k sub-blocks with a size of s / k according to the encoding parameter and distribute them in all blocks of the same stripe. At this time, s / k serves as the boundary of SZ / CZ of this stripe.
[0192] (3) For all the remaining objects, they can be considered as small objects and aggregated. This operation is a typical binning, and the capacity of the "bins" is c or multiple c - s / k. At this time, a binning approximation algorithm can be used for processing.
[0193] In the embodiments of the present application, the logic of the stripe organization algorithm is: split large objects -> allocate sub-blocks of large objects -> perform an approximate binning algorithm on small objects.
[0194] In the data processing algorithm provided by the embodiments of the present application, a simple replica redundancy strategy will be adopted to solve the fault tolerance problem in the generation process of HC stripes. During the generation process of HC stripes, such as Fig.10As shown in the figure, the calculation task of the check block is delegated to the check node, and then the data copy is distributed to nk data nodes during data transmission, so that it forms (n-k+1)-way copies, that is, there are nk copies based on the original data, so as to ensure the fault tolerance of the stripe organization process. After the data distribution is completed, the data on each check block can be verified and calculated by itself, and the copy mode is converted to EC mode. Among them, when the data of a node is lost, since it has copies on other nodes, its data is not permanently lost, and the copy can be directly copied and used.
[0195] Metadata Management
[0196] For mixed encoding of large and small objects, the data processing method provided in the embodiment of the present application adopts a multi-level, closed index structure to achieve unified indexing of large and small objects. The multi-level is reflected in three different indexes: object, stripe, and block index, and the closedness is reflected in the ability to locate other index items associated with any index item. Existing indexing technologies are all based on a single Self Coding / Cross Coding and cannot support mixed loads of large and small objects. Compared with existing indexing technologies, the unified index proposed in the embodiment of the present application expands the index items on the basis of the Cross Coding mode index, making it compatible with large objects in the Self Coding mode, and at the same time adds additional information to make the multi-level indexes interrelated, achieve the closure of the index, and then provide comprehensive index information for various system operations, such as downgraded reads and repairs.
[0197] In the data processing method provided in the embodiment of the present application, a multi-level index structure will be designed for the HC stripe to uniformly manage the metadata of large and small objects. The index of the HC stripe will consist of three hash tables: object index, stripe index and storage block index. Fig.11 As shown, the three indexes are linked to each other. The erasure code system can query any object / strip / storage block information related to any index, thereby meeting the query requirements of operations such as read / write / repair / degraded read.
[0198] like Fig.11 As shown in Figure 2, the object index (table) is a hash table that maps the object key to a 64-bit bucket. Each bucket contains the values of four parameters: Stripe ID, Chunk Idx, Size, and Offset. The meanings of the four parameters are shown in Table 2. Through these four parameters, the specific physical storage location of the object can be located. At the same time, by analyzing the relationship between Size and Offset, the size of the current object can be determined without the help of additional tags.
[0199] Table 2. Definition of object parameters
[0200] Bit value describe 0-31 Stripe ID A serial number used to refer to the stripe 32-35 Chunk Idx Sequence number of the block within the stripe (0 to n-1) 36-51 Size The size of the object 52-63 Offset The offset of the object in the block
[0201] like Fig.11 As shown, the stripe index (table) is a hash table that maps from Stripe ID to stripe metadata. The stripe metadata will store a mapping array from the block number Chunk Idx in the stripe to the block ID, thereby recording the block organization of the stripe, so that the location of the target storage block can be located through the Stripe ID and Chunk Idx.
[0202] like Fig.11 As shown in Figure 1, the block index (table) is a hash table that maps Chunk ID to block metadata. In addition to storing the physical address of the block, the block metadata also stores a pointer list to record the organization of objects in the block. The structure of the linked list represents the object composition of the block, and Fig.11 As shown in the figure, the content of the linked list is a pointer (Ptr) to the corresponding object metadata. This enables quick understanding of the organization of storage blocks and implements indexing from the physical storage layer to object information (this design also provides an index basis for optimizing the degraded read performance of small objects)
[0203] Below, Fig. 9 As an example, the access process of object data under HC coding is described in detail. Assume that the storage blocks in the erasure code system are a, b, and c from top to bottom, and the keys of the cache objects are 0, 1, 2, 3, and 4 from top to bottom and from left to right. The stripe ID is 0, and the capacity limit of a single data block in the erasure code system is 1MB.
[0204] For an object with a size of 1KB in CZ. The object's Key: 3 is used to obtain index A: <0,1,1KB,44KB> in the object index. The stripe number 0 in index A is used to locate the index of Stripe0, and the element No. 1 in the mapping array in the index of Stripe0 is queried to locate block a. Thus, the physical block address where the object is stored is located according to block ID a. At this time, since the object's Offset+Size<1MB, it can be determined that the object is a small object and is in CZ, so only 1KB of data needs to be accessed from the starting address + offset of block a.
[0205] For an object with a size of 1.9MB in SZ, similarly, the index A can be obtained through the Key4 of the object: <0,0,1.9MB,50KB>. Since the Offset+Size of the object>1MB, it can be determined that the object is a large object and is in SZ. Therefore, the stripe index is located through the stripe number 0, and all storage blocks containing the large object, namely ab, are determined according to the mapping array in the stripe index item. Then, all the remaining data can be accessed from the starting address + offset of blocks a and b respectively.
[0206] The above-mentioned process access demonstrates the advantages of the metadata design provided by the embodiment of the present application: the same metadata structure can manage the data in SZ and CZ at the same time, and also realizes the unification of the access process.
[0207] When deploying EC for small objects, neither the existing CC mode nor the HC mode proposed in the embodiment of the present application can avoid the serious read amplification problem caused by the downgraded read of small objects. This is because there are often multiple small objects in a stripe in the CC / HC mode. At this time, when a downgraded read operation is performed on a single small object, the entire stripe will be decoded and calculated. At this time, the read amplification ratio will reach (k*block size) / object size. If Fig. 9 Taking the downgraded read of 1KB data as an example, the read amplification ratio will reach an astonishing 2MB / 1KB=2000.
[0208] Degraded Read Operations
[0209] In order to optimize this read amplification ratio, the embodiment of the present application proposes an extended technical solution: through block indexes to specifically read other object data aligned with the read object to be degraded, to achieve partial decoding of the stripe, thereby reducing unnecessary data reading overhead and lowering the read amplification ratio.
[0210] by Fig.12 The (4, 3) EC is taken as an example to illustrate how to achieve local decoding of the stripe through selective reading. Here, it is assumed that object 1 is currently inaccessible and a downgraded read operation is required. In the traditional downgraded read process, the erasure code system will arbitrarily select k (3) storage blocks that do not contain object 1 from the stripe, namely Chunk 1, Chunk 2, Chunk 3, and then perform decoding calculations, restore Chunk 0, and then truncate object 1. In an embodiment of the present application, by analyzing the index items of each block, objects whose offsets coincide with object 1, namely objects 4, 5, 7 and the check block, are selected, and then a local decoding operation is performed to restore object 1. In this process, access to useless object data 3, 6, and 8 can be avoided, thereby optimizing the downgraded read ratio of small objects.
[0211] It should be noted that although the above example only shows the downgraded read optimization solution in CC mode, its concept is also applicable in CZ of HC.
[0212] The data processing method provided in the embodiment of the present application has the following technical effects:
[0213] As for the benefits of large objects, since all large objects are stored in SZ, the access mode of large objects is consistent with the SC mode, and its read and write performance can reach the performance level of a single SC mode. When accessing large objects through the index process provided in the embodiment of the present application, the erasure code system will also perform parallel I / O, which will improve the read and write throughput of objects while reducing the tail latency of object access. The characteristics and advantages of this access mode have been verified in practice in EC-Cache. At the same time, since the SZ-CZ boundaries in the HC stripes are aligned, the downgraded read of large objects is the same as in the SC mode, and there is no problem of read amplification.
[0214] As for the benefits of small objects, since all small objects are stored in CZ, the access mode of small objects is consistent with the CC mode, and its read and write performance can reach the performance level of a single CC mode. By accessing small objects through the index process provided in the embodiment of the present application, the erasure code system will only start a single I / O, thereby avoiding the bottleneck caused by the startup overhead of multiple I / Os and reducing the access performance of small objects. In addition, due to the additional design of block indexes, there can also be additional designs to optimize the problem of degraded read I / O amplification of small objects.
[0215] Combined with the above benefits for large objects and small objects, the data processing method provided in the embodiment of the present application has the following performance advantages: HC strips can be flexibly organized according to different object sizes so that all types of objects can adopt the optimal encoding mode, thereby obtaining performance benefits and avoiding performance losses in some business scenarios.
[0216] The unified framework design of data storage in this application has good flexibility and is compatible with most object load scenarios. At the same time, there is no need to maintain two different frameworks to manage large / small objects respectively, which makes the EC framework have great advantages in development, expansion, and operation and maintenance.
[0217] An electronic device according to an embodiment of the present application, such as Fig.13 As shown, the electronic device 1300 includes:
[0218] The first splitting unit 1301 is configured to split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe;
[0219] An allocating unit 1302 is configured to allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;
[0220] The verification unit 1303 is configured to verify the first number of first data blocks to obtain a second number of first verification blocks if the first number of first data blocks meet the verification condition, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
[0221] In some embodiments, the electronic device 1300 further includes: a classification unit configured to:
[0222] For an object in the cache area, the size of the object is obtained to obtain a second data volume, and based on the relationship between the first data volume, the third data volume and the second data volume, the object type of the object is determined. An object with the first type of object is the first object, and an object with the second type of object is the second object.
[0223] In some embodiments, the classification unit is further configured to:
[0224] If the second data amount is greater than the first data amount and less than the third data amount, determining that the object type of the object is the first type;
[0225] If the second data amount is smaller than the first data amount, it is determined that the object type of the object is a second type.
[0226] In some embodiments, the classification unit is further configured to:
[0227] If the second data volume is greater than the third data volume, split the object according to the third data volume to obtain at least two split objects;
[0228] If the data volume of the split object is greater than the first data volume and less than the third data volume, treating the split object as a first object of a first type;
[0229] If the data volume of the split object is smaller than the first data volume, the split object is treated as a second object of a second type.
[0230] In some embodiments, the electronic device 1300 further includes a second splitting unit configured to, if the data amount of the split object is the third data amount, divide the split object into the first number of second sub-blocks, and the size of the second sub-block is the first data amount;
[0231] The verification unit 1303 is further configured to verify the first number of second sub-blocks to obtain a second number of second verification blocks, and the first number of second sub-blocks and the second number of second verification blocks constitute a second stripe.
[0232] In some embodiments, the first splitting unit 1301 is further configured to:
[0233] The first object is split based on the first number to obtain a first number of first sub-blocks, and the data amounts of different first sub-blocks in the first number of sub-blocks are the same.
[0234] In some embodiments, the electronic device 1300 further includes: an access unit configured to:
[0235] Determine target object metadata based on the keyword of the third object; the target object metadata is object metadata of the third object, and the object metadata includes: stripe index, block index, object size and offset;
[0236] Determine target stripe metadata based on the stripe index in the target object metadata, where the target stripe metadata is the stripe metadata of the target stripe where the third object is located;
[0237] Based on the block index, object size and offset in the target object metadata, and the target stripe metadata, data of the third object is obtained.
[0238] In some embodiments, the access unit is further configured to:
[0239] In the case where the third object is accessible, determining target data block metadata based on the block index in the target object metadata and the target stripe metadata, the target data block metadata being the block metadata of the target data block where the third object is located;
[0240] Determining an address of the target data block in the target data block metadata;
[0241] Based on the address of the target data block and the object size and offset in the target object metadata, the data of the third object is read.
[0242] In some embodiments, the access unit is further configured to:
[0243] In the case that the third object is inaccessible, determining a fourth number of reference block metadata based on the block index in the target object metadata and the target stripe metadata; the reference block metadata is block metadata of a reference block, the reference block is a block in the target stripe that does not include the third object, and the fourth number is the number of reference blocks included in the target stripe;
[0244] For each reference block metadata of the fourth number of reference block metadata, determining reference data at a target position range in the corresponding reference block based on the reference block metadata;
[0245] The data of the reference object included in the fourth number of reference blocks is repaired to obtain the data of the third object.
[0246] In some embodiments, the acquisition unit is further configured to:
[0247] Determine a pointer of a candidate object included in the reference block metadata, and determine corresponding candidate object metadata, wherein the candidate object metadata is object metadata of the candidate object, and the candidate object is an object included in the reference block corresponding to the reference block metadata;
[0248] Determining a position of a corresponding candidate based on the candidate object metadata of each candidate object in the reference block;
[0249] Based on the target position of the third object in the target data block and the positions of the candidate objects in the reference block, the reference object is selected from the candidate objects included in the reference block, and the reference data includes the data of the reference objects in the reference block.
[0250] In practical applications, the above-mentioned first splitting unit, allocation determination unit, verification unit, classification unit, second splitting unit, access unit and other modules can be implemented by a processor located on the electronic device, specifically a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA).
[0251] Those skilled in the art should understand that the relevant description of the above-mentioned electronic device in the embodiment of the present application can be understood by referring to the relevant description of the data processing method in the embodiment of the present application.
[0252] Fig.14 A schematic diagram of the structure of an optional electronic device provided in an embodiment of the present application, such as Fig.14 As shown, an embodiment of the present application provides an electronic device 1400, including an electronic chip 1401, which can be implemented as a data processing algorithm as described in one or more of the above embodiments.
[0253] An embodiment of the present application provides an electronic device, Fig.15A schematic diagram of the structure of another optional electronic device provided in an embodiment of the present application, such as Fig.15 As shown, an embodiment of the present application provides an electronic device 1500, including:
[0254] A processor 1501 and a storage medium 1502 storing executable instructions of the processor 1501, wherein the storage medium 1502 relies on the processor 1501 to perform operations via a communication bus 1503, and when the instructions are executed by the processor 1501, the data processing method performed in one or more of the above embodiments is executed.
[0255] It should be noted that in actual application, the various components in the terminal are coupled together through the communication bus 1503. It is understandable that the communication bus 1503 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 1503 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.15 Various buses are labeled as communication buses 1503.
[0256] An embodiment of the present application provides a computer storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the steps of the data processing method described in one or more of the above embodiments.
[0257] An embodiment of the present application provides a schematic structural diagram of an electronic device 1600. Fig.16 The electronic device 1600 shown includes a processor 1610. The processor 1610 is configured to:
[0258] Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe;
[0259] Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume;
[0260] When the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
[0261] In the embodiment of the present application, the processor 1610 can call and run a computer program from the memory to implement the data processing method in the embodiment of the present application.
[0262] Alternatively, if Fig.16 As shown, the electronic device 1600 may further include a memory 1620. The processor 1610 may call and run a computer program from the memory 1620 to implement the data processing method in the embodiment of the present application.
[0263] The memory 1620 may be a separate device independent of the processor 1610 , or may be integrated into the processor 1610 .
[0264] Alternatively, if Fig.16 As shown, the electronic device 1600 may further include a transceiver 1630, and the processor 1610 may control the transceiver 1630 to communicate with other devices, specifically, to receive signals sent by other devices. Here, the transceiver may include at least two antennas.
[0265] Optionally, the electronic device 1600 can implement the corresponding processes implemented by the electronic device in each method of the embodiment of the present application, and for the sake of brevity, it will not be repeated here. It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Arra y, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0266] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0267] It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiments of the present application may also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct RAM bus random access memory (Direct RAM, DR RAM), etc. That is to say, the memory in the embodiments of the present application is intended to include but not limited to these and any other suitable types of memory.
[0268] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.
[0269] Optionally, the computer-readable storage medium can be applied to the electronic device in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not repeated here.
[0270] An embodiment of the present application also provides a computer program product, including computer program instructions.
[0271] Optionally, the computer program can be applied to the electronic device in the embodiments of the present application. When the computer program runs on a computer, the computer executes the corresponding processes implemented by the electronic device in the various methods of the embodiments of the present application. For the sake of brevity, they are not described here.
[0272] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0273] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0274] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0275] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0276] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0277] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0278] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A data processing method, characterized in that: The method comprises: Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe; Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume; When the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
2. The method according to claim 1, characterized in that The method further comprises: For an object in the cache area, the size of the object is obtained to obtain a second data volume, and based on the relationship between the first data volume, the third data volume and the second data volume, the object type of the object is determined. An object with the first type of object is the first object, and an object with the second type of object is the second object.
3. The method according to claim 2, characterized in that The determining the object type of the object according to the relationship between the first data volume, the third data volume, and the second data volume includes: If the second data amount is greater than the first data amount and less than the third data amount, determining that the object type of the object is the first type; If the second data amount is smaller than the first data amount, it is determined that the object type of the object is a second type.
4. The method according to claim 3, characterized in that: The determining the object type of the object according to the relationship between the first data volume, the third data volume, and the second data volume also includes: If the second data volume is greater than the third data volume, split the object according to the third data volume to obtain at least two split objects; If the data volume of the split object is greater than the first data volume and less than the third data volume, treating the split object as a first object of a first type; If the data volume of the split object is smaller than the first data volume, the split object is treated as a second object of a second type.
5. The method according to claim 4, characterized in that The method further comprises: If the data amount of the split object is the third data amount, dividing the split object into the first number of second sub-blocks, and the size of the second sub-blocks is the first data amount; The first number of second sub-blocks are checked to obtain a second number of second check blocks, and the first number of second sub-blocks and the second number of second check blocks constitute a second stripe.
6. The method according to claim 1, characterized in that The step of splitting the first object to obtain a first number of first sub-blocks includes: The first object is split based on the first number to obtain a first number of first sub-blocks, and the data amounts of different first sub-blocks in the first number of sub-blocks are the same.
7. The method according to claim 1, characterized in that The method further comprises: Determine target object metadata based on the keyword of the third object; the target object metadata is object metadata of the third object, and the object metadata includes: stripe index, block index, object size and offset; Determine target stripe metadata based on the stripe index in the target object metadata, where the target stripe metadata is the stripe metadata of the target stripe where the third object is located; Based on the block index, object size and offset in the target object metadata, and the target stripe metadata, data of the third object is obtained.
8. The method according to claim 7, characterized in that The obtaining of the data of the third object based on the block index, the object size and the offset in the target object metadata and the target stripe metadata includes: In the case where the third object is accessible, determining target data block metadata based on the block index in the target object metadata and the target stripe metadata, the target data block metadata being the block metadata of the target data block where the third object is located; Determining an address of the target data block in the target data block metadata; Based on the address of the target data block and the object size and offset in the target object metadata, the data of the third object is read.
9. The method according to claim 7, characterized in that: The obtaining of the data of the third object based on the block index, the object size and the offset in the target object metadata and the target stripe metadata includes: In the case that the third object is inaccessible, determining a fourth number of reference block metadata based on the block index in the target object metadata and the target stripe metadata; the reference block metadata is block metadata of a reference block, the reference block is a block in the target stripe that does not include the third object, and the fourth number is the number of reference blocks included in the target stripe; For each reference block metadata of the fourth number of reference block metadata, determining reference data at a target position range in the corresponding reference block based on the reference block metadata; The data of the reference object included in the fourth number of reference blocks is repaired to obtain the data of the third object.
10. The method according to claim 9, characterized in that The step of determining reference data at a target position range in a corresponding reference block based on the reference block metadata includes: Determine a pointer of a candidate object included in the reference block metadata, and determine corresponding candidate object metadata, wherein the candidate object metadata is object metadata of the candidate object, and the candidate object is an object included in the reference block corresponding to the reference block metadata; Determining a position of a corresponding candidate based on the candidate object metadata of each candidate object in the reference block; Based on the target position of the third object in the target data block and the positions of the candidate objects in the reference block, the reference object is selected from the candidate objects included in the reference block, and the reference data includes the data of the reference objects in the reference block.
11. An electronic device, characterized in that: The electronic device comprises: A first splitting unit is configured to split the first object in the cache area to obtain a first number of first sub-blocks, and store the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe; an allocating unit, configured to allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume; The verification unit is configured to verify the first number of first data blocks to obtain a second number of first verification blocks if the first number of first data blocks meet the verification condition, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
12. An electronic device, comprising a processor, characterized in that: The processor is configured to: Splitting a first object in a cache area to obtain a first number of first sub-blocks, and storing the first number of first sub-blocks as the first number of first data blocks respectively; wherein the data volume of the first object is greater than the first data volume, and the first number is the number of data blocks included in one stripe; Allocate a second object in the cache area to the first number of first data blocks, wherein the data volume of the second object is smaller than the first data volume; When the first number of first data blocks meet the verification condition, the first number of first data blocks are verified to obtain a second number of first verification blocks, and the first number of first data blocks and the second number of first verification blocks constitute a first stripe.
13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps in the data processing method according to any one of claims 1 to 10 are implemented.
14. A storage medium storing an executable program, characterized in that: When the executable program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.
15. A chip, comprising a processor, characterized in that: The processor is configured to execute the data processing method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Data recovery method and equipment
CN108170555A
Data storage method and system and storage medium
CN111400083A
Storage method based on self-adaptive storage redundancy strategy
CN111414271A
Method and system for task tracking and allocation
US20100293021A1
System and method for handling data storage on storage devices
US20200026427A1
