Data reading method and device and data storage method and device

Abstract

The invention provides a data reading method and device and a data storage method and device, and relates to the field of data processing, in particular to the technical fields of data reading, data storage and the like. According to the specific implementation scheme, the data reading method comprises the steps that a data reading request is received, and the data reading request comprises the first logic offset and the first data length of data to be read; querying the first data structure according to the first logic offset to obtain a first sorting position of the first logic offset in the allocated and used logic offset; querying the second data structure according to the first sorting position to obtain a first physical offset of the to-be-read data; and performing data reading according to the first physical offset and the first data length.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, specifically to technical fields such as data reading and data storage, and in particular to a data reading method and device and a data storage method and device. Background Art

[0002] Block storage is a storage method that stores and manages data in fixed-size blocks. In block storage systems, indexes play a vital role, recording the mapping relationship between logical addresses and physical addresses. This mapping mechanism is the key to ensuring that data can be accessed efficiently and accurately.

[0003] Since the index volume of block storage is large, it is usually impossible to cache all of it in memory, and only some indexes can be cached through LRU (Least Recently Used). Therefore, it is worth studying how to reduce the amount of index data and cache more indexes in limited memory space to improve cache hit rate and speed up search efficiency. Summary of the invention

[0004] The present disclosure provides a data reading method and device, and a data storage method and device.

[0005] According to a first aspect of the present disclosure, there is provided a data reading method, comprising:

[0006] Receiving a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read;

[0007] Querying a first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use;

[0008] Querying a second data structure according to the first sorting position to obtain a first physical offset of the data to be read;

[0009] Data is read according to the first physical offset and the first data length.

[0010] According to a second aspect of the present disclosure, there is provided a data storage method, comprising:

[0011] In response to a data storage request, writing the data to be stored in the target storage block, wherein the data storage request includes a second logical offset and a second data length of the data to be stored;

[0012] updating the first data structure according to the second logical offset;

[0013] The second data structure is updated according to the second data length and the third sorted position of the second logical offset in the logical offsets that have been allocated for use.

[0014] According to a third aspect of the present disclosure, there is provided a data reading device, comprising:

[0015] A receiving module, configured to receive a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read;

[0016] A first query module, configured to query a first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use;

[0017] A second query module, configured to query a second data structure according to the first sorting position to obtain a first physical offset of the data to be read;

[0018] A reading module is used to read data according to the first physical offset and the first data length.

[0019] According to a fourth aspect of the present disclosure, there is provided a data storage device, comprising:

[0020] A storage module, configured to write the data to be stored in the target storage block in response to a data storage request, wherein the data storage request includes a second logical offset and a second data length of the data to be stored;

[0021] A first updating module, configured to update a first data structure according to the second logical offset;

[0022] The second updating module is used to update the second data structure according to the second data length and the third sorting position of the second logical offset in the logical offsets that have been allocated for use.

[0023] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0024] at least one processor; and

[0025] a memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data reading method proposed in the first aspect of the present disclosure, or execute the data storage method proposed in the second aspect of the present disclosure.

[0027] According to the sixth aspect of the present disclosure, a non-transitory computer-readable storage medium of computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the data reading method proposed in the first aspect of the present disclosure, or to execute the data storage method proposed in the second aspect of the present disclosure.

[0028] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the data reading method proposed in the first aspect of the present disclosure, or implements the data storage method proposed in the second aspect of the present disclosure.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0031] Figure 1 A schematic diagram of the existing index organization scheme;

[0032] Figure 2 A schematic diagram of a flow chart of a data reading method provided in the first embodiment of the present disclosure;

[0033] Figure 3 A flowchart of a data reading method provided in the second embodiment of the present disclosure;

[0034] Figure 4 A schematic diagram of a bitmap storage logic offset provided by an embodiment of the present disclosure;

[0035] Figure 5 A flowchart of a data reading method provided in Embodiment 3 of the present disclosure;

[0036] Figure 6 A schematic diagram of the data length corresponding to the logical offset that has been allocated for use in the array storage provided by the embodiment of the present disclosure;

[0037] Figure 7 A flowchart of a data reading method provided in Embodiment 4 of the present disclosure;

[0038] Figure 8 A flowchart of a data storage method provided in Embodiment 5 of the present disclosure;

[0039] Fig. 9 A flowchart of a data storage method provided in Embodiment 6 of the present disclosure;

[0040] Fig.10 A flowchart of a data storage method provided in Embodiment 7 of the present disclosure;

[0041] Fig.11 This is a structural diagram of a data reading device provided in Embodiment 8 of the present disclosure;

[0042] Fig.12 A schematic diagram of the structure of a data storage device provided in Embodiment 9 of the present disclosure;

[0043] Fig.13 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0044] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] The I / O (Input / Output) operation of block storage is generally divided into two steps: first check the index, and then read the data according to the index. The address space of block storage is generally divided into two layers: logical address space and physical address space. For example, for a 1TB cloud disk, the user sees a 1TB logical address space. The user can read and write data in this 1TB space, and the actual data exists in the physical address space. The index records the mapping relationship between the logical address (the storage address of the data from the user's perspective) and the physical address (the actual storage address of the data, that is, the location where the data is actually stored on the storage medium). Therefore, the organization of the index determines the efficiency of searching the index, which ultimately affects the IO latency performance.

[0046] In the related art, the KV (Key-Value) method is used to record [logical_offset->(physical_offset, length)] (used to describe the mapping relationship between data in the logical address space and the physical address space, that is, the index of the data), where logical_offset represents the logical offset, physical_offset represents the physical offset, and length represents the data length.

[0047] For example, suppose there are 3 data, and the indexes are: [0K->(0K, 8K)], [12K->(8K, 4K)], [16K->(12K, 4K)], then the schematic diagram of index storage is as follows Figure 1As shown, [0K->(0K, 8K)] indicates a data block whose logical offset starts from 0K, whose physical offset also starts from 0K, and whose length is 8K; [12K->(8K, 4K)] indicates a data block whose logical offset starts from 12K, whose physical offset starts from 8K, and whose length is 4K; [16K->(12K, 4K)] indicates a data block whose logical offset starts from 16K, whose physical offset starts from 12K, and whose length is 4K.

[0048] However, this approach has the following disadvantages:

[0049] 1. The index takes up a lot of space. Logical_offset 4B, physical_offset 8B, length 4B, a total of 16B.

[0050] 2. Low search efficiency. If the logical_offset you are looking for does not exist, you need to traverse all KVs that overlap with the range (logical_offset, logical_offset+length), which is called range query. This method is much less efficient than point query.

[0051] Therefore, in order to address at least one of the above problems, the present disclosure provides a data reading method and device and a data storage method and device.

[0052] The following describes the data reading method, device, and data storage method, device according to the embodiments of the present disclosure with reference to the accompanying drawings.

[0053] It should be noted that the execution subject of the data reading method of this embodiment is a data reading device, and the execution subject of the data storage method is a data storage device. The data reading device or the data storage device can be implemented by software and / or hardware, or can be applied to electronic devices. The electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens, and / or display screens.

[0054] Figure 2 This is a flow chart of the data reading method provided in the first embodiment of the present disclosure.

[0055] like Figure 2 As shown, the data reading method may include the following steps S201 to S204:

[0056] Step S201: receiving a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read.

[0057] The data read request is used to request to read the data to be read, and the number of the data to be read requested to be read by the data read request may be one or more.

[0058] The first logical offset is the logical offset of the data to be read, and is used to indicate the position of the data to be read in the logical address space.

[0059] The first data length is the data length of the data to be read, and is used to indicate the number of bytes occupied by the data to be read.

[0060] In the disclosed embodiment, any device can send a data read request to the data reading device when it needs to read data. Accordingly, the data reading device can receive the data read request to locate and read the data based on the first logical offset and the first data length in the data read request.

[0061] Step S202: query the first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use.

[0062] The first data structure is used to store logical offsets. It should be noted that the logical offsets stored in the first data structure at least include the logical offsets that have been allocated for use, that is, the first data structure can store only the logical offsets that have been allocated for use, or can store both the logical offsets that have been allocated for use and the logical offsets that have not been allocated for use, and this is not limited in the present disclosure.

[0063] Among them, the allocated logical offset means that the storage space (or data block) pointed to by the logical offset has been allocated to certain data for use, and the data can be located based on the logical offset when reading the data; the unallocated logical offset means that the storage space (or data block) pointed to by the logical offset has not yet been allocated for use.

[0064] It should be noted that in the present disclosure, the numerical value of the logical offset that has been allocated for use can be directly stored in the first data structure, or different logical offsets can be represented by different positions in the first data structure, and the numerical values ​​at different positions can be used to indicate whether the corresponding logical offset has been allocated for use. This disclosure does not impose any restrictions on this.

[0065] As a possible implementation, when any data is stored, the logical offset of the data can be stored in the first data structure accordingly. Optionally, in the case where the numerical value of the logical offset that has been allocated for use is directly stored in the first data structure, the numerical value of the logical offset of the data can be stored in the first data structure accordingly according to the storage order of the data; in the case where different logical offsets are represented by different positions in the first data structure, and the numerical values ​​of different positions are used to indicate whether the corresponding logical offset has been allocated for use, when any data is stored, the position of the logical offset of the data in the first data structure can be determined first, and then the numerical value of the position can be set to the numerical value used to indicate that the corresponding logical offset has been allocated for use.

[0066] In the embodiment of the present disclosure, one or more allocated logical offsets are stored in the first data structure, so that the first data structure can be queried based on the first logical offset to obtain the first sorted position of the first logical offset among all allocated logical offsets stored in the first data structure.

[0067] Step S203: query the second data structure according to the first sorting position to obtain a first physical offset of the data to be read.

[0068] Wherein, the second data structure is used to store the data length corresponding to the logical offset that has been allocated for use. It should be noted that the data length stored in the second data structure is in a one-to-one correspondence with the logical offset stored in the first data structure. The number of logical offsets stored in the first data structure may be one or more. Correspondingly, the number of data lengths stored in the second data structure may be at least one. Moreover, the sorting position of any allocated logical offset stored in the first data structure among all the allocated logical offsets stored in the first data structure is the same as the sorting position of the data length corresponding to the allocated logical offset stored in the second data structure among all the data lengths stored in the second data structure.

[0069] As a possible implementation, when any data is stored, the data length of the data can be stored in the second data structure accordingly. Optionally, if the logical offset is stored in the first data structure in accordance with the storage order of the data, then when the data length is to be stored, the data length can also be stored in the second data structure in accordance with the storage order of the data; if the logical offset is stored when any data is stored, the position of the logical offset of the data in the first data structure is first determined, and then the value of the position is set to a value indicating that the corresponding logical offset has been allocated for use, then when the data length is to be stored, the position of the data length in the second data structure can be determined according to the sorting position of the logical offset of the data in all the allocated logical offsets stored in the first data structure, and then the value of the data length of the data is stored in the position.

[0070] The first physical offset is the physical offset of the data to be read, and is used to indicate the position of the data to be read in the physical address space.

[0071] In the embodiment of the present disclosure, since the sorting position of any allocated logical offset stored in the first data structure among all allocated logical offsets stored in the first data structure is the same as the sorting position of the data length corresponding to the allocated logical offset stored in the second data structure among all data lengths stored in the second data structure, the second data structure can be queried based on the first sorting position of all allocated logical offsets stored in the first data structure by the first logical offset, and the data length sorted before the first sorting position in the second data structure can be obtained, and then the first physical offset of the data to be read can be determined based on the data length sorted before the first sorting position in the second data structure.

[0072] It should be noted that in the present disclosure, data is stored continuously in the physical address space. Therefore, for any data, the physical offset of the data can be determined according to the sorting position of the data length of the data in all data lengths stored in the second data structure, and the data length stored in the second data structure. For example, assuming that the data length of the first data is 8K and the physical offset is 0K, if the data length of the second data is 4K, the physical offset of the second data is 8K (0K (physical offset of the first data) + 8K (data length of the first data)), if the data length of the third data is 4K, the physical offset of the third data is 12K (0K (physical offset of the first data) + 8K (data length of the first data) + 4K (data length of the second data)), and so on.

[0073] Step S204: reading data according to the first physical offset and the first data length.

[0074] In the disclosed embodiment, data can be read according to the first physical offset and the first data length. Specifically, the position of the data to be read in the physical address space can be determined according to the first physical offset, so as to read data of the first data length from the position.

[0075] The data reading method of the disclosed embodiment adopts a first data structure to store a logical offset, and adopts a second data structure to store a data length corresponding to an allocated logical offset. Thus, when reading data, the first data structure is queried according to the first logical offset of the data to be read in the data reading request, and the sorting position of the first logical offset in all the allocated logical offsets stored in the first data structure is obtained. Then, according to the sorting position, the second data structure is queried to determine the first physical offset of the data to be read, thereby accelerating the search efficiency and reducing the amount of indexed data, so that more indexes can be cached in a limited memory space.

[0076] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, and are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0077] In order to clearly explain how to obtain the first sorting position of the first logical offset in the allocated logical offsets in any embodiment of the present disclosure, the present disclosure further proposes a data reading method.

[0078] Figure 3 This is a flow chart of the data reading method provided in the second embodiment of the present disclosure.

[0079] like Figure 3 As shown, the page display method may include the following steps S301 to S305:

[0080] Step S301: receiving a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read.

[0081] For the explanation of step S301, reference may be made to the relevant description in any embodiment of the present disclosure, and no further details will be given here.

[0082] Step S302: Acquire a first data structure, wherein the first data structure includes first values ​​arranged in sequence, and the first values ​​are used to indicate whether a corresponding logical offset has been allocated for use.

[0083] The first data structure includes one or more first numerical values ​​arranged in sequence, one first numerical value corresponds to one logical offset, and each first numerical value is used to indicate whether the corresponding logical offset has been allocated for use.

[0084] As an example, in block storage, since the size of a data block is usually fixed, such as 4KB, the offset, logical offset, and physical offset of all I / O operations will be integer multiples of the block size, that is, 4K alignment. Therefore, in the first data structure, the first value at the first position may correspond to a logical offset of 0K, the first value at the second position may correspond to a logical offset of 4K, the first value at the third position may correspond to a logical offset of 8K, and so on. Assuming that the first value is 1, it indicates that the corresponding logical offset has been allocated for use, and the first value is 0, it indicates that the corresponding logical offset has not been allocated for use. Figure 1 The indexes of the three data in the first data structure are: [0K->(0K, 8K)], [12K->(8K, 4K)], and [16K->(12K, 4K)], and the first numerical values ​​of the sequential order included in the first data structure can be: 1, 0, 0, 1, 1.

[0085] In order to reduce the storage space occupied by the index, as a possible implementation method, the first data structure can be a bitmap, each bit in the bitmap corresponds to a logical offset (logical_offset), and the value of each bit is used to indicate whether the corresponding logical_offset has been allocated for use.

[0086] As an example, the bitmap stores logical offsets such as Figure 4 As shown, Figure 4 Bitmap is used for storage Figure 1 The logical offset of the three indexes in . It should be noted that Figure 1 The logical_offset value is 0K, 4K, 8K, ..., 20K, a total of 6 values. Correspondingly, there are 6 bits in the bitmap, among which the bit with index 0 corresponds to the logical_offset value of 0K (the index in the bitmap starts from 0), the bit with index 1 corresponds to the logical_offset value of 4K, the bit with index 2 corresponds to the logical_offset value of 8K, and so on. Figure 1The logical offsets of the three indexes are 0K, 12K, and 16K respectively. Therefore, in the bitmap, only the bits with indexes 0, 3, and 4 are set to 1 to indicate that the corresponding logical_offset has been allocated for use, and the remaining bits are set to 0 to indicate that the corresponding logical_offset has not been allocated for use.

[0087] It should be noted that the above only uses the first data structure as a bitmap as an example. In actual application, the first data structure can also be other data structures, and the embodiments of the present disclosure do not limit this.

[0088] Step S303: query the first value in the first data structure according to the first logical offset to obtain the first sorting position of the first logical offset in the logical offsets that have been allocated for use.

[0089] In an embodiment of the present disclosure, after obtaining the first data structure, the first value in the first data structure can be queried based on the first logical offset, thereby obtaining the first sorted position of the first logical offset among all the allocated logical offsets in the first data structure when the first logical offset has been allocated for use.

[0090] In any one of the embodiments of the present disclosure, the position of the first logical offset in the first data structure can be determined based on the first logical offset and the corresponding relationship between the logical offset and the first numerical value in the first data structure. After that, the first numerical value at the position can be queried to determine whether the first logical offset has been allocated for use, and if the first logical offset has been allocated for use, the first numerical value in the first data structure can be queried to determine the first sorting position of the first logical offset in the logical offsets that have been allocated for use.

[0091] Therefore, by directly locating the position of the first logical offset in the first data structure based on the first logical offset and the corresponding relationship between the logical offset and the first numerical value in the first data structure, and querying the numerical value at this position to determine whether the first logical offset has been allocated for use, the query efficiency is greatly improved, especially when processing a large number of logical offsets.

[0092] Step S304: query the second data structure according to the first sorting position to obtain a first physical offset of the data to be read.

[0093] Step S305: reading data according to the first physical offset and the first data length.

[0094] For explanations of steps S304 to S305, reference may be made to the relevant descriptions in any embodiment of the present disclosure, and will not be repeated here.

[0095] The data reading method of the disclosed embodiment adopts the first data structure to store the logical offset, thereby reducing the storage space occupied by the index and realizing rapid checking with extremely low overhead whether any given logical offset has been allocated and used, thereby improving query efficiency.

[0096] In order to clearly explain how to obtain the first physical offset of the data to be read in any embodiment of the present disclosure, the present disclosure also proposes a data reading method.

[0097] Figure 5 This is a flow chart of the data reading method provided in the third embodiment of the present disclosure.

[0098] like Figure 5 As shown, the page display method may include the following steps S501 to S505:

[0099] Step S501: receiving a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read.

[0100] Step S502: query the first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use.

[0101] For explanations of steps S501 to S502, reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0102] Step S503: Acquire a second data structure, wherein the second data structure includes second values ​​arranged in sequence, and the second values ​​are used to indicate the data length corresponding to the logical offset that has been allocated for use.

[0103] The second data structure includes one or more second numerical values ​​arranged in sequence, one second numerical value corresponds to a logical offset that has been allocated for use, and each second numerical value is used to indicate the data length corresponding to the corresponding logical offset that has been allocated for use.

[0104] As an example, for the example in which the first numerical value of the sequential sorting included in the first data structure in step 302 can be: 1, 0, 0, 1, 1, the second numerical value of the sequential sorting included in the second data structure can be: 8K (the logical offset corresponding to 8K that has been allocated for use is the logical offset corresponding to the first numerical value at the first position in the first data structure), 4K (the logical offset corresponding to 4K that has been allocated for use is the logical offset corresponding to the first numerical value at the fourth position in the first data structure), 4K (the logical offset corresponding to 4K that has been allocated for use is the logical offset corresponding to the first numerical value at the fifth position in the first data structure).

[0105] In order to reduce the storage space occupied by the index, as a possible implementation method, the second data structure can be an array, in which one element corresponds to a logical offset that has been allocated for use, and the value of each element is used to indicate the data length corresponding to the corresponding logical offset that has been allocated for use.

[0106] As an example, the array stores the data length corresponding to the logical offset that has been allocated for use, such as Figure 6 As shown, Figure 6 The length array is used to store Figure 1 The data length of the three indexes in . It should be noted that Figure 1 Only three data indexes are stored in it: [0K->(0K, 8K)], [12K->(8K, 4K)], [16K->(12K, 4K)], so there are only three logical offsets allocated for use, namely 0K, 12K and 16K. Correspondingly, only three data lengths [8K, 4K, 4K] are recorded in the length array.

[0107] It should be noted that the above only uses the second data structure as an array as an example. In actual application, the second data structure can also be other data structures, and the embodiments of the present disclosure do not limit this.

[0108] Step S504: According to the first sorting position, query the second data structure for a second value that is sorted before the first sorting position to obtain a first physical offset of the data to be read.

[0109] In the embodiment of the present disclosure, after acquiring the second data structure, the second value sorted before the first sorting position in the second data structure can be queried according to the first sorting position to determine the first physical offset of the data to be read based on the queried second value.

[0110] In any one of the embodiments of the present disclosure, the second data structure can be queried to determine whether there is a second value sorted before the first sort position in the second data structure. Then, if the second value sorted before the first sort position does not exist in the second data structure, the first physical offset of the data to be read is determined to be the target value, or, if the second value sorted before the first sort position exists in the second data structure, the first physical offset of the data to be read is determined based on the target value and the second value sorted before the first sort position in the second data structure.

[0111] The target value is the starting offset of the data continuously stored in the physical address space. For example, the target value may be 0K.

[0112] Therefore, by storing the logical offset and data length of the same index at the same sorting position in the first data structure and the second data structure respectively, the physical offset of the data to be read can be quickly determined based on the first sorting position after determining the first sorting position of the first logical offset among all the allocated and used logical offsets in the first data structure.

[0113] In any embodiment of the present disclosure, when there is a second numerical value sorted before the first sorting position in the second data structure, the sum of the target value and the second numerical value sorted before the first sorting position in the second data structure can be determined as the first physical offset of the data to be read.

[0114] Therefore, the target value and the data length of at least one data stored before the first sorting position can be used as "landmarks" to quickly locate the first physical offset, thereby improving data access efficiency.

[0115] Step S505: reading data according to the first physical offset and the first data length.

[0116] For the explanation of step S505, reference may be made to the relevant description in any embodiment of the present disclosure, and no further details will be given here.

[0117] The data reading method of the disclosed embodiment adopts a second data structure to store the data length corresponding to the allocated logical offset, thereby reducing the storage space occupied by the index and quickly determining the physical offset of the data to be read with extremely low overhead, thereby improving query efficiency.

[0118] It should be noted that, in the present disclosure, the first logic offset may not be allocated for use. Figure 7 , a detailed description is given of the process of data reading when the first logical offset is not allocated for use.

[0119] Figure 7 This is a flow chart of the data reading method provided in the fourth embodiment of the present disclosure.

[0120] like Figure 7 As shown, the data reading method may include the following steps S701 to S709:

[0121] Step S701: receiving a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read.

[0122] Step S702: Acquire a first data structure, wherein the first data structure includes first values ​​arranged in sequence, and the first values ​​are used to indicate whether a corresponding logical offset has been allocated for use.

[0123] Step S703: Determine the position of the first logical offset in the first data structure according to the first logical offset and the corresponding relationship between the logical offset and the first value in the first data structure.

[0124] Step S704: query the first value at the position to determine whether the first logical offset has been allocated for use.

[0125] For explanations of steps S701 to S704, reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0126] Step S705 , when the first logical offset has not been allocated for use, a candidate logical offset corresponding to the first logical offset is determined from the logical offsets that have been allocated for use according to the first value in the first data structure.

[0127] The candidate logical offset corresponding to the first logical offset refers to the allocated logical offset that overlaps with the range of (first logical offset, first logical offset+first data length). And the number of candidate logical offsets corresponding to the first logical offset may be one or more.

[0128] In the embodiment of the present disclosure, if the first logical offset has not been allocated for use, a range query is required. Optionally, a candidate logical offset corresponding to the first logical offset may be determined from the logical offsets that have been allocated for use based on the first value in the first data structure.

[0129] As an example, for Figure 1 The indexes of the three data in are: [0K->(0K, 8K)], [12K->(8K, 4K)], [16K->(12K, 4K)], assuming that the following is used Figure 4 The bitmap shown stores the logical offsets of these three indexes, using Figure 6The length array shown stores the data lengths of the three indexes. If you want to read data with a logical_offset of 4K and a data length of 4K, you can determine that the index of the logical_offset of the data to be read in the bitmap is 1 (4K / 4K=1) based on the logical_offset of 4K. Then query the bit with bitmap index 1. Since the value of the bit with bitmap index 1 is 0, it means that the logical_offset at this position has not been allocated for use. At this time, a range query is required: query the bits corresponding to the logical offsets (logical offsets greater than 4K and less than 8K) that may cover the range of (4K, 8K (4K (logical_offset) + 4K (data length))). Thus, you can query the first bit that is 1 to the left, find the bit with index 0, and determine the logical offset (0K) corresponding to the bit with index 0 as the candidate logical offset.

[0130] It should be noted that the reason why the first 1 bit is not queried to the right is that the logical offset corresponding to the first bit queried to the right is 8K, which will not overwrite the data to be read. If the data to be read is data with a logical_offset of 4K and a data length of 8K, and the value of the bit with a bitmap index of 1 is 0, when performing a range query, it is necessary to query the first 1 bit to the left and right, where the first 1 bit queried to the right should be less than 12K (4K (logical_offset) + 8K (data length)), and there is no value restriction for the first 1 bit queried to the left. This is because even if the logical offset corresponding to the first 1 bit queried to the left is not within the range, as long as the data stored based on the logical offset and the data length corresponding to the logical offset can overwrite the data to be read, the physical_offset of the data to be read can be determined.

[0131] Therefore, a point query method can be used to replace the range query in the prior art, which has higher query efficiency.

[0132] Step S706, query the second data structure according to the second sorting position of the candidate logical offset in the allocated logical offsets to obtain the candidate physical offset, wherein the second data structure includes sequentially arranged second values, and the second values ​​are used to indicate the data length corresponding to the allocated logical offset.

[0133] In the embodiment of the present disclosure, after the candidate logical offset corresponding to the first logical offset is determined, the second data structure may be queried according to the second sorting position of the candidate logical offset in the allocated logical offsets to determine the candidate physical offset.

[0134] In any embodiment of the present disclosure, the second numerical value sorted before the second sorting position in the second data structure may be queried according to the second sorting position to obtain the candidate physical offset. That is, the second numerical value sorted before the second sorting position in the second data structure may be queried according to the second sorting position of the candidate logical offset in the allocated logical offsets to determine the candidate physical offset based on the queried second numerical value.

[0135] In any one of the embodiments of the present disclosure, the second data structure can be queried based on the second sorting position of the candidate logical offset in the allocated logical offsets to determine whether there is a second value sorted before the second sorting position in the second data structure. Then, if the second value sorted before the second sorting position does not exist in the second data structure, the candidate physical offset is determined to be the target value, or, if the second value sorted before the second sorting position exists in the second data structure, the candidate physical offset is determined based on the target value and the second value sorted before the second sorting position in the second data structure.

[0136] The target value is the starting offset of the data continuously stored in the physical address space. For example, the target value may be 0K.

[0137] Therefore, by storing the logical offset and data length of the same index at the same sorting position in the first data structure and the second data structure respectively, the candidate physical offset can be quickly determined based on the second sorting position when the second sorting position of the candidate logical offset among all allocated and used logical offsets in the first data structure is determined.

[0138] In any embodiment of the present disclosure, when there is a second value sorted before the second sorting position in the second data structure, the sum of the target value and the second value sorted before the second sorting position in the second data structure may be determined as the candidate physical offset.

[0139] Therefore, the target value and the data length of at least one data stored before the second sorting position can be used as "landmarks" to quickly locate the candidate physical offset, thereby improving data access efficiency.

[0140] Step S707, determining whether the candidate data stored based on the candidate logical offset and the data length corresponding to the candidate logical offset contains the data to be read according to the data length corresponding to the candidate physical offset and the candidate logical offset, and the first logical offset and the first data length of the data to be read.

[0141] In the embodiment of the present disclosure, the second data structure is queried according to the second sorted position of the candidate logical offset in the logical offsets that have been allocated for use. In addition to obtaining the candidate physical offset, the data length corresponding to the candidate logical offset can also be obtained. Therefore, based on the candidate physical offset and the data length corresponding to the candidate logical offset, and the first logical offset and the first data length of the data to be read, it can be determined whether the candidate data stored based on the candidate logical offset and the data length corresponding to the candidate logical offset contains the data to be read.

[0142] As an example, for the example of data to be read in step S705 with a logical_offset of 4K and a data length of 4K, after the logical offset (0K) corresponding to the bit with an index of 0 is determined as the candidate logical offset, it can be determined that the sorting position of 0K in all the allocated logical offsets in the bitmap is 1. Then, based on position 1, the length array is queried to determine that the candidate physical offset is 0K (in this example, the starting offset of the data continuously stored in the physical address space is 0K, that is, the target value is 0K), and the data length corresponding to the candidate logical offset is 8K (the first value in the length array is 8K), which just covers the data to be read (data with a logical_offset of 4K and a data length of 4K). Therefore, it can be determined that the candidate data stored based on the candidate logical offset and the data length corresponding to the candidate logical offset contains the data to be read.

[0143] Step S708: When the candidate data includes the data to be read, determine the first physical offset of the data to be read according to the candidate physical offset and the first logical offset.

[0144] In any embodiment of the present disclosure, when the candidate data includes the data to be read, the sum of the candidate physical offset and the first logical offset may be determined as the first physical offset of the data to be read.

[0145] As an example, for the above example of data to be read with a logical_offset of 4K and a data length of 4K, it can be determined that the first physical offset of the data to be read is 0K (candidate physical offset) + 4K (first logical offset of the data to be read) = 4K.

[0146] Step S709: reading data according to the first physical offset and the first data length.

[0147] For the explanation of step S709, reference may be made to the relevant description in any embodiment of the present disclosure, and no further details will be given here.

[0148] The data reading method of the disclosed embodiment determines a candidate logical offset corresponding to the first logical offset from the logical offsets that have been allocated when the first logical offset has not been allocated, and determines the first physical offset of the data to be read based on the candidate logical offset and the data length of the candidate logical offset. This method can replace the range query in the prior art with a point query, maximize the cache hit rate of the index, and accelerate the efficiency of index search.

[0149] In any embodiment of the present disclosure, combined with the I / O characteristics of block storage - in block storage, since the size of data blocks is usually fixed, such as 4KB, the offset, logical offset and physical offset of all I / O operations will be integer multiples of the block size, that is, 4K alignment, an efficient index organization scheme is proposed - a bitmap is used to represent all logical_offsets, valid (allocated and used) is 1, and invalid (not allocated and used) is 0. A length array is used to represent the data length corresponding to the valid logical_offset.

[0150] by Figure 1 , Figure 4 and Figure 6 For example, suppose there are 3 data, and the indexes are: [0K->(0K, 8K)], [12K->(8K, 4K)], [16K->(12K, 4K)], press Figure 1 The index organization scheme shown records 3 indexes and occupies the following space:

[0151] 16B (logical_offset 4B, physical_offset 8B, length 4B, total 16B)*3=48B

[0152] According to the index organization scheme provided by the embodiment of the present disclosure, only Figure 4 As shown, use 3 bits to record the effective logical_offset, and then Figure 6 As shown, the length array is used to record three data lengths [8K, 4K, 4K]. The physical_offset is calculated through the length array, for example, the second physical_offset = 8K, the third physical_offset = 8K + 4K = 12K. At this time, three indexes are recorded, and the space occupied is:

[0153] 1B (bitmap) + 4B (1 data length 4B) * 3 = 13B

[0154] The storage space saved is:

[0155] 1-13B / 48B=73%

[0156] In actual online situations, the logical address space of a block is 64MB, so the value of logical_offset is any 4K-aligned value between 0 and 64MB. The space occupied by the bitmap is as follows:

[0157] 64MB / 4K / 8=2KB

[0158] Use a length array to represent the data length corresponding to the valid logical_offset. The maximum space occupied by the length array is as follows:

[0159] 64MB / 4K*4B=64KB

[0160] Total occupancy: 2KB+64KB=66KB

[0161] according to Figure 1 The index organization scheme shown here takes up:

[0162] 64MB / 4KB*16B=256KB

[0163] The storage space saved is:

[0164] 1-66KB / 256KB=74%

[0165] Let's look at index search. For example, if you want to read data with a logical_offset of 4K and a length of 4K, first check the bitmap index of 1 and find that it is 0 (indexes start at 0), indicating that the logical_offset at that position does not exist. Then search the left for the first bit that is 1 and find the bit with index 0. Based on the index of 0, find the first value of the length array, which is 8K, which just covers the data to be read. You can get physical_offset of 0+4K=4K and length=4K.

[0166] In summary, the following two points ensure that the index organization scheme provided by the embodiment of the present disclosure is a more efficient index organization scheme:

[0167] First, the index space is saved by 74%. The same memory space can cache 3.8 times more index entries than before, and the cache hit rate will also be 3.8 times higher than before.

[0168] Second: Since the bitmap occupies a small space and can reside in memory, the logical_offset to be searched can be locked according to the bitmap, and the point query method can be used instead of the range query, which is more efficient.

[0169] The above embodiment describes a data reading process. In order to clearly illustrate the above embodiment, the present disclosure also proposes a data storage method.

[0170] Figure 8 This is a flowchart of the data storage method provided in Embodiment 5 of the present disclosure.

[0171] like Figure 8 As shown, the data storage method may include the following steps S801 to S803:

[0172] Step S801 : writing data to be stored in a target storage block in response to a data storage request, wherein the data storage request includes a second logical offset and a second data length of the data to be stored.

[0173] The data storage request is used to request the storage of data to be stored, and the number of data to be stored requested to be stored by the data storage request may be one or more.

[0174] The second logical offset is the logical offset of the data to be stored, and is used to indicate the position of the data to be stored in the logical address space.

[0175] The second data length is the data length of the data to be stored, and is used to indicate the number of bytes occupied by the data to be stored.

[0176] In the embodiment of the present disclosure, any device may send a data storage request to the data storage device when data needs to be stored. Accordingly, the data storage device may receive the data storage request, determine the target storage block based on the second logical offset and the second data length in the data storage request, and write the data to be stored in the target storage block.

[0177] Step S802: Update the first data structure according to the second logical offset.

[0178] The explanation of the first data structure can refer to the relevant description in step S202, which will not be repeated here.

[0179] In the embodiments of the present disclosure, in the case where the numerical value of the logical offset that has been allocated for use is directly stored in the first data structure, the numerical value of the second logical offset can be stored in the first data structure in accordance with the storage order of the data to update the first data structure; in the case where different logical offsets are represented by different positions in the first data structure, and the numerical values ​​at different positions are used to indicate whether the corresponding logical offset has been allocated for use, the position of the second logical offset in the first data structure can be determined first, and then the numerical value of the position can be set to a numerical value used to indicate that the corresponding logical offset has been allocated for use to update the first data structure.

[0180] Step S803: Update the second data structure according to the second data length and the first sorting position of the second logical offset in the logical offsets that have been allocated for use.

[0181] The explanation of the second data structure can refer to the relevant description in step S203, which will not be repeated here.

[0182] In the embodiment of the present disclosure, if the second data structure is updated in accordance with the storage order of the data, the numerical value of the logical offset of the data is stored in the second data structure accordingly, then when the second data structure is updated, the numerical value of the second data length can also be stored in the second data structure in accordance with the storage order of the data. In this way, it can also be ensured that the sorting position of any allocated and used logical offset stored in the first data structure among all allocated and used logical offsets stored in the first data structure is the same as the sorting position of the data length corresponding to the allocated and used logical offset stored in the second data structure among all data lengths stored in the second data structure. If the second data structure is updated, When the second data structure is to be updated, the position of the second logical offset in the first data structure is first determined, and then the value of the position is set to a value used to indicate that the corresponding logical offset has been allocated for use. When the second data structure is updated, the position of the second data length in the second data structure can be determined based on the sorting position of the second logical offset in all the allocated logical offsets stored in the first data structure (the sorting position of the second logical offset in all the allocated logical offsets stored in the first data structure is the same as the sorting position of the second data length in all the data lengths stored in the second data structure), and then the value of the second data length is stored in this position.

[0183] The data storage method of the disclosed embodiment adopts a first data structure to store the logical offset and adopts a second data structure to store the data length corresponding to the logical offset that has been allocated for use, thereby greatly reducing the storage space occupied by the index, maximizing the cache hit rate of the index, and accelerating the efficiency of index search.

[0184] In order to clearly explain how to update the first data structure in any embodiment of the present disclosure, the present disclosure also proposes a data storage method.

[0185] Fig. 9 This is a flowchart of the data storage method provided in Embodiment 6 of the present disclosure.

[0186] like Fig. 9 As shown, the data storage method may include the following steps S901 to S904:

[0187] Step S901 : writing the data to be stored in the target storage block in response to a data storage request, wherein the data storage request includes a second logical offset and a second data length of the data to be stored.

[0188] Step S902: Acquire a first data structure, wherein the first data structure includes first values ​​arranged in sequence, and the first values ​​are used to indicate whether a corresponding logical offset has been allocated for use.

[0189] For explanations of steps S901 to S902, reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0190] Step S903: Update the first numerical value corresponding to the second logical offset in the first data structure to a first set value.

[0191] The first setting value is used to indicate that the corresponding logic offset has been allocated for use.

[0192] Step S904: update the second data structure according to the second data length and the first sorting position of the second logical offset in the logical offsets that have been allocated for use.

[0193] For the explanation of step S904, reference may be made to the relevant description in any embodiment of the present disclosure, and will not be repeated here.

[0194] The data storage method of the disclosed embodiment can realize the use of the first data structure to store the second logical offset of the data to be stored by updating the first numerical value corresponding to the second logical offset in the first data structure to a first set value used to indicate that the corresponding logical offset has been allocated for use. Since the first data structure includes one or more first numerical values ​​arranged in sequence, more logical offset information can be cached in a limited memory space, thereby reducing the amount of indexed data.

[0195] In order to clearly explain how to update the second data structure in any embodiment of the present disclosure, the present disclosure also proposes a data storage method.

[0196] Fig.10 This is a flow chart of the data storage method provided in Embodiment 7 of the present disclosure.

[0197] like Fig.10 As shown, the data storage method may include the following steps S1001 to S1004:

[0198] Step S1001 , in response to a data storage request, writing data to be stored in a target storage block, wherein the data storage request includes a second logical offset and a second data length of the data to be stored.

[0199] Step S1002: Update the first data structure according to the second logical offset.

[0200] Step S1003: Acquire a second data structure, wherein the second data structure includes second numerical values ​​arranged in sequence, and the second numerical values ​​are used to indicate the data length corresponding to the logical offset that has been allocated for use.

[0201] For explanations of steps S1001 to S1003 , reference may be made to the relevant descriptions in any embodiment of the present disclosure and will not be repeated here.

[0202] Step S1004: according to the third sorting position of the second logical offset in the allocated logical offsets, update the second value of the third sorting position in the second data structure to the second data length.

[0203] The data storage method of the disclosed embodiment can store the logical offset and data length of the same index at the same sorting position in the first data structure and the second data structure respectively by updating the second value of the third sorting position in the second data structure to the second data length. Therefore, when the sorting position of the logical offset of the data to be read among all the allocated and used logical offsets in the first data structure is determined, the physical offset of the data to be read can be quickly determined based on the sorting position, thereby improving query efficiency.

[0204] With the above Figures 2 to 7 Corresponding to the data reading method provided in the embodiment, the present disclosure also provides a data reading device. Figures 2 to 7 The data reading method provided in the embodiment corresponds to the embodiment, so the implementation of the data reading method is also applicable to the data reading device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0205] Fig.11 This is a structural diagram of the data reading device provided in Embodiment 8 of the present disclosure.

[0206] like Fig.11As shown, the data reading device 1100 may include: a receiving module 1110 , a first query module 1120 , a second query module 1130 and a reading module 1140 .

[0207] The receiving module 1110 is used to receive a data read request, wherein the data read request includes a first logical offset and a first data length of the data to be read;

[0208] A first query module 1120, configured to query the first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use;

[0209] A second query module 1130 is used to query the second data structure according to the first sorting position to obtain a first physical offset of the data to be read;

[0210] The reading module 1140 is used to read data according to the first physical offset and the first data length.

[0211] In a possible implementation of the embodiment of the present disclosure, the first query module 1120 includes: a first acquisition unit, used to acquire a first data structure, wherein the first data structure includes first numerical values ​​arranged in sequence, and the first numerical values ​​are used to indicate whether the corresponding logical offset has been allocated for use; a first query unit, used to query the first numerical value in the first data structure according to the first logical offset, and obtain the first sorting position of the first logical offset in the logical offsets that have been allocated for use.

[0212] In a possible implementation of the embodiment of the present disclosure, the first query unit is further used to: determine the position of the first logical offset in the first data structure based on the first logical offset and the corresponding relationship between the logical offset and the first numerical value in the first data structure; query the first numerical value at the position to determine whether the first logical offset has been allocated for use; and if the first logical offset has been allocated for use, query the first numerical value in the first data structure to determine the first sorting position of the first logical offset in the logical offsets that have been allocated for use.

[0213] In a possible implementation of the embodiment of the present disclosure, the second query module 1130 includes: a second acquisition unit, used to acquire a second data structure, wherein the second data structure includes sequentially arranged second numerical values, and the second numerical values ​​are used to indicate the data length corresponding to the logical offset that has been allocated for use; a second query unit, used to query the second numerical value sorted before the first sorting position in the second data structure according to the first sorting position, to obtain the first physical offset of the data to be read.

[0214] In a possible implementation of the embodiment of the present disclosure, the second query unit is further used to: query the second data structure to determine whether there is a second value sorted before the first sorting position in the second data structure; if there is no second value sorted before the first sorting position in the second data structure, determine the physical offset corresponding to the data to be read as the target value; if there is a second value sorted before the first sorting position in the second data structure, determine the first physical offset of the data to be read according to the target value and the second value sorted before the first sorting position in the second data structure.

[0215] In a possible implementation of the embodiment of the present disclosure, the second query unit is further used to: when there is a second numerical value sorted before the first sorting position in the second data structure, determine the sum of the target value and the second numerical value sorted before the first sorting position in the second data structure as the first physical offset of the data to be read.

[0216] In a possible implementation of the embodiment of the present disclosure, the data reading device 1100 further includes:

[0217] A first determining module, configured to determine, when the first logical offset has not been allocated for use, a candidate logical offset corresponding to the first logical offset from the logical offsets that have been allocated for use according to a first value in the first data structure;

[0218] A third query module is used to query the second data structure according to the second sorting position of the candidate logical offset in the allocated logical offsets to obtain the candidate physical offset, wherein the second data structure includes sequentially arranged second values, and the second value is used to indicate the data length corresponding to the allocated logical offset;

[0219] A second determination module, configured to determine whether the candidate data stored based on the candidate logical offset and the data length corresponding to the candidate logical offset includes the data to be read, according to the data length corresponding to the candidate physical offset and the candidate logical offset, and the first logical offset and the first data length of the data to be read;

[0220] The third determination module is used to determine the first physical offset of the data to be read according to the candidate physical offset and the first logical offset when the candidate data includes the data to be read.

[0221] In a possible implementation of the embodiment of the present disclosure, the third query module is further used to: query the second data structure according to the second sorting position of the candidate logical offset in the logical offsets that have been allocated for use, so as to determine whether there is a second numerical value sorted before the second sorting position in the second data structure; in the case where there is a second numerical value sorted before the second sorting position in the second data structure, determine the candidate physical offset according to the second numerical value sorted before the second sorting position in the second data structure; in the case where there is no second numerical value sorted before the second sorting position in the second data structure, determine the candidate physical offset as the target value.

[0222] The data reading device of the disclosed embodiment adopts a first data structure to store logical offsets, and adopts a second data structure to store data lengths corresponding to the allocated logical offsets. Thus, when reading data, the first data structure is queried according to the first logical offset of the data to be read in the data reading request, and the sorting position of the first logical offset in all the allocated logical offsets stored in the first data structure is obtained. Then, according to the sorting position, the second data structure is queried to determine the first physical offset of the data to be read, thereby accelerating the search efficiency and reducing the amount of indexed data, so that more indexes can be cached in a limited memory space.

[0223] With the above Figures 8 to 10 Corresponding to the data storage method provided in the embodiment, the present disclosure also provides a data storage device. Figures 8 to 10 The data storage method provided in the embodiment corresponds to the data storage method, so the implementation of the data storage method is also applicable to the data storage device provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0224] Fig.12 This is a schematic diagram of the structure of the data storage device provided in the ninth embodiment of the present disclosure.

[0225] like Fig.12 As shown, the data storage device 1200 may include: a storage module 1210 , a first update module 1220 , and a second update module 1230 .

[0226] The storage module 1210 is used to write the data to be stored in the target storage block in response to the data storage request, wherein the data storage request includes the second logical offset and the second data length of the data to be stored;

[0227] A first updating module 1220, configured to update the first data structure according to the second logical offset;

[0228] The second updating module 1230 is used to update the second data structure according to the second data length and the third sorting position of the second logical offset in the allocated logical offsets.

[0229] In a possible implementation of the embodiment of the present disclosure, the first update module 1220 is also used to: obtain a first data structure, wherein the first data structure includes first numerical values ​​arranged in sequence, and the first numerical values ​​are used to indicate whether the corresponding logical offset has been allocated for use; and update the first numerical value corresponding to the second logical offset in the first data structure to a first set value.

[0230] In a possible implementation of the embodiment of the present disclosure, the second update module 1230 is also used to: obtain a second data structure, wherein the second data structure includes second numerical values ​​arranged in sequence, and the second numerical value is used to indicate the data length corresponding to the logical offset that has been allocated for use; according to the third sorting position of the second logical offset in the logical offsets that have been allocated for use, update the second numerical value at the third sorting position in the second data structure to the second data length.

[0231] The data storage device of the disclosed embodiment adopts a first data structure to store the logical offset and adopts a second data structure to store the data length corresponding to the logical offset that has been allocated for use, thereby greatly reducing the storage space occupied by the index, maximizing the cache hit rate of the index, and accelerating the efficiency of index search.

[0232] In order to implement the above embodiments, the present disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data reading method proposed in any of the above embodiments of the present disclosure, or execute the data storage method proposed in any of the above embodiments of the present disclosure.

[0233] In order to implement the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data reading method proposed in any of the above embodiments of the present disclosure, or to execute the data storage method proposed in any of the above embodiments of the present disclosure.

[0234] In order to implement the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the data reading method proposed in any of the above embodiments of the present disclosure, or implements the data storage method proposed in any of the above embodiments of the present disclosure.

[0235] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0236] Fig.13 A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present disclosure is shown. Among them, the electronic device may include the server and client in the above-mentioned embodiment. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0237] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1302 or a computer program loaded from a storage unit 1307 to a RAM (Random Access Memory) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An I / O (Input / Output) interface 1305 is also connected to the bus 1304.

[0238] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0239] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, CPU (Central Processing Unit), GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the above-mentioned data reading method or the above-mentioned data storage method. For example, in some embodiments, the above-mentioned data reading method or the above-mentioned data storage method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the data reading method or the data storage method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to perform the above data reading method or the above data storage method in any other appropriate manner (e.g., by means of firmware).

[0240] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0241] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0242] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0243] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0244] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0245] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0246] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0247] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools, including servers, operating systems, networks, software, applications, and storage devices, and can deploy and manage resources in an on-demand, self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as model training.

[0248] According to the technical solution of the embodiment of the present disclosure, a first data structure is used to store logical offsets, and a second data structure is used to store data lengths corresponding to allocated logical offsets. Thus, when data is read, the first data structure is queried according to the first logical offset of the data to be read in the data read request, and the sorting position of the first logical offset in all allocated logical offsets stored in the first data structure is obtained. Then, according to the sorting position, the second data structure is queried to determine the first physical offset of the data to be read, thereby accelerating the search efficiency and reducing the amount of indexed data, so that more indexes can be cached in a limited memory space.

[0249] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0250] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data reading method, comprising: Receiving a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read; Querying a first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use; Querying a second data structure according to the first sorting position to obtain a first physical offset of the data to be read; Data is read according to the first physical offset and the first data length.

2. The data reading method according to claim 1, wherein: The step of querying the first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use includes: Acquire a first data structure, wherein the first data structure includes first values ​​arranged in sequence, and the first values ​​are used to indicate whether a corresponding logical offset has been allocated for use; According to the first logical offset, a first value in the first data structure is queried to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use.

3. The data reading method according to claim 2, wherein: The step of querying the first value in the first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use includes: Determine the position of the first logical offset in the first data structure according to the first logical offset and the corresponding relationship between the logical offset and the first value in the first data structure; querying a first value at the position to determine whether the first logical offset has been allocated for use; In the case where the first logical offset has been allocated for use, a first value in the first data structure is queried to determine a first sorting position of the first logical offset in the logical offsets that have been allocated for use.

4. The data reading method according to claim 1, wherein: The querying the second data structure according to the first sorting position to obtain the first physical offset of the data to be read includes: Acquire a second data structure, wherein the second data structure includes second values ​​arranged in sequence, and the second values ​​are used to indicate the data length corresponding to the logical offset that has been allocated for use; According to the first sorting position, a second value sorted before the first sorting position in the second data structure is queried to obtain a first physical offset of the data to be read.

5. The data reading method according to claim 4, wherein: The step of querying, according to the first sorting position, a second value sorted before the first sorting position in the second data structure to obtain a first physical offset of the data to be read includes: querying the second data structure to determine whether there is a second value sorted before the first sort position in the second data structure; If there is no second value sorted before the first sorting position in the second data structure, determining the first physical offset of the to-be-read data as a target value; In a case where there is a second value sorted before the first sorting position in the second data structure, a first physical offset of the to-be-read data is determined according to the target value and the second value sorted before the first sorting position in the second data structure.

6. The data reading method according to claim 5, wherein: In a case where there is a second value sorted before the first sorting position in the second data structure, determining the first physical offset of the to-be-read data according to the target value and the second value sorted before the first sorting position in the second data structure includes: When there is a second value sorted before the first sorting position in the second data structure, the sum of the target value and the second value sorted before the first sorting position in the second data structure is determined as the first physical offset of the data to be read.

7. The data reading method according to claim 3, wherein: The method further comprises: In a case where the first logical offset has not been allocated for use, determining a candidate logical offset corresponding to the first logical offset from the logical offsets that have been allocated for use according to the first value in the first data structure; Querying a second data structure according to a second sorting position of the candidate logical offset in the allocated logical offsets to obtain a candidate physical offset, wherein the second data structure includes second values ​​arranged in sequence, and the second values ​​are used to indicate a data length corresponding to the allocated logical offset; Determine, according to the candidate physical offset and the data length corresponding to the candidate logical offset, and the first logical offset and the first data length of the data to be read, whether the candidate data stored based on the candidate logical offset and the data length corresponding to the candidate logical offset includes the data to be read; In a case where the candidate data includes the data to be read, a first physical offset of the data to be read is determined according to the candidate physical offset and the first logical offset.

8. The data reading method according to claim 7, wherein: The step of querying the second data structure according to the second sort position of the candidate logical offset in the allocated logical offsets to obtain the candidate physical offset includes: querying the second data structure according to a second sorting position of the candidate logical offset in the logical offsets that have been allocated for use, to determine whether there is a second value sorted before the second sorting position in the second data structure; In a case where there is a second value sorted before the second sorting position in the second data structure, determining the candidate physical offset according to the second value sorted before the second sorting position in the second data structure; In a case where there is no second value sorted before the second sort position in the second data structure, the candidate physical offset is determined to be a target value.

9. A data storage method, comprising: In response to a data storage request, writing the data to be stored in the target storage block, wherein the data storage request includes a second logical offset and a second data length of the data to be stored; updating the first data structure according to the second logical offset; The second data structure is updated according to the second data length and the third sorted position of the second logical offset in the logical offsets that have been allocated for use.

10. The data storage method according to claim 9, wherein: The updating of the first data structure according to the second logical offset includes: Acquire a first data structure, wherein the first data structure includes first values ​​arranged in sequence, and the first values ​​are used to indicate whether a corresponding logical offset has been allocated for use; The first numerical value corresponding to the second logical offset in the first data structure is updated to a first set value.

11. The data storage method according to claim 9, wherein: The updating of the second data structure according to the second data length and the third sorting position of the second logical offset in the logical offsets that have been allocated for use comprises: Acquire a second data structure, wherein the second data structure includes second values ​​arranged in sequence, and the second values ​​are used to indicate the data length corresponding to the logical offset that has been allocated for use; According to the third sorting position of the second logical offset in the logical offsets that have been allocated for use, the second value of the third sorting position in the second data structure is updated to the second data length.

12. A data reading device, comprising: A receiving module, configured to receive a data read request, wherein the data read request includes a first logical offset and a first data length of data to be read; A first query module, configured to query a first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use; A second query module, configured to query a second data structure according to the first sorting position to obtain a first physical offset of the data to be read; A reading module is used to read data according to the first physical offset and the first data length.

13. The data reading device according to claim 12, wherein: The first query module includes: A first acquisition unit, configured to acquire a first data structure, wherein the first data structure includes first values ​​arranged in sequence, and the first values ​​are used to indicate whether a corresponding logical offset has been allocated for use; The first query unit is used to query a first value in the first data structure according to the first logical offset to obtain a first sorting position of the first logical offset in the logical offsets that have been allocated for use.

14. The data reading device according to claim 13, wherein: The first query unit is further configured to: Determine the position of the first logical offset in the first data structure according to the first logical offset and the corresponding relationship between the logical offset and the first value in the first data structure; querying a first value at the position to determine whether the first logical offset has been allocated for use; In the case where the first logical offset has been allocated for use, a first value in the first data structure is queried to determine a first sorting position of the first logical offset in the logical offsets that have been allocated for use.

15. The data reading device according to claim 12, wherein: The second query module includes: A second acquisition unit, configured to acquire a second data structure, wherein the second data structure includes second values ​​arranged in sequence, and the second values ​​are used to indicate a data length corresponding to a logical offset that has been allocated for use; The second query unit is used to query the second value sorted before the first sorting position in the second data structure according to the first sorting position, so as to obtain a first physical offset of the data to be read.

16. The data reading device according to claim 15, wherein: The second query unit is further used for: querying the second data structure to determine whether there is a second value sorted before the first sort position in the second data structure; If there is no second value sorted before the first sorting position in the second data structure, determining the physical offset corresponding to the to-be-read data as a target value; In a case where there is a second value sorted before the first sorting position in the second data structure, a first physical offset of the to-be-read data is determined according to the target value and the second value sorted before the first sorting position in the second data structure.

17. The data reading device according to claim 16, wherein: The second query unit is further used for: When there is a second value sorted before the first sorting position in the second data structure, the sum of the target value and the second value sorted before the first sorting position in the second data structure is determined as the first physical offset of the data to be read.

18. The data reading device according to claim 14, wherein: The device further comprises: A first determining module, configured to determine, when the first logical offset has not been allocated for use, a candidate logical offset corresponding to the first logical offset from the logical offsets that have been allocated for use according to a first value in the first data structure; A third query module is used to query a second data structure according to a second sorting position of the candidate logical offset in the allocated logical offsets to obtain a candidate physical offset, wherein the second data structure includes second values ​​arranged in sequence, and the second values ​​are used to indicate a data length corresponding to the allocated logical offset; A second determination module, configured to determine whether the candidate data stored based on the candidate logical offset and the data length corresponding to the candidate logical offset includes the data to be read, according to the data length corresponding to the candidate physical offset and the candidate logical offset, and the first logical offset and the first data length of the data to be read; A third determining module is configured to determine a first physical offset of the data to be read according to the candidate physical offset and the first logical offset when the candidate data includes the data to be read.

19. The data reading device according to claim 18, wherein: The third query module is also used for: querying the second data structure according to a second sorting position of the candidate logical offset in the logical offsets that have been allocated for use, to determine whether there is a second value sorted before the second sorting position in the second data structure; In a case where there is a second value sorted before the second sorting position in the second data structure, determining the candidate physical offset according to the second value sorted before the second sorting position in the second data structure; In a case where there is no second value sorted before the second sort position in the second data structure, the candidate physical offset is determined to be a target value.

20. A data storage device, comprising: A storage module, configured to write the data to be stored in the target storage block in response to a data storage request, wherein the data storage request includes a second logical offset and a second data length of the data to be stored; A first updating module, configured to update a first data structure according to the second logical offset; The second updating module is used to update the second data structure according to the second data length and the third sorting position of the second logical offset in the logical offsets that have been allocated for use.

21. The data storage device according to claim 20, wherein: The first update module is further used for: Acquire a first data structure, wherein the first data structure includes first values ​​arranged in sequence, and the first values ​​are used to indicate whether a corresponding logical offset has been allocated for use; The first numerical value corresponding to the second logical offset in the first data structure is updated to a first set value.

22. The data storage device according to claim 20, wherein: The second update module is further used for: Acquire a second data structure, wherein the second data structure includes second values ​​arranged in sequence, and the second values ​​are used to indicate the data length corresponding to the logical offset that has been allocated for use; According to the third sorting position of the second logical offset in the logical offsets that have been allocated for use, the second value of the third sorting position in the second data structure is updated to the second data length.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data reading method described in any one of claims 1-8, or execute the data storage method described in any one of claims 9-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the data reading method according to any one of claims 1-8, or to execute the data storage method according to any one of claims 9-11.

25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data reading method according to any one of claims 1 to 8, or implements the data storage method according to any one of claims 9 to 11.

Images