Data storage method, data reading method and computer device

Abstract

The application is suitable for the computer technical field, and provides a data storage method, a data reading method and a computer device, including: obtaining data to be stored, wherein a table corresponding to the data to be stored is a temporary table; storing the data to be stored in a log-structured merge tree or storing the data to be stored in a memory; and recording an index of the stored data in the log-structured merge tree. The above method can improve the response speed of the computer device.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, and particularly relates to a data storage method, a data reading method, a storage device, a computer device and a computer readable storage medium. BACKGROUND

[0002] At present, a log-structured merge tree (LSMTree) is one of mainstream storage manners for developing a storage engine.

[0003] The LSMTree mainly comprises a write-ahead logging (WAL), a memory table (Memtable), an immutable memory table (Immutable Memtable) and a sorted string table (SSTable). Among them:

[0004] The WAL is a commonly used means for guaranteeing persistence in a database. In the LSMTree structure, if data is first written into the Memtable and sorted in the memory, and then sequentially written into a hard disk to form an SSTable file after a certain amount of data is written, the data in the Memtable will be lost in the process of converting the Memtable into the SSTable if the system crashes. In order to guarantee the persistence of the data, the LSMTree sequentially writes the data into a log (Log) and then writes the data into the Memtable. In this way, even if the system crashes, the data in the Memtable can be recovered through the Log.

[0005] The Memtable is located in the memory.

[0006] The Immutable Memtable structure is completely same as the Memtable structure, and also exists in the memory, but it is only readable and not writable. When the data written into the Memtable reaches a threshold value, the Memtable will be converted into the Immutable Memtable, and the Immutable Memtable will be written into a file by a background thread to generate the SSTable.

[0007] The SSTable is an ordered storage record generated by the Immutable Memtable.

[0008] Although the LSMTree has greatly improved the write performance by converting random Input / Output (I / O) into sequential I / O, it still needs to occupy certain I / O resources, and the occupation of the I / O resources will affect the performance of the computer device. Therefore, further optimization can be combined with actual application scenarios to improve the performance by reducing I / O. SUMMARY

[0009] Embodiments of the present application provide a data storage method, which can solve the problem of slow response speed of a computer device.

[0010] In a first aspect, embodiments of the present application provide a data storage method, comprising:

[0011] obtaining data to be stored, wherein a table corresponding to the data to be stored is a temporary table;

[0012] storing the data to be stored in a log-structured merge tree or in memory;

[0013] recording an index of the stored data in the log-structured merge tree.

[0014] In a second aspect, embodiments of the present application provide a data reading method, comprising:

[0015] receiving a data reading instruction, wherein the reading instruction carries a type identifier of a table corresponding to data to be read, and carries an index value of the data to be read;

[0016] identifying whether the table is a temporary table according to the type identifier of the table;

[0017] if the table is a temporary table, reading corresponding data from the log-structured merge tree or from memory and a hard disk according to the index value of the data to be read and the log-structured merge tree, wherein the log-structured merge tree records indexes of data stored therein or stored in memory.

[0018] In a third aspect, embodiments of the present application provide a data storage device, comprising:

[0019] a data to be stored obtaining module configured to obtain data to be stored, wherein a table corresponding to the data to be stored is a temporary table;

[0020] a data storage module configured to store the data to be stored in a log-structured merge tree or in memory;

[0021] a corresponding relationship recording module configured to record an index of the stored data in the log-structured merge tree.

[0022] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to the first aspect or the second aspect when executing the computer program.

[0023] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the method according to the first aspect or the second aspect.

[0024] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer device, causes the computer device to execute the method according to the first aspect or the second aspect.

[0025] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0026] In the embodiment of the present application, if it is needed to store data in the temporary table, the data is stored in the log-structured merge tree or in the memory, and then an index of the stored data or the data stored in the memory is recorded in the log-structured merge tree. Since no pre-write log operation is performed on the data to be stored when the data is stored based on the log-structured merge tree, but a data storage operation is directly performed, the I / O resources occupied by the pre-write log operation can be reduced, and thus the write performance of the computer device is improved. In addition, since the table targeted by the embodiment of the present application is a temporary table, and the data in the temporary table does not need to be guaranteed to be persistent, even if the pre-write log operation is not performed, no bad effect will be caused. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0028] Figure 1 is a flowchart of a data storage method provided by an embodiment of the present application;

[0029] Figure 2 is a flowchart of another data storage method provided by an embodiment of the present application;

[0030] Figure 3 is an application schematic diagram of the data storage method provided by an embodiment of the present application;

[0031] Figure 4 is a flowchart of a data reading method provided by an embodiment of the present application;

[0032] Figure 5 is a structural block diagram of a data storage device provided by another embodiment of the present application;

[0033] Figure 6 is a structural block diagram of a data reading device provided by an embodiment of the present application;

[0034] Figure 7 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc. in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0036] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", when used in this specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0037] It is also to be understood that the terminology "and / or" when used in this specification and in the following claims, refers to at least one of the items, or any combination of one or more of the items, and includes all possible combinations of one or more of the items.

[0038] In addition, in the description of the application in this specification and in the following claims, the terms "first", "second", "third", etc. are used only to distinguish different descriptions, and cannot be understood as indicating or implying relative importance.

[0039] The description of the reference "one embodiment" or "some embodiments" and the like in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in yet some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized.

[0040] Example One:

[0041] In the prior art, when data of a table needs to be stored, a log-structured merge tree can be used to store the data. However, when the data is stored by using the log-structured merge tree, the same storage manner is used to store data of any type of table, for example, the data is sequentially written into a log, then written into a Memtable, and finally, when the data written into the Memtable reaches a threshold, the data is converted into data of an Immutable Memtable, and then, a background thread asynchronously writes the data of the Immutable Memtable into a file to generate an SSTable. Since the above data storage process needs to occupy certain I / O resources, and the I / O resources are the main performance bottleneck of a database, the existing log-structured merge tree still reduces the performance of the database when storing data.

[0042] During the running of a database, many operations need to use temporary tables, such as intermediate tables used in association or sorting. Since the temporary tables are only useful for the current operation and can be released after the operation is completed, in order to save I / O resources and improve performance, the existing log-structured merge tree is improved, and data is stored based on the improved log-structured merge tree. Specifically, in the embodiments of the present application, it is first determined whether the table corresponding to the data to be stored is a temporary table. If it is a temporary table, a memory table is directly generated according to the data to be stored, that is, the step of sequentially writing the data into a log is no longer performed. Since the step of sequentially writing the data into a log (WAL) is no longer performed, and sequentially writing the data into a log will occupy some I / O resources, therefore, when the data is stored by using the method of the embodiments of the present application, the impact on the performance of the computer device can be further reduced.

[0043] The data storage method provided by the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0044] Figure 1 A flowchart of a data storage method provided by the embodiments of the present application is shown, and the data storage method is applied in a computer device, which is described in detail as follows.

[0045] In step S11, data to be stored is obtained, and the table corresponding to the data to be stored is a temporary table.

[0046] In the embodiments, the data to be stored is data to be filled into a temporary table, and the data to be stored is related to a scenario in which the temporary table needs to be generated.

[0047] In some embodiments of the present application, before step S11, an empty temporary table is generated, that is, a temporary table without data stored therein is generated. Specifically, when a specified operation is performed, the temporary table will be generated. The specified operation includes but is not limited to an association operation and a sorting operation.

[0048] For example, in a distributed database, an association needs to be performed on two tables, but the distribution rules of the two tables are different, at this time, the tables need to be redistributed (redistribution refers to an operation of uniformly re-distributing the distribution rules of the two tables to the same distribution rule), and the table generated by the redistribution is a temporary table, at this time, the data to be stored is the data involved in the redistribution.

[0049] For another example, if a table needs to be sorted, a temporary table with an index on the sorting field can be established, at this time, the data to be stored is the data corresponding to the sorting field. The data to be stored is written into the temporary table, and the ordered data can be obtained by sequentially reading the temporary table.

[0050] Step S12, storing the data to be stored in the log-structured merge tree, or storing the data to be stored in the memory.

[0051] The memory table mentioned above refers to a "Memtable" in the log-structured merge tree, and the Memtable is located in the memory.

[0052] In the embodiment, the data to be stored can be stored in the log-structured merge tree, such as the memory table in the log-structured merge tree, to obtain a memory table with data. Alternatively, the data to be stored can be stored only in the memory, but not in the memory table in the log-structured merge tree.

[0053] However, in either case, the data to be stored does not need to be pre-written in the log first, but a data storage operation (such as storing in the memory table or storing in the memory) is directly performed. This is because the table to which the embodiment of the present application is directed is a temporary table, and the data in the temporary table does not need to be guaranteed to be persistent, so even if the pre-written log is not performed, there will be no bad effect, and after the pre-written log is not performed, the occupation of the I / O resources of the computer device can be reduced, thereby facilitating the improvement of the processing speed of the computer device.

[0054] Step S13, recording the index of the stored data in the log-structured merge tree.

[0055] The index of the stored data corresponds to the stored data one by one, and the stored data includes at least one of the data stored in the memory table and the data stored in the memory.

[0056] In the embodiment, in order to quickly find the stored data, the index corresponding to the stored data is recorded in the memory table, and the index value can be "1", "2", "3", etc. Of course, other identifiers can also be used for marking, which is determined by the table structure definition and implementation, and each index value only needs to correspond to one stored data.

[0057] In the embodiment, if the data needs to be stored in the temporary table, the data is stored in the memory table of the log-structured merge tree or in the memory, and the index of the data stored in the memory table or the memory is recorded in the memory table. Since no pre-write log operation is performed on the data to be stored when the data is stored based on the log-structured merge tree, but the data storage operation is directly performed, the I / O resource occupied by the pre-write log operation can be reduced, and thus the processing speed of the computer device is improved. In addition, since the table targeted by the embodiment is the temporary table, and the data in the temporary table does not need to be guaranteed to be persistent, no bad effect is caused even if the pre-write log is not performed.

[0058] Figure 2 A flowchart of another data storage method provided by the embodiment is shown. In the embodiment, step S21 is the same as step S11 described above, and thus is not described herein.

[0059] In step S21, the data to be stored is obtained, and the table corresponding to the data to be stored is a temporary table.

[0060] In step S22, the data to be stored is stored in the memory.

[0061] The memory described above refers to a specific data structure of the "memory" stored in the computer device.

[0062] In some embodiments, the data to be stored is sequentially stored in the memory by a list, for example, each record (i.e., the data of each record) corresponds to a serial number, and the order of the records (i.e., the data of the records) in the list can be determined by comparing the values of the serial numbers. The serial number herein corresponds to the position information in step S23, and the serial number can be used as the unique identifier of the data. Since the serial number (i.e., the unique identifier of the data) corresponds to the stored data, the corresponding data can be accurately found by using the serial number.

[0063] In step S23, the position information of the data stored in the memory is extracted.

[0064] In the embodiment, after the data to be stored is stored in the memory, the position information of the data in the memory is determined, and the position information is extracted.

[0065] In step S24, the correspondence between the index value of the data stored in the memory and the position information of the data stored in the memory is recorded in the memory table of the log-structured merge tree.

[0066] Specifically, the index is the correspondence between the index value and the position information.

[0067] In this embodiment, the index value and the position information of the data are stored in the key-value structure, that is, the "key" corresponds to the index value of the data, and the "value" is used to save the position information of the data stored in the memory.

[0068] Since the index value and the position information of the data are stored in the key-value structure, the stored data can be accurately found according to the index value and the position information of the data.

[0069] In this embodiment, since whether the memory table of the log-structured merge tree is converted into the read-only memory table is only related to the size of the memory table, and the space occupied by the data is greater than the space occupied by the corresponding relationship of the memory table record, the data to be stored is not stored in the memory table of the log-structured merge tree, and only the corresponding relationship between the index value of the data stored in the memory and the position information of the data stored in the memory is recorded in the memory table, so that the amount of data stored in the log-structured merge tree can be reduced, thereby reducing the write amplification. In addition, since the memory table of the log-structured merge tree records the corresponding relationship between the index value of the data stored in the memory and the position information of the data stored in the memory, the position information of the data stored in the memory can be determined according to the index value of the data, and then the corresponding data in the memory can be accurately found according to the position information.

[0070] In some embodiments, the data storage method further includes:

[0071] The first memory table threshold is used to indicate the maximum value of the amount of data that the memory table can currently store.

[0072] Correspondingly, step S22 includes:

[0073] A1, determining the sum of the size of the data stored in the memory and the size of the memory table to obtain a sum value.

[0074] It should be noted that the data stored in the memory here refers to the data to be stored in the memory.

[0075] A2, if the sum value is not greater than the preset temporary table size, storing the data to be stored in the memory, and the preset temporary table size is less than the first memory table threshold.

[0076] The preset temporary table size is related to the memory size of the computer device itself, and generally, the preset temporary table size is positively correlated with the memory size of the computer device itself.

[0077] In the embodiment, considering that the original memory table can store less data, and once the data stored in the memory table reaches the maximum data amount that the memory table can store, the memory table will be converted into a read-only memory table, and then an ordered string table is generated and written into the hard disk (i.e., no longer stored in the memory), therefore, in order to improve the utilization of the memory as much as possible and ensure that the occupied memory does not affect the operation of the computer device, a first memory table threshold is set for the memory table, the first memory table threshold is greater than the original threshold corresponding to the memory table, and a temporary table size is set, and the temporary table size is less than the first memory table threshold. In this way, as long as the sum (i.e., the sum value) of the space occupied by the data stored in the memory and the corresponding relationship of the memory table is not greater than the preset temporary table size, the data is continuously stored in the memory, and the corresponding relationship of the index value and the position information of the data newly stored in the memory is continuously stored in the memory table of the log-structured merge tree.

[0078] In some embodiments, the data storage method further includes:

[0079] B1, if the sum value is greater than the preset temporary table size, the first memory table threshold is reduced to a second memory table threshold, and the unstored data is stored in the hard disk, and the position information of the data stored in the hard disk is extracted, wherein the second memory table threshold is less than the preset temporary table size, and the unstored data refers to the data to be stored that has not been stored.

[0080] Specifically, when the data to be stored is not stored in the memory, and the sum value is greater than the preset temporary table size, the data to be stored is stored in the hard disk.

[0081] In the embodiment, the unstored data can be stored in the hard disk in the form of a heap file (HeapFile). Of course, if the HeapFile form is used for storage, the position information of the data stored in the hard disk is represented by an offset.

[0082] The first memory table threshold is usually set to several hundred megabytes (MB) or several gigabytes (GB), and the second memory table threshold is usually set to several MB, such as 4 MB.

[0083] B2, a memory table is newly created in the log-structured merge tree, and the corresponding relationship of the index value of the data stored in the hard disk and the position information of the data stored in the hard disk is recorded in the newly created memory table.

[0084] In the embodiment, since the corresponding relationship of the index value of the data stored in the hard disk and the position information of the data stored in the hard disk is recorded, the position information of the data stored in the hard disk can be determined according to the index value of the data stored in the hard disk in the future.

[0085] In the above B1-B2, when the sum of the space occupied by the data stored in the memory and the corresponding relationship stored in the memory table is greater than the preset temporary table size, the data not stored is stored in the hard disk, so that the memory of the computer device can be avoided from being excessively occupied. Meanwhile, when the sum of the space occupied by the data stored in the memory and the corresponding relationship stored in the memory table is not greater than the preset temporary table size, the data is stored in the memory as much as possible, and the storage in the memory does not need to occupy the I / O resource of the computer device, so that the storage path of the data is selected by setting the size of the temporary table, the utilization rate of the memory can be effectively improved, and the response speed of the computer device is improved. In addition, since the corresponding ordered string table needs to be generated according to the newly created memory table in the subsequent process, and a larger memory table needs to occupy more memory, after the size of the first memory table (i.e., the memory table recording the corresponding relationship between the index value of the data stored in the memory and the position information of the data stored in the memory) reaches the first memory table threshold, the first memory table threshold is modified to a smaller second memory table threshold, so that the memory overhead can be effectively reduced, and the efficiency of converting to the ordered string table is improved.

[0086] In some embodiments, the data storage method provided by the embodiments of the present application further includes:

[0087] C1, if the sum is greater than the preset temporary table size, the memory table recording the corresponding relationship between the index value of the data stored in the memory and the position information of the data stored in the memory is converted into a read-only memory table, and will not be converted into an ordered string table.

[0088] C2, if the size of the newly created memory table is greater than the second memory table threshold, the newly created memory table is converted into a read-only memory table, and an ordered string table is generated according to the converted read-only memory table.

[0089] Specifically, when the size of the newly created memory table (Memtable) is greater than the second memory table threshold, the newly created memory table (Memtable) is converted into a read-only memory table (Immutable Memtable), and then an ordered string table corresponding to the read-only memory table is generated. The ordered string table is an ordered record of the data written by the Immutable Memtable. Of course, if there is still data to be stored, a new empty memory table is generated again, and the step B2 and the subsequent steps are returned until all the data to be stored is stored in the hard disk.

[0090] In this embodiment, the data originally stored in the memory continues to be retained in the memory for subsequent reading, and the memory table recording the correspondence between the index value of the data stored in the memory and the position information of the data is converted into a read-only memory table, so that the information recorded in the memory table is not changed. That is, it is ensured that after the size of the temporary table exceeds the preset size, only the newly written data is written to the hard disk, and the old data (data already stored in the memory) is still saved in the memory. Since the data is still retained in the memory, the read performance of the computer device can be improved when reading data from the memory subsequently. Moreover, since the data in the memory is not written to the hard disk, the I / O resources of the computer device do not need to be occupied. In addition, since the maximum amount of data stored in the memory table has been adjusted to the second memory table threshold, subsequent conversion operations will be performed only when it is determined that the size of the newly created memory table is greater than the second memory table threshold. Since the second memory table threshold is small, the memory occupation can be reduced, and the memory table can be quickly converted into a read-only memory table, and then quickly converted into an ordered string table.

[0091] In some embodiments, if multiple ordered string tables are generated, after step C2, it includes: judging whether there is overlapping data between the data stored in each ordered string table, if there is overlapping data, generating a new ordered string table according to the ordered string table with overlapping data, and there is no overlapping data between the new ordered string tables. For example, assuming that the upper limit of the Memtable is 5 rows of records. After the Memtable writes 1, 3, 2, 5, and 7, the Memtable will be converted into an Immutable Memtable, and the Immutable Memtable generates an SSTable1, which is [1, 2, 3, 5, 7]. Continue to write 9, 8, 19, 6, and 4 in the new Memtable, and the generated SSTable2 is saved as [4, 6, 8, 9, 19]. Since there is overlapping data "4, 5, 6, 7" between the two SSTable data intervals, in order to query "4" now, it is necessary to query SSTable1 and SSTable2 at the same time to determine whether there is record 4. If record 9 is queried, only SSTable2 needs to be queried. In order to improve the query efficiency, the asynchronous compression (Compaction) thread will periodically merge the SSTable files to generate globally ordered new SSTable files. For example, here, SSTable1 and SSTable2 are merged to regenerate two new SSTable files [1, 2, 3, 4, 5] and [6, 7, 8, 9, 19], and the old SSTable files will be deleted.

[0092] In some embodiments, in order to distinguish, the location information of the data stored in the memory is marked with a different identifier than the location information of the data stored in the hard disk.

[0093] For example, the location information of the data stored in the memory is marked with a serial number, and the location information of the data stored in the hard disk is marked with an offset. Of course, if both are marked with the same identifier, a flag bit can be added for distinction, such as when the flag bit is 0, it indicates that the corresponding identifier is the location information of the data stored in the memory, and when the flag bit is 1, it indicates that the corresponding identifier is the location information of the data stored in the hard disk.

[0094] In order to more clearly describe the data storage method provided by the embodiments of the present application, the following will be described in combination with Figure 3 .

[0095] 1. First, record the data into the memory to obtain a recorded serial number value.

[0096] 2. Extract the index value of the data stored in the memory from the record, use the index value of the data stored in the memory as the key, and use the above-mentioned serial number value as the value, and insert it into the memory table (Memtable) of the log-structured merge tree. At this time, the threshold of the Memtable is the first memory table threshold.

[0097] 3. If the sum of the size of the data stored in the memory and the size of the memory table is greater than the preset temporary table size, the log-structured merge tree adjusts the threshold of the Memtable to the second memory table threshold. Wherein, the first memory table threshold is greater than the preset temporary table size, the original Memtable becomes an immutable memory table (Immutable Memtable), and will not be converted into a sorted string table (SSTable).

[0098] 4. Record the data into the heap file (HeapFile) to obtain a recorded offset.

[0099] 5. Extract the index value of the data stored in the hard disk from the record, use the index value of the data stored in the hard disk as the key, and use the above-mentioned offset as the value, and insert it into the newly created memory table (Memtable) of the log-structured merge tree. If the size of the newly created Memtable is greater than the second memory table threshold, it is converted into a read-only memory table, and asynchronous compaction is performed to obtain a merged sorted string table.

[0100] It should be understood that the size of the serial number of each step in the above-mentioned embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0101] Example Two:

[0102] The embodiment describes a method for reading the data stored in the first embodiment.

[0103] Figure 4 A flow chart of a method for reading data provided by the embodiment is shown, which is described in detail as follows.

[0104] In step S41, a reading instruction of data is received, the reading instruction carries a type identifier of a table corresponding to the data to be read, and carries an index value of the data to be read.

[0105] Specifically, different types of tables have different type identifiers.

[0106] In step S42, it is identified whether the table corresponding to the data to be read is a temporary table according to the type identifier of the table.

[0107] In the embodiment, the way of reading data is related to the way of storing data, and the embodiment is for storing data of a temporary table, thus, it is needed to identify whether the data to be read is data of a temporary table, so as to select a corresponding reading way according to the identification result.

[0108] In step S43, if the table corresponding to the data to be read is a temporary table, the corresponding data is read from a log-structured merge tree or from a memory according to the index value of the data to be read and the log-structured merge tree, wherein the log-structured merge tree records indexes of the data stored therein or stored in the memory.

[0109] In the embodiment, since the log-structured merge tree contains indexes of data, the index value of the data to be read can be compared with index values of each data contained in the log-structured merge tree, so as to identify the corresponding data, and then select to read the corresponding data from the log-structured merge tree or from the memory. For example, if the data is stored in the memory at that time, the data is read from the memory, otherwise, the data is directly read from the log-structured merge tree.

[0110] In the embodiment, since no pre-write log operation is performed on the data to be stored when the data is stored based on the log-structured merge tree, but a data storage operation is directly performed, the I / O resources occupied by the pre-write log operation can be reduced, so as to improve the write performance of the computer device. That is, the whole data storage and reading process can reduce the occupation of I / O resources, so as to improve the write performance of the computer device.

[0111] In some embodiments, if the data to be stored is stored in the memory and the correspondence between the index value of the data stored in the memory and the location information of the data stored in the memory is recorded in the in-memory table of the log-structured merge tree, then step S43 reads the corresponding data from the memory according to the index value of the data to be read and the log-structured merge tree, including:

[0112] According to the index value of the data to be read, the location information of the data stored in the memory is matched from the in-memory table of the log-structured merge tree, and then the corresponding data is read from the memory according to the location information.

[0113] In some embodiments, if the data is stored in the memory and the hard disk, and the correspondence between the index value of the data stored in the memory and the location information of the data stored in the memory is recorded in the in-memory table of the log-structured merge tree, and the correspondence between the index value of the data stored in the hard disk and the location information of the data stored in the hard disk is recorded in the newly created in-memory table, the data reading method provided by the embodiments of the present application further includes:

[0114] If the index value of the index in the in-memory table of the log-structured merge tree is the same as the index value of the data to be read, and the location information corresponding to the index value points to the memory, then the corresponding data is read from the memory according to the corresponding location information; if the location information corresponding to the index value points to the hard disk, then the corresponding data is read from the hard disk according to the corresponding location information.

[0115] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0116] Example Three:

[0117] According to the data storage method of the above embodiment one, Figure 5 The structure block diagram of the data storage device provided by the embodiments of the present application is shown, only the part related to the embodiments of the present application is shown for convenience of description.

[0118] Referring to Figure 5 The data storage device 5 includes: a data to be stored acquisition module 51, a data storage module 52, and a correspondence recording module 53. Among them:

[0119] The data to be stored acquisition module 51 is used to acquire the data to be stored, and the table corresponding to the data to be stored is a temporary table.

[0120] In some embodiments, the data storage device 5 includes:

[0121] a temporary table generation module configured to generate an empty temporary table.

[0122] a data storage module 52 configured to store the data to be stored in a log-structured merge tree, or store the data to be stored in the memory.

[0123] an index value recording module 53 configured to record an index of the stored data in the log-structured merge tree.

[0124] In the embodiments of the present application, if the data needs to be stored in the temporary table, the data is stored in the log-structured merge tree, or stored in the memory and the hard disk, and an index of the stored data or the data in the memory is recorded in the log-structured tree. Since no pre-write log operation is performed on the data to be stored when the data is stored based on the log-structured merge tree, but a data storage operation is directly performed, the I / O resources occupied by the pre-write log operation can be reduced, thereby improving the write performance of the computer device. In addition, since the table targeted by the embodiments of the present application is a temporary table, and the data in the temporary table does not need to be guaranteed to be persistent, even if the pre-write log operation is not performed, no bad effect will be caused.

[0125] In some embodiments, if the data to be stored is stored in the memory, the data storage device 5 further comprises:

[0126] a position information extraction module configured to extract position information of the data stored in the memory.

[0127] Correspondingly, the corresponding relationship recording module 53 is specifically configured to:

[0128] record, in the memory table of the log-structured merge tree, a corresponding relationship between the index of the data stored in the memory and the position information of the data stored in the memory.

[0129] In some embodiments, the data storage device 5 further comprises:

[0130] a first memory table threshold setting module configured to set a first memory table threshold, the first memory table threshold being used to indicate a maximum value of an amount of data that can be currently stored in the memory table.

[0131] When the data storage module 52 stores the data to be stored in the memory, the data storage module 52 comprises:

[0132] a sum determining unit configured to determine a sum of a size of the data stored in the memory and a size of the memory table, to obtain a sum value.

[0133] a memory storage unit configured to store the data to be stored in the memory if the sum value is not greater than a preset temporary table size, the preset temporary table size being smaller than the first memory table threshold.

[0134] In some embodiments, the data storage device 5 further comprises:

[0135] a hard disk storage module, configured to, if the sum value is greater than the preset temporary table size, decrease the first memory table threshold to a second memory table threshold, and store the un-stored data in the hard disk and extract the location information of the data stored in the hard disk, wherein the second memory table threshold is smaller than the preset temporary table size, and the un-stored data refers to the data to be stored which has not been stored.

[0136] a memory table creation module, configured to create a memory table in the log-structured merge tree, and record the correspondence between the index value of the data stored in the memory and the location information of the data stored in the memory in the created memory table.

[0137] In some embodiments, the data storage device 5 further comprises:

[0138] a read-only memory table conversion module, configured to, if the sum value is greater than the preset temporary table size, convert the memory table recording the correspondence between the index value of the data stored in the memory and the location information of the data stored in the memory into a read-only memory table, and not convert the memory table into an ordered string table.

[0139] an ordered string table generation module, configured to, if the size of the created memory table is greater than the second memory table threshold, convert the created memory table into a read-only memory table, and generate an ordered string table according to the read-only memory table obtained by the conversion.

[0140] In some embodiments, different identification marks are used to store the location information of the data stored in the memory and the location information of the data stored in the hard disk.

[0141] In some embodiments, the data storage device 5 further comprises:

[0142] an overlapping data processing module, configured to, if multiple ordered string tables are generated, determine whether there is overlapping data between the data stored in each ordered string table, if there is overlapping data, generate a new ordered string table according to the ordered string table having the overlapping data, and the new ordered string table does not have overlapping data.

[0143] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought by them can be referred to the method embodiments part, which will not be repeated here.

[0144] Example Four:

[0145] the data reading method corresponding to the above-mentioned embodiment two, Figure 6A structure block diagram of the data reading apparatus provided by the embodiment of the present application is shown. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0146] The data reading apparatus 6 comprises a reading instruction receiving module 61, a table type judging module 62, and a data reading module 63. Among them:

[0147] The reading instruction receiving module 61 is configured to receive a data reading instruction, wherein the data reading instruction carries a table type identifier of a table corresponding to data to be read, and carries an index value of the data to be read.

[0148] The table type judging module 62 is configured to identify whether the table corresponding to the data to be read is a temporary table according to the table type identifier.

[0149] The data reading module 63 is configured to read corresponding data from a log-structured merge tree or read corresponding data from a memory according to the index value of the data to be read and the log-structured merge tree, if the table corresponding to the data to be read is a temporary table, wherein the log-structured merge tree records index values of data stored in the log-structured merge tree or data stored in the memory.

[0150] In the embodiment of the present application, since no pre-write log operation is performed on the data to be stored when the data is stored based on the log-structured merge tree, but the data storage operation is directly performed, the I / O resources occupied by the pre-write log operation can be reduced, thereby improving the write performance of the computer device. That is, the entire data storage and reading process can reduce the occupation of I / O resources, thereby improving the write performance of the computer device.

[0151] In some embodiments, when the data reading module 63 reads corresponding data from the memory according to the index value of the data to be read and the log-structured merge tree, it is specifically configured to:

[0152] According to the index value of the data to be read, the position information of the data stored in the memory is matched from the memory table in the log-structured merge tree, and then the corresponding data is read from the memory according to the position information.

[0153] In some embodiments, the data reading apparatus 6 further comprises:

[0154] The hybrid reading module is configured to read corresponding data from the memory according to the corresponding position information if there is an index value same as the index value of the data to be read in the index of the memory table of the log-structured merge tree, and the position information corresponding to the index value points to the memory; and read corresponding data from the hard disk according to the corresponding position information if the position information corresponding to the index value points to the hard disk.

[0155] Example Five:

[0156] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 7 As shown, the computer device 7 of this embodiment includes: at least one processor 70 ( Figure 7 The diagram shows only one processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, which, when executed, performs the steps in any of the above-described method embodiments.

[0157] The computer device 7 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This computer device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that... Figure 7 The computer device 7 is merely an example and does not constitute a limitation on the computer device 7. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0158] The processor 70 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] The memory 71 can be an internal storage unit of the computer device 7 in some embodiments, for example, a hard disk or a memory of the computer device 7. The memory 71 can also be an external storage device of the computer device 7 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 7. Further, the memory 71 can include both an internal storage unit and an external storage device of the computer device 7. The memory 71 is used to store an operating system, an application program, a boot loader, data, and other programs, etc., for example, program codes of the computer program, etc. The memory 71 can also be used to temporarily store data that has been output or is to be output.

[0160] It should be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0161] The embodiments of the present application also provide a network device, which comprises at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the steps in any of the method embodiments described above when executing the computer program.

[0162] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps in any of the method embodiments described above.

[0163] The embodiments of the present application provide a computer program product, which, when running on a mobile terminal, enables the mobile terminal to implement the steps in any of the method embodiments described above.

[0164] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above-mentioned embodiment methods through a computer program to instruct relevant hardware to complete, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / computer equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0165] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0166] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0167] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between each other can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0169] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data storage method characterized by comprising: The method comprises the following steps: generating a temporary table when a specified operation is executed, the specified operation comprising an association operation and a sorting operation, and the data to be stored being data corresponding to the specified operation to be filled in the temporary table, the data filled in the temporary table not needing to be guaranteed to be persistent; storing the data to be stored in a log-structured merge tree or in a memory, without needing to perform a pre-write log operation on the data to be stored, but directly performing a data storage operation; recording an index of the stored data in the log-structured merge tree; if the data to be stored is stored in the memory, the data storage method further comprises the following steps: extracting position information of the data stored in the memory; the step of recording the index of the stored data in the log-structured merge tree comprises the following steps: recording, in a memory table of the log-structured merge tree, a correspondence between an index value of the data stored in the memory and the position information of the data stored in the memory; setting a first memory table threshold of the log-structured merge tree, the first memory table threshold being used to indicate a maximum value of an amount of data that the memory table can currently store; the step of storing the data to be stored in the memory comprises the following steps: determining a sum of a size of the data stored in the memory and a size of the memory table to obtain a sum value; if the sum value is not greater than a preset temporary table size, storing the data to be stored in the memory, the preset temporary table size being smaller than the first memory table threshold, wherein the preset temporary table size is positively correlated with a memory size of the computer device itself.

2. The data storage method of claim 1, wherein, the data storage method further comprises the following steps: if the sum value is greater than the preset temporary table size, reducing the first memory table threshold to a second memory table threshold, storing data not stored in a hard disk, and extracting position information of the data stored in the hard disk, wherein the second memory table threshold is smaller than the preset temporary table size, and the data not stored refers to the data to be stored that has not been stored; creating a memory table in the log-structured merge tree, and recording, in the created memory table, a correspondence between an index value of the data stored in the hard disk and the position information of the data stored in the hard disk.

3. The data storage method of claim 2, wherein, the data storage method further comprises the following steps: if the sum value is greater than the preset temporary table size, converting the memory table recording the correspondence between the index value of the data stored in the memory and the position information of the data stored in the memory into a read-only memory table, and not converting the memory table into an ordered string table; if a size of the created memory table is greater than the second memory table threshold, converting the created memory table into a read-only memory table, and generating an ordered string table according to the read-only memory table obtained by the conversion.

4. The data storage method of claim 2, wherein, The position information of the data stored in the memory and the position information of the data stored in the hard disk are marked with different identifiers.

5. A data reading method for reading data stored by a data storage method according to any one of claims 1 to 4, characterized by, The method comprises the following steps: receive a read instruction of data, the read instruction carrying a type identifier of a table corresponding to the data to be read, and carrying an index value of the data to be read, the data to be read being data corresponding to a specified operation filled in a temporary table, the data filled in the temporary table not needing to be guaranteed for persistence, the specified operation including an association operation and a sorting operation, wherein a size of the temporary table is positively correlated with a memory size of the computer device itself; identify whether the table corresponding to the data to be read is a temporary table according to the type identifier of the table; if the table corresponding to the data to be read is a temporary table, read corresponding data from a log-structured merge tree according to the index value of the data to be read and the log-structured merge tree, or read the corresponding data from a memory, wherein the log-structured merge tree records indexes of data stored therein or stored in the memory, and in either case, a pre-write log operation is not needed to be performed on the data to be stored, but a data storage operation is directly performed.

6. A storage device for data, characterized by comprise: a data to be stored obtaining module, configured to generate a temporary table when a specified operation is performed, and obtain data to be stored, the specified operation including an association operation and a sorting operation, the data to be stored being data corresponding to the specified operation filled in the temporary table, the data filled in the temporary table not needing to be guaranteed for persistence; a data storage module, configured to store the data to be stored in a log-structured merge tree, or store the data to be stored in a memory, wherein in either case, a pre-write log operation is not needed to be performed on the data to be stored, but a data storage operation is directly performed; a corresponding relationship recording module, configured to record indexes of the data stored in the log-structured merge tree; if the data to be stored is stored in the memory, the data storage method further comprises: extracting position information of the data stored in the memory; the recording of the indexes of the data stored in the log-structured merge tree comprises: recording, in a memory table of the log-structured merge tree, a corresponding relationship between the index value of the data stored in the memory and the position information of the data stored in the memory; setting a first memory table threshold value of the log-structured merge tree, the first memory table threshold value being used to indicate a maximum value of an amount of data currently able to be stored in the memory table; the storing of the data to be stored in the memory comprises: determining a sum of a size of the data stored in the memory and a size of the memory table to obtain a sum value; if the sum value is not greater than a preset temporary table size, storing the data to be stored in the memory, the preset temporary table size being smaller than the first memory table threshold value, wherein the preset temporary table size is positively correlated with a memory size of the computer device itself.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the method of any one of claims 1 to 5.

Images