Data writing method and device, data reading method and device

Abstract

One or more embodiments of the present specification provide a data writing method and device, as well as a data reading method and device, which are applied to a storage system. The method includes: in response to a write instruction corresponding to the data to be written, writing the data to be written into a temporary storage file in the memory; determining whether the amount of the data to be written in the temporary storage file reaches a preset quantity threshold; if so, performing statistics on the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes the value range of the data to be written; writing the statistical information and the data to be written in the temporary storage file into the persistent storage carried by the storage system, so as to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system.

Description

Technical Field

[0001] The present specification relates to the field of computer software technology, and in particular to a data writing method and device, and a data reading method and device. Background Art

[0002] Some storage systems have the problem of read amplification. For example, the LSM storage system is a type of storage system that stores data based on the log structured merge tree (LSM tree) data structure, which can achieve extremely efficient data writing operations. However, when reading data, the LSM storage system needs to read the data of multiple files in the memory and persistent storage to return the final result, which causes multiple file reads and writes in one query operation, causing the problem of read amplification. Therefore, how to solve the problem of read amplification in storage systems is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0003] In view of this, one or more embodiments of the present specification provide a data writing method and device, and a data reading method and device to solve the problems existing in the related art.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of an embodiment of this specification, a data writing method is provided, which is applied to a storage system, and the method includes:

[0006] In response to a write instruction corresponding to the data to be written, writing the data to be written into a temporary storage file in the memory;

[0007] Determine whether the amount of data to be written in the temporary storage file reaches a preset amount threshold;

[0008] If yes, statistics are collected for the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes a value range of the data to be written;

[0009] The statistical information and the data to be written in the temporary storage file are written into the persistent storage carried by the storage system to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system.

[0010] According to a second aspect of an embodiment of this specification, a data reading method is provided, which is applied to a storage system, and the method includes:

[0011] Obtaining a read instruction for target data to be read; wherein the read instruction includes a data query range;

[0012] In response to the read instruction, the query index stored in the memory of the storage system is traversed, target files whose statistical information intersects with the data query range are filtered out from temporary storage files and persistent storage files stored in the storage system, and the target data is read from the filtered target files.

[0013] According to a third aspect of an embodiment of this specification, a data writing device is provided, which is applied to a storage system, and the device includes:

[0014] A first writing module, configured to write the data to be written into a temporary storage file in the memory in response to a write instruction corresponding to the data to be written;

[0015] A statistical information module is used to determine whether the amount of data to be written in the temporary storage file reaches a preset amount threshold;

[0016] If yes, statistics are collected for the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes a value range of the data to be written;

[0017] The second writing module writes the statistical information and the data to be written in the temporary storage file into the persistent storage carried by the storage system, so as to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system.

[0018] According to a fourth aspect of an embodiment of this specification, a data reading device is provided, which is applied to a storage system, and the device includes:

[0019] An instruction acquisition module, used to acquire a read instruction for target data to be read; wherein the read instruction includes a data query range;

[0020] A target file screening module is used to respond to the read instruction, traverse the query index stored in the memory of the storage system, screen out target files whose statistical information intersects with the data query range from temporary storage files and persistent storage files stored in the storage system, and read the target data from the screened target files.

[0021] According to a fifth aspect of the embodiments of this specification, there is provided an electronic device, including a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0022] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0023] According to a sixth aspect of the embodiments of this specification, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-readable instructions, and the machine-readable instructions implement the above method when called and executed by a processor.

[0024] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0025] By performing statistics on the data to be written stored in the temporary storage file, statistical information corresponding to the temporary storage file is obtained, and the statistical information is used as a query index when performing a range query on the data stored in the storage system. When reading data, the target file is located through the query index in the memory, which can reduce the number of file reads and writes, and can realize the query of data within a certain range. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the architecture of a data writing and data reading system provided by an exemplary embodiment.

[0027] Figure 2 is a flow chart of a data writing method provided by an exemplary embodiment.

[0028] Figure 3 It is a schematic diagram of a set of persistent storage files with strictly increasing statistical information provided by an exemplary embodiment.

[0029] Figure 4 It is a system architecture diagram of a storage system provided by an exemplary embodiment.

[0030] Figure 5 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment.

[0031] Figure 6 is a block diagram of a data writing device provided by an exemplary embodiment.

[0032] Figure 7 is a block diagram of a data reading device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0034] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0035] Figure 1 FIG. 1 is a schematic diagram of the architecture of a data writing method and a data reading method provided by an exemplary embodiment. Figure 1 As shown, the system may include a network 10, a server 11, and several electronic devices, such as a mobile phone 12, a mobile phone 13, and a mobile phone 14.

[0036] The server 11 may be a physical server including an independent host, or the server 11 may be a virtual server, a cloud server, etc. hosted by a host cluster. The mobile phones 12-14 are just one type of electronic device that the user can use. In fact, the user may also use electronic devices such as tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and one or more embodiments of this specification do not limit this. The network 10 may include various types of wired or wireless networks.

[0037] In one embodiment, the server 11 can cooperate with the mobile phones 12-14; wherein the mobile phones 12-14 can receive instructions and upload the received instructions to the server 11 through the network 10, and then the server 11 processes the received instructions based on the data writing method and data reading method of this specification to realize the reading and writing of data. In another embodiment, the mobile phones 12-14 can independently implement the data writing method and data reading method of this specification; wherein the mobile phones 12-14 receive instructions and process the received call traffic based on the data writing method and data reading method of this specification to realize the reading and writing of data.

[0038] The LSM storage system is a type of storage system that stores data based on the log structured merge tree (LSMtree) data structure. It can implement extremely efficient data writing operations. However, when reading data, the LSM storage system needs to read data from multiple files in the memory and persistent storage to return the final result, which causes multiple file reads and writes in one query operation, resulting in the problem of read amplification.

[0039] The LSM storage system first writes the data into a temporary storage file in the memory, sorts the data in the temporary storage file, and when the amount of data in the temporary storage file reaches a certain threshold, writes the data in the memory sequentially to the disk, generating a file each time it is written, thereby obtaining a set of ordered files.

[0040] The MYSQL storage system writes data randomly to disk. Compared with the sequential writing method of the LSM storage system, the MYSQL storage system needs to spend more performance to determine the writing order of data on the disk. Therefore, the sequential writing method of the LSM storage system has higher data writing performance.

[0041] However, since the LSM storage system needs to store data in multiple files, when reading data, it needs to read data from multiple files in memory and persistent storage to return the final result, which means that one query operation needs to read multiple files, causing the problem of read amplification.

[0042] Influenced by this demand, the industry generally uses Bloom Filter to filter files, and through filtering, reduce the files that do not contain the query key. Bloom filter consists of a very long bit vector and multiple hash functions, and all bits are set to 0 initially. Use multiple hash functions to calculate multiple hash values ​​corresponding to each element in the set, and correspond the multiple hash values ​​to multiple positions of the bit vector, marking them as 1. The same method is used when querying, and the query element is mapped to the bit vector through each hash function. If any of the positions is not 1, it can be judged that the element must not exist in the set; conversely, if all positions are 1, the element may be in the set, but there will be a certain misjudgment rate.

[0043] Although using Bloom filters can reduce the number of files that do not contain the query key, there is a possibility of misjudgment, and unnecessary files can still be queried, wasting system performance; and Bloom filters can only be used to optimize point queries and do not support range queries.

[0044] In view of this, this specification proposes a solution for collecting statistical information of the data to be written when writing data, so that a range search can be performed based on the statistical information when reading data.

[0045] The concept of this specification is:

[0046] When receiving a write instruction corresponding to the data to be written, the write instruction can be responded to to write the data to be written into a temporary storage file in the memory;

[0047] Then, it may be determined whether the amount of data to be written in the temporary storage file reaches a preset amount threshold;

[0048] If the amount of data to be written reaches a preset amount threshold, statistics may be performed on the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes a value range of the data to be written;

[0049] Furthermore, the statistical information and the data to be written in the temporary storage file can be written into the persistent storage carried by the storage system to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system.

[0050] Through the above technical solution, the target file can be located through the query index in the memory when reading data, the number of file reading and writing times can be reduced, and the data within a certain range can be queried.

[0051] The detection scheme of the call link of this specification is described in detail below with reference to the accompanying drawings.

[0052] See also Figure 2 , Figure 2 This is a flow chart of a data writing method proposed in this specification, and the method is applied to a storage system. The storage system may include temporary storage files in the memory and persistent storage files in the persistent storage. The solution can be used in real-time communication (Instant Messaging, referred to as IM), security systems, the Internet of Things and other scenarios.

[0053] like Figure 2 As shown, the method includes the following execution steps:

[0054] Step 202, in response to a write instruction corresponding to the data to be written, writing the data to be written into a temporary storage file in the memory;

[0055] In one embodiment shown, the above storage system may be an LSM storage system, which is a type of storage system that stores data based on a log structured merge tree (LSM tree) data structure.

[0056] The above-mentioned write instruction may be an instruction initiated by a user to write the data to be written into the storage system. The write instruction may first sequentially write the data to be written into a temporary storage file in the memory of the storage system, and the storage system may sort the data to be written in the temporary storage file, and may package the sorted data into a file, so as to sequentially write the data into the persistent storage file.

[0057] For example, in an application scenario shown, taking the LSM storage system as an example, the above-mentioned write instruction can be a write instruction for real-time communication. In response to the write instruction initiated by the user, the write instruction can sequentially write the images, texts, files and other data sent during real-time communication into a temporary storage file in the memory of the LSM storage system. The LSM storage system can sort the data in the temporary storage file and package the sorted data into a file to realize the sequential writing of the data into the persistent storage file.

[0058] In another application scenario shown, taking the LSM storage system as an example, the write instruction may be an instruction for writing the image recorded by the camera of the security system into the memory. In this case, in response to the write instruction, the write instruction may sequentially write the image data recorded by the camera of the security system into a temporary storage file in the memory of the LSM storage system, and the LSM storage system may sort the data in the temporary storage file, and may package the sorted data into a file, so as to sequentially write the data into the persistent storage file.

[0059] The data to be written may be sequential data. For example, the data to be written may have a time sequence or a sequence number sequence, so that the data to be written may be sorted in a certain order when being written into the temporary storage file.

[0060] Step 204, determining whether the amount of data to be written in the temporary storage file reaches a preset amount threshold;

[0061] After the data to be written is written into the temporary storage file in the memory, the storage system can automatically calculate the amount of data to be written that has been written into the temporary storage file in the memory, and can compare the amount with a preset quantity threshold to determine whether the amount of data stored in the current temporary storage file exceeds the preset quantity threshold.

[0062] Step 206: if yes, statistics are collected for the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes the value range of the data to be written;

[0063] If the amount of data stored in the current temporary storage file exceeds a preset threshold, the storage system may count the data to be written stored in the temporary storage file to obtain statistical information of the data to be written; wherein the statistical information includes a value range of the data to be written.

[0064] The above statistical information can be used to perform range queries on files when reading data, and the target file can be accurately hit based on the statistical information, thereby achieving the effect of reducing the number of file reads and writes.

[0065] In one embodiment shown, the storage system can collect statistics on the maximum value, minimum value, and file sequence number of the data to be written stored in the temporary storage file. The statistical information may include independent maximum value, minimum value, and file sequence number, or a combination of maximum value, minimum value, and file sequence number.

[0066] In one example, statistical information can be identified only by the maximum value, that is, the value range of the statistical information is an open interval less than or equal to the maximum value to be expected. When reading data, the file can be filtered by the maximum value; similarly, statistical information can be identified only by the minimum value, that is, the value range of the statistical information is an open interval greater than or equal to the minimum value. When reading data, the file can be filtered by the maximum value; statistical information can also be identified by both the minimum value and the maximum value, that is, the statistical information can be a value range limited by the maximum value and the minimum value. When reading data, the file can be filtered by the value range.

[0067] In another example, the statistical information can be identified by the file number, and when reading data, the files can be filtered by the file number. A mapping relationship between the file number and other statistical information of the file can also be established, so as to perform a range query directly by the file number.

[0068] In practical applications, the above two methods can be used simultaneously. And since the data to be written usually has multiple groups, when counting the data to be written stored in the temporary storage file, the data to be written stored in the temporary storage file can be grouped according to the key value corresponding to the data to be written.

[0069] In one embodiment shown, the data to be written stored in the temporary storage file can be grouped according to the key value corresponding to the data to be written; statistics can be performed on the data in at least part of the groups obtained by grouping to obtain statistical information corresponding to the temporary storage file. For example, statistics can be performed on the data in only one group to obtain corresponding statistical information, or statistics can be performed on the data in multiple groups to obtain corresponding statistical information, so as to implement range query when reading data.

[0070] In another embodiment shown, only data in the group whose key value is the primary key may be counted to obtain statistical information corresponding to the temporary storage file, so as to implement range query when reading data.

[0071] In another embodiment shown, since there may be no order between the data of some groups, the data in the groups with sequentiality may be statistically counted to obtain statistical information corresponding to the temporary storage file to implement range query when reading data.

[0072] Step 208, write the statistical information and the data to be written in the temporary storage file into the persistent storage carried by the storage system, so as to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system.

[0073] After obtaining the statistical information in the temporary storage file, the sorted data to be written and the statistical information can be written together into the persistent storage equipped with the storage system, and a persistent storage file corresponding to the temporary storage file can be generated in the persistent storage, and a file sequence number can be added to the generated persistent storage file.

[0074] In one embodiment shown, after statistics are taken for the data to be written in the temporary storage file, the statistical information corresponding to the temporary storage file can be used as a query index corresponding to the temporary storage file and can be saved in the memory. The statistical information can be saved in the temporary storage file or in other storage files in the memory, and this specification does not limit this. When the statistical information and the data to be written in the temporary storage file are written to the persistent storage, the statistical information in the multiple persistent storage files stored in the persistent storage can be used as the query index corresponding to the persistent storage file, and an index tree corresponding to the multiple persistent storage files can be constructed and saved in the memory. In this way, the query index used for range query can be saved in the memory, which improves the speed of reading the query index.

[0075] In another embodiment shown, after the statistical information corresponding to the temporary storage file and the data to be written in the temporary storage file are written to the carried persistent storage, the statistical information and the data to be written corresponding to the temporary storage file stored in the memory can be deleted; after the temporary storage file is cleared, the temporary storage file can continue to respond to write instructions, write the data to be written into the temporary storage file, and can count the statistical information of the newly written data; when the number of data to be written in the temporary storage file exceeds the preset number threshold again, the newly written statistical information and data can be written to the persistent storage carried by the LSM.

[0076] Among them, it should be noted that, in actual applications, taking the LSM storage system as an example, since the write instruction can be an online write instruction sent by other terminals to the device equipped with the LSM storage system through the network, the response time requirement is relatively high; and the above-mentioned data writing scheme is to write the data to be written in the temporary storage file in the memory, and then count the data to be written in the temporary storage file. Therefore, even if the statistical information increases the amount of data, it will not affect the performance of online writing, and the storage space occupied by the statistical information is small, and it will not affect the process of writing data to persistent storage.

[0077] Obtain the statistical information corresponding to the temporary storage file and the persistent storage file. After establishing the query index, you can use the query index to perform range query or point query on the file.

[0078] For example, in one embodiment shown, taking the LSM storage system as an example, a read instruction for the target data to be read can be obtained; wherein the read instruction includes a data query range; after obtaining the read instruction, in response to the read instruction, the query index stored in the memory can be first traversed, and the target files whose statistical information intersects with the data query range can be filtered out from the temporary storage files and persistent storage files stored in the LSM storage system, and the target data can be read from the filtered target files.

[0079] In an example, after obtaining a read instruction containing a data query range for the target data to be read, the query index stored in the memory can be traversed first, and target files whose value ranges corresponding to the statistical information intersect with the data query range can be filtered out from temporary storage files and persistent storage files stored in the LSM storage system, and the target data can be read from the filtered target files.

[0080] In an embodiment shown, in a scenario where the value of the statistical information is strictly increasing, that is, the data order between multiple files is consistent with the order of the files, the query range in the read instruction can be matched with the statistical information corresponding to the persistent storage file stored in the persistent storage latest, to determine whether the file stored before it is within the query range; if the query range is within the value range of the statistical information corresponding to the persistent storage file stored in the persistent storage latest, it can be determined that the persistent storage file stored in the persistent storage latest is the target data file, and none of the persistent storage files stored before contains the target data; if the minimum value of the query range is not within the value range of the statistical information corresponding to the persistent storage file stored in the persistent storage latest, it can be determined whether the minimum value is in the persistent storage file stored before. It is possible to search forward in the order opposite to the order in which the persistent storage files are generated until the persistent storage file corresponding to the statistical information hit by the minimum value can be determined; similarly, the persistent storage file corresponding to the statistical information hit by the maximum value can be determined; multiple persistent storage files stored continuously between the two are used as target data files.

[0081] like Figure 3 As shown in the figure, taking the statistical information of the grouping with the key value as the primary key as an example, assuming that the primary key is (UserID, SeqNum), UserID can represent the user number, SeqNum can represent the sequence number, and the primary key (UserID, SeqNum) can represent that each user maintains a sequence number. All the data to be written can be stored in several persistent storage files in the persistent storage, for example, 10, and the value range of the sequence number can be used as the statistical information of the persistent storage file. In the scenario where the sequence number is strictly increasing, the value range of the sequence number SeqNum of user User0 continues to increase as the persistent storage file is generated, that is, the statistical information continues to increase from File0 to File9, where File9 is the latest generated persistent storage file. Now to query the data whose SeqNum of User0 is greater than 97, according to the statistical information, you only need to query the statistical information of file File9. According to the statistical information of File9, you can know that the SeqNum of User0 of all files generated before File9 is less than 90, so File0~File8 can be directly discarded at File9. At this point, although there are a total of 10 files, the target data file can be found by querying the statistical information of only the latest file.

[0082] It should be noted that the specific manner of writing data into the storage system and reading data from the storage system in response to the write instruction is not particularly limited in this specification;

[0083] In one embodiment shown, the write instruction may be sent via a real-time communication client (eg, dingtalk); the storage system may be carried by a server.

[0084] In this scenario, the real-time communication client can specifically respond to a write instruction and send the data to be written to the storage system mounted on the server through the network; or, it can respond to a read instruction and obtain the target data from the storage system mounted on the server, and send the target data to the real-time communication client.

[0085] Of course, in actual applications, in addition to the above scenarios, other scenarios can also be used; for example, security systems, the Internet of Things, etc., which will not be listed one by one in this manual.

[0086] See also Figure 4 , Figure 4 This is a system architecture diagram of the storage system shown in this specification.

[0087] like Figure 4 As shown, the system architecture of the storage system includes:

[0088] The memory 410 can be used to store the temporary storage file 420 and the index tree 430; wherein the index tree can be stored in the memory to speed up the reading of the index tree, thereby realizing the speed of speeding up the reading of the target file.

[0089] The persistent storage 440 can be used to store persistent storage files 450, and can merge multiple stored persistent storage files 350 to reduce the number of internal persistent storage files, thereby reducing the number of file reads and writes.

[0090] When used for writing data, the LSM storage system can respond to a write instruction and write the data to be written sequentially into the temporary storage file 420 of the memory 410 of the LSM storage system. The LSM storage system can sort the data to be written in the temporary storage file 420, and can package the sorted data into a file to achieve sequential writing of the data into the persistent storage file. If the amount of data currently stored in the temporary storage file 420 exceeds a preset quantity threshold, the LSM storage system can count the data to be written stored in the temporary storage file 420 to obtain statistical information of the data to be written. After obtaining the statistical information in the temporary storage file 420, the sorted data to be written and the statistical information can be written together into the persistent storage 440 equipped with the LSM storage system, and a persistent storage file 450 corresponding to the temporary storage file 420 can be generated in the persistent storage 440, and a file sequence number can be added to the generated persistent storage file 450.

[0091] When used to read data, a read instruction containing a query range for the target data can be obtained. After obtaining the read instruction, in response to the read instruction, the query index stored in the memory 410 can be first traversed, and the query index can include a query index constructed by statistical information of the temporary storage file 420, and an index tree 430 constructed by statistical information of the persistent storage file 450; the target files whose statistical information intersects with the data query range are filtered out from the temporary storage file 420 and the persistent storage file 450 stored in the LSM storage system, and the target data can be read from the filtered target files.

[0092] In an exemplary embodiment of the present specification, a device capable of implementing the above method is also provided.

[0093] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, an internal memory 508, and a non-volatile memory 510, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 509 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0094] Please refer to Figure 6 In a software implementation, a data writing device is provided, which is applied to a storage system, and the device includes:

[0095] A first writing module 610, configured to write the data to be written into a temporary storage file in the memory in response to a write instruction corresponding to the data to be written;

[0096] A statistical information module 620 is used to determine whether the amount of data to be written in the temporary storage file reaches a preset amount threshold;

[0097] If yes, statistics are collected for the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes a value range of the data to be written;

[0098] The second writing module 630 writes the statistical information and the data to be written in the temporary storage file into the persistent storage carried by the storage system, so as to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system.

[0099] Optionally, also include:

[0100] Call the query index generation module 640 to use the statistical information corresponding to the temporary storage file as the query index corresponding to the temporary storage file, and save it in the memory; use the statistical information in the multiple persistent storage files stored in the persistent storage as the query index corresponding to the persistent storage file, build an index tree corresponding to the multiple persistent storage files, and save it in the memory.

[0101] Optionally, also include:

[0102] The deletion module 650 is used to delete the statistical information corresponding to the temporary storage file stored in the memory after writing the statistical information corresponding to the temporary storage file and the data to be written in the temporary storage file to the mounted persistent storage.

[0103] Optionally, the statistical information module 620 further includes:

[0104] The grouping statistics module 622 is used to group the data to be written stored in the temporary storage file according to the key value corresponding to the data to be written; and to perform statistics on the data in at least part of the groups obtained by the grouping to obtain statistical information corresponding to the temporary storage file.

[0105] Optionally, the statistical information module 620 further includes:

[0106] The primary key statistics module 624 is used to collect statistics on the data in the group corresponding to the primary key of the data to be written in each group obtained by grouping, so as to obtain statistical information corresponding to the temporary storage file.

[0107] Optionally, also include:

[0108] The data reading module 660 is used to obtain a read instruction for the target data to be read; wherein the read instruction includes a data query range;

[0109] In response to the read instruction, the query index stored in the memory of the storage system is traversed, target files whose statistical information intersects with the data query range are filtered out from temporary storage files and persistent storage files stored in the storage system, and the target data is read from the filtered target files.

[0110] Optionally, the data reading module 660 includes:

[0111] The strictly incremental module 662 is used to match the query range with the statistical information corresponding to the persistent storage file stored in the persistent storage latest, and determine whether the file stored before it is within the query range; if not, determine that the persistent storage file stored in the persistent storage latest is the target data file.

[0112] Optional, including:

[0113] The statistical information also includes at least one of the maximum value, minimum value, and file sequence number of the data to be written.

[0114] Please refer to Figure 7 In a software implementation, a data reading device is provided, which is applied to a storage system, and the device includes:

[0115] The data reading module 710 is used to obtain a read instruction for the target data to be read; wherein the read instruction includes a data query range;

[0116] In response to the read instruction, the query index stored in the memory of the storage system is traversed, target files whose statistical information intersects with the data query range are filtered out from temporary storage files and persistent storage files stored in the storage system, and the target data is read from the filtered target files.

[0117] Optionally, the data reading module 710 includes:

[0118] The strictly incremental module 712 is used to match the query range with the statistical information corresponding to the persistent storage file stored in the persistent storage latest, and determine whether the file stored before it is within the query range; if not, determine that the persistent storage file stored in the persistent storage latest is the target data file.

[0119] Optional, including:

[0120] The statistical information also includes at least one of the maximum value, minimum value, and file sequence number of the data to be written.

[0121] Optionally, also include:

[0122] A first writing module 720, configured to write the data to be written into a temporary storage file in the memory in response to a write instruction corresponding to the data to be written;

[0123] A statistical information module 730 is used to determine whether the amount of data to be written in the temporary storage file reaches a preset amount threshold;

[0124] If yes, statistics are collected for the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes a value range of the data to be written;

[0125] The second writing module 740 writes the statistical information and the data to be written in the temporary storage file into the persistent storage carried by the storage system, so as to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system.

[0126] Optionally, also include:

[0127] Call the query index generation module 750, which is used to use the statistical information corresponding to the temporary storage file as the query index corresponding to the temporary storage file, and save it in the memory; use the statistical information in the multiple persistent storage files stored in the persistent storage as the query index corresponding to the persistent storage file, build an index tree corresponding to the multiple persistent storage files, and save it in the memory.

[0128] Optionally, also include:

[0129] The deletion module 760 is used to delete the statistical information corresponding to the temporary storage file stored in the memory after writing the statistical information corresponding to the temporary storage file and the data to be written in the temporary storage file to the mounted persistent storage.

[0130] Optionally, the statistical information module 730 further includes:

[0131] The grouping statistics module 732 is used to group the data to be written stored in the temporary storage file according to the key value corresponding to the data to be written; and to perform statistics on the data in at least part of the groups obtained by the grouping to obtain statistical information corresponding to the temporary storage file.

[0132] Optionally, the statistical information module 730 further includes:

[0133] The primary key statistics module 734 is used to collect statistics on the data in the group corresponding to the primary key of the data to be written in each group obtained by grouping, so as to obtain statistical information corresponding to the temporary storage file.

[0134] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0135] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0136] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0137] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0138] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0139] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0140] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0141] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0142] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A data writing method, applied to a storage system, comprising: In response to a write instruction corresponding to the data to be written, writing the data to be written into a temporary storage file in the memory; Determine whether the amount of data to be written in the temporary storage file reaches a preset amount threshold; If yes, statistics are collected for the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes a value range of the data to be written; Writing the statistical information and the data to be written in the temporary storage file into the persistent storage carried by the storage system, so as to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system; The statistical information corresponding to the temporary storage file is used as a query index corresponding to the temporary storage file and is saved in the memory; the statistical information in the multiple persistent storage files stored in the persistent storage is used as a query index corresponding to the persistent storage file to construct an index tree corresponding to the multiple persistent storage files and save it in the memory.

2. The method according to claim 1, further comprising: After the statistical information corresponding to the temporary storage file and the data to be written in the temporary storage file are written to the mounted persistent storage, the statistical information corresponding to the temporary storage file stored in the memory is deleted.

3. According to the method of claim 1, the step of performing statistics on the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file comprises: Grouping the data to be written stored in the temporary storage file according to the key values ​​corresponding to the data to be written; Statistics are performed on data in at least part of the groups obtained by grouping to obtain statistical information corresponding to the temporary storage file.

4. The method according to claim 3, wherein the key value comprises a primary key; The collecting statistics of data in at least part of the groups obtained by grouping includes: Statistics are performed on the data in the group corresponding to the primary key of the data to be written in each group obtained by grouping, so as to obtain statistical information corresponding to the temporary storage file.

5. The method according to claim 1, further comprising: Obtaining a read instruction for target data to be read; wherein the read instruction includes a data query range; In response to the read instruction, the query index stored in the memory of the storage system is traversed, target files whose statistical information intersects with the data query range are filtered out from temporary storage files and persistent storage files stored in the storage system, and the target data is read from the filtered target files.

6. The method according to claim 5, in a scenario where the value of the statistical information is strictly increasing, the method further comprises: Matching the query range with the statistical information corresponding to the persistent storage file stored in the persistent storage at the latest, and determining whether the file stored before it is within the query range; If not, then the persistent storage file that is stored in the persistent storage most recently is determined to be the target data file.

7. The method of claim 1, wherein the storage system comprises an LSM storage system.

8. A data reading method, applied to a storage system, the method comprising: Obtaining a read instruction for target data to be read; wherein the read instruction includes a data query range; In response to the read instruction, the query index stored in the memory of the storage system is traversed, target files whose statistical information intersects with the data query range are screened out from the temporary storage files and persistent storage files stored in the storage system, and the target data is read from the screened out target files; wherein the query index includes statistical information obtained by performing statistics on the data to be written stored in the temporary storage files in the current memory, and an index tree constructed based on statistical information obtained by performing statistics on the data to be written stored in the temporary storage files corresponding to the persistent storage files; the statistical information includes the value range of the corresponding data to be written.

9. The method according to claim 8, in a scenario where the value of the statistical information is strictly increasing, the method further comprises: Matching the query range with the statistical information corresponding to the persistent storage file stored in the persistent storage at the latest, and determining whether the file stored before it is within the query range; If not, then the persistent storage file that is stored in the persistent storage most recently is determined to be the target data file.

10. A data writing device, applied to a storage system, comprising: A first writing module, configured to write the data to be written into a temporary storage file in the memory in response to a write instruction corresponding to the data to be written; A statistical information module is used to determine whether the amount of data to be written in the temporary storage file reaches a preset amount threshold; If yes, statistics are collected for the data to be written stored in the temporary storage file to obtain statistical information corresponding to the temporary storage file; wherein the statistical information includes a value range of the data to be written; A second writing module writes the statistical information and the data to be written in the temporary storage file into a persistent storage carried by the storage system, so as to generate a persistent storage file corresponding to the temporary storage file in the persistent storage; wherein the statistical information is used as a query index when performing a range query on the data stored in the storage system; A query index generation module is called to use the statistical information corresponding to the temporary storage file as the query index corresponding to the temporary storage file, and save it in the memory; the statistical information in the multiple persistent storage files stored in the persistent storage is used as the query index corresponding to the persistent storage file, to construct an index tree corresponding to the multiple persistent storage files, and save it in the memory.

11. A data reading device, applied to a storage system, comprising: An instruction acquisition module, used to acquire a read instruction for target data to be read; wherein the read instruction includes a data query range; A target file screening module is used to respond to the read instruction, traverse the query index stored in the memory of the storage system, screen out target files whose statistical information intersects with the data query range from the temporary storage files and persistent storage files stored in the storage system, and read the target data from the screened target files; wherein the query index includes statistical information obtained by counting the data to be written stored in the temporary storage files in the current memory, and an index tree constructed based on statistical information obtained by counting the data to be written stored in the temporary storage files corresponding to the persistent storage files; the statistical information includes the value range of the corresponding data to be written.

12. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method as described in any one of claims 1-7 or 8-9 by running the executable instructions.

13. A machine-readable storage medium having machine-readable instructions stored thereon, wherein the instructions, when executed by a processor, implement the method according to any one of claims 1-7 or 8-9.

Images