Data storage method, device, equipment and computer readable storage medium

Abstract

The application relates to a data storage method, device and equipment and a computer readable storage medium, which comprises the following steps: receiving a data write request, wherein the data write request carries to-be-stored data; allocating the to-be-stored data to at least one shard to obtain at least one data shard; generating index data and storage data corresponding to each data shard; adding the index data corresponding to each data shard to a Lucene storage and adding the storage data corresponding to each data shard to a Rocksdb storage to generate Lucene data and Rocksdb data of each data shard; respectively replicating the Lucene data and the Rocksdb data of each data shard based on a raft protocol to obtain multiple copy data of each data shard; and respectively storing the multiple copy data of each data shard on multiple storage nodes.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data storage method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the development of technologies such as the Internet, the Internet of Things, and big data, the volume of data is exploding, placing higher demands on the performance, scalability, and reliability of data storage. Traditional single-machine storage systems can no longer meet these needs, making distributed storage systems a superior solution. Distributed storage systems can distribute data across multiple nodes, achieving high availability, load balancing, and fault tolerance. Furthermore, storage capacity and performance can be dynamically adjusted by adding or removing nodes.

[0003] Solving the data consistency problem in local storage and implementing distributed storage is an urgent issue that needs to be addressed. Summary of the Invention

[0004] This application provides a data storage method, apparatus, device, and computer-readable storage medium that can solve the data consistency problem in localized storage, thereby realizing localized distributed storage.

[0005] In a first aspect, this application provides a data storage method, comprising: receiving a data write request, the data write request carrying data to be stored, the data write request being used to request that the data to be stored be written to local storage; allocating the data to be stored to at least one data shard, thereby obtaining at least one data shard; generating index data and storage data corresponding to each data shard, wherein the index data corresponding to the target data shard includes an index of each data item in the target data shard and an identifier of the data corresponding to each index, and the storage data corresponding to the target data shard includes each data item in the target data shard and an identifier of each data item, wherein the target data shard is the at least one Any one of the data shards; add the index data corresponding to each data shard to Lucene storage, and add the storage data corresponding to each data shard to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard. The Lucene data and Rocksdb data of each data shard correspond to the same data in the data to be stored; based on the Raft protocol, replicate the Lucene data and Rocksdb data of each data shard to obtain multiple replica data of each data shard; store the multiple replica data of each data shard on multiple storage nodes.

[0006] In some embodiments of this application, allocating the data to be stored to at least one shard to obtain at least one data shard includes: calculating the consistent hash value of the identifier of each data in the data to be stored based on a consistent hash algorithm; allocating data with the same consistent hash value in the data to be stored to the same shard to obtain at least one shard data.

[0007] In some embodiments of this application, adding the index data corresponding to each data shard to Lucene storage and adding the storage data corresponding to each data shard to Rocksdb storage includes: adding the index data corresponding to each data shard to Lucene storage and adding the storage data corresponding to each data shard to Rocksdb storage based on a two-phase commit protocol.

[0008] In some embodiments of this application, the data to be stored is batch data; the process of adding the index data corresponding to each data shard to Lucene storage and adding the storage data corresponding to each data shard to Rocksdb storage based on a two-phase commit protocol includes: in the first phase of the two-phase commit protocol, adding the index data corresponding to each data shard to a temporary index directory and adding the storage data corresponding to each data shard to an SST file; in the second phase of the two-phase commit protocol, adding the index data corresponding to each data shard from the temporary index directory to the Lucene storage and adding the index data corresponding to each data shard from the SST file to the Rocksdb storage.

[0009] In some embodiments of this application, the data to be stored is real-time data; the process of adding the index data corresponding to each data shard to Lucene storage and adding the storage data corresponding to each data shard to Rocksdb storage based on a two-phase commit protocol includes: in the first phase of the two-phase commit protocol, adding the index data corresponding to each data shard to a temporary memory space and adding the storage data corresponding to each data shard to the temporary memory space; in the second phase of the two-phase commit protocol, adding the index data corresponding to each data shard from the temporary memory space to the Lucene storage and adding the index data corresponding to each data shard from the temporary memory space to the Rocksdb storage.

[0010] In some embodiments of this application, the process of replicating the Lucene data and Rocksdb data of each data shard based on the Raft protocol to obtain multiple replicas of each data shard includes: generating a RowLog corresponding to the target data shard from the Lucene data and Rocksdb data of the target data shard, wherein the RowLog carries the target data shard; and replicating the RowLog based on the Raft protocol to obtain multiple replicas of the RowLog, thereby obtaining multiple replicas of each data shard.

[0011] In some embodiments of this application, the method further includes: receiving a data retrieval request, the data retrieval request carrying retrieval conditions, the data retrieval request being used to request the output of data in the local storage that matches the retrieval conditions; extracting a target index matching the retrieval conditions from Lucene data on the plurality of storage nodes, the Lucene data on the plurality of storage nodes including an index of each piece of data in the local storage and an identifier of the data corresponding to each index; obtaining a target identifier of the data corresponding to the target index from the Lucene data on the plurality of storage nodes; reading the target data corresponding to the target identifier from Rocksdb data on the plurality of storage nodes, the Rocksdb data on the plurality of storage nodes including each piece of data in the local storage and an identifier of each piece of data; and outputting the target data.

[0012] Secondly, this application provides a data storage device, comprising: a receiving module for receiving a data write request, the data write request carrying data to be stored, the data write request being used to request that the data to be stored be written to local storage; an allocation module for allocating the data to be stored to at least one shard, thereby obtaining at least one data shard; and a generation module for generating index data and storage data corresponding to each data shard, wherein the index data corresponding to the target data shard includes an index of each data entry in the target data shard and an identifier of the data corresponding to each index, and the storage data corresponding to the target data shard includes each data entry in the target data shard and an identifier of each data entry, the target data shard being... The system includes at least one data shard; it adds the index data corresponding to each data shard to Lucene storage and adds the storage data corresponding to each data shard to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard, with the Lucene data and Rocksdb data of each data shard corresponding to the same data in the data to be stored; a replication module is used to replicate the Lucene data and Rocksdb data of each data shard according to the Raft protocol to obtain multiple replica data of each data shard; and a storage module is used to store the multiple replica data of each data shard on multiple storage nodes.

[0013] In some embodiments of this application, the allocation module is specifically used to calculate the consistent hash value of the identifier of each data in the data to be stored based on the consistent hash algorithm; and to allocate data with the same consistent hash value in the data to be stored to the same shard to obtain at least one shard of data.

[0014] In some embodiments of this application, the generation module is specifically used to add the index data corresponding to each data shard to the Lucene storage and the storage data corresponding to each data shard to the Rocksdb storage based on a two-phase commit protocol.

[0015] In some embodiments of this application, the data to be stored is batch data; the generation module is specifically used to add the index data corresponding to each data shard to a temporary index directory and add the storage data corresponding to each data shard to an SST file in the first stage of the two-phase commit protocol; in the second stage of the two-phase commit protocol, add the index data corresponding to each data shard from the temporary index directory to the Lucene storage and add the index data corresponding to each data shard from the SST file to the Rocksdb storage.

[0016] In some embodiments of this application, the data to be stored is real-time data; the generation module is specifically used to add the index data corresponding to each data shard to a temporary memory space and add the storage data corresponding to each data shard to the temporary memory space in the first stage of the two-phase commit protocol; and to add the index data corresponding to each data shard from the temporary memory space to the Lucene storage and add the index data corresponding to each data shard from the temporary memory space to the Rocksdb storage in the second stage of the two-phase commit protocol.

[0017] In some embodiments of this application, the replication module is specifically used to generate RowLog logs corresponding to the target data shard from the Lucene data and Rocksdb data of the target data shard. The RowLog logs carry the target data shards. Based on the Raft protocol, the RowLog logs are replicated to obtain multiple copies of the RowLog logs, thereby obtaining multiple copies of the data shards.

[0018] In some embodiments of this application, the receiving module is further configured to receive a data retrieval request, the data retrieval request carrying retrieval conditions, the data retrieval request being used to request the output of data in the local storage that matches the retrieval conditions; the apparatus further includes: an extraction module, configured to extract a target index matching the retrieval conditions from Lucene data on the plurality of storage nodes, the Lucene data on the plurality of storage nodes including an index of each piece of data in the local storage, and an identifier of the data corresponding to each index; an acquisition module, configured to acquire a target identifier of the data corresponding to the target index from the Lucene data on the plurality of storage nodes; a reading module, configured to read the target data corresponding to the target identifier from Rocksdb data on the plurality of storage nodes, the Rocksdb data on the plurality of storage nodes including each piece of data in the local storage, and an identifier of each piece of data; and an output module, configured to output the target data.

[0019] Thirdly, this application provides an electronic device, including: a processor, the processor being configured to execute a computer program stored in a memory, the computer program being executed by the processor to implement the steps of any of the data storage methods provided in the first aspect.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the data storage methods provided in the first aspect.

[0021] A fifth aspect of this application provides a computer program product, wherein the computer program product includes a computer program or instructions, and when the computer program product is run on a processor, the processor executes the computer program or instructions to implement the steps of the data storage method as described in the first aspect.

[0022] A sixth aspect of this application provides a chip including a processor, a memory, and a communication interface. The communication interface is coupled to the processor. The memory is used to store a program or instructions that can run on the processor. The processor is used to execute the program or instructions to implement the steps of the data storage method as described in the first aspect.

[0023] The technical solution provided in this application embodiment has the following advantages compared with the prior art: In this application embodiment, a data write request is received, the data write request carries the data to be stored, and the data write request is used to request that the data to be stored be written to local storage; the data to be stored is allocated to at least one shard to obtain at least one data shard; index data and storage data corresponding to each data shard are generated, the index data corresponding to the target data shard includes the index of each data in the target data shard and the identifier of the data corresponding to each index, and the storage data corresponding to the target data shard includes each data in the target data shard and the identifier of each data. The data shard is any one of the at least one data shards. The index data corresponding to each data shard is added to Lucene storage, and the storage data corresponding to each data shard is added to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard. The Lucene data and Rocksdb data of each data shard correspond to the same data in the data to be stored. Based on the Raft protocol, the Lucene data and Rocksdb data of each data shard are replicated to obtain multiple replicas of each data shard. These multiple replicas of each data shard are stored on multiple storage nodes. Thus, in this embodiment, flexible data retrieval and storage can be achieved based on Lucene and Rocksdb. Furthermore, based on the Raft protocol, data is synchronously replicated, enabling data redundancy through multiple replicas. By sharding the data, data consistency issues in local storage are resolved, achieving distributed data storage. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a data storage method provided in this application;

[0027] Figure 2 A schematic diagram of a data storage structure provided in this application;

[0028] Figure 3 A schematic diagram of the structure of a data storage device provided in this application;

[0029] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in this application. Detailed Implementation

[0030] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more.

[0033] Distributed storage systems also face challenges such as data consistency, data migration, data compression, and data search. To address these issues, efficient data management technologies are needed, such as consistency protocols, index structures, and storage engines. Currently, several mature open-source projects provide implementations of these technologies, such as Ratis, Lucene, and RocksDB.

[0034] This application provides a hybrid storage solution based on Ratis, Lucene, and RocksDB to achieve high-performance, highly scalable, and highly reliable data storage services. This solution leverages the respective advantages of these three open-source technologies—Ratis, Lucene, and RocksDB—to implement distributed replication, consistency, indexing, searching, storage, and access functions, thereby improving data availability, retrievability, analyzability, and compressibility. This solution also addresses the challenges of traditional distributed storage systems in areas such as data consistency, data migration, data compression, and data searching, enhancing the performance and efficiency of distributed storage systems. This solution can be applied to various data-intensive scenarios, such as the Internet, the Internet of Things, and big data, meeting diverse data storage needs.

[0035] Distributed storage systems also face challenges such as data consistency, data migration, data compression, and data search. To address these issues, efficient data management technologies are needed, such as consistency protocols, index structures, and storage engines. Currently, several mature open-source projects provide implementations of these technologies, such as Ratis, Lucene, and RocksDB.

[0036] Ratis is a distributed consistency framework based on the Raft protocol, providing strong consistency, high availability, and fault tolerance data replication services. Ratis supports various storage types, such as memory, files, and databases, and can be used as an underlying component of distributed storage systems to achieve data consistency and reliability.

[0037] Lucene is an open-source full-text search engine library that provides powerful indexing and search capabilities, as well as text analysis capabilities in multiple languages. Lucene can efficiently index and retrieve documents of any size, supports various query syntaxes and sorting rules, and can be applied to various search scenarios, such as website search, database search, and desktop search.

[0038] RocksDB is an embedded persistent key-value storage engine based on a log-structured merged tree (LSM-tree) that provides high-performance, low-latency, and high-compression data storage services. Utilizing the high read / write performance of flash memory and high-speed hard drives, RocksDB enables rapid data insertion, updating, and deletion. It also supports data snapshots, backups, compression, and filtering, and can serve as a storage layer in distributed storage systems for efficient data storage and access.

[0039] Hybrid storage based on Ratis, Lucene, and RocksDB is a distributed storage solution that combines these three technologies to provide a high-performance, highly scalable, and highly reliable data storage service.

[0040] In this embodiment, a data write request is received, carrying data to be stored, which requests that the data to be stored be written to local storage; the data to be stored is allocated to at least one shard to obtain at least one data shard; index data and storage data corresponding to each data shard are generated, wherein the index data corresponding to the target data shard includes the index of each data in the target data shard and the identifier of the data corresponding to each index, and the storage data corresponding to the target data shard includes each data in the target data shard and the identifier of each data, and the target data shard is any one of the at least one data shard; the index data corresponding to each data shard is added to Lucene storage, and the storage data corresponding to each data shard is added to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard, wherein the Lucene data and Rocksdb data of each data shard correspond to the same data in the data to be stored; based on the Raft protocol, the Lucene data and Rocksdb data of each data shard are replicated to obtain multiple replica data of each data shard; and the multiple replica data of each data shard are stored on multiple storage nodes. Thus, in this embodiment of the application, flexible data retrieval and storage can be achieved based on Lucene and Rocksdb, and data can be synchronously replicated based on the Raft protocol to achieve data redundancy with multiple copies. Furthermore, by slicing the data, the data consistency problem in local storage is resolved, thereby realizing distributed data storage.

[0041] This application is applied to data storage scenarios. The electronic devices in the embodiments of this application can be tablet computers, laptop computers, handheld computers, etc., which can be determined according to the actual situation and are not limited here.

[0042] The technical solution of this application will be explained in detail below through several specific embodiments.

[0043] Figure 1 A flowchart illustrating a data storage method provided in this application is shown below. Figure 1 As shown, the data storage method may include steps 101 to 106 as described below.

[0044] 101. Receive data write request.

[0045] The data write request carries the data to be stored, and is used to request that the data to be stored be written to local storage.

[0046] In some embodiments of this application, the data write request can be a data write request sent by the receiving client device, or a data write request input by the user through peripheral devices such as a keyboard, mouse, and microphone. The specific request can be determined according to the actual situation and is not limited here.

[0047] 102. Allocate the data to be stored to at least one shard to obtain at least one data shard.

[0048] In some embodiments of this application, the data to be stored can be sharded based on hash sharding. Specifically, the data to be stored can be sharded based on the key-value pairs of each data in the data to be stored and a consistent hash algorithm to obtain at least one data shard.

[0049] Here, the key-value pairs are key-value pairs, where each piece of data is identified by a key, and each piece of data itself is represented by a value.

[0050] For example, step 102 can be implemented by steps 102a and 102b below.

[0051] 102a. Based on the consistent hashing algorithm, calculate the consistent hash value of the identifier of each data in the data to be stored.

[0052] 102b. Assign data with the same consistent hash value in the data to be stored to the same shard, and obtain at least one shard of data.

[0053] In this embodiment, each piece of data in the data to be stored has a key-value mapping. Based on the consistent hashing algorithm, the hash value of the key of each piece of data is calculated. Assuming that the hash value calculated for a piece of data is 4, then this piece of data is mapped to data shard 4; assuming that the hash value calculated for a piece of data is 2, then this piece of data is mapped to data shard 2. Each piece of data on the same data shard has the same hash value.

[0054] In this embodiment of the application, the hash sharding method using key-value pairs and consistent hashing algorithm can quickly shard the data to be stored, thereby improving the storage efficiency of the data to be stored.

[0055] In some embodiments of this application, the data to be stored can also be fragmented based on range fragmentation, random fragmentation, etc. For details, please refer to the relevant technologies, which will not be elaborated here.

[0056] 103. Generate index data and storage data corresponding to each data shard.

[0057] The index data corresponding to the target data shard includes the index of each data in the target data shard and the identifier of the data corresponding to each index. The storage data corresponding to the target data shard includes each data in the target data shard and the identifier of each data. The target data shard is any one of the at least one data shard.

[0058] The index data corresponding to any data shard includes the index of each data record in that data shard and the identifier of the data record corresponding to each index. The storage data corresponding to any data shard includes each data record in that data shard and the identifier of each data record.

[0059] For example, the process of generating index data corresponding to a data shard may include: first generating an index for each piece of data in the data shard, then obtaining the identifier for each piece of data in the data shard, and finally establishing a correspondence between the index and the identifier for each piece of data to generate the index data corresponding to the data shard.

[0060] 104. Add the index data corresponding to each data shard to Lucene storage, and add the storage data corresponding to each data shard to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard.

[0061] In this data shard, the Lucene data and Rocksdb data in each data shard correspond to the same data in the data to be stored.

[0062] It is understandable that the stored data corresponding to each index in the Lucene data store of each data shard is the same data in the data to be stored as the stored data in the Rocksdb data store of each data shard.

[0063] In some embodiments of this application, step 104 can be specifically implemented by step 104a.

[0064] 104a. Based on the two-phase commit protocol, add the index data corresponding to each data shard to Lucene storage, and add the storage data corresponding to each data shard to Rocksdb storage.

[0065] Based on the two-phase commit (2PC) protocol, the index data and storage data corresponding to the data shards are added to the Lucene data and the Rocksdb data respectively. This ensures data consistency between the storage data and the index data.

[0066] Specifically, the process of adding the index data and stored data corresponding to the data shards to the Lucene data and the Rocksdb data respectively based on the two-phase commit protocol can be divided into two phases: the first phase is called the voting phase or pre-commit phase, and the second phase is called the execution phase or commit phase. In the first phase, the transaction control mechanisms of Rocksdb and Lucene each confirm whether the corresponding data can be added to their respective stored data. In the second phase, if both Rocksdb and Lucene confirm that the corresponding data can be added to their respective stored data, then the corresponding data will be added to their respective stored data; otherwise, if Rocksdb and / or Lucene cannot add the corresponding data to their own stored data, then neither Rocksdb nor Lucene will add the data.

[0067] Because storing data requires simultaneously enabling write methods in both RocksDB and Lucene, and leveraging their respective transaction control mechanisms, data is written to both components concurrently. If either the RocksDB or Lucene write fails, the Lucene data will become mismatched with the RocksDB data. In the above embodiment, a two-phase commit protocol is used to add the index data and storage data corresponding to the data shards to the Lucene data and the RocksDB data respectively, thus ensuring consistency between the Lucene and RocksDB data.

[0068] In some embodiments of this application, the data to be stored is batch data; the above step 104a can be specifically implemented through the following steps 104a1 and 104a2.

[0069] 104a1. In the first phase of the two-phase commit protocol, the index data corresponding to each data shard is added to the temporary index directory, and the storage data corresponding to each data shard is added to the SST file.

[0070] 104a2. In the second phase of the two-phase commit protocol, the index data corresponding to each data shard is added from the temporary index directory to the Lucene storage, and the index data corresponding to each data shard is added from the SST file to the Rocksdb storage.

[0071] In this embodiment of the application, for batch data storage, intermediate data is generated, such as an SST file generated by Rocksdb and a temporary index directory generated by Lucene. Then, the data is loaded separately. In order to prevent rollback after loading failure, a two-phase commit protocol is defined for control, which can achieve consistency of batch data.

[0072] In some embodiments of this application, the data to be stored is real-time data; step 104a above can be specifically implemented through steps 104a3 and 104a4 below.

[0073] 104a3. In the first phase of the two-phase commit protocol, the index data corresponding to each data shard is added to the temporary memory space, and the storage data corresponding to each data shard is added to the temporary memory space.

[0074] 104a4. In the second phase of the two-phase commit protocol, the index data corresponding to each data shard is added from the temporary memory space to the Lucene storage, and the index data corresponding to each data shard is added from the temporary memory space to the Rocksdb storage.

[0075] In this embodiment of the application, a temporary memory space can be allocated for storing real-time data to perform data transaction operations. Once the data transaction data is committed to the corresponding data storage, the consistency of real-time data can be achieved.

[0076] 105. Based on the Raft protocol, the Lucene data and Rocksdb data of each data shard are replicated to obtain multiple replicas of each data shard.

[0077] In some embodiments of this application, step 105 can be specifically implemented by the following steps 105a and 105b.

[0078] 105a. Generate RowLog logs corresponding to the target data shards from the Lucene data and Rocksdb data of the target data shard.

[0079] The RowLog contains the target data shard.

[0080] 105b. Based on the Raft protocol, the RowLog is replicated to obtain multiple copies of the RowLog data, thereby obtaining multiple copies of each data shard.

[0081] Among them, multiple replicas of data are usually an odd number of replicas, such as 3, 5, 7, etc. The specific number can be determined according to the actual situation, and there is no limit here.

[0082] In this embodiment, to ensure data consistency among multiple replicas, the hybrid storage defines a RowLog log format. Each field in the RowLog is defined as a Cell, which comes in various types, including int, long, double, string, date, timestamp, list, map, and 56 other types. Each type has its own serialization and deserialization methods. Data replica consistency is achieved by combining the RowLog with Ratis.

[0083] In this embodiment, data consistency between stored data and indexed data is achieved by using RowLog logging and defining a two-phase commit protocol. If the two-phase commit fails, a rollback is performed based on the log.

[0084] 106. Store the multiple replicas of each data shard on multiple storage nodes.

[0085] For example, such as Figure 2 As shown, the data to be stored is divided into three shards: shard 1, shard 2, and shard 3. Each shard is replicated into three copies, which are stored on four nodes: machine 1 (shardserver 1), machine 2 (shardserver 2), machine 3 (shardserver 3), and machine 4 (shardserver 4). These four nodes form a Raft group.

[0086] In this embodiment of the application, when there is a data write request, the data replication layer will allocate the data to one or more shards according to the data key-value pairs and consistent hashing algorithm, and replicate the data to multiple nodes where the shards are located through the Ratis Raft protocol, so as to achieve strong consistency and high availability of data.

[0087] In some embodiments of this application, the data storage method provided in this application may further include the following steps 107 to 111.

[0088] 107. Receive data retrieval requests.

[0089] The data retrieval request carries retrieval criteria, and is used to request the output of data in the local storage that matches the retrieval criteria.

[0090] Optionally, receiving a data retrieval request can be either receiving a data retrieval request sent by a terminal device via a network interface, or receiving a data retrieval request input by a user via peripheral devices such as a keyboard, mouse, microphone, or touch input module.

[0091] Since the actual function of the search conditions in this embodiment is to extract target indexes that match the search conditions from Lucene data, and Lucene provides a flexible query interface, the search conditions can be flexibly set according to user needs. For example, the search conditions can be set to a specified field. Another example is that the search conditions can be set to a combination of multiple query conditions.

[0092] 108. Extract the target index that matches the search criteria from the Lucene data on these multiple storage nodes.

[0093] The Lucene data on these multiple storage nodes includes the index of each piece of data in the local storage, as well as the identifier of the data corresponding to each index.

[0094] In this embodiment, Lucene data refers to data stored in Lucene, comprising two parts: an index and an identifier. The index includes the index of each piece of data in local storage, and the identifier includes the identifier of each piece of data in local storage. A unique correspondence exists between the index and the identifier of each piece of data. The content and data structure of the index can be the same as those of the index stored in Lucene in the prior art. Specifically, the identifier of the data can be information such as the data's identity document (ID).

[0095] In some embodiments of this application, prior to step S108 above, the data retrieval method provided in this application further includes: converting the retrieval conditions into retrieval conditions of Lucene's retrieval syntax.

[0096] That is, before step 108 above, it is first determined whether the search conditions are search conditions of Lucene's search syntax. If they are not search conditions of Lucene's search syntax, the search conditions are converted to search conditions of Lucene's search syntax, and then step 108 above is executed.

[0097] 109. Obtain the target identifier of the data corresponding to the target index from the Lucene data on the multiple storage nodes.

[0098] After obtaining the target index, obtain the identifier of the data corresponding to each target index one by one, and use the identifier of the data corresponding to each target index as the target identifier to execute subsequent steps.

[0099] 110. Read the target data corresponding to the target identifier from the Rocksdb data on the multiple storage nodes.

[0100] The Rocksdb data on these multiple storage nodes includes each piece of data in the local storage, as well as the identifier of each piece of data.

[0101] It is understood that the Rocksdb data in this application embodiment refers to the data stored in Rocksdb. Rocksdb is a key-value store that stores the identifier of each data item as the key and the corresponding data itself as the value. Since Rocksdb supports quickly finding the target value by key, the corresponding target data can be quickly located and read according to the target identifier in the above step 110.

[0102] 111. Output the target data.

[0103] In some embodiments of this application, the output target data may be target data sent to the terminal device through a network interface, target data written to a specified location on a specified disk, or target data that has been processed and displayed for visualization.

[0104] In this embodiment, upon receiving a data retrieval request, the target index matching the retrieval criteria can be extracted from the Lucene data first. Then, the identifier of the data corresponding to the target index is obtained based on the target index. Next, the target data corresponding to the target index is read from the Rocksdb data, and finally, the target data is output. Since the Lucene data only includes the data index and the data identifier, and does not need to store the data itself in local storage, the locally stored data will not consume excessive resources and significantly impact Lucene's performance. Therefore, this embodiment can support flexible setting of retrieval criteria and quickly retrieve the index of the target data matching the retrieval criteria. Furthermore, because Rocksdb can store a large amount of data and supports quickly locating and searching for target data from the stored data by identifier, this embodiment can also quickly read the target data from the Rocksdb data based on the identifier of the target data. Therefore, this embodiment can flexibly and quickly extract target data from local storage.

[0105] When a data retrieval request is received, the data replication layer locates the shard and node where the data is located based on the data's key-value pairs and consistent hashing algorithm. Then, it reads the data from the node through Ratis' Raft protocol, achieving fast data access and fault tolerance.

[0106] Suppose there is a distributed storage system consisting of three nodes, each running Ratis, Lucene, and RocksDB. The system's data replication layer, data indexing layer, and data storage layer are all based on a hybrid storage scheme of Ratis + Lucene + RocksDB, implemented as follows:

[0107] Data Replication Layer: This layer uses Ratis as the data replication framework, dividing the data into three shards. Each shard has one master node and two slave nodes, and each node stores a copy of the shard. When a data write request occurs, the data replication layer allocates the data to a shard based on the data's key-value pairs and a consistent hash algorithm. Then, using Ratis's Raft protocol, it replicates the data to the three nodes residing in the shard, achieving strong consistency and high availability. When a data read request occurs, the data replication layer locates the shard and node containing the data based on the data's key-value pairs and a consistent hash algorithm. Then, using Ratis's Raft protocol, it reads the data from the node, achieving fast data access and fault tolerance. When a data migration or expansion request occurs, the data replication layer, according to user instructions, uses Ratis's Raft protocol to migrate data from one shard to another, or increase or decrease the number of shards, achieving dynamic data migration and expansion.

[0108] Data Indexing Layer: This layer uses Lucene as the data indexing library to perform full-text indexing and searching of data, improving data retrievability and usability, while also supporting multi-dimensional data analysis and aggregation. When a data indexing request is received, the data indexing layer uses Lucene's text analyzer to perform word segmentation, standardization, filtering, and other processing based on the data's content and type, generating an inverted index and storing it in the data storage layer, thus achieving full-text indexing. When a data search request is received, the data indexing layer uses Lucene's query analyzer to perform word segmentation, standardization, filtering, and other processing based on the user's query statement, generating query objects, and then retrieving matching data from the data storage layer based on the inverted index, thus achieving full-text search. When a data analysis request is received, the data indexing layer uses Lucene's aggregation framework to perform grouping, sorting, and statistical operations on the data according to the user's analytical needs, and retrieves relevant data from the data storage layer, thus achieving multi-dimensional data analysis and aggregation.

[0109] Data Storage Layer: This layer uses RocksDB as the data storage engine for efficient data storage and access, improving data performance and compression ratio. It also supports data snapshots, backups, and compression. When a data storage request is received, the data storage layer uses RocksDB's LSM-tree structure based on the data's key-value pairs for efficient data storage, enabling fast data insertion, updates, and deletion. When a data access request is received, the data storage layer uses RocksDB's LSM-tree structure based on the data's key-value pairs for efficient data access, achieving low latency and high throughput. When a data compression request is received, the data storage layer uses RocksDB's compression algorithm based on the data type and characteristics to compress the data, achieving a high compression ratio and saving space. When a data snapshot request is received, the data storage layer uses RocksDB's snapshot mechanism to take a snapshot of the data at a user-specified point in time, enabling historical versioning and recovery capabilities. When a data backup request is received, the data storage layer uses RocksDB's backup mechanism to back up the data according to the user-specified target, enabling remote data storage and disaster recovery capabilities.

[0110] This application leverages the advantages of three open-source technologies: Ratis, Lucene, and RocksDB. Through deeper integration and binding of these three technologies, a synergistic effect greater than the sum of its parts (1+1>2) is achieved, enabling hybrid data storage and improving data availability, retrieval, and analyzability. This solution supports various data types and scenarios, such as text, images, video, audio, structured and unstructured data, and can be applied to various data-intensive scenarios, such as the Internet, the Internet of Things, and big data, meeting diverse data storage needs.

[0111] This application's embodiments leverage the respective advantages of three open-source technologies: Ratis, Lucene, and RocksDB. These technologies respectively implement functions such as distributed data replication, consistency, indexing, searching, storage, and access, improving data availability, retrieval, analyzability, and compressibility, thus meeting diverse data storage needs.

[0112] This application's embodiments separate data replication, indexing, and storage into three layers, achieving data decoupling and modularization, improving data flexibility and scalability, while also reducing data management complexity and cost.

[0113] This application supports various data types and scenarios, such as text, images, videos, audio, structured data, and unstructured data. It can be applied to various data-intensive scenarios, such as the Internet, the Internet of Things, and big data, thereby improving the performance and efficiency of distributed storage systems and enhancing the value and significance of data.

[0114] This application also provides a data storage device. Figure 3 A schematic diagram of a data storage device provided in this application is shown below. Figure 3 As shown, the data storage device includes: a receiving module 301, used to receive a data write request, the data write request carrying data to be stored, the data write request being used to request that the data to be stored be written to local storage; an allocation module 302, used to allocate the data to be stored to at least one shard, to obtain at least one data shard; and a generation module 303, used to generate index data and storage data corresponding to each data shard, wherein the index data corresponding to the target data shard includes the index of each data item in the target data shard and the identifier of the data corresponding to each index, and the storage data corresponding to the target data shard includes each data item in the target data shard and the identifier of each data item, and the target data shard is the at least one Any one of the data shards; add the index data corresponding to each data shard to Lucene storage, and add the storage data corresponding to each data shard to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard, with the Lucene data and Rocksdb data of each data shard corresponding to the same data in the data to be stored; replication module 304 is used to replicate the Lucene data and Rocksdb data of each data shard according to the Raft protocol to obtain multiple replica data of each data shard; storage module 305 is used to store the multiple replica data of each data shard on multiple storage nodes respectively.

[0115] In some embodiments of this application, the allocation module 302 is specifically used to calculate the consistent hash value of the identifier of each data in the data to be stored based on the consistent hash algorithm; and to allocate the data with the same consistent hash value in the data to be stored to the same shard to obtain at least one shard of data.

[0116] In some embodiments of this application, the generation module 303 is specifically used to add the index data corresponding to each data shard to the Lucene storage and add the storage data corresponding to each data shard to the Rocksdb storage based on the two-phase commit protocol.

[0117] In some embodiments of this application, the data to be stored is batch data; the generation module 303 is specifically used to add the index data corresponding to each data shard to a temporary index directory and add the storage data corresponding to each data shard to an SST file in the first stage of the two-phase commit protocol; in the second stage of the two-phase commit protocol, add the index data corresponding to each data shard from the temporary index directory to the Lucene storage and add the index data corresponding to each data shard from the SST file to the Rocksdb storage.

[0118] In some embodiments of this application, the data to be stored is real-time data; the generation module 303 is specifically used to add the index data corresponding to each data shard to the temporary memory space and add the storage data corresponding to each data shard to the temporary memory space in the first stage of the two-phase commit protocol; and to add the index data corresponding to each data shard from the temporary memory space to the Lucene storage and add the index data corresponding to each data shard from the temporary memory space to the Rocksdb storage in the second stage of the two-phase commit protocol.

[0119] In some embodiments of this application, the replication module 304 is specifically used to generate a RowLog corresponding to the target data shard from the Lucene data and Rocksdb data of the target data shard. The RowLog carries the target data shard. Based on the Raft protocol, the RowLog is replicated to obtain multiple copies of the RowLog, thereby obtaining multiple copies of the data shard.

[0120] In some embodiments of this application, the receiving module 301 is further configured to receive a data retrieval request, the data retrieval request carrying retrieval conditions, the data retrieval request being used to request the output of data in the local storage that matches the retrieval conditions; the apparatus further includes: an extraction module, configured to extract a target index matching the retrieval conditions from Lucene data on the plurality of storage nodes, the Lucene data on the plurality of storage nodes including an index of each piece of data in the local storage, and an identifier of the data corresponding to each index; an acquisition module, configured to acquire a target identifier of the data corresponding to the target index from the Lucene data on the plurality of storage nodes; a reading module, configured to read the target data corresponding to the target identifier from Rocksdb data on the plurality of storage nodes, the Rocksdb data on the plurality of storage nodes including each piece of data in the local storage, and an identifier of each piece of data; and an output module, configured to output the target data.

[0121] In this embodiment, each module can implement the data storage method provided in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0122] like Figure 4 As shown, this application embodiment also provides an electronic device 400, which can be the electronic device described above. The electronic device 400 includes: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the computer program is executed by the processor 401, it implements the various processes performed by the data storage method described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0123] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data storage method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0124] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0125] This application provides a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the above-described data storage method.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0127] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A data storage method, characterized in that, The method includes: Receive a data write request, the data write request carrying data to be stored, the data write request being used to request that the data to be stored be written to local storage; The data to be stored is allocated to at least one shard to obtain at least one data shard; Generate index data and storage data corresponding to each data shard. The index data corresponding to the target data shard includes the index of each data in the target data shard and the identifier of the data corresponding to each index. The storage data corresponding to the target data shard includes each data in the target data shard and the identifier of each data. The target data shard is any one of the at least one data shard. Add the index data corresponding to each data shard to Lucene storage, and add the storage data corresponding to each data shard to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard. The Lucene data and Rocksdb data for each data shard correspond to the same data in the data to be stored. The Lucene data and Rocksdb data of the target data shard are used to generate RowLog logs corresponding to the target data shards. The RowLog logs carry the target data shards. Each field of the RowLog log is defined as a Cell. The Cell includes multiple data types, and each data type has its own corresponding serialization and deserialization methods. The RowLog is used to store the commit status of the two-phase commit protocol. When the data commit of the two-phase commit protocol fails, a rollback is performed based on the RowLog. Based on the Raft protocol, the RowLog is replicated to obtain multiple copies of the RowLog, thereby obtaining multiple copies of each data shard. The multiple copies of each data shard are stored on multiple storage nodes respectively.

2. The method according to claim 1, characterized in that, The step of allocating the data to be stored to at least one shard to obtain at least one data shard includes: Based on the consistent hashing algorithm, calculate the consistent hash value of the identifier of each data in the data to be stored; Data with the same consistent hash value in the data to be stored are assigned to the same shard to obtain at least one shard of data.

3. The method according to claim 1, characterized in that, The step of adding the index data corresponding to each data shard to Lucene storage and adding the storage data corresponding to each data shard to Rocksdb storage includes: Based on the two-phase commit protocol, the index data corresponding to each data shard is added to Lucene storage, and the storage data corresponding to each data shard is added to Rocksdb storage.

4. The method according to claim 3, characterized in that, The data to be stored is batch data; the process of adding the index data corresponding to each data shard to Lucene storage and adding the storage data corresponding to each data shard to RocksDB storage based on the two-phase commit protocol includes: In the first phase of the two-phase commit protocol, the index data corresponding to each data fragment is added to the temporary index directory, and the storage data corresponding to each data fragment is added to the SST file. In the second phase of the two-phase commit protocol, the index data corresponding to each data shard is added from the temporary index directory to the Lucene storage, and the index data corresponding to each data shard is added from the SST file to the Rocksdb storage.

5. The method according to claim 3, characterized in that, The data to be stored is real-time data; the process of adding the index data corresponding to each data shard to Lucene storage and adding the storage data corresponding to each data shard to RocksDB storage based on the two-phase commit protocol includes: In the first phase of the two-phase commit protocol, the index data corresponding to each data shard is added to the temporary memory space, and the storage data corresponding to each data shard is added to the temporary memory space. In the second phase of the two-phase commit protocol, the index data corresponding to each data shard is added from the temporary memory space to the Lucene storage, and the index data corresponding to each data shard is added from the temporary memory space to the Rocksdb storage.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Receive a data retrieval request, the data retrieval request carrying retrieval conditions, the data retrieval request being used to request the output of data in the local storage that matches the retrieval conditions; Extract target indexes that match the search criteria from the Lucene data on the multiple storage nodes. The Lucene data on the multiple storage nodes includes the index of each piece of data in the local storage and the identifier of the data corresponding to each index. Obtain the target identifier of the data corresponding to the target index from the Lucene data on the multiple storage nodes; Read the target data corresponding to the target identifier from the Rocksdb data on the multiple storage nodes, wherein the Rocksdb data on the multiple storage nodes includes each piece of data in the local storage and the identifier of each piece of data; Output the target data.

7. A data storage device, characterized in that, The device includes: A receiving module is used to receive a data write request, the data write request carrying data to be stored, the data write request being used to request that the data to be stored be written to local storage. The allocation module is used to allocate the data to be stored to at least one shard, thereby obtaining at least one data shard. The generation module is used to generate index data and storage data corresponding to each data shard. The index data corresponding to the target data shard includes the index of each data in the target data shard and the identifier of the data corresponding to each index. The storage data corresponding to the target data shard includes each data in the target data shard and the identifier of each data. The target data shard is any one of the at least one data shard. Add the index data corresponding to each data shard to Lucene storage, and add the storage data corresponding to each data shard to Rocksdb storage to generate Lucene data and Rocksdb data for each data shard. The Lucene data and Rocksdb data for each data shard correspond to the same data in the data to be stored. The replication module is used to generate RowLog logs corresponding to the target data shards from the Lucene data and Rocksdb data of the target data shards. The RowLog logs carry the target data shards. Each field of the RowLog logs is defined as a Cell, and the Cells include multiple data types, each of which has its own serialization and deserialization methods. The RowLog is used to store the commit status of the two-phase commit protocol. When the data commit of the two-phase commit protocol fails, a rollback is performed based on the RowLog. Based on the Raft protocol, the RowLog is replicated to obtain multiple copies of the RowLog, thereby obtaining multiple copies of each data shard. The storage module is used to store multiple copies of the data shards on multiple storage nodes respectively.

8. An electronic device, characterized in that, include: A processor for executing a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the data storage method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data storage method according to any one of claims 1-6.

Images