A data management method, device, computing equipment and storage medium

Abstract

The present application relates to the technical field of database, in particular to a kind of data management method, device, computing device and storage medium. Among them, method includes: obtaining several documents to be stored;Each document is processed to generate index list in the corresponding index, and each document is stored in the corresponding shard of the index;When receiving query request, each node in the index will query request be issued to each shard managed by the node;Each shard queries each document stored in the shard according to the index list and the query request to generate the result set of the shard;All the generated result sets are summarized and sorted to obtain the query result. The present application can improve the query speed of massive data, and can meet the management application requirements of hundreds of millions of data.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of database technology, and in particular to a data management method, apparatus, computing device and storage medium. Background Technology

[0002] In simulation technology experiments, a single experiment can generate hundreds of millions of data points. These data, generated by the simulation software, need to be stored in a database, and the data content needs to be quickly retrieved and accessed. However, due to the massive amount of data, traditional database management methods cannot meet the requirements.

[0003] Therefore, a new data management method is urgently needed. Summary of the Invention

[0004] To address the problem that traditional database management methods cannot meet the requirements for rapid querying and reading of massive amounts of data, embodiments of the present invention provide a data management method, apparatus, computing device, and storage medium.

[0005] In a first aspect, embodiments of the present invention provide a data management method, including:

[0006] Retrieve several documents to be stored;

[0007] Each document is segmented into words to generate an index list in the corresponding index, and each document is stored in the corresponding segment of the index.

[0008] When a query request is received, each node in the index distributes the query request to each shard managed by that node.

[0009] Each of the shards queries each of the documents stored in that shard according to the index list and the query request to generate the result set of that shard;

[0010] Summarize and sort all the generated result sets to obtain the query results.

[0011] Preferably, the index list includes several word segments, the document ID corresponding to each word segment, the position information of each word segment in the corresponding document, and the weight ratio of each word segment in the corresponding document.

[0012] Preferably, the step of performing word segmentation on each document to generate an index list in the corresponding index, and storing each document in the corresponding segment of the index, includes:

[0013] Each document is written into memory sequentially to generate a translation file that gradually increases in size in the corresponding index;

[0014] As the document is continuously written, the document written within that period is segmented into words at regular intervals. After establishing an index relationship in the index list based on the segmentation results, the document written within that period is divided into segments.

[0015] After a set time, a disk write operation is performed to write each segment in the translated file to the corresponding slice of the index.

[0016] Preferred options also include:

[0017] Between each two adjacent disk write operations, all operations between those two disk write operations are recorded so that the progress can be restored based on the records after a failure.

[0018] Preferably, after performing a disk write operation to write each segment of the translation file into the corresponding shard of the index, the method further includes merging each segment in the corresponding shard.

[0019] Preferably, each of the aforementioned segments corresponds to a deleted file;

[0020] When a deletion request is received, the document is marked as deleted in the deletion file until the multiple segments are merged, at which point the document is deleted from the corresponding segment.

[0021] Preferably, each of the shards queries each document stored in that shard according to the index list and the query request to generate a result set for that shard, including:

[0022] For each shard, execute:

[0023] Based on the query request, determine the target word segmentation in the query request;

[0024] Based on the target word segmentation and the index list, determine the document ID corresponding to the target word segmentation;

[0025] Based on the document ID corresponding to the target word, query each document stored in the segment to determine the target document corresponding to the target word in the segment, and sort the target documents queried in the segment according to the weight ratio of the target word in each target document.

[0026] Based on the predetermined return sort number, the result set of the segment is generated using the target document with the corresponding sort number.

[0027] Secondly, embodiments of the present invention also provide a data management device, comprising:

[0028] The retrieval unit is used to retrieve several documents to be stored.

[0029] A storage unit is used to perform word segmentation on each document to generate an index list in the corresponding index, and to store each document in the corresponding segment of the index;

[0030] The distribution unit is used so that when a query request is received, each node in the index distributes the query request to each shard managed by that node.

[0031] The query unit is configured to enable each of the shards to query each of the documents stored in the shard according to the index list and the query request, so as to generate the result set of the shard;

[0032] The summary unit is used to summarize and sort all the generated result sets to obtain the query results.

[0033] Thirdly, embodiments of the present invention also provide a computing device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of this specification.

[0034] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.

[0035] This invention provides a data management method, apparatus, computing device, and storage medium. First, each document to be stored is segmented to generate an index list in the corresponding index, and each document is stored in the corresponding shard of that index. Then, when a query request is received, each node in the index sends the query request to each shard managed by that node. Next, each shard queries each document stored in that shard according to the index list and the query request, generating a result set for that shard. Finally, all generated result sets are summarized and sorted to obtain the final query result, thereby improving the query speed for massive amounts of data and meeting the management requirements of hundreds of millions of data entries. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1This is a flowchart of a data management method provided in an embodiment of the present invention;

[0038] Figure 2 This is a hardware architecture diagram of an electronic device provided in an embodiment of the present invention;

[0039] Figure 3 This is a structural diagram of a data management device provided in an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0041] As mentioned earlier, in simulation technology experiments, a single experiment can generate hundreds of millions of data points. These data are generated by the simulation software and need to be stored in a database to enable rapid querying and retrieval of the data content. However, due to the massive amount of data, traditional database management methods cannot meet the requirements.

[0042] To solve the above technical problems, the inventors may consider performing word segmentation on each document to be stored, and then distributing each document to different segments. When a query request is received, each segment can query the documents stored in its own segment, thereby improving the query speed.

[0043] The following describes the specific implementation of the above concept.

[0044] Please refer to Figure 1 This invention provides a data management method, which includes:

[0045] Step 100: Obtain several documents to be stored;

[0046] Step 102: Perform word segmentation on each document to generate an index list in the corresponding index, and store each document in the corresponding segment of the index;

[0047] Step 104: When a query request is received, each node in the index will distribute the query request to each shard managed by that node;

[0048] Step 106: Each shard queries each document stored in that shard based on the index list and the query request to generate the result set for that shard;

[0049] Step 108: Summarize and sort all the generated result sets to obtain the query results.

[0050] In this embodiment of the invention, firstly, each document to be stored is segmented to generate an index list in the corresponding index, and each document is stored in the corresponding shard of the index. Then, when a query request is received, each node in the index sends the query request to each shard managed by that node. Next, each shard queries each document stored in that shard according to the index list and the query request, generating a result set for that shard. Finally, all the generated result sets are summarized and sorted to obtain the final query result, thereby improving the query speed of massive amounts of data and meeting the management application requirements of hundreds of millions of data entries.

[0051] The following description Figure 1 The execution method of each step is shown.

[0052] For steps 100 and 102:

[0053] In this embodiment, the entire solution is implemented using Java programming language and can run on a PC. The Elasticsearch search server is used for document storage and retrieval.

[0054] In some implementations, the index list includes several word segments, the document ID corresponding to each word segment, the position information of each word segment in the corresponding document, and the weight ratio of each word segment in the corresponding document.

[0055] In this embodiment, each document corresponds to an ID. The inverted index segments each document according to a specified syntax and then maintains an index list that lists all the segments appearing in the documents, the document ID corresponding to each segment, the position information of each segment in the document, and the weight ratio of each segment in the corresponding document, i.e., the frequency of occurrence. The index list is a specific storage form for implementing the "segmentation-document matrix".

[0056] In some implementations, step 102 may include steps S1-S3:

[0057] Step S1: Write each document into memory sequentially to generate a translation file that gradually increases in size in the corresponding index;

[0058] Step S2: As documents are continuously written, the documents written within that period are segmented into words at regular intervals. After establishing an index relationship in the index list based on the segmentation results, the documents written within that period are divided into segments.

[0059] Step S3: After setting the time, perform a disk write operation to write each segment in the translation file to the corresponding fragment of the index.

[0060] In this embodiment, in step S1, each document is first written sequentially into a translog file (transaction.log) in memory. The translog file grows larger as documents are continuously written. At this time, if a query request is issued, these new documents cannot be indexed. In step S2, in this embodiment, the time interval is set to 1 second. Every 1 second, the documents written within this 1 second are segmented, and an index relationship between the documents within this 1 second and each segment is established in the index list. Then, the documents within this 1 second are written into the file system cache of the translog file, and a segment is generated in the file system cache. At this time, the documents in the segment can be searched, but they have not yet been written to the hard disk and may be lost due to a system crash. In step S3, since new documents are continuously written, steps S1 and S2 will be repeatedly executed, continuously generating new segment files, and the translog file will become larger and larger. Therefore, at a set time or when the translog file reaches a certain storage size, a disk write operation is performed to write the segment in the file system cache of the translog file to the corresponding fragment of the index on the disk, and the translog will be deleted (a new translog will be generated thereafter).

[0061] It's important to note that an index contains multiple nodes that manage the shards within that index. Documents are stored in each shard, and each shard is assigned to a specific node. There are different types of nodes, one of which is a coordinating node. This coordinating node determines which shard each translog file should be written to based on the amount of documents in the shards managed by each node, thus balancing the load across the nodes. When nodes are added or removed, the coordinating node migrates shards between nodes to ensure a more even distribution of document data.

[0062] In some implementations, it also includes:

[0063] Between each two adjacent disk write operations, all operations between those two disk write operations are recorded so that the progress can be restored based on the records after a failure.

[0064] In this embodiment, a translog file is introduced to record all operations between each two adjacent disk write operations, so that when the machine recovers from a failure or restarts, it can be restored based on the translog file.

[0065] In some implementations, after performing a disk write operation to write each segment of the translation file to the corresponding shard of the index, the method further includes merging each segment in the corresponding shard.

[0066] In this embodiment, since new segment files are constantly being generated, when querying each segment, all segments in that segment are queried in turn, which greatly affects the search performance. Therefore, every fixed time interval, each segment is merged into a new larger segment, and all merged old segments are cleared.

[0067] In some implementations, each fragment corresponds to a deleted file;

[0068] When a deletion request is received, the document is marked as deleted in the deletion file until multiple segments are merged, at which point the document is deleted from the corresponding segment.

[0069] In this embodiment, the index list cannot be modified; therefore, update and delete operations are not performed directly on the original index list. Each segment on each shard maintains a deletion file, which records deleted documents. Whenever a user initiates a delete request, the document is not actually deleted, and the index list remains unchanged. Instead, the deletion file marks the document as deleted. Therefore, deleted documents can still be retrieved, but they are filtered out in the returned results. Only when the segment merging process is initiated are those documents marked as deleted truly deleted.

[0070] When updating a document, the original document is first located to obtain its version number. Then, the updated document is written into memory, which means a new document is written, while the old document is marked as deleted.

[0071] Regarding step 104:

[0072] In step 104, when a node receives a query request, that node becomes the coordinating node. The coordinating node distributes the query request to every other node, so that each node can distribute the query request to every shard it manages.

[0073] Regarding step 106:

[0074] In some implementations, step 106 may include:

[0075] For each shard, execute:

[0076] Based on the query request, determine the target word segmentation in the query request;

[0077] Based on the target word segmentation and the index list, determine the document ID corresponding to the target word segmentation;

[0078] Based on the document ID corresponding to the target word, query each document stored in the segment to determine the target document corresponding to the target word in the segment, and sort the target documents queried in the segment according to the weight ratio of the target word in each target document.

[0079] Based on the predetermined return sort number, the result set of the segment is generated using the target document with the corresponding sort number.

[0080] In this embodiment, each shard is a Lucene instance, and the shard itself is a complete search engine. Therefore, for each shard, upon receiving a query request, the document ID corresponding to the target word in the index list is determined based on the target word in the query request. The shard then queries each document stored within it to determine the target document corresponding to the target word in that shard, and sorts the retrieved target documents in that shard according to the weight ratio of the target word in each target document. The client can request a result set with a number of elements starting from the "from" position and a length of "size". In this case, the shard creates a priority queue of size "from + size", places the information of the target documents corresponding to the sorted number in the priority queue, and thus generates the result set for that shard.

[0081] Regarding step 108:

[0082] In this step, each shard returns the result set obtained in step 106 to the coordinating node. The coordinating node summarizes all result sets and sorts them according to the weight ratio of the target word in the target document in each result, thus obtaining the final query result.

[0083] After the query is completed, if the client still needs to return these target documents, the coordinating node will send a retrieval request to the shard containing the target document. The shard retrieves the corresponding target document and returns it to the coordinating node, which then returns the target document to the client.

[0084] Finally, the performance of the solution in this embodiment is compared with that of a traditional MySQL database, and the comparison results are shown in Table 1 below.

[0085] Table 1

[0086] mysql This plan Version 5.7 6.2.2 Data volume 10 million 10 million Query speed 6.011s 0.203s New speed 0.008s 0.253s Modify speed 13.016s 0.051s Deletion speed 12.215s 0.052s

[0087] As shown in Table 1 above, this solution achieves a query speed 30 times faster than MySQL with the same data volume. Modification and deletion speeds are also dozens of times faster than MySQL.

[0088] The solution in this embodiment is also compared with the performance of the traditional HBase database, and the comparison results are shown in Table 2 below.

[0089] Table 2

[0090]

[0091] As shown in Table 2 above, both this solution and the HBase database contain 270 million records. When performing a full table scan query, the traditional HBase query takes 90 times longer than the query time of this solution.

[0092] In summary, this solution can improve the query speed of massive amounts of data and meet the management and application requirements of hundreds of millions of data entries.

[0093] like Figure 2 , Figure 3 As shown, this embodiment of the invention provides a data management device. The device embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, as... Figure 2 The diagram shown is a hardware architecture diagram of a computing device containing a frame rate switching image compression device provided in an embodiment of the present invention. Except for... Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the computing device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 3 As shown, a device in a logical sense is formed by the CPU of the computing device in which it is located reading the corresponding computer program from the non-volatile memory into the memory for execution.

[0094] like Figure 3 As shown, this embodiment provides a data management device, including:

[0095] Acquisition unit 301 is used to acquire several documents to be stored;

[0096] Storage unit 302 is used to perform word segmentation on each document to generate an index list in the corresponding index and store each document in the corresponding segment of the index;

[0097] The distribution unit 303 is used to distribute the query request to each shard managed by the node in the index when a query request is received.

[0098] The query unit 304 is used to enable each shard to query each document stored in the shard according to the index list and the query request, so as to generate the result set of the shard;

[0099] Summary unit 305 is used to summarize and sort all the generated result sets to obtain the query results.

[0100] In one embodiment of the present invention, the index list in the storage unit 302 includes several word segments, the document ID corresponding to each word segment, the position information of each word segment in the corresponding document, and the weight ratio of each word segment in the corresponding document.

[0101] In one embodiment of the present invention, the storage unit 302 is used to perform:

[0102] Each document is written into memory sequentially to generate a progressively larger translation file in the corresponding index.

[0103] As documents are continuously written, the documents written within that period are segmented into words at regular intervals. After establishing an index relationship in the index list based on the segmentation results, the documents written within that period are divided into segments.

[0104] After a set time, a disk write operation is performed to write each segment of the translation file to the corresponding slice of the index.

[0105] In one embodiment of the present invention, the storage unit 302 is further configured to:

[0106] Between each two adjacent disk write operations, all operations between those two disk write operations are recorded so that the progress can be restored based on the records after a failure.

[0107] In one embodiment of the present invention, after performing a disk write operation to write each segment of the translation file into the corresponding slice of the index, the storage unit 302 is also used to merge each segment in the corresponding slice.

[0108] In one embodiment of the present invention, in the storage unit 302, each segment corresponds to a deleted file;

[0109] When a deletion request is received, the document is marked as deleted in the deletion file until multiple segments are merged, at which point the document is deleted from the corresponding segment.

[0110] In one embodiment of the present invention, the query unit 304 is used to perform:

[0111] For each shard, execute:

[0112] Based on the query request, determine the target word segmentation in the query request;

[0113] Based on the target word segmentation and the index list, determine the document ID corresponding to the target word segmentation;

[0114] Based on the document ID corresponding to the target word, query each document stored in the segment to determine the target document corresponding to the target word in the segment, and sort the target documents queried in the segment according to the weight ratio of the target word in each target document.

[0115] Based on the predetermined return sort number, the result set of the segment is generated using the target document with the corresponding sort number.

[0116] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0117] This invention also provides a computing device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a data management method according to any embodiment of this invention.

[0118] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform a data management method according to any embodiment of this invention.

[0119] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0120] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0121] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0122] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0123] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0124] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0125] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data management method, characterized in that, include: Retrieve several documents to be stored; Each document is segmented into words to generate an index list in the corresponding index, and each document is stored in the corresponding segment of the index. When a query request is received, each node in the index distributes the query request to each shard managed by that node. An index contains multiple nodes, which manage at least one shard in the index. There are several types of nodes, one of which is a coordinating node, which balances the load of each node based on the amount of documents in the shards managed by each node. When nodes are added or removed, the coordinating node will migrate shards between nodes to ensure a balanced distribution of document data. Each of the shards queries each of the documents stored in that shard according to the index list and the query request to generate the result set of that shard; Summarize and sort all generated result sets to obtain the query results; The step of performing word segmentation on each document to generate an index list in the corresponding index, and storing each document in the corresponding segment of the index, includes: Each document is sequentially written into a translation file in memory to generate a progressively larger translation file in the corresponding index; the documents in the translation file cannot be indexed. As the documents are continuously written, the documents written within that period are segmented into words at regular intervals. After establishing an index relationship in the index list based on the segmentation results, the documents written to the translation file within that period are written into the file system cache of the translation file, and a segment is generated in the file system cache. The documents in the file system cache can be indexed. After a set time, a disk write operation is performed to write each segment in the translation file to the corresponding slice of the index on the disk; After performing a disk write operation to write each segment of the translation file into the corresponding slice of the index, the method further includes merging each segment in the corresponding slice; Each of the aforementioned fragments corresponds to one deleted file; When a deletion request is received, the document is marked as deleted in the deletion file until the multiple segments are merged, at which point the document is deleted from the corresponding segment.

2. The method according to claim 1, characterized in that, The index list includes several word segments, the document ID corresponding to each word segment, the position information of each word segment in the corresponding document, and the weight ratio of each word segment in the corresponding document.

3. The method according to claim 1, characterized in that, Also includes: Between each two adjacent disk write operations, all operations between those two adjacent disk write operations are recorded so that the progress can be restored based on the records after a failure.

4. The method according to claim 2, characterized in that, Each of the shards queries each document stored in that shard according to the index list and the query request to generate a result set for that shard, including: For each shard, execute: Based on the query request, determine the target word segmentation in the query request; Based on the target word segmentation and the index list, determine the document ID corresponding to the target word segmentation; Based on the document ID corresponding to the target word, query each document stored in the segment to determine the target document corresponding to the target word in the segment, and sort the target documents queried in the segment according to the weight ratio of the target word in each target document. Based on the predetermined return sort number, the result set of the segment is generated using the target document with the corresponding sort number.

5. A data management apparatus for implementing the method as described in any one of claims 1-4, characterized in that, include: The retrieval unit is used to retrieve several documents to be stored. A storage unit is used to perform word segmentation on each document to generate an index list in the corresponding index, and to store each document in the corresponding segment of the index; The distribution unit is used so that when a query request is received, each node in the index distributes the query request to each shard managed by that node. The query unit is configured to enable each of the shards to query each of the documents stored in the shard according to the index list and the query request, so as to generate the result set of the shard; The summary unit is used to summarize and sort all the generated result sets to obtain the query results.

6. A computing device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-4.

Images