A Method, Device, Equipment and Medium for Expanding a Full-Text Retrieval System

Abstract

An embodiment of the present invention discloses a method, device, equipment and medium for expanding a full-text retrieval system. The system includes an original shard for storing data and a text index corresponding to the data. The method includes: determining a shard set corresponding to the original shard and including at least one expanded shard according to a detected expansion instruction; determining an expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard, where the original storage range is a set of text indexes of data stored in the original shard before expansion, and the expanded storage range is a set of text indexes of data stored in the expanded shard after expansion; for each expanded shard, storing the data corresponding to the expanded storage range of the expanded shard into the expanded shard. By determining the expanded storage range of each expanded shard according to the original storage range of the original shard and performing data migration based on the expanded storage range, the difficulty of data migration during the expansion of the full-text retrieval system is reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data processing, and in particular, to a method, apparatus, device, and medium for expanding the capacity of a full-text retrieval system. Background Art

[0002] The capacity of a system often determines the storage service capacity of the storage system. During the use of the system, the accumulated data reading and writing will surely exhaust the space of the file system. At this time, capacity expansion is the main method to increase the capacity of the file system.

[0003] There are mainly two ways to expand the existing system: one is to increase the storage capacity on the data storage node, and the other is to add new storage nodes. However, due to the limited computing power of the storage node and the transmission capacity of the network resources, the capacity expansion effect achieved by simply increasing the storage capacity on the storage node is relatively not obvious. Usually, adding new storage nodes is more effective. When adding new storage nodes, some data in the original storage nodes of the system needs to be migrated to the expanded storage nodes. Therefore, which data in the original storage nodes is used as the migration data is the key to realizing system capacity expansion.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following technical problems in the prior art: In order to achieve efficient retrieval, the full-text retrieval system generally uses the inverted index technology to organize and store the data of documents. Using the inverted index technology to organize and store the data of documents makes it difficult to reversely restore the data contained in each document, that is, for each storage node, it is difficult to determine which data in the storage node belongs to the same text. And in order to ensure the accuracy and feasibility of retrieval, the data (keywords, etc.) under the same text need to be stored in the same storage node. Therefore, the data migration during system capacity expansion is difficult. Summary of the Invention

[0005] Embodiments of the present invention provide a method, apparatus, device, and medium for expanding the capacity of a full-text retrieval system to reduce the complexity and data migration difficulty during system capacity expansion.

[0006] In a first aspect, embodiments of the present invention provide a method for expanding the capacity of a full-text retrieval system. The system includes an original shard for storing data and a text index corresponding to the data. The method includes:

[0007] Determine a shard set corresponding to the original shard according to a detected capacity expansion instruction. The shard set includes at least one expanded shard;

[0008] Determine the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard. The original storage range is the set of text indexes where the original shard stores data before expansion, and the expanded storage range is the set of text indexes where the expanded shard stores data after expansion;

[0009] For each of the expanded shards, store the data corresponding to the expanded storage range of the expanded shard into the expanded shard.

[0010] In a second aspect, an embodiment of the present invention further provides an expansion device for a full-text retrieval system. The system includes original shards for storing data and text indexes corresponding to the data. The device includes:

[0011] A shard set determination module, configured to determine a shard set corresponding to the original shard according to a detected expansion instruction. The shard set includes at least one expanded shard;

[0012] A storage range determination module, configured to determine the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard. The original storage range is the set of text indexes where the original shard stores data before expansion, and the expanded storage range is the set of text indexes where the expanded shard stores data after expansion;

[0013] An expanded shard storage module, configured to, for each of the expanded shards, store the data corresponding to the expanded storage range of the expanded shard into the expanded shard.

[0014] In a third aspect, an embodiment of the present invention further provides a computer device, and the device includes:

[0015] One or more processors;

[0016] A storage device, configured to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the full-text retrieval system expansion method provided in any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the full-text retrieval system expansion method provided in any embodiment of the present invention.

[0019] In an embodiment of the present invention, a shard set including at least one expanded shard corresponding to an original shard is determined according to a detected expansion instruction; an expanded storage range of each expanded shard in the shard set is determined according to an original storage range corresponding to the original shard, where the original storage range is a text index set in which the original shard stores data before expansion, and the expanded storage range is a text index set in which the expanded shard stores data after expansion; for each expanded shard, the data corresponding to the expanded storage range of the expanded shard is stored in the expanded shard. By determining the expanded storage range of each expanded shard according to the original storage range of the original shard and migrating data based on the expanded storage range, the complexity and data migration difficulty during the expansion of the full-text retrieval system are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of a method for expanding a full-text retrieval system provided in Embodiment 1 of the present invention;

[0021] Figure 2 is a flowchart of a method for expanding a full-text retrieval system provided in Embodiment 2 of the present invention;

[0022] Figure 3a is a flowchart of a method for expanding a full-text retrieval system provided in Embodiment 3 of the present invention;

[0023] Figure 3b is an expansion schematic diagram in a method for expanding a full-text retrieval provided in Embodiment 3 of the present invention;

[0024] Figure 4 is a schematic structural diagram of a device for expanding a full-text retrieval system provided in Embodiment 4 of the present invention;

[0025] Figure 5 is a schematic structural diagram of a computer device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the drawings rather than all the structures.

[0027] Embodiment 1

[0028] Figure 1This is a flow chart of a method for expanding a full-text search system provided in the first embodiment of the present invention. This embodiment is applicable to the situation when expanding a full-text search system, wherein the system includes original shards for storing data and text indexes corresponding to the data. The method can be executed by a full-text search system expansion device, which can be implemented in software and / or hardware. For example, the full-text search system expansion device can be configured in a computer device. Figure 1 As shown, the method includes:

[0029] S110. Determine a shard set corresponding to the original shard according to the detected expansion instruction, where the shard set includes at least one expanded shard.

[0030] In a full-text retrieval system, in order to increase the data storage capacity and query performance of the system, data is generally stored in multiple devices in a dispersed manner. Data sharding storage is a technology for distributing and storing data on multiple storage devices. In the present embodiment, a shard is a data storage node. It can be understood that each device corresponds to a shard, or multiple shards can be distributed on the same device. System expansion is to expand an original shard into multiple shards, and migrate part of the data of the original shard to the expanded shard, so that the amount of data stored in each shard is more balanced. In the present embodiment, when an expansion instruction is detected, the original shard that needs to be expanded is determined, and the shard set corresponding to the original shard that needs to be expanded is determined, that is, the corresponding relationship between the original shard and the shard after expansion is determined. Optionally, some original shards can be selected from all original shards for expansion, or all original shards can be expanded.

[0031] In this embodiment, the shard set corresponding to the original shard includes at least one expanded shard. Optionally, the shard set may only include at least two expanded shards, that is, the original shard is expanded into at least two expanded shards. Exemplarily, if the original shard is shard A, and the expanded shards are shards B and shard C, then shard A is expanded into shard B and shard C. Optionally, the shard set may also include the original shard and at least one expanded shard, that is, the original shard is expanded into the original shard and at least one expanded shard. Exemplarily, if the original shard is shard A, and the expanded shard is shard D, then shard A is expanded into shard A and shard D.

[0032] S120. Determine the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard.

[0033] In this embodiment, considering that the size of the text index is much smaller than the size of the data corresponding to the text, the storage of the text index will not affect the system performance. Therefore, the data and the text index corresponding to the data are stored in the original shards correspondingly. Wherein, the original storage range is the set of text indexes of the data stored in the original shards before expansion, and the expanded storage range is the set of text indexes of the data stored in the expanded shards after expansion. To determine the data migration method, the expanded storage range of each expanded shard in the shard set is determined according to the original storage range corresponding to the original shard, so that the expanded storage range of each shard (including the expanded shard, or the original shard and the expanded shard) in the shard set after expansion meets the expanded storage requirements. Optionally, the expanded storage requirement may be that there is no intersection between the expanded storage ranges of each shard in the shard set, and the union of the expanded storage ranges of each shard in the shard set is the original storage range of the original shard.

[0034] It can be understood that if the shard set corresponding to the original shard only contains expanded shards, then the expanded storage ranges of the expanded shards in the shard set meet the expanded storage requirements. If the shard set corresponding to the original shard contains the original shard and the expanded shard, then the expanded storage ranges of the original shard and the expanded shard meet the expanded storage requirements.

[0035] Exemplarily, if the original storage range of the original shard A is [0, 1000], and the shard set corresponding to the original shard includes the original shard A and the expanded shard D, then the expanded storage range corresponding to the original shard A after expansion can be [0, 499), and the expanded storage range corresponding to the expanded shard D can be [499, 1000]. Among them, there is no intersection between the expanded storage range [0, 499) of the original shard A and the expanded storage range [499, 1000] of the expanded shard D, and the union of the expanded storage range [0, 499) of the original shard A and the expanded storage range [499, 1000] of the expanded shard D is the original storage range [0, 1000] of the original shard A.

[0036] In an implementation manner of the present invention, determining the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard includes: dividing the original storage range into at least two consecutive split storage ranges, and the number of the split storage ranges is equal to the number of shards in the shard set; respectively using each split storage range as the expanded storage range of each shard in the shard set.

[0037] Optionally, the number of split storage ranges can be determined according to the number of shards in the shard set. Based on the number of split storage ranges, the original storage range is divided into consecutive split storage ranges, and each split storage range is used as the extended storage range for each shard in the shard set. Exemplarily, if the shard set contains shard A, shard B, and shard C, that is, the number of shards in the shard set is 3, and the original storage range is [M, N], then the original storage range is divided into 3 consecutive split storage ranges, that is, [M, N] is divided into [M, P), [P, Q), and [Q, N]. [M, P) is used as the extended storage range for shard A, [P, Q) is used as the extended storage range for shard B, and [Q, N] is used as the extended storage range for shard C. Preferably, the original storage range is evenly divided into at least two split storage ranges with equal ranges, so that the storage ranges of each shard after expansion are equal, and the data stored in each shard after expansion is evenly distributed.

[0038] S130. For each extended shard, store the data corresponding to the extended storage range of the extended shard into the extended shard.

[0039] In this embodiment, after determining the extended storage range of each extended shard, for each extended shard, store the data corresponding to the extended storage range of the extended shard into the extended shard, and the system expansion can be completed. Since the extended storage range corresponds to the text index set stored by the extended shard, and in this embodiment, the data corresponding to the same text index is the data under the same text, therefore, in this embodiment, by migrating the data according to the text index, it is easy to determine the data to be migrated, and it is ensured that the data under the same text is stored in the same shard after the data migration.

[0040] In the embodiment of the present invention, a shard set including at least one extended shard corresponding to the original shard is determined according to the detected expansion instruction; the extended storage range of each extended shard in the shard set is determined according to the original storage range corresponding to the original shard, where the original storage range is the text index set stored by the original shard before expansion, and the extended storage range is the text index set stored by the extended shard after expansion; for each extended shard, store the data corresponding to the extended storage range of the extended shard into the extended shard. By determining the extended storage range of each extended shard according to the original storage range of the original shard and migrating the data based on the extended storage range, the complexity and data migration difficulty during the expansion of the full-text retrieval system are reduced.

[0041] Based on the above solution, after storing the data corresponding to the expansion storage range of the expansion shard into the expansion shard, the following steps are further included: obtaining the data to be written included in the received data write instruction, where the data to be written includes the text identifier and keyword corresponding to the data to be written; determining the text index of the data to be written according to the text identifier, and determining the write shard corresponding to the data to be written according to the text index; storing the text index, the text identifier, and the keyword correspondingly into the write shard.

[0042] In this embodiment, when storing data after the expansion is completed, the write shard corresponding to the data to be written is determined according to the text index of the data to be written. Specifically, when a data write instruction is received, first obtain the data to be written included in the data write instruction, such as the text identifier (web link, text title, etc.) of the data to be written and the keyword corresponding to this text, then generate a text index that is uniquely corresponding to the data to be written according to the text identifier, determine the write shard of the data to be written according to the text index, and then store the text index, the text identifier, and the keyword correspondingly into the write shard. In this embodiment, the method for generating the text index is not limited, as long as it is ensured that the same text identifier can generate a uniquely corresponding text index.

[0043] In an implementation manner of the present invention, the determining the text index of the data to be written according to the text identifier and determining the write shard corresponding to the data to be written according to the text index includes: determining the hash value corresponding to the text identifier according to the hash algorithm, and determining the text index corresponding to the text identifier according to the hash value; using the expansion shard corresponding to the expansion storage range containing the text index as the write shard corresponding to the data to be written.

[0044] Optionally, the hash value corresponding to the text identifier of the data to be written can be generated through the hash algorithm, the text index of the data to be written is generated after taking the modulus of the hash value, and the shard corresponding to the expansion storage range containing the text index is used as the write shard corresponding to the data to be written. Exemplarily, if the text index of the data to be written is 400 and the expansion storage range of shard C is [0, 500], then shard C is used as the write shard corresponding to the data to be written, and the data to be written is written into shard C.

[0045] Embodiment Two

[0046] Figure 2 is a flowchart of a method for expanding a full-text retrieval system provided in Embodiment Two of the present invention. This embodiment is further optimized based on the above embodiment. As Figure 2 shown, the method includes:

[0047] S210. Determine the shard set corresponding to the original shard according to the detected expansion instruction, where the shard set includes at least one expanded shard.

[0048] S220. Determine the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard.

[0049] S230. Write the original data stored in the original shard to each expanded shard, and delete the data to be deleted in the expanded shard.

[0050] In this embodiment, the storage of the data corresponding to the expanded storage range of the expanded shard to the expanded shard is specified. Wherein, the data to be deleted in the expanded shard is other data in the original data except the data corresponding to the expanded storage range of the expanded shard.

[0051] In a full-text retrieval system, it is impossible to directly write part of the data in a shard to another shard. Therefore, if partial data migration is to be achieved, all data needs to be written first and then the data that does not need to be migrated is deleted. Exemplarily, if part of the data in shard A needs to be migrated to shard B, then after writing all the data in shard A to shard B, the data that does not need to be migrated is deleted from shard B.

[0052] In this embodiment, the original data stored in the original shard is written to each expanded shard, and the data to be deleted in the expanded shard is deleted, so that the data stored in the expanded shard is the data corresponding to the expanded storage range of the expanded shard. Exemplarily, assume that the original storage range of the original shard is [0, 1000], and the expanded storage range of expanded shard B is [0, 500). Then, after writing all the original data stored in the original shard to expanded shard B, the data corresponding to the storage range [500, 1000] is deleted as the data to be deleted in the expanded shard.

[0053] S240. After the deletion of the data to be deleted in the expanded shard is completed, delete the original data to be deleted in the original shard.

[0054] To ensure that data is not lost during migration, or can be re-migrated if lost during migration, after all expanded shards store the data corresponding to their expanded storage ranges, then process the data in the original shard.

[0055] In this embodiment, the original data to be deleted is the other data in the original data except the data corresponding to the extended storage range of the original shard. The shard set corresponding to the original shard may only contain extended shards, or may contain both the original shard and the extended shards. When the shard set only contains extended shards, the extended storage range corresponding to the original shard is empty, and all the data in the original shard is deleted. When the shard set contains both the original shard and the extended shards, the original data to be deleted is determined according to the extended storage range of the original shard and deleted. Exemplarily, assume that the original storage range of the original shard is [0, 1000], and the extended storage range of the original shard is [500, 1000]. After the extended data to be deleted in the extended shard is deleted, the data corresponding to the extended storage range [0, 500) is used as the original data to be deleted and deleted.

[0056] The technical solution of the embodiment of the present invention concretizes the storage of the data corresponding to the extended storage range of the extended shard into the extended shard. By writing the original data stored in the original shard into each extended shard and deleting the extended data to be deleted in the extended shard, after the extended data to be deleted in the extended shard is deleted, the original data to be deleted in the original shard is deleted, ensuring that the data will not be lost during the migration process, or the migration can be completed again after the data is lost.

[0057] Based on the above solution, deleting the extended data to be deleted in the extended shard includes:

[0058] For each extended shard, determine the text index corresponding to the extended data to be deleted in the extended shard according to the extended storage range of the extended shard, and generate a data deletion logic according to the text index of the extended data to be deleted, so as to delete the extended data to be deleted according to the data deletion logic.

[0059] In this embodiment, considering that the deletion of the extended data to be deleted takes a long time, after the original data in the original shard is written into the extended shard, the data deletion logic can be generated first to mark and delete the extended data to be deleted, so that the extended shard does not have to wait for the physical deletion of the extended data to be deleted to perform the next operation (such as deleting the original data to be deleted in the original shard), which speeds up the system expansion speed. Optionally, for each extended shard, determine the text index of the extended data to be deleted in the extended shard according to the extended storage range of the extended shard, and generate a data generation logic according to the text index of the extended data to be deleted, so that the extended shard deletes the data according to the text index.

[0060] Based on the above solution, the method further includes:

[0061] After detecting an expansion instruction, determine the write expansion shard corresponding to the data write instruction according to the received data write instruction, and generate a data write log according to the write expansion shard and the data write instruction;

[0062] After the expansion pending deletion data in the write expansion shard is completely deleted, write the data write log to the write expansion shard, so that the write expansion shard performs a write operation corresponding to the data write log.

[0063] In this embodiment, after detecting an expansion instruction, before the expansion pending deletion data in the expansion shard is completely deleted, the expansion shard is not added to the topology structure and does not perform read and write operations. Only the original shard performs read and write operations. In the above stage, when a data write instruction is received, determine the write shard corresponding to the data write instruction according to the expanded expansion storage range (which can be the original shard or the expansion shard). When the write shard corresponding to the data write instruction is the original shard, the write operation can be directly performed in the original shard. When the write shard corresponding to the data write instruction is the expansion shard, generate a data write log corresponding to the data write execution in the original shard, and after the expansion pending deletion data in the expansion shard is completely deleted, write the data write log to the write expansion shard, so that the write expansion shard performs a write operation corresponding to the data write log. Specifically, the method for determining the write shard corresponding to the data write instruction can be seen in the above embodiment and will not be elaborated here.

[0064] Exemplarily, assume that the original shard is shard A and the write expansion shard corresponding to the data write instruction is shard B. After receiving the data write instruction, generate a data write log corresponding to the data write instruction in shard A. After the expansion pending deletion data in shard B is completely deleted, write the data write log to shard B, so that shard B performs a write operation corresponding to the data write log.

[0065] Embodiment III

[0066] Figure 3a It is a flowchart of a method for expanding a full-text retrieval system provided in Embodiment III of the present invention. On the basis of the above embodiment, this embodiment provides a preferred embodiment. This embodiment is described by taking the expansion of shard 1 into shard 1_1 and shard 1_2 as an example. As Figure 3a shown, the method includes:

[0067] S310. Copy the storage files of shard 1 to shard 1_1 and shard 1_2.

[0068] In this embodiment, when storing a document, calculate the hash value of the document identifier, generate the index value of the document by means of modulo operation, etc., determine the shard to which the document belongs according to the storage range of each shard, write the document into the corresponding shard, and the shard stores the index value together while processing the document. Exemplarily, the document index ranges corresponding to all shards in the system can be (0 to 65535) or (0 to 4294967296).

[0069] When expansion is needed, complete a data snapshot of the storage file of Shard 1 by means of hard link, and copy the storage file of Shard 1 to Shard 1_1 and Shard 1_2.

[0070] S320. Start Shard 1_1 and Shard 1_2, and delete the documents that do not belong to their respective document index ranges according to the document index ranges each responsible for by the newly split shards.

[0071] When a shard needs to be expanded into two shards, calculate the intermediate value of the document index range responsible for by this shard, divide the document index range into two document index ranges based on the intermediate value, and use the two divided document index ranges as the document index ranges respectively responsible for by the two expanded shards. For each expanded shard, delete the documents that do not belong to the document index range of this shard.

[0072] Figure 3b It is the expansion schematic diagram in a full-text retrieval expansion method provided by Embodiment 3 of the present invention. As Figure 3b shown, the document index range responsible for by Shard 1 is (0 to 1000), the document index range responsible for by Shard 2 is (1000 to 2000),..., the document index range responsible for by Shard N is (X to Y), and each shard stores the documents within the document index range it is responsible for. Taking the splitting of Shard 1 into Shard 1_1 and Shard 1_2 as an example, calculate the intermediate value of the document index range responsible for by Shard 1 as 500, then divide the document index range into (0 to 500) and (500 to 1000), use (0 to 500) as the document index range responsible for by Shard 1_1, use (500 to 1000) as the document index range responsible for by Shard 1_2, and delete the documents with the document index range within (500 to 1000) in Shard 1_1, and delete the documents with the document index range within (0 to 500) in Shard 1_2.

[0073] It should be noted that the deletion mentioned in this stage is not physical deletion, but is quickly completed by means of marked deletion. The marked deletion is very fast, which speeds up the system expansion speed.

[0074] S330. Shard 1 writes the data into the logs and appends them to Shard 1_1 and Shard 1_2 respectively.

[0075] During the period from receiving the expansion instruction to the completion of the marked deletion of shard 1_1 and shard 1_2, shard 1 can receive new read and write operations to ensure uninterrupted service. When performing a read operation, it directly reads from the original shard. When performing a write operation, it determines the corresponding write shard according to the write operation instruction and generates a data write log. After the marked deletion of shard 1_1 and shard 1_2 is completed, shard 1 sends the generated data write log to the corresponding shard so that shard 1_1 and shard 1_2 can append the write operation.

[0076] S340. After the log append is completed, add shard 1_1 and shard 1_2 to the shard topology, and the split is completed.

[0077] After the data write logs of shard 1_1 and shard 1_2 are appended, add shard 1_1 and shard 1_2 to the shard topology structure, make shard 1_1 and shard 1_2 visible to the client, and perform read and write operations, and delete the original shard 1 according to specific policies.

[0078] The technical solution provided in this embodiment does not need to reconstruct the original index through technical means such as reverse tracing of the original document or inverted index. It directly performs data splitting at the file level, binds the document index to the document, and cleverly solves the problem of dynamic system expansion through shard splitting without stopping the system by means of snapshot file replication and log appending.

[0079] Embodiment 4

[0080] Figure 4 It is a schematic structural diagram of a full-text retrieval system expansion device provided in Embodiment 4 of the present invention. The system includes an original shard for storing data and the text index corresponding to the data. The full-text retrieval system expansion device can be implemented in software and / or hardware. For example, the full-text retrieval system expansion device can be configured in a computer device. As Figure 4 shown, the device includes a shard set determination module 410, a storage range determination module 420, and an expanded shard storage module 430, where:

[0081] The shard set determination module 410 is used to determine the shard set corresponding to the original shard according to the detected expansion instruction, and the shard set includes at least one expanded shard;

[0082] The storage range determination module 420 is used to determine the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard. The original storage range is the text index set in which the original shard stores data before expansion, and the expanded storage range is the text index set in which the expanded shard stores data after expansion;

[0083] The expanded shard storage module 430 is used to store the data corresponding to the expanded storage range of each expanded shard into the expanded shard.

[0084] In the embodiment of the present invention, the shard set determination module determines the shard set corresponding to the original shard according to the detected expansion instruction, and the shard set includes at least one expanded shard; the storage range determination module determines the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard, the original storage range is the text index set of the data stored in the original shard before expansion, and the expanded storage range is the text index set of the data stored in the expanded shard after expansion; the expanded shard storage module stores the data corresponding to the expanded storage range of the expanded shard into the expanded shard for each expanded shard. By determining the expanded storage range of each expanded shard according to the original storage range of the original shard and migrating the data based on the expanded storage range, the complexity and data migration difficulty during the expansion of the full-text retrieval system are reduced.

[0085] Based on the above solution, the storage range determination module 420 is specifically used for:

[0086] Dividing the original storage range into at least two consecutive split storage ranges, and the number of the split storage ranges is equal to the number of shards in the shard set;

[0087] Respectively using each split storage range as the expanded storage range of each shard in the shard set.

[0088] Based on the above solution, the expanded shard storage module 430 is specifically used for:

[0089] Writing the original data stored in the original shard into each expanded shard, and deleting the data to be deleted in the expanded shard, where the data to be deleted in the expanded shard is other data in the original data except the data corresponding to the expanded storage range of the expanded shard.

[0090] Based on the above solution, the shard set further includes the original shard, and the expanded shard storage module 430 is further used for:

[0091] After the deletion of the data to be deleted in the expanded shard is completed, deleting the original data to be deleted in the original shard, where the original data to be deleted is other data in the original data except the data corresponding to the expanded storage range of the original shard.

[0092] Based on the above solution, the expanded shard storage module 430 is specifically used for:

[0093] For each of the expanded shards, determine the text index corresponding to the data to be deleted in the expansion of the expanded shard according to the expansion storage range of the expanded shard, and generate a data deletion logic based on the text index of the data to be deleted in the expansion, so as to delete the data to be deleted in the expansion according to the data deletion logic.

[0094] Based on the above solution, the device further includes a data append module, which is used for:

[0095] After detecting an expansion instruction, determine the expanded shard corresponding to the data write instruction according to the received data write instruction, and generate a data write log based on the expanded shard and the data write instruction;

[0096] After the deletion of the data to be deleted in the expansion in the expanded shard to be written is completed, write the data write log into the expanded shard to be written, so that the expanded shard to be written performs a write operation corresponding to the data write log.

[0097] Based on the above solution, the device further includes a data write module, including:

[0098] A write data acquisition unit, configured to acquire the data to be written included in the received data write instruction after storing the data corresponding to the expansion storage range of the expanded shard into the expanded shard, where the data to be written includes a text identifier and a keyword corresponding to the data to be written;

[0099] A write shard determination unit, configured to determine the text index of the data to be written according to the text identifier, and determine the write shard corresponding to the data to be written according to the text index;

[0100] A write data write unit, configured to store the text index, the text identifier, and the keyword correspondingly into the write shard.

[0101] Based on the above solution, the write shard determination unit is specifically configured to:

[0102] Determine the hash value corresponding to the text identifier according to the hash algorithm, and determine the text index corresponding to the text identifier according to the hash value;

[0103] Take the expanded shard corresponding to the expansion storage range including the text index as the write shard corresponding to the data to be written.

[0104] The full-text retrieval system expansion device provided by the embodiments of the present invention can execute the full-text retrieval system expansion method provided by any embodiment, and has the corresponding functional modules and beneficial effects for executing the method.

[0105] Embodiment Five

[0106] Figure 5 It is a schematic structural diagram of the computer device provided in the fifth embodiment of the present invention. Figure 5 It shows a block diagram of an exemplary computer device 512 suitable for implementing the embodiments of the present invention. Figure 5 The displayed computer device 512 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0107] As Figure 5 shown, the computer device 512 is presented in the form of a general-purpose computing device. The components of the computer device 512 may include but are not limited to: one or more processors 516, a system memory 528, and a bus 518 connecting different system components (including the system memory 528 and the processor 516).

[0108] The bus 518 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor 516, or a local bus using any of the various bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0109] The computer device 512 typically includes a variety of computer system-readable media. These media can be any available media accessible by the computer device 512, including volatile and non-volatile media, removable and non-removable media.

[0110] The system memory 528 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 530 and / or cache memory 532. The computer device 512 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage device 534 may be used for reading and writing non-removable, non-volatile magnetic media ( Figure 5 not shown, commonly referred to as a "hard disk drive"). Although Figure 5 not shown in the figure, a disk drive for reading and writing removable non-volatile disks (such as "floppy disks") and an optical disk drive for reading and writing removable non-volatile optical disks (such as CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 518 through one or more data media interfaces. The memory 528 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0111] A program / utilities 540 having a set (at least one) of program modules 542 can be stored, for example, in a memory 528. Such program modules 542 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules 542 generally execute the functions and / or methods in the embodiments described in the present invention.

[0112] The computer device 512 can also communicate with one or more external devices 514 (such as a keyboard, a pointing device, a display 524, etc.), and can also communicate with one or more devices that enable a user to interact with the computer device 512, and / or communicate with any device that enables the computer device 512 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 522. Moreover, the computer device 512 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 520. As shown in the figure, the network adapter 520 communicates with other modules of the computer device 512 through a bus 518. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computer device 512, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0113] The processor 516 executes various functional applications and data processing by running programs stored in the system memory 528, for example, implementing the full-text retrieval system expansion method provided by the embodiments of the present invention. The method includes:

[0114] Determining a shard set corresponding to the original shard according to the detected expansion instruction, where the shard set includes at least one expanded shard;

[0115] Determining the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard, where the original storage range is the text index set of the data stored in the original shard before expansion, and the expanded storage range is the text index set of the data stored in the expanded shard after expansion;

[0116] For each of the expanded shards, storing the data corresponding to the expanded storage range of the expanded shard into the expanded shard.

[0117] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the full-text retrieval system expansion method provided by any embodiment of the present invention.

[0118] Embodiment Six

[0119] Embodiment Six of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the full-text retrieval system expansion method provided in the embodiments of the present invention. The method includes:

[0120] Determine a shard set corresponding to the original shard according to the detected expansion instruction, where the shard set includes at least one expanded shard;

[0121] Determine the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard. The original storage range is the text index set of the data stored in the original shard before expansion, and the expanded storage range is the text index set of the data stored in the expanded shard after expansion;

[0122] For each expanded shard, store the data corresponding to the expanded storage range of the expanded shard into the expanded shard.

[0123] Of course, the computer program stored on the computer-readable storage medium provided in the embodiments of the present invention is not limited to the above method operations, and can also execute related operations in the full-text retrieval system expansion method provided in any embodiment of the present invention.

[0124] The computer storage medium of the embodiments of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0125] A computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0126] The program code contained on a computer-readable medium can be transmitted by any appropriate medium, including - but not limited to - wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the above.

[0127] The computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network - including a local area network (LAN) or a wide area network (WAN) - or, it can be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0128] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, it can also include more other equivalent embodiments, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for expanding a full-text retrieval system, characterized in that, The system includes original shards for storing data and text indexes corresponding to the data. The method includes: Determining a shard set corresponding to the original shard according to a detected expansion instruction, where the shard set includes at least one expanded shard; Determining the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard. The original storage range is the set of text indexes of the data stored in the original shard before expansion, and the expanded storage range is the set of text indexes of the data stored in the expanded shard after expansion; For each of the expanded shards, storing the data corresponding to the expanded storage range of the expanded shard into the expanded shard; Wherein, the storing the data corresponding to the expanded storage range of the expanded shard into the expanded shard includes: Writing the original data stored in the original shard into each of the expanded shards, and deleting the data to be deleted in the expansion in the expanded shard. The data to be deleted in the expansion is other data in the original data except the data corresponding to the expanded storage range of the expanded shard; The method further includes: After detecting an expansion instruction and before the deletion of the data to be deleted in the expansion of the expanded shard is completed, the expanded shard does not join the topology structure. When a data write instruction is received, determine the write shard corresponding to the data write instruction according to the expanded storage range after expansion; when the write shard corresponding to the data write instruction is the original shard, directly perform a write operation in the original shard; when the write shard corresponding to the data write instruction is an expanded shard, generate a data write log corresponding to the data write execution in the original shard, and after the deletion of the data to be deleted in the expansion of the expanded shard is completed, write the data write log into the expanded shard so that the expanded shard performs the write operation corresponding to the data write log; The deleting the data to be deleted in the expansion in the expanded shard includes: For each of the expanded shards, determining the text indexes corresponding to the data to be deleted in the expansion of the expanded shard according to the expanded storage range of the expanded shard, and generating a data deletion logic according to the text indexes of the data to be deleted in the expansion to delete the data to be deleted in the expansion according to the data deletion logic.

2. The method according to claim 1, characterized in that, The determining the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard includes: Dividing the original storage range into at least two consecutive split storage ranges, where the number of split storage ranges is equal to the number of shards in the shard set; Respectively taking each of the split storage ranges as the expanded storage range of each shard in the shard set.

3. The method according to claim 1, characterized in that, The shard set further includes the original shard. The method further includes: After the deletion of the data to be deleted in the expansion in the expanded shard is completed, deleting the original data to be deleted in the original shard. The original data to be deleted is other data in the original data except the data corresponding to the expanded storage range of the original shard.

4. The method according to claim 1, characterized in that, After storing the data corresponding to the expanded storage range of the expanded shard into the expanded shard, it further includes: Obtain the data to be written included in the received data write instruction, where the data to be written includes a text identifier and a keyword corresponding to the data to be written; Determine the text index of the data to be written according to the text identifier, and determine the write shard corresponding to the data to be written according to the text index; Correspondingly store the text index, the text identifier, and the keyword into the write shard.

5. The method according to claim 4, characterized in that, The determining the text index of the data to be written according to the text identifier, and determining the write shard corresponding to the data to be written according to the text index includes: Determine the hash value corresponding to the text identifier according to the hash algorithm, and determine the text index corresponding to the text identifier according to the hash value; Use the expanded shard corresponding to the expanded storage range including the text index as the write shard corresponding to the data to be written.

6. A device for expanding a full-text retrieval system, characterized in that, The system includes an original shard for storing data and the text index corresponding to the data, and the device includes: A shard set determination module, configured to determine a shard set corresponding to the original shard according to a detected expansion instruction, where the shard set includes at least one expanded shard; A storage range determination module, configured to determine the expanded storage range of each expanded shard in the shard set according to the original storage range corresponding to the original shard, where the original storage range is a set of text indexes of data stored in the original shard before expansion, and the expanded storage range is a set of text indexes of data stored in the expanded shard after expansion; An expanded shard storage module, configured to, for each expanded shard, store the data corresponding to the expanded storage range of the expanded shard into the expanded shard; Among them, the expanded shard storage module is specifically configured to: Write the original data stored in the original shard into each expanded shard, and delete the data to be deleted in the expansion of the expanded shard, where the data to be deleted in the expansion is other data in the original data except the data corresponding to the expanded storage range of the expanded shard; The device further includes a data append module, configured to: After detecting an expansion instruction and before the deletion of the data to be deleted in the expansion of the expanded shard is completed, the expanded shard has not joined the topology. When a data write instruction is received, determine the write shard corresponding to the data write instruction according to the expanded storage range after expansion; when the write shard corresponding to the data write instruction is the original shard, directly perform a write operation in the original shard; when the write shard corresponding to the data write instruction is an expanded shard, generate a data write log corresponding to the execution of the data write in the original shard, and after the deletion of the data to be deleted in the expansion of the expanded shard is completed, write the data write log into the expanded shard so that the expanded shard performs a write operation corresponding to the data write log; The expanded shard storage module is specifically configured to: For each expanded shard, determine the text index corresponding to the data to be deleted in the expansion of the expanded shard according to the expanded storage range of the expanded shard, and generate a data deletion logic according to the text index of the data to be deleted in the expansion, so as to delete the data to be deleted in the expansion according to the data deletion logic; Among them, the data deletion logic is used to mark and delete the data to be deleted during expansion.

7. A computer device, characterized in that, The device includes: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the full-text retrieval system expansion method according to any one of claims 1-5.

8. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the program is executed by the processor, it implements the full-text retrieval system expansion method according to any one of claims 1-5.

Images