Method, device and computer program product for managing index table

By dynamically adjusting the number of index tables, the performance and capacity limitations of traditional full-text search engines when processing streaming data are solved, and efficient processing and low-latency search of unbounded data are achieved.

CN112579726BActive Publication Date: 2024-12-06EMC IP HLDG CO LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910932404.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-09-29
Publication Date
2024-12-06
Estimated Expiration
2039-09-29

AI Technical Summary

Technical Problem

When traditional full-text search engines process streaming data, they are unable to effectively process unbounded data, process data increments at low latency and efficiently delete expired data, resulting in performance and capacity limitations.

Method used

By dynamically adjusting the number of index tables, merging or deleting the index tables based on the number of requests and data volumes of received indexes, ensuring that the index tables are not too large or too small, thereby improving the capacity and performance of the search system.

Benefits of technology

It realizes efficient index establishment and search of streaming data, improves the scalability and processing capabilities of the system, and ensures low-latency data processing and efficient data clearing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112579726B_ABST
    Figure CN112579726B_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure relate to methods, devices, and computer program products for managing index tables. A method for managing index tables is proposed, comprising: determining the number of index creation requests for documents received within a predetermined time; obtaining information related to a processing index table in an index processing system, the processing index table being used to store indexes generated according to the index creation requests, the index processing system also comprising a receiving index table, the receiving index table being used to store at least a portion of the indexes in the processing index table; and adjusting the number of processing index tables in the index processing system based on the number of index creation requests and the acquired information. Through the embodiments of the present disclosure, the index creation throughput of the index processing system can be improved when a large amount of data arrives, and the problem of unlimited growth of the index table can be avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of information processing, and more particularly to a method, device, and computer program product for managing an index table. Background Art

[0002] Currently, most popular full-text search engines use full-text search engine toolkits that rely on inverted index tables (also referred to as "index tables" in this article) to complete indexing and searching. However, an inverted index table can only be associated with a limited number of documents, so it cannot handle high indexing request throughput. And when there are too many entries in the inverted index table, indexing and searching will be very slow.

[0003] Streaming data is unbounded data that is continuously generated by various sources. Streaming data includes a wide variety of data, such as log files, sensor data, social network data, etc. Streaming data is usually unbounded, append-only, and has unpredictable size. Streaming data always needs to be processed in real time. Due to the characteristics of streaming data, an efficient streaming search engine must have the following capabilities: the ability to process unbounded data; the ability to process sudden surges in data with low latency; and the ability to efficiently delete expired data. However, traditional full-text search engines still have some limitations in terms of capacity and performance when processing streaming data. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, an apparatus, and a computer program product for managing an index table.

[0005] In a first aspect of the present disclosure, a method for managing an index table is provided. The method includes: determining the number of index creation requests for documents received within a predetermined time; obtaining information related to a processing index table in an index processing system, the processing index table being used to store indexes generated according to the index creation requests, the index processing system further including a receiving index table, the receiving index table being used to store at least a portion of the indexes in the processing index table; and adjusting the number of processing index tables in the index processing system based on the number of index creation requests and the obtained information.

[0006] In a second aspect of the present disclosure, a device for managing an index table is provided. The device includes: at least one processing unit; at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the device performs an action, the action including: determining the number of index establishment requests for documents received within a predetermined time; obtaining information related to a processing index table in an index processing system, the processing index table is used to store indexes generated according to the index establishment request, the index processing system also includes a receiving index table, the receiving index table is used to store the index of at least a part of the processing index table; and adjusting the number of processing index tables in the index processing system based on the number of index establishment requests and the obtained information.

[0007] In a third aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored in a non-transient computer storage medium and includes machine executable instructions. The machine executable instructions, when executed by a device, cause the device to perform any step of the method described in the first aspect of the present disclosure.

[0008] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0010] Figure 1 A block diagram illustrating an example index processing system in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A schematic diagram showing a process of establishing an index table from a document applicable to an embodiment of the present disclosure;

[0012] Figure 3 A flowchart of an example method for managing an index table according to an embodiment of the present disclosure is shown;

[0013] Figure 4 A schematic diagram showing an example arrangement of a processing index table and a receiving index table in an index processing system according to an embodiment of the present disclosure;

[0014] Figure 5A schematic diagram showing statistics corresponding to a processing index table and a receiving index table according to an embodiment of the present disclosure is shown;

[0015] Figure 6 A schematic diagram showing an example arrangement of a processing index table and a receiving index table in an index processing system according to an embodiment of the present disclosure;

[0016] Figure 7 A flowchart of a method for generating a new processing index table according to an embodiment of the present disclosure is shown;

[0017] Figure 8 A flowchart of a method for deleting an existing processing index table according to an embodiment of the present disclosure is shown;

[0018] Fig. 9 A schematic diagram showing an example arrangement of an index table according to an embodiment of the present disclosure; and

[0019] Fig.10 A schematic block diagram of an example device that can be used to implement embodiments of the present disclosure is shown.

[0020] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0022] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] Currently, most popular full-text search engines use full-text search engine toolkits (e.g., Lucene) that rely on inverted index tables to complete indexing and searching. An inverted index table is a data structure that stores mappings from terms to document IDs. During indexing, documents are converted into a collection of terms by an analyzer, which performs language detection, tokenization, lemmatization, stop word filtering, etc. Then an index table is created using the terms as keys and the document IDs as values. During searching, keywords are also linguistically analyzed using the same analyzer, and the terms obtained are then used to retrieve documents associated with the term.

[0024] The documents associated with the index table are not in continuous disk space. In order to avoid random I / O access, adding new documents does not directly update the main index table, but creates small index tables (also called "fragments"). The search engine toolkit (for example, Lucene) will merge several small index tables of similar size into a single larger small index table through a background merge process, and then delete these small index tables. When there are too many larger small index tables of the same size, the process is repeated for the larger small index tables. When deleting documents, the documents associated with the main index table are not deleted directly, but the documents associated with the small index table are marked as "deleted" and then deleted during the merge process. This merge process is similar to merge sort, which utilizes sequential I / O and can greatly improve performance.

[0025] Since the search system searches the index table instead of searching the documents directly, it can achieve fast search response. Using the index table can help find the required results from billions of documents with low latency. However, currently, a single index table can only be associated with a limited number of documents. Typically, a single index table can be associated with a maximum of a certain number (e.g., 2 31 +1) Each document is associated with the index. Therefore, it cannot handle high indexing request throughput. Moreover, when there are too many entries in the index table, indexing and searching will be slow.

[0026] To achieve better scalability and availability, modern search engines (such as Elastic Search and Solr) divide a single index table into multiple smaller index tables (also known as "shards") and distribute these smaller index tables across cluster nodes. However, this is not without cost. Properly increasing the number of smaller index tables can improve indexing throughput and search performance, but too many smaller index tables may lead to poor search performance. Since the data being searched exists in all smaller index tables, the search engine must perform a search, obtain results, and merge the obtained results for each smaller index table. Therefore, if there are too many smaller index tables, such as more than the number of cluster nodes, the search will be performed multiple times on the same node, and this will also affect the merging process. On the other hand, too few smaller index tables will also reduce performance because all nodes are not fully utilized.

[0027] In a streaming search system, as mentioned above, since the amount of streaming data is variable and may suddenly increase due to various reasons, an efficient streaming search system must have the ability to process unbounded data, the ability to process suddenly increased data with low latency, and the ability to efficiently delete expired data. Therefore, in a streaming search system, it is important to ensure that a single index table is not too large, the number of index tables can be dynamically increased to handle the sudden increase in data, and can be dynamically reduced when old data is cleared or the workload decreases without additional system overhead.

[0028] Modern search engines use multiple index tables to increase scalability and capacity, but the number of index tables is determined when the index is created and cannot be changed. Since the index table stores the mapping from terms to documents, it is impossible to split the index table directly according to the document.

[0029] In the traditional method of splitting an existing index table, a new index table with a subset of the document ID set is first created, and then the association of the documents corresponding to the subset with the existing index table is deleted, and these documents are associated with the new index table. This work is equivalent to re-indexing these documents. For example, assuming that 1,000 documents can be indexed per second, it takes 12 days to re-index 1 billion documents. During this period, both index building and search performance will be affected. When processing streaming data with unbounded size and unpredictable arrival time, a fixed number of index tables has limited capacity and may cause performance issues.

[0030] The embodiments of the present disclosure propose a solution for managing index tables to solve one or more of the above problems and other potential problems. The solution dynamically adjusts the number of index tables to index streaming data, thereby improving the capacity and performance of the search system.

[0031] Figure 1 1 shows a block diagram of an example index processing system 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, the index processing system 100 includes a control device 110 and one or more index processing devices 120-1 to 120-4 (collectively referred to as index processing devices 120). The various methods according to the embodiments of the present disclosure can be implemented at the control device 110. It should be understood that the structure of the index processing system 100 is described for exemplary purposes only, and does not imply any limitation on the scope of the present disclosure. For example, the embodiments of the present disclosure can also be applied to systems different from the index processing system 100. It should be understood that the specific numbers of the above-mentioned devices are given for illustrative purposes only, and do not imply any limitation on the scope of the present disclosure. For example, the embodiments of the present disclosure can also be applied to more or fewer devices.

[0032] The control device 110 is a device that receives requests and controls the processing of the requests. The request may be, for example, a request to index a document, a request to search for a keyword, a request to obtain a document, a request to delete a document, etc. The control device 110 may be, for example, a computer, a virtual machine, a server, etc. The control device 110 may also be used to receive streaming data, or the streaming data may also be received by a separate data storage device (not shown), and the present disclosure is not limited in this respect.

[0033] The index processing device 120 is a device that processes various requests according to the scheduling of the control device 110. In the index processing device 120, the processing index table and the receiving index table according to the embodiment of the present disclosure can be stored. In some embodiments, the index processing device 120 can be a physical node such as a computer, a server, etc. Alternatively, in other embodiments, the index processing device 120 can be a distributed resource unit distributed on multiple physical nodes, such as a pod, a container, etc. In a preferred embodiment, the index processing device 120 can be a Kubernetes pod because it can be dynamically generated.

[0034] The control device 110 and the index processing device 120 communicate with each other via a network, which may be, for example, the Internet, an intranet, or the like.

[0035] Figure 2 FIG. 2 is a schematic diagram showing a process 200 for creating an index table from a document applicable to an embodiment of the present disclosure. Figure 2As shown, during index establishment, the control device causes the index processing device 120 to analyze the text in the documents 210-1, 210-2, and 210-3, filter the stop words 220, and then obtain the index table 230. In the index table 230, each entry includes the obtained term and the ID of the document containing the term. For example, the word "bright" appears in document 1 210-1 and document 3 210-3, and the entry associated with the word "bright" includes the term "bright" and document IDs 1 and 3. During the search, when the keyword searched by the user is "bright", the control device 110 causes the index processing device 120 to retrieve documents 1 and 3 associated with "bright" and present them to the user.

[0036] Figure 3 FIG. 3 is a flowchart of an exemplary method 300 for managing an index table according to an embodiment of the present disclosure. For example, the method 300 may be performed as follows: Figure 1 It should be understood that the method 300 may also be performed by other devices, and the scope of the present disclosure is not limited in this respect. It should also be understood that the method 300 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0037] The following combination Figures 1 to 3 The method 300 will be described in detail.

[0038] When the control device 110 receives an index creation request for a document, the control device 110 assigns the index creation request to an index processing device (e.g., index processing device 120-1) in the specific index processing device set 120 for processing according to a predetermined allocation strategy. Figure 2 As shown, an index is created for the associated document according to the index creation request, and the created index is stored in the processing index table in the index processing device 120-1.

[0039] At 310 , the control device 110 determines the number of indexing requests for documents received within a predetermined time.

[0040] In some embodiments, the control device 110 periodically determines the number of index creation requests received in a previous period of time. Alternatively or additionally, the control device 110 may also determine the number of index creation requests received in a previous period of time when a specific event occurs, for example, according to a user's request. It is easy to understand that, when the predetermined time length remains unchanged, the more requests are received, the higher the real-time processing speed requirement of the index processing system.

[0041] Figure 4A schematic diagram of an example arrangement 400 of processing index tables and receiving index tables in an index processing system according to an embodiment of the present disclosure is shown. In the arrangement 400, processing index tables 420-1, 420-2, 420-3 (collectively referred to as processing index tables 420) and receiving index tables (430) according to an embodiment of the present disclosure are stored in index processing devices 410-1, 410-2, 410-3 (collectively referred to as index processing devices 410). It should be understood that the arrangement 400 is described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure. It should be understood that the specific number of index tables shown is given for illustrative purposes only and does not imply any limitation on the scope of the present disclosure. For example, there may be more or fewer index tables in an index processing system according to an embodiment of the present disclosure.

[0042] Return now Figure 3 At 320 , the control device 110 obtains information related to the processing index table 420 in the index processing system 100 .

[0043] In some embodiments, the control device 110 may obtain the number of documents associated with each entry in the processing index table 420. Alternatively or additionally, the control device 110 may obtain the data volume of each entry in the processing index table 420. Alternatively or additionally, the control device 110 may obtain the data volume of the document associated with each entry in the processing index table 420. It should be noted that the information obtained listed herein is illustrative and not restrictive.

[0044] The index processing system 100 also has a receiving index table 430, which is used to store at least a portion of the indexes in the processing index table 420. When the total data volume of a portion of the entries in the processing index table 420 and the corresponding documents exceeds a predetermined first threshold data volume, the portion of the entries in the processing index table 420 is merged with the entries in the receiving index table 430, the documents associated with the portion of the entries in the processing index table 420 are associated with the receiving index table 430, and the portion is deleted from the processing index table 420.

[0045] The receiving index table 430 and the processing index table 420 may be located in the same index processing device 410 , or may be located in a different index processing device 410 .

[0046] At 330 , the control device 110 adjusts the number of processed index tables 420 according to the number of index creation requests and the acquired information.

[0047] In some embodiments, when the amount of data to be indexed is not large, there may be only one processing index table. For example, there is a processing index table 420-1 in the index processing device 410-1, and the indexes generated by the index processing device 410-1 according to the index establishment request are stored in the processing index table 420-1. When the amount of data to be indexed increases, for example, when the number of index establishment requests determined by the control device 110 exceeds the first threshold number, a new processing index table is generated. As mentioned above, since the index table can only be associated with a limited number of documents, the generation of a new processing index table helps to index a larger number of documents.

[0048] In a preferred embodiment, a new processing index table is generated at another index processing device that is different from the index processing device where the existing processing index table is located. For example, a new processing index table 420-2 is generated at the index processing device 410-2. As discussed above, the index processing device 410 can be a distributed resource unit distributed on multiple physical nodes. Therefore, by using different index processing devices to process multiple requests received in parallel, and each index processing device 410 evenly distributes the multiple requests that it needs to process to each node it contains for parallel processing, it is possible to fully utilize system resources and improve the processing capacity of the system. Additionally, in a preferred embodiment, each index processing device 410 contains at most one processing index table.

[0049] As the amount of data to be indexed increases, for example, when the number of indexing requests determined by the control device 110 exceeds a predetermined threshold number that is higher than the first threshold number, another new processing index table is generated, such as Figure 4 The processing index table 420-3 in . And so on.

[0050] Meanwhile, as time goes by, as mentioned above, when the control device 110 determines that the total data volume of a part of the entries in the processing index table 420 and the corresponding documents exceeds the predetermined first threshold data volume according to the obtained information, the part of the entries in the processing index table 420 is merged with the entries in the receiving index table 430, the documents associated with the part of the entries in the processing index table 420 are associated with the receiving index table 430, and the part is deleted from the processing index table 420. In this way, the processing index table 420 will not continue to grow, which is beneficial to the subsequent indexing process.

[0051] On the other hand, when the amount of data to be indexed decreases, for example, when the number of index creation requests determined by the control device 110 is lower than the second threshold number, a processing index table is deleted from the existing processing index tables. Figure 4As shown, when the number of index creation requests determined by the control device 110 is lower than the second threshold number, one is selected from the processing index tables 420-1, 420-2 and 420-3 for deletion.

[0052] Alternatively, in other embodiments, when the amount of data to be indexed is not large, there may be no processing index table 420, and there may be only one receiving index table 430. In this case, the receiving index table 430 may also store the index generated according to the index creation request, but when the control device 110 determines that the number of index creation requests exceeds the first threshold number and generates a processing index table, the receiving index table 430 no longer stores the index generated according to the index creation request.

[0053] The indexing process is CPU and IO intensive. Through the above exemplary embodiment, the number of processing index tables is dynamically adjusted according to the number of indexing requests received within a predetermined time, so that when a large amount of data arrives, there are more processing index tables running in multiple index processing devices, thereby utilizing more resources and significantly improving indexing throughput. In addition, this is also beneficial to the search process. Since the processing index tables are generally small and distributed in various index processing devices, searching the processing index tables will return results faster, thereby improving search efficiency.

[0054] Figure 5 A schematic diagram 500 showing statistics corresponding to a processing index table and a receiving index table according to an embodiment of the present disclosure is shown. Figure 5 A data volume statistics graph 510 of each portion corresponding to the received index table and a data volume statistics graph 520 of each portion corresponding to each portion of the processed index table are shown. As mentioned above, the data volume of each portion refers to the total data volume of the entries of each portion in the index table and the corresponding documents. It should be understood that the diagram 500 is described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure. It should be understood that the specific numbers of the portions of the index table shown are given for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0055] The control device 110 determines to process the excess portion in the index table. In some embodiments, the excess portion may be defined as: when the total data volume of the data volume of one or more documents and the data volume of the entries corresponding to the one or more documents exceeds a first threshold data volume, the entries corresponding to the one or more documents are called the excess portion. Figure 5 As shown, the statistical graph 520 corresponding to the processing index table indicates that the portion corresponding to the column 530 is determined as the over-limit portion.

[0056] Then, the control device 110 splits the determined overrun entries from the processing index table, marks the entries and the documents associated with the entries as "deleted", copies the entries and adds them to the receiving index table, associates the documents associated with the entries with the receiving index table, and then deletes the entries from the processing index table (for example, this process can be implemented through the IndexSplitter API and the IndexWriter.addIndexes API in the Lucene toolkit). Figure 5 As shown, after the overrun portion is merged with the receiving index table, a column 530 corresponding to the overrun portion appears in the statistical graph 510 corresponding to the receiving index table. The overrun portion in the merging process can still be used for search requests, and when the process is completed, the overrun portion is deleted from the processing index table. If a search request occurs during this merging process, the same document may be returned by the receiving index table and a processing index table, and the returned results need to be deduplicated.

[0057] In a preferred embodiment, the first threshold data amount is set to a relatively small amount so that the merging process does not result in large network traffic.

[0058] Through the above exemplary embodiments, the excess part in the processing index table is merged into the receiving index table, and the processing index table will not continue to grow, so as to ensure that the subsequent index establishment process maintains a faster processing speed.

[0059] Figure 6 A schematic diagram of an example arrangement 600 of processing index tables and receiving index tables in an index processing system according to an embodiment of the present disclosure is shown. In the arrangement 600, the processing index tables 620-1, 620-2, 620-3 (collectively referred to as processing index tables 620) and the receiving index table (630) according to an embodiment of the present disclosure are stored in index processing devices 610-1, 610-2, 610-3 (collectively referred to as index processing devices 610). It should be understood that the arrangement 600 is described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure. It should be understood that the specific number of index tables shown is given for illustrative purposes only and does not imply any limitation on the scope of the present disclosure. For example, there may be more or fewer index tables in the index processing system according to an embodiment of the present disclosure.

[0060] Similar to Figure 4, the processing index table 620 is used to store the index generated according to the index establishment request, and the receiving index table 630 is used to store the index of at least a part of the processing index table. Since the receiving index table 630 receives the excess part from all the processing index tables 620, for a system running for a long time, the amount of data in the receiving index table 630 and the associated documents will become very large over time. Since each search runs a single thread on a single index table, the search delay is related to the size of the index table. An index table that is too large will cause insufficient system memory, resulting in poor search and merge performance.

[0061] In some embodiments, when the control device 110 determines that the amount of data corresponding to the reception index table 630 exceeds the second threshold amount of data, the reception index table 630 no longer receives the excess portion of the processing index table 620, and generates a new reception index table 640. Additionally, in some embodiments, the control device 110 may mark the reception index table 630 as "stop receiving". The "stop receiving" reception index table 630 may be used for searching and data clearing, but is no longer used for merging.

[0062] In some preferred embodiments, the control device 110 generates a new reception index table 640 at an index processing device different from the reception index table 630. Alternatively, the control device 110 may determine an index processing device with the least number of "stop receiving" reception index tables, and then generate a new reception index table 640 at the index processing device. In this way, the "stop receiving" reception index tables with a large amount of data are distributed in different index processing devices to balance system resources. Additionally, in some preferred embodiments, the second threshold data amount is set to 50G.

[0063] Through the above exemplary embodiments, each receiving index table in the system will not grow too large, thereby maintaining good search performance and merging performance. In addition, when data needs to be cleared, only the receiving index table that "stops receiving" needs to be cleared, without affecting other active index tables, that is, the processing index table that is being used for index establishment requests or the receiving index table used for merging.

[0064] Figure 7 FIG. 7 is a flowchart of a method 700 for generating a new processing index table according to an embodiment of the present disclosure. For example, the method 700 may be performed as follows: Figure 1 It should be understood that the method 700 may also be performed by other devices, and the scope of the present disclosure is not limited in this respect. It should also be understood that the method 700 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0065] In some embodiments, when a new processing index table needs to be generated and where a new processing index table is generated can be determined by the number of index establishment requests within a predetermined time determined by the control device 110, the number of processing index tables allowed to be stored by each index processing device predetermined by the control device 110, and the maximum number of index processing devices allowed by the system predetermined by the control device 110.

[0066] Specifically, at 710, the control device 110 determines whether the number of index creation requests for documents received within a predetermined time exceeds a first threshold. If it exceeds the first threshold, it means that there is a lot of data to be processed, and a new processing index table needs to be added to improve index throughput.

[0067] At 720, the control device 110 determines the number of processing index tables in each existing index processing device. At 730, the control device 110 determines whether the number of processing index tables in each existing index processing device exceeds a second predetermined threshold. The second predetermined threshold is the number of processing index tables allowed to be stored in each index processing device in order to better balance the allocation of system resources.

[0068] If there are existing index processing devices whose number of stored processing index tables is lower than the second predetermined threshold, the control device 110 selects one of the existing index processing devices at 760 to generate a new processing index table at 770. Additionally, in some embodiments, the control device 110 selects an existing index processing device having a smaller number of stored processing index tables to generate a new processing index table.

[0069] If the number of processing index tables in each existing index processing device exceeds the second predetermined threshold, it indicates that the existing index processing devices can no longer store new processing index tables and a new index processing device needs to be allocated.

[0070] At 740, the control device 110 determines whether the number of existing index processing devices is lower than a third predetermined threshold. The third predetermined threshold is a predetermined maximum number of index processing devices allowed by the system.

[0071] If the number of existing index processing devices is lower than the third predetermined threshold, the control device 110 allocates a new index processing device at 750 to generate a new processing index table at 770 .

[0072] In the above exemplary embodiments, balanced allocation of system resources can be achieved by setting the second predetermined threshold and the third predetermined threshold.

[0073] Figure 8 FIG. 8 is a flowchart of a method 800 for deleting an existing processing index table according to an embodiment of the present disclosure. For example, the method 800 may be performed as follows: Figure 1It should be understood that the method 800 may also be performed by other devices, and the scope of the present disclosure is not limited in this respect. It should also be understood that the method 800 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0074] In some embodiments, when an existing processing index table needs to be deleted and where to delete the existing processing index table can be determined by the number of index establishment requests within a predetermined time determined by the control device 110, the number of index tables stored in the index processing device, and the smaller number of index processing devices allowed by the system predetermined by the control device 110.

[0075] Specifically, at 810, the control device 110 determines whether the number of index creation requests for documents received within a predetermined time is lower than a second threshold. If it is lower than the second threshold, it means that the current data to be processed is decreasing, and the processing index table can be reduced to reduce the system load.

[0076] At 820, the control device 110 selects a target index processing device from the existing index processing devices to delete the processing index table. In some embodiments, the control device 110 selects an index processing device from the index processing devices whose total number of stored processing index tables and received index tables exceeds a first predetermined threshold as the target index processing device.

[0077] At 830, the control device 110 deletes a processing index table in the target index processing device. In some embodiments, the control device 110 randomly deletes a processing index table in the target index processing device. Alternatively, in other embodiments, the control device 110 deletes a processing index table based on the acquired information related to the processing index table, such as the number of documents associated with the processing index table, the amount of data of each entry in the processing index table, the amount of data of the documents associated with the processing index table, etc.

[0078] In some embodiments, the control device 110 marks the determined processing index table to be deleted as "being removed", then merges all entries in the processing index table to be deleted into the receiving index table, associates the documents associated with the processing index table to be deleted with the receiving index table, and then deletes the processing index table.

[0079] At 840 , the control device 110 determines whether other index tables exist in the target index processing apparatus after deleting the processed index table.

[0080] If no other index table exists, it means that the target index processing device is empty. Then, the control device 110 determines at 850 whether the total number of index processing devices in the system exceeds a fourth predetermined threshold, which is the smaller number of index processing devices allowed by the system.

[0081] If the total number of index processing devices in the system exceeds a fourth predetermined threshold, the control device 110 removes the target index processing device that is already empty at 860 to save system resources.

[0082] Fig. 9 A diagram of an example arrangement 900 of index tables according to an embodiment of the present disclosure is shown. It should be understood that the arrangement 900 is described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure. It should be understood that the specific number of index tables or index processing devices shown is given for illustrative purposes only and does not imply any limitation on the scope of the present disclosure. For example, there may be more or fewer index tables or index processing devices in an index processing system according to an embodiment of the present disclosure.

[0083] like Fig. 9 As shown, index table sets 910-1 to 910-3 (collectively referred to as index table sets 910) are respectively index table sets established for streaming data arriving at different times according to an embodiment of the present disclosure. For the streaming data that arrives, the index processing system 100 can determine the number of index table subsets 930 to be established, which is fixed, and the documents in the streaming data are assigned to the index processing device 920 that is to store the index table subsets 930. The document ID can be used to determine in which index table subset 930 the document is. The index table subset 930 has a processing index table and a receiving index table 940 according to an embodiment of the present disclosure. As discussed above, since the processing index table and the receiving index table 940 are dynamically adjusted according to the speed at which the streaming data arrives and the amount of index table data, it is unknown in which index table a specific document is located, and the user needs to send a search request to obtain it. The processing index table and the receiving index table 940 are dynamically adjusted by the system, but are not visible to the user. In a preferred embodiment, consistent hashing or any hashing method is used to assign documents to specific index tables. Since all data associated with a processing index table will sooner or later be merged into the receiving index table, the distribution path table can be safely changed without moving data between processing index tables.

[0084] In a preferred embodiment, in order to fully utilize system resources, the processing index table and the receiving index table 940 are evenly distributed in the nodes. As mentioned above, the index processing device 920 can be a Kubernetes pod, managed by a microservice running in the Kubernetes pod.

[0085] In the index table arrangement according to the embodiment of the present disclosure, operations such as searching, acquiring documents, deleting documents, and clearing data can be performed on the index table.

[0086] For a search request, a term search or aggregate search is performed on all index tables. Each index table will perform the search process in parallel, and the index processing system will merge the results of all index tables and return the results to the user. A document usually appears in one index table. However, in rare cases, a document may appear in the receiving index table and a processing index table. In this case, the results should be deduplicated during the result merge.

[0087] For the Get Document Request, the Get Document Request is to obtain a specific document by document ID. The allocation table for the index table subset 930 is fixed, so we can use the document ID to locate a specific index table subset. However, since the allocation table for the processing index table and the receiving index table changes from time to time, it is difficult to determine which index table the document belongs to. Therefore, the Get Document Request should be sent to all index tables within the index table subset 930, and the document should be returned when any index table is hit.

[0088] For a delete document request, the delete request is made in the index table where the data is located. In some embodiments, a bitmap can be used to mark documents associated with the index table as "deleted". When a merge occurs, these documents will be permanently deleted. Since the index table subset 930 has a consistent allocation table, the delete request should be able to locate the index table subset 930 and then be sent to all index tables within the index table subset. In the case of processing streaming data, since streaming data is append-only, deleting specific documents in a streaming search system is not a common use case. Instead, a data purge request is always used.

[0089] For data purge requests, all index tables should be checked for old data. If all data in an index table needs to be purged, simply delete the entire index table. Otherwise, mark the documents associated with the index table as "deleted" and the index processing system will handle the actual removal when the index tables are merged.

[0090] Fig.10 1 shows a schematic block diagram of an example device 1000 that can be used to implement an embodiment of the present disclosure. Figure 1 The control device 110 shown may be implemented by the device 1000. Fig.10As shown, the device 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1002 or computer program instructions loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0091] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0092] The various processes and processing described above, such as methods 300, 700, and 800, may be performed by the processing device 1001. For example, in some embodiments, the methods 300, 700, and 800 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the CPU 1001, one or more actions of the methods 300, 700, and 800 described above may be performed.

[0093] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0094] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0095] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0096] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0097] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions can be provided to a processing device of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing device of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0099] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0100] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0101] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for managing an index table, include: receiving streaming data of a document, wherein the streaming data arrives at an index processing system according to an arrival rate; determining a number of indexing requests for documents received within a predetermined time, the indexing requests being received within a predetermined time period; Acquire information related to a processing index table in the index processing system, the processing index table being used to store indexes generated according to the index establishment request, the index processing system further comprising a receiving index table, the receiving index table being used to store at least a portion of the indexes in the processing index table; as well as Based on the number of the index creation requests and the acquired information, adjusting the number of processing index tables in the index processing system; wherein, for the first part of the streaming data, the index request for the first index processing device causes the first index processing device to index the documents of the first part into a first processing index table generated by the first index processing device; wherein, in response to the number of index establishment requests being greater than a first threshold number, for a second portion of the streaming data, a second index processing device is allocated to perform indexing of documents of the second portion into a second processing index table generated by the second index processing device, and the indexing of the documents of the second portion by the second index processing device and the indexing of the documents of the first portion into the first processing index table by the first index processing device are performed in parallel; and Among them, based on the comparison between the data amount corresponding to the receiving index table and the second threshold data amount, it is determined whether the first processing index table and the second processing index table are to be merged into a first receiving index table for searching, or whether a second receiving index table and a third receiving index table are to be generated from the first processing index table and the second processing index table respectively for searching.

2. The method according to claim 1, wherein obtaining information related to the processing index table comprises obtaining at least one of the following: the number of documents associated with each entry in the processing index table; The amount of data of each entry in the processing index table; The data volume of the document associated with each entry in the processing index table.

3. The method according to claim 1, wherein the number of processing index tables in the index processing system is adjusted include: In response to the number of index creation requests being lower than a second threshold number, a processing index table is deleted from existing processing index tables, wherein the second threshold number is lower than the first threshold number.

4. The method according to claim 3, wherein a processing index table is deleted from an existing processing index table. include: Selecting a target index processing device from a plurality of index processing devices storing the existing processing index tables in the index processing system, wherein the total number of processing index tables and receiving index tables stored in the target index processing device exceeds a first predetermined threshold; as well as One of the processing index tables stored in the target index processing device is deleted.

5. The method according to claim 3, wherein a processing index table is deleted from the existing processing index table include: Merging all entries in the processing index table to be deleted with the entries in the receiving index table; as well as Associating the document associated with the to-be-deleted processing index table with the receiving index table; as well as Delete the processing index table to be deleted.

6. The method according to claim 1, further comprising: include: Determine and process an excess portion in the index table, wherein the amount of data corresponding to the excess portion exceeds a first threshold amount of data; merging the entries in the excess portion with the entries in the received index table; Associating the document associated with the overrun portion with the received index table; as well as The excess portion is deleted from the processing index table.

7. The method according to claim 1, further comprising: include: In response to determining that the amount of data corresponding to the reception index table exceeds a second threshold amount of data, adding new entries to the reception index table is stopped.

8. The method according to claim 3, wherein the generation of the second processing index table include: In response to determining that the number of processed index tables stored in each of the existing index processing devices in the index processing system exceeds a second predetermined threshold, and the number of the existing index processing devices is lower than a third predetermined threshold, allocating the second index processing device; as well as At the second index processing device, the second processing index table is generated.

9. A device for managing an index table, include: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the apparatus to perform actions comprising: receiving streaming data of documents, wherein the streaming data arrives at the index processing system according to an arrival rate; determining a number of indexing requests for documents received within a predetermined time, the indexing requests being received within a predetermined time period; Acquire information related to a processing index table in an index processing system, the processing index table being used to store indexes generated according to the index establishment request, the index processing system further comprising a receiving index table, the receiving index table being used to store at least a portion of the indexes in the processing index table; and Based on the number of the index creation requests and the acquired information, adjusting the number of processing index tables in the index processing system; wherein, for the first part of the streaming data, the index request for the first index processing device causes the first index processing device to index the documents of the first part into a first processing index table generated by the first index processing device; wherein, in response to the number of index establishment requests being greater than a first threshold number, for a second portion of the streaming data, a second index processing device is allocated to perform indexing of documents of the second portion into a second processing index table generated by the second index processing device, and the indexing of the documents of the second portion by the second index processing device and the indexing of the documents of the first portion into the first processing index table by the first index processing device are performed in parallel; and Among them, based on the comparison between the data amount corresponding to the receiving index table and the second threshold data amount, it is determined whether the first processing index table and the second processing index table are to be merged into a first receiving index table for search, or whether a second receiving index table and a third receiving index table are to be generated from the first processing index table and the second processing index table respectively for search.

10. The device according to claim 9, wherein obtaining information related to the processing index table comprises obtaining at least one of the following: The number of documents associated with the index stored in the processing index table; The amount of data of each entry in the processing index table; The data volume of the document associated with each entry in the processing index table.

11. The apparatus according to claim 9, wherein the number of processing index tables in the index processing system is adjusted include: In response to the number of index creation requests exceeding a first threshold number, generating a new processing index table; as well as In response to the number of index creation requests being lower than a second threshold number, a processing index table is deleted from existing processing index tables, wherein the second threshold number is lower than the first threshold number.

12. The apparatus according to claim 11, wherein a processing index table is deleted from an existing processing index table. include: Selecting a target index processing device from a plurality of index processing devices storing the existing processing index tables in the index processing system, wherein the total number of processing index tables and receiving index tables stored in the target index processing device exceeds a first predetermined threshold; as well as One of the processing index tables stored in the target index processing device is deleted.

13. The apparatus according to claim 11, wherein a processing index table is deleted from an existing processing index table. include: Merging all entries in the processing index table to be deleted with the entries in the receiving index table; as well as Associating the document associated with the to-be-deleted processing index table with the receiving index table; as well as Delete the processing index table to be deleted.

14. The apparatus of claim 9, wherein the action further include: Determine and process an excess portion in the index table, wherein the amount of data corresponding to the excess portion exceeds a first threshold amount of data; merging the entries in the excess portion with the entries in the received index table; Associating the document associated with the overrun portion with the received index table; as well as The excess portion is deleted from the processing index table.

15. The apparatus of claim 9, wherein the action further include: In response to determining that the amount of data corresponding to the received index table exceeds a second threshold amount of data, stopping adding new entries to the received index table; as well as A new reception index table is generated at another index processing device different from the index processing device where the reception index table is located.

16. The apparatus according to claim 11, wherein the generation of the second processing index table include: In response to determining that the number of processed index tables stored in each of the existing index processing devices in the index processing system exceeds a second predetermined threshold, and the number of the existing index processing devices is lower than a third predetermined threshold, allocating the second index processing device; as well as At the second index processing device, the processing index table is generated.

17. A computer program product tangibly stored in a non-transitory computer storage medium and comprising machine executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1-8.