A retrieval method and device, electronic equipment and computer readable storage medium

Abstract

The present disclosure provides a retrieval method and device, electronic equipment and computer readable storage medium, the method comprising: loading a first index corresponding to a quantity of to-be-retrieved data in a running memory; generating index data corresponding to incrementally to-be-retrieved data obtained in real time; inserting the index data into the running memory to generate a second index node of the incrementally to-be-retrieved data in the running memory, obtaining a second index comprising the first index node and the second index node; and retrieving the full amount of to-be-retrieved data through the second index stored in the running memory to obtain a retrieval result. The first index of the quantity of to-be-retrieved data can be loaded in the running memory, and the corresponding index data can be generated according to the incrementally to-be-retrieved data, the index data is inserted into the running memory to obtain the second index, so that each retrieval task can retrieve the latest to-be-retrieved data through the second index, and the retrieval timeliness for the newly added to-be-retrieved data is high.

Description

Technical Field

[0001] This disclosure relates to the field of retrieval technology, and in particular to a retrieval method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of retrieval technology, current retrieval solutions typically do not directly search the data to be retrieved in order to improve retrieval efficiency. Instead, an index is built based on the data to be retrieved, and then the data to be retrieved is searched. In actual business scenarios, new data to be retrieved will be continuously generated over time, i.e., incremental data to be retrieved. It is necessary to fully consider the incremental data to be retrieved during the retrieval process to improve the completeness of the retrieval results.

[0003] In related technologies, a dual-index approach is typically used. First, when the retrieval service is running, an index corresponding to the existing data to be retrieved is generated based on the existing data to be retrieved and stored in the first memory space. The retrieval service is then provided through the index in the first memory space. After a period of time, a new index is generated based on the incremental data to be retrieved accumulated during this period and the aforementioned existing data to be retrieved. This new index is stored in the second memory space, and the retrieval service is then provided based on the index in the second memory space. After a period of time, the index in the first memory space is updated again, and this process is repeated.

[0004] The above retrieval scheme requires generating a complete index corresponding to the entire set of data to be retrieved each time the index is updated, which consumes a lot of resources and results in a very low index update frequency. This leads to poor timeliness in retrieving the latest data to be retrieved. In addition, the above retrieval scheme requires maintaining two sets of indexes in memory at the same time, resulting in a high memory usage rate. Summary of the Invention

[0005] This disclosure provides a retrieval method, apparatus, electronic device, and computer-readable storage medium to address the problem of poor timeliness in retrieving newly added data in related technologies.

[0006] In a first aspect, embodiments of this disclosure provide a retrieval method applied to a retrieval server, the method comprising:

[0007] A first index corresponding to the existing data to be retrieved is loaded into the running memory. The first index contains multiple first index nodes. The existing data to be retrieved represents the data to be retrieved generated before the first index is loaded into the running memory. The first index nodes are generated based on the existing data to be retrieved.

[0008] Generate index data corresponding to the incremental data to be retrieved in real time, wherein the incremental data to be retrieved represents the data to be retrieved generated after loading the first index into the running memory;

[0009] The index data is inserted into the running memory to generate a second index node for the incremental data to be retrieved in the running memory, resulting in a second index containing the first index node and the second index node;

[0010] The full set of data to be retrieved is retrieved by using the second index stored in the running memory, and the retrieval results are obtained. The full set of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0011] In one optional implementation, loading the first index corresponding to the existing data to be retrieved into the running memory includes:

[0012] The first index corresponding to the existing data to be retrieved is obtained from the index server, and the index server is used to generate the first index based on the existing data to be retrieved.

[0013] Based on the resource usage of the first index, apply for index storage space with a resource capacity in the running memory, wherein the resource capacity is greater than the resource usage;

[0014] The first index is loaded into the index storage space.

[0015] In an optional implementation, the method further includes:

[0016] Obtain the estimated runtime of the retrieval server, the previous runtime, and the historical resource usage of the first historical index loaded during the previous runtime of the retrieval server; wherein, the estimated runtime represents the estimated duration for which the retrieval server will need to continuously provide retrieval services;

[0017] Based on the previous running time, the historical resource usage, and the resource usage, determine the historical resource usage change rate of the first index;

[0018] The resource capacity is determined based on the estimated running time, the historical rate of change in resource usage, and the resource usage.

[0019] In one optional implementation, generating the index data corresponding to the incremental data to be retrieved in real time includes:

[0020] Determine the adjacent node information corresponding to the incremental data to be retrieved from the index storage space;

[0021] Based on the incremental data to be retrieved and the information of the adjacent nodes, index data corresponding to the incremental data to be retrieved is generated.

[0022] In one optional implementation, the indexes in the index storage space are stored in the index storage space in the form of skip lists, and the method further includes:

[0023] The index storage space is set to a lock-free programming mode so that the index storage space is readable during the insertion of index data.

[0024] In one optional implementation, inserting the index data into the runtime memory includes:

[0025] The index data is inserted into the free storage space of the index storage space, where the free storage space represents the storage space in the index storage space that does not contain any index data.

[0026] In one optional implementation, the step of retrieving the full amount of data to be retrieved using a second index stored in the running memory to obtain retrieval results includes:

[0027] Obtain the search keywords, and match the third index node from the second index that meets the preset similarity condition with the search keywords;

[0028] From the second index, determine a preset number of fourth index nodes whose distance from the third index node satisfies a preset distance condition;

[0029] The first data to be retrieved for generating the third index node and the second data to be retrieved for generating the fourth index node are obtained, and the retrieval results composed of the first data to be retrieved and the second data to be retrieved are obtained to complete the retrieval operation on the full amount of data to be retrieved.

[0030] In an optional implementation, the method further includes:

[0031] The incremental data to be retrieved is obtained from the write-ahead log, which is stored by the storage service. The write-ahead log is used to delete the incremental data to be retrieved when the number of reads of the incremental data to be retrieved reaches a preset number.

[0032] Secondly, this disclosure provides a retrieval method applied to a storage service client, the method comprising:

[0033] The existing data to be retrieved is written to the write-ahead log so that the index server can obtain the existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0034] The incremental data to be retrieved in real time is written to the write-ahead log so that the retrieval server can obtain the incremental data to be retrieved from the write-ahead log, generate index data corresponding to the incremental data to be retrieved, and insert the index data into the running memory loaded with the first index to generate a second index node corresponding to the incremental data to be retrieved in the running memory, thus obtaining a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved through the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0035] Thirdly, this disclosure provides a retrieval method applied to an indexing server, the method comprising:

[0036] The existing data to be retrieved is obtained from the write-ahead log, and a first index corresponding to the existing data to be retrieved is generated. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0037] The first index is stored in a distributed file system so that the retrieval server can obtain the first index from the distributed file system and load the first index into the running memory to generate index data corresponding to the incremental data to be retrieved in real time. The index data is then inserted into the running memory to generate a second index node corresponding to the incremental data to be retrieved in the running memory, resulting in a second index containing the first index node and the second index node. The full set of data to be retrieved is then retrieved using the second index stored in the running memory to obtain the retrieval results. The full set of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0038] Fourthly, embodiments of this disclosure provide a retrieval device, the device comprising:

[0039] The index loading module is configured to load a first index corresponding to the existing data to be retrieved in the running memory. The first index contains multiple first index nodes. The existing data to be retrieved represents the data to be retrieved generated before the first index is loaded in the running memory. The first index nodes are generated based on the existing data to be retrieved.

[0040] The index data module is configured to generate index data corresponding to the incremental data to be retrieved in real time, wherein the incremental data to be retrieved represents the data to be retrieved generated after loading the first index into the running memory;

[0041] The node insertion module is configured to insert the index data into the running memory to generate a second index node for the incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node.

[0042] The data retrieval module is configured to retrieve the full set of data to be retrieved using a second index stored in the running memory, and to obtain retrieval results. The full set of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0043] In one optional implementation, the index loading module includes:

[0044] The first index submodule is configured to obtain the first index corresponding to the existing data to be retrieved from the index server, and the index server is used to generate the first index based on the existing data to be retrieved.

[0045] The index space submodule is configured to request an index storage space with a resource capacity in the running memory based on the resource occupancy of the first index, wherein the resource capacity is greater than the resource occupancy.

[0046] The index loading submodule is configured to load the first index into the index storage space.

[0047] In one alternative embodiment, the device further includes:

[0048] The acquisition submodule is configured to acquire the estimated runtime of the retrieval server, the previous runtime, and the historical resource usage of the historical first index loaded during the previous runtime of the retrieval server; wherein, the estimated runtime represents the estimated time that the retrieval server needs to continuously provide retrieval services;

[0049] The historical resource usage change rate submodule is configured to determine the historical resource usage change rate of the first index based on the previous running time, the historical resource usage, and the resource usage.

[0050] The space size submodule is configured to determine the resource capacity based on the estimated runtime, the historical resource usage change rate, and the resource usage.

[0051] In one optional implementation, the index data module includes:

[0052] The adjacent node information submodule is configured to determine the adjacent node information corresponding to the incremental data to be retrieved from the index storage space;

[0053] The index data submodule is configured to generate index data corresponding to the incremental data to be retrieved based on the incremental data to be retrieved and the neighboring node information.

[0054] In one optional implementation, the indexes in the index storage space are stored in the index storage space in the form of skip lists, and the apparatus further includes:

[0055] The storage mode setting module is configured to set the storage mode of the index storage space to lock-free programming mode, so that the index storage space is readable during the insertion of the index data.

[0056] In one optional implementation, the node insertion module includes:

[0057] The node insertion submodule is configured to insert the index data into the free storage space of the index storage space, where the free storage space represents the storage space in the index storage space that does not contain any index data.

[0058] In one optional implementation, the data retrieval module includes:

[0059] The third index node submodule is configured to obtain search keywords and match third index nodes from the second index that meet preset similarity conditions with the search keywords;

[0060] The fourth index node submodule is configured to determine a preset number of fourth index nodes from the second index that satisfy a preset distance condition with respect to the third index node;

[0061] The retrieval results submodule is configured to obtain the first data to be retrieved to generate the third index node and the second data to be retrieved to generate the fourth index node, and to obtain the retrieval results composed of the first data to be retrieved and the second data to be retrieved, so as to complete the retrieval operation on the full amount of data to be retrieved.

[0062] In one alternative embodiment, the device further includes:

[0063] The incremental data to be retrieved module is configured to retrieve incremental data to be retrieved stored by the storage service client from the write-ahead log. The write-ahead log is used to delete the incremental data to be retrieved when the number of times the incremental data to be retrieved is read reaches a preset number.

[0064] Fifthly, embodiments of this disclosure provide a retrieval device, the device comprising:

[0065] The existing data writing module is configured to write existing data to be retrieved into a write-ahead log, so that the index server can obtain the existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0066] The incremental data writing module is configured to write the incremental data to be retrieved in real time into the write-ahead log, so that the retrieval server can obtain the incremental data to be retrieved from the write-ahead log, generate index data corresponding to the incremental data to be retrieved, and insert the index data into the running memory loaded with the first index to generate a second index node corresponding to the incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved through the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0067] Sixthly, embodiments of this disclosure provide a retrieval device, the device comprising:

[0068] The existing data to be retrieved module is configured to retrieve existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0069] An index storage module is configured to store the first index in a distributed file system, so that the retrieval server obtains the first index from the distributed file system, loads the first index into the running memory, generates index data corresponding to the incremental data to be retrieved in real time, inserts the index data into the running memory to generate a second index node corresponding to the incremental data to be retrieved in the running memory, and obtains a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved using the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0070] In a seventh aspect, embodiments of this disclosure also provide an electronic device, including a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the retrieval method.

[0071] Eighthly, embodiments of this disclosure also provide a storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the retrieval method.

[0072] Ninthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the retrieval method.

[0073] In this embodiment, the method includes: loading a first index corresponding to existing data to be retrieved into the running memory, the first index containing multiple first index nodes, the existing data to be retrieved representing the data to be retrieved generated before the first index is loaded into the running memory, and the first index nodes being generated based on the existing data to be retrieved; generating index data corresponding to incremental data to be retrieved in real time, the incremental data to be retrieved representing the data to be retrieved generated after the first index is loaded into the running memory; inserting the index data into the running memory to generate a second index node for the incremental data to be retrieved in the running memory, resulting in a second index containing the first index node and the second index node; and retrieving the full amount of data to be retrieved using the second index stored in the running memory to obtain retrieval results, the full amount of data to be retrieved including existing data to be retrieved and incremental data to be retrieved. In this embodiment of the disclosure, a first index of existing data to be retrieved can be loaded into the running memory, and after receiving incremental data to be retrieved, corresponding index data is generated based on the incremental data to be retrieved. The index data is then continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, resulting in high retrieval timeliness for newly added data to be retrieved. Furthermore, only one index needs to be maintained in the running memory, significantly reducing the memory usage of the retrieval service.

[0074] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0075] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0076] Figure 1 This is a flowchart of the steps of a retrieval method provided in an embodiment of the present disclosure;

[0077] Figure 2 This is a schematic diagram of a second index provided in this disclosure;

[0078] Figure 3 This is a schematic diagram of a dual-index scheme provided in an embodiment of this disclosure;

[0079] Figure 4 This is a schematic diagram of an index merging scheme provided in an embodiment of this disclosure;

[0080] Figure 5 This is a schematic diagram of a layered, navigable small world provided in an embodiment of this disclosure;

[0081] Figure 6 This is a flowchart of another retrieval method provided in an embodiment of the present disclosure;

[0082] Figure 7 This is a node storage loading intent provided in an embodiment of the present disclosure;

[0083] Figure 8 This is a schematic diagram of an index node storage structure provided in an embodiment of this disclosure;

[0084] Figure 9 This is a flowchart illustrating the steps of another retrieval method provided in this embodiment of the disclosure;

[0085] Figure 10 This is a flowchart of another retrieval method provided in this embodiment of the present disclosure;

[0086] Figure 11 This is a schematic diagram of a retrieval system provided in an embodiment of this disclosure;

[0087] Figure 12 This is a block diagram of a retrieval device provided in an embodiment of this disclosure;

[0088] Figure 13 This is a block diagram of another retrieval device provided in an embodiment of this disclosure;

[0089] Figure 14 This is a block diagram of yet another retrieval device provided in the embodiments of this disclosure;

[0090] Figure 15 This is a block diagram of an electronic device according to an embodiment of the present disclosure;

[0091] Figure 16 This is a block diagram of an electronic device according to another embodiment of the present disclosure. Detailed Implementation

[0092] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0093] Figure 1 This is a flowchart illustrating the steps of a retrieval method provided in an embodiment of this disclosure, applicable to electronic devices, such as... Figure 1 As shown, the method may include:

[0094] Step 101: Load the first index corresponding to the existing data to be retrieved into the running memory. The first index contains multiple first index nodes. The existing data to be retrieved represents the data to be retrieved generated before loading the first index into the running memory. The first index nodes are generated based on the existing data to be retrieved.

[0095] In this embodiment of the disclosure, the data to be retrieved can be media data, such as text, images, and videos, or feature data generated from media data, such as feature vectors. For example, in one application scenario, a vector retrieval service can be provided to a customer: the customer retrieves multiple images as needed, and feature vectors corresponding to each image are generated. The customer provides these feature vectors as the data to be retrieved. Thus, when the customer needs to retrieve these images, they can generate retrieval vectors according to the retrieval request. The retrieval method of this embodiment can retrieve the data to be retrieved composed of the feature vectors of the images using the retrieval vectors, and return the retrieval results of the image feature vectors to the user. The customer can then determine the corresponding image based on these retrieval results to obtain retrieval results for multiple images.

[0096] It should be noted that the embodiments of this disclosure provide a basic search service. Customers can flexibly choose the data type to be searched according to their own needs. The embodiments of this disclosure do not specifically limit the data type of the data to be searched.

[0097] In actual production, as time goes on and business operations continue, the data to be retrieved will continuously increase. When the retrieval service is running (e.g., when the retrieval server starts up), a first index of the already determined existing data to be retrieved can be loaded into the runtime memory, and retrieval services can be provided for the existing data to be retrieved through the first index in the runtime memory. Here, the existing data to be retrieved refers to the data to be retrieved that has been determined during the runtime of the retrieval service.

[0098] In this embodiment, the first index can be constructed based on Approximate Nearest Neighbor (ANN) retrieval. To meet the requirements of approximate nearest neighbor retrieval, the first index can include index types that support node insertion, such as graph indexes and tree indexes. Graph indexes can include Hierarchical Navigable Small World (HNSW), Navigable Small World (NSW), and Navigating Spreading-out Graph (NSG); tree indexes can include k-dimensional trees (kd-tree), ball trees, etc. These index types differ in accuracy, efficiency, and data type compatibility. Those skilled in the art can flexibly choose the required index type based on the performance of the retrieval server and the data type of the data to be retrieved. This embodiment does not impose specific limitations on this.

[0099] Running memory, also known as main memory, refers to the memory required for program execution. It is used to temporarily store data that needs to be exchanged with the CPU at high speed, and generally refers to Random Access Memory (RAM). By loading the index into running memory, the speed at which the search service reads the index can be improved, thereby increasing the search efficiency of the search service.

[0100] Step 102: Generate index data corresponding to the incremental data to be retrieved in real time. The incremental data to be retrieved refers to the data to be retrieved generated after loading the first index into the running memory.

[0101] After the retrieval service is running, new data to be retrieved will continuously be generated over time. For example, if the data to be retrieved consists of feature vectors of various videos on a video website, as time goes on, users will continuously upload new videos to the website, generating new feature vectors and thus continuously producing new data to be retrieved. In this embodiment, the data to be retrieved obtained after adding the first index to the runtime memory can be referred to as incremental data to be retrieved.

[0102] For example, if the search service starts running at 9:00 AM, and the number of existing data entries to be searched is 100, then the first index generated from these 100 entries will be loaded into the runtime memory. If, at 9:10 AM, a customer sends a new data entry to be searched, then this newly sent data entry will be identified as the incremental data entry to be searched, and the number of existing data entries to be searched will be 100. If, at 9:20 AM, a customer sends another new data entry to be searched, then this newly sent data entry will also be identified as the incremental data entry to be searched.

[0103] After obtaining the incremental data to be retrieved, index data corresponding to the incremental data to be retrieved can be generated based on the first index. The index data can include the incremental data to be retrieved, as well as information about adjacent index nodes. It should be noted that, since the incremental data to be retrieved increases continuously over time, whenever a new incremental data to be retrieved is obtained, all the data to be retrieved prior to that incremental data can be referred to as existing data to be retrieved. The index formed by these existing data can be the first index relative to the most recently received incremental data to be retrieved.

[0104] Step 103: Insert the index data into the running memory to generate a second index node for the incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node.

[0105] In this embodiment of the disclosure, in order to maintain high retrieval performance, the retrieval service needs to read the index from the running memory and perform retrieval based on the index in the running memory. Therefore, in order to enable users to retrieve the newly added incremental data to be retrieved and improve the timeliness of the retrieval service, after generating the index data corresponding to the incremental data to be retrieved, the index data can be inserted into the running memory, so that it serves as a second index node in memory. Together with the first index node in the first index that already exists in memory, it constitutes the second index. Since the second index is already in the running memory, it can be retrieved directly. While maintaining high retrieval efficiency, it also enables the incremental data to be retrieved.

[0106] Reference Figure 2 , Figure 2 A schematic diagram of a second index provided in this disclosure is shown, such as Figure 2 As shown, nodes 1 to 5 are the first index nodes. If the second index node (node ​​6) generated from the incremental data to be retrieved is adjacent to nodes 2 and 4, then the second index node is inserted between nodes 2 and 4, resulting in the following: Figure 2 The second index shown allows for the retrieval of both existing and incremental data.

[0107] Step 104: Search the full set of data to be searched using the second index stored in the running memory to obtain search results. The full set of data to be searched includes the existing data to be searched and the incremental data to be searched.

[0108] In this embodiment of the disclosure, all currently received data to be retrieved (including existing data to be retrieved and incremental data to be retrieved) can be referred to as the full data to be retrieved. Since the second index includes not only the first index node corresponding to the existing data to be retrieved, but also the second node corresponding to the incremental data to be retrieved, the full data to be retrieved can be retrieved by running the second index in memory.

[0109] It should be noted that in the embodiments of this application, the index node can directly contain the data to be retrieved (direct storage). However, since the index node is loaded in memory and memory resources are relatively precious, the data to be retrieved and its corresponding index node can be stored separately to avoid the data to be retrieved directly occupying memory space. That is, the index node can also only contain the storage location information of the data to be retrieved (indirect storage), and the data to be retrieved can be obtained from the storage location of the data to be retrieved through the storage location information in the index node.

[0110] Specifically, after obtaining the search terms that the customer needs to retrieve, the search terms can be matched with the second index. Based on the second index, one or more data items with the highest matching degree with the search terms can be determined to obtain the search results, thereby realizing the retrieval operation of the full set of data to be retrieved.

[0111] Reference Figure 3 , Figure 3 A schematic diagram of a dual-index scheme provided in an embodiment of this disclosure is shown, such as... Figure 3 As shown, in the current related technologies, firstly, index 1 is generated in buffer 1 based on the existing data to be retrieved, then index 1 is loaded and retrieval services are provided through index 1. After a period of time, a new index 2 is generated in buffer 2 based on the existing data to be retrieved and the incremental data to be retrieved generated during this period, then index 2 is loaded and retrieval services are provided through index 2, and then the index in buffer 1 is cleared, and so on.

[0112] It can be seen that the retrieval method provided in this embodiment has higher timeliness and requires relatively less computing resources compared to the indexing scheme described above.

[0113] Reference Figure 4 , Figure 4 A schematic diagram of an index merging scheme provided by an embodiment of this disclosure is shown, such as... Figure 4As shown, in current related technologies, a large index is first generated based on existing data to be retrieved. After a period of time, a small index is generated based on incremental data to be retrieved generated during this period. Therefore, the retrieval party can search the existing data to be retrieved using the large index, and search the incremental data to be retrieved that has already generated a small index using the small index. It can also search the incremental data to be retrieved that has not yet generated a small index. It should be noted that since searching the incremental data to be retrieved without generating a small index cannot be done using the index, only traditional search methods can be used. The search process consumes a lot of computing resources; therefore, this search is usually optional, and technicians need to make a trade-off between timeliness and efficiency.

[0114] As can be seen, the retrieval method provided in this embodiment, compared with the above indexing scheme, can insert the incremental data to be retrieved into the running memory at the first moment to obtain a complete second index. Compared with the above index merging scheme, it ensures higher retrieval timeliness without sacrificing retrieval efficiency.

[0115] It should be noted that, in this embodiment, the index can be a hierarchical navigable small-world graph. That is, the first index can be a first hierarchical navigable small-world graph, and the first index node can be a first graph node. The second index can be a second hierarchical navigable small-world graph, and the second index node can be a second graph node. This allows loading the first hierarchical navigable small-world graph corresponding to the existing data to be retrieved into the runtime memory. The first hierarchical navigable small-world graph contains multiple first graph nodes. Second graph nodes corresponding to the incremental data to be retrieved in real-time are generated. The second graph nodes are inserted into the runtime memory to obtain a second hierarchical navigable small-world graph containing both first and second graph nodes. The entire dataset to be retrieved is then retrieved using the second hierarchical navigable small-world graph stored in the runtime memory to obtain the retrieval results.

[0116] Reference Figure 5 , Figure 5 This illustration shows a schematic diagram of a layered, navigable small world provided by an embodiment of this disclosure, such as... Figure 5 As shown, since the hierarchical navigable small-world graph introduces the concept of hierarchy, a connected graph containing all nodes can be constructed in the 0th layer. As the number of layers increases, the number of nodes in each layer gradually decreases, and the layers are connected by the same points. When querying, the search can start from the highest layer, find the matching node of the highest layer from the relatively small number of nodes in the highest layer, and then search downwards based on the matching node of the highest layer, finally determining the closest node from the 0th layer. There is no need to traverse all nodes, which can improve the search efficiency.

[0117] This disclosure provides a retrieval method, comprising: loading a first index corresponding to existing data to be retrieved into runtime memory, the first index containing multiple first index nodes, the existing data to be retrieved representing the data to be retrieved generated before loading the first index into runtime memory, and the first index nodes being generated based on the existing data to be retrieved; generating index data corresponding to incremental data to be retrieved in real time, the incremental data to be retrieved representing the data to be retrieved generated after loading the first index into runtime memory; inserting the index data into runtime memory to generate a second index node for the incremental data to be retrieved in runtime memory, resulting in a second index containing the first index node and the second index node; and retrieving the full amount of data to be retrieved using the second index stored in runtime memory to obtain retrieval results, the full amount of data to be retrieved including existing data to be retrieved and incremental data to be retrieved. In this embodiment of the present disclosure, a first index of existing data to be retrieved can be loaded into the running memory, and after receiving incremental data to be retrieved, a corresponding second index node is generated based on the incremental data to be retrieved. The second index node is continuously inserted into the first index, so that each retrieval task can retrieve the latest data to be retrieved. This has high retrieval timeliness for newly added data to be retrieved, and only one index needs to be maintained in the running memory, which significantly reduces the memory usage of the retrieval service.

[0118] This disclosure provides another retrieval method, applied to a retrieval server, as described below. Figure 6 , Figure 6 This is a flowchart of another retrieval method provided in this disclosure embodiment, such as... Figure 6 As shown, the method may include:

[0119] Step 201: Obtain the first index corresponding to the existing data to be retrieved from the index server. The index server is used to generate the first index based on the existing data to be retrieved.

[0120] In this embodiment of the disclosure, the retrieval server can obtain the first index corresponding to the existing data to be retrieved from the index server, and the index server can generate the corresponding first index based on the existing data to be retrieved.

[0121] In one implementation, after the retrieval server is started, it can send a request to the index server to obtain the first index. The index server can respond to the request by sending the first index to the retrieval server, thereby enabling the retrieval server to obtain the first index.

[0122] In another implementation, the index server can store the first index in a common storage space that can be accessed by both the retrieval server and the index server, so that the retrieval server can actively retrieve the first index from the common storage space, which helps to improve the efficiency of system operation.

[0123] It should be noted that the retrieval server and the indexing server can be different physical machines or different service programs running on the same physical machine. Technicians can flexibly deploy the retrieval server and the indexing server according to actual needs, and this disclosure does not impose specific limitations on this.

[0124] Step 202: Based on the resource usage of the first index, apply for index storage space with a resource capacity in the running memory, wherein the resource capacity is greater than the resource usage.

[0125] When loading the first index into the running memory, storage space in the running memory needs to be allocated. Since the second index node corresponding to the incremental data to be retrieved needs to be added to the running memory in subsequent steps, when allocating the index storage space to store the first index, it is necessary to allocate storage space exceeding the resource usage of the first index to avoid subsequent memory overflow issues.

[0126] Specifically, a preset multiplier of the resource usage of the first index can be used as the resource capacity of the requested index storage space. For example, if the resource usage of the first index is 2GB and the preset multiplier is 2, then 4GB of index storage space will be requested in the running memory. Alternatively, the sum of the first index and a preset increment can be used as the resource capacity of the requested index storage space. For example, if the resource usage of the first index is 2GB and the preset increment is 3, then 5GB of index storage space will be requested in the running memory. Here, the preset multiplier is greater than 1, and the preset increment is greater than 0. It should be noted that those skilled in the art can also choose other methods to determine the resource capacity; this disclosure does not specifically limit the method of determining the resource capacity.

[0127] In this embodiment of the disclosure, when loading the first index into the running memory, an index storage space greater than the resource usage of the first index can be requested in memory, so that more index nodes can be inserted into the first index in the running memory, avoiding the problem of memory overflow caused by subsequent insertion of index nodes.

[0128] Optionally, in this embodiment of the disclosure, when processing a retrieval task, it is necessary to read the index in the index storage space. When processing incremental data to be retrieved, it is necessary to write a second index node into the index storage space. To enable the index storage space to perform both write and read operations simultaneously, the storage method of the index storage space can be set to lock-free programming when applying for the index storage space, so that the index storage space is in a readable state during the insertion of the index data. It should be noted that when the storage method of the index storage space is set to lock-free, the index can be loaded into the index storage space in the form of a skip list to meet the data requirements of the lock-free method.

[0129] By setting the index storage space to lock-free mode and storing the indexes in the index storage space as skip lists, simultaneous read and write operations can be performed on the indexes in the index storage space, thereby improving the concurrent read and write capabilities of the index storage space and thus improving retrieval efficiency.

[0130] Optionally, step 202 may include:

[0131] Sub-step 2021: Obtain the estimated running time of the retrieval server, the previous running time, and the historical resource usage of the historical first index loaded during the previous running of the retrieval server; wherein, the estimated running time represents the time that the retrieval server is expected to continuously provide retrieval services.

[0132] Determining resource capacity in a fixed manner may result in the index storage space being insufficient to meet actual requirements, leading to problems of being too large or too small. If the index storage space is too large, it will result in wasted storage space; if the index storage space is too small, it may cause system errors. Therefore, in this embodiment of the disclosure, the required index storage space resource capacity can also be accurately calculated based on the estimated running time of the retrieval server, the previous running time, and the historical resource usage of the historical first index corresponding to the previous running time of the retrieval server.

[0133] Specifically, the first step is to obtain the estimated runtime, previous runtime, and historical resource usage of the first historical index corresponding to the previous runtime of the retrieval server. The estimated runtime indicates the expected duration for which the current indexing service will continue to provide indexing services; this time can be specified by the client or determined according to certain rules. The previous runtime indicates the duration for which the indexing server last continuously provided indexing services, and the historical first index indicates the first index loaded into memory when the indexing server last booted up.

[0134] Sub-step 2022: Based on the previous running time, the historical resource usage, and the resource usage, determine the historical resource usage change rate of the first index.

[0135] By analyzing the previous running time, the historical resource usage of the first index, and the resource usage of the first index, the rate of change in the historical resource usage of the first index during the previous running period of the server can be determined.

[0136] For example, if the previous running time was 50 hours (h), the historical resource usage was 5GB, and the resource usage was 6GB, then the historical resource usage change rate can be calculated as (6-5) / 50 = 0.02GB / h.

[0137] Sub-step 2023: Determine the resource capacity based on the estimated running time, the historical resource usage change rate, and the resource usage.

[0138] After obtaining the historical resource usage change rate, the expected increase in the entire index's resource usage during the current run can be determined by using the historical resource usage change rate and the expected running time. The sum of the expected increase in resource usage and the actual resource usage is then used to determine the resource capacity, thereby achieving accurate determination of the resource capacity. This avoids situations where the resource capacity is too large, leading to memory waste, or where the resource capacity is too small, leading to memory overflow, thus improving the stability and efficiency of system operation.

[0139] For example, if the historical resource usage change rate is 0.02 GB / h, the expected running time is 100 hours, and the resource usage is 6 GB, then the expected increase in resource usage is 2 GB, and the resource capacity is 8 GB.

[0140] To further improve the stability of system operation, an additional redundant capacity can be added on top of the expected increase in resource consumption and the sum of resource consumption, resulting in a resource capacity. This ensures that the index storage space requested based on the resource capacity contains a certain amount of redundancy to cope with sudden data increments during operation.

[0141] Step 203: Load the first index into the index storage space.

[0142] In this embodiment of the disclosure, after allocating index storage space from the running memory, the first index can be loaded into the index storage space of the running memory. The running memory can be the running memory of the physical machine where the retrieval server resides, or it can be the running memory of other storage servers.

[0143] Step 204: Obtain incremental data to be retrieved from the write-ahead log, which is stored by the storage service client. The write-ahead log is used to delete the incremental data to be retrieved when the number of reads of the incremental data to be retrieved reaches a preset number.

[0144] In this embodiment of the disclosure, the complete retrieval system may include a storage service server in addition to a retrieval server and an index server. The storage service server can obtain incremental data to be retrieved from clients and store the incremental data in a write-ahead log, so that the retrieval server can retrieve the incremental data from the write-ahead log. To improve task processing capabilities, the retrieval system may include multiple servers of each type.

[0145] Write-Ahead Logging (WAL) is a technique used in database systems to provide atomicity and durability. WAL ensures that any changes to the database are written to the log before they actually take effect. WAL offers several advantages: reads and writes can be performed concurrently without blocking each other; performance is generally better; and disk I / O behavior is more predictable.

[0146] In this embodiment, to improve retrieval performance, multiple retrieval servers can be set up simultaneously to respond to multiple retrieval commands, thereby increasing the number of retrieval tasks that can be processed per unit time. Therefore, by setting up a write-ahead log, data blocking caused by direct communication between the storage service server and the index server can be avoided, thus improving the overall system efficiency. Furthermore, the write-ahead log can count the number of reads of its data and delete the corresponding data when the number of reads of a certain data reaches a preset number, thereby promptly clearing useless data and avoiding space waste.

[0147] Specifically, based on the number of servers in the system, the target number of times the incremental data to be retrieved stored in the write-ahead log needs to be read can be determined, and the read count of the storage folder for the incremental data to be retrieved in the write-ahead log can be configured to the target number. This will enable the write-ahead log to automatically delete the incremental data to be retrieved when the number of reads reaches the target number, thereby improving the space utilization of the write-ahead log.

[0148] Reference Figure 7 , Figure 7 This illustration shows an intent to load a node storage according to an embodiment of the present disclosure, such as... Figure 7 As shown, the retrieval server can obtain incremental data to be retrieved from the preset log, generate the second node information corresponding to the incremental data to be retrieved, and then insert the second node information into the empty position of the index storage space.

[0149] Step 205: Determine the neighboring node information corresponding to the incremental data to be retrieved from the index storage space.

[0150] After obtaining the incremental data to be retrieved, the adjacent node information corresponding to the incremental data to be retrieved can be determined from the index storage space so as to generate the second index node corresponding to the incremental data to be retrieved in the subsequent process.

[0151] It should be noted that the method for determining adjacent node information will differ depending on the index type. Taking a hierarchical navigable small world graph as an example, the method for determining adjacent node information will be explained as follows:

[0152] In the hierarchical navigable microworld, each node in each layer records the corresponding data to be retrieved and information about adjacent nodes. First, the target layer into which the second index node corresponding to the incremental data to be retrieved needs to be inserted is determined (this can be determined randomly). Then, the nodes adjacent to the incremental data to be retrieved in the target layer and in each layer below the target layer are determined. Based on the correspondence between these adjacent nodes and the layers, the information about adjacent nodes corresponding to the data to be retrieved is established.

[0153] like Figure 5 As shown, the hierarchical navigable small world contains three layers. If the highest layer corresponding to an incremental data to be retrieved is determined to be layer 1, then the neighboring nodes of the incremental data to be retrieved in layer 1 can be determined based on the similarity between the incremental data to be retrieved and the data to be retrieved in each node of layer 1; and the neighboring nodes of the incremental data to be retrieved in layer 0 can be determined based on the similarity between the incremental data to be retrieved and the data to be retrieved in each node of layer 0. Then, the neighboring node information containing the neighboring nodes in layer 1 and the neighboring nodes in layer 2 is obtained.

[0154] It should be noted that in this embodiment of the disclosure, incremental data to be retrieved will be continuously received during system operation. In order to improve the accuracy of node insertion, the adjacent node information of the most recently received incremental data to be retrieved can be determined according to all index nodes stored in the index storage space. The index nodes stored in the index storage space include all first index nodes in the first index, as well as the index nodes corresponding to all incremental data to be retrieved (except for the most recently obtained incremental data to be retrieved) after the first index is loaded into the running memory.

[0155] Step 206: Based on the incremental data to be retrieved and the neighboring node information, generate index data corresponding to the incremental data to be retrieved.

[0156] After determining the neighboring node information corresponding to the incremental data to be retrieved, index data corresponding to the incremental data to be retrieved can be generated based on the incremental data to be retrieved and the neighboring node information.

[0157] It should be noted that the specific method for generating index data will differ depending on the index type. Taking a hierarchical navigable small world graph as an example, the method for generating index data will be explained as follows:

[0158] The hierarchical navigable small-world graph is stored in runtime memory using two tables. One table records the first node information, consisting of the node's adjacent nodes at level 0 and its attributes. The other table records the second node information, consisting of the node's adjacent nodes at other levels. Therefore, when the index is a hierarchical navigable small-world graph, the index data corresponding to the incremental data to be retrieved can be composed of the aforementioned first and second node information.

[0159] Reference Figure 8 , Figure 8 A schematic diagram of an index node storage structure provided in an embodiment of this disclosure is shown, such as... Figure 8 As shown, the Level 0 data (data_level0_memory) records the underlying index node data contained in Level 0 of the hierarchical navigable small world (for example, nodes 0 and 1 in the figure represent Level 0 data of one index node each). Based on the data structure in Level 0 data, Level 0 node information corresponding to incremental data to be retrieved can be generated. Specifically, Level 0 node information may include the number (size) of neighboring nodes in that level, a status bit (flag), a reserved bit (reserved), neighboring node identifiers (neighbors), the data to be retrieved (data), and the node's identifier (label). The LinkLists is a two-dimensional array where each row represents the neighbor relationships of a node (e.g., node 0, node 1, node 2) from Level 1 to the maximum level. The data structure of each node at each level mainly includes: the number (size) of neighboring nodes in that level, a reserved bit (reserved), and neighboring node identifiers (neighbors).

[0160] In this embodiment of the disclosure, the index data corresponding to the latest incremental data to be retrieved can be determined based on the complete index corresponding to the existing full amount of data to be retrieved. When the index data is added to memory as a second index node, it can not only form an adjacent relationship with the first index node, but also form an adjacent relationship with the index node corresponding to the incremental data to be retrieved added before the current time, thereby improving the completeness and accuracy of subsequent retrieval results.

[0161] Step 207: Insert the index data into the free storage space of the index storage space, where the free storage space represents the storage space in the index storage space that does not contain any index data.

[0162] Since an index storage space is allocated in the running memory to store the index, after obtaining the index data, the index data can be inserted into the index storage space as a second index node to obtain a second index composed of all the index nodes stored in the index storage space.

[0163] Specifically, the index data can be inserted as a second index node into the free storage space adjacent to the end index node of the index storage space to obtain a second index composed of all index nodes of the index storage space. The free storage space refers to the storage space in the index storage space where no index node is stored.

[0164] In this embodiment of the disclosure, a second index node can be added to the index storage space by means of continuous storage. The address in the index storage space can be read continuously to obtain the complete second index, so that the second index stored in the index storage space has high reading efficiency when it is used.

[0165] Step 208: Search the full set of data to be searched using the second index stored in the running memory to obtain search results. The full set of data to be searched includes the existing data to be searched and the incremental data to be searched.

[0166] The following methods can be used to retrieve the full amount of data to be retrieved through the second index.

[0167] The system retrieves the search keywords and matches the third index node from the second index with the highest similarity to the search keywords. It then determines a predetermined number of fourth index nodes from the second index that are closest to the third index node. Based on the third and fourth index nodes, it retrieves the search results corresponding to the search keywords to complete the retrieval of all data to be searched.

[0168] Specifically, please refer to the following sub-steps 2081 to 2083:

[0169] Sub-step 2081: Obtain the search keywords and match the third index node from the second index that meets the preset similarity condition with the search keywords.

[0170] The retrieval server can obtain search keywords and match them in the second index with third index nodes whose similarity to the search keywords meets preset similarity criteria. These preset similarity criteria may include, but are not limited to, highest similarity or similarity exceeding a preset similarity threshold. The search keywords must be of the same data type as the data to be retrieved; for example, if the data to be retrieved is a vector, the search keywords must also be vectors.

[0171] Sub-step 2082: From the second index, determine a preset number of fourth index nodes whose distance from the third index node satisfies a preset distance condition.

[0172] Typically, search results need to contain a lot of data to be retrieved. Therefore, after determining the third index node, a preset number of fourth index nodes that are within a preset distance condition from the second index can be matched based on the third index node. The preset distance condition may include, but is not limited to, nearest, or less than a preset distance threshold.

[0173] The distance between two index nodes can be determined by the number of nodes required to connect the two index nodes, such as... Figure 2 As shown, if node 2 and node 5 are directly connected, the distance between node 2 and node 5 can be represented as 0. If node 2 and node 4 are indirectly connected through node 6, the distance between node 2 and node 4 can be represented as 1.

[0174] The preset quantity can be determined based on the quantity that needs to be retrieved. This embodiment of the disclosure does not impose a specific limit on the preset quantity, and those skilled in the art can adjust it flexibly according to actual needs.

[0175] For example, in such Figure 2 In the second index shown, if the third index node determined based on the search keywords is node 4, then the distance between the third index node and nodes 5 and 6 is 0, the distance between the third index node and nodes 2 and 3 is 1, and the distance between the third index node and node 1 is 2. With a preset number of 4, nodes 2, 3, 5, and 6 can be determined as the fourth index node.

[0176] Sub-step 2083: Obtain the first data to be retrieved to generate the third index node and the second data to be retrieved to generate the fourth index node, and obtain the retrieval results composed of the first data to be retrieved and the second data to be retrieved, so as to complete the retrieval operation on the full amount of data to be retrieved.

[0177] Since the index nodes contain the corresponding data to be retrieved in a direct or indirect manner, after determining the third and fourth index nodes, the corresponding data to be retrieved can be determined based on each third and fourth index node, thereby obtaining the retrieval results composed of this data.

[0178] The entire dataset can be retrieved using search terms and a second index, generating search results containing the required amount of data, resulting in highly complete and customizable search results.

[0179] This disclosure provides another retrieval method, comprising: loading a first index corresponding to existing data to be retrieved into runtime memory, the first index containing multiple first index nodes, the existing data to be retrieved representing the data to be retrieved generated before loading the first index into runtime memory, and the first index nodes being generated based on the existing data to be retrieved; generating index data corresponding to incremental data to be retrieved in real time, the incremental data to be retrieved representing the data to be retrieved generated after loading the first index into runtime memory; inserting the index data into runtime memory to generate a second index node for the incremental data to be retrieved in runtime memory, resulting in a second index containing the first index node and the second index node; and retrieving the full amount of data to be retrieved using the second index stored in runtime memory to obtain retrieval results, the full amount of data to be retrieved including both existing data to be retrieved and incremental data to be retrieved. In this embodiment of the disclosure, a first index of existing data to be retrieved can be loaded into the running memory, and after receiving incremental data to be retrieved, corresponding index data is generated based on the incremental data to be retrieved. The index data is then continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, resulting in high retrieval timeliness for newly added data to be retrieved. Furthermore, only one index needs to be maintained in the running memory, significantly reducing the memory usage of the retrieval service.

[0180] This disclosure provides yet another retrieval method, applied to a storage service client, as described below. Figure 9 , Figure 9 This is a flowchart illustrating the steps of another retrieval method provided in this disclosure embodiment, such as... Figure 9 As shown, the method may include:

[0181] Step 301: Write the existing data to be retrieved into the write-ahead log so that the index server can obtain the existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0182] Step 302: Write the incremental data to be retrieved in real time into the write-ahead log so that the retrieval server can obtain the incremental data to be retrieved from the write-ahead log, generate index data corresponding to the incremental data to be retrieved, and insert the index data into the running memory loaded with the first index to generate a second index node corresponding to the incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved through the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0183] In this embodiment, another retrieval method is disclosed, which can load a first index of existing data to be retrieved into the running memory, and generate corresponding index data based on the incremental data to be retrieved after receiving incremental data to be retrieved. The index data is then continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, resulting in high retrieval timeliness for newly added data to be retrieved. Furthermore, only one index needs to be maintained in the running memory, significantly reducing the memory usage of the retrieval service.

[0184] This disclosure provides another retrieval method applied to an indexing server, as described below. Figure 10 , Figure 10 This is a flowchart of another retrieval method provided in this disclosure embodiment, such as... Figure 10 As shown, the method may include:

[0185] Step 401: Obtain existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0186] Step 402: The first index is stored in a distributed file system so that the retrieval server can obtain the first index from the distributed file system and load the first index into the running memory to generate index data corresponding to the incremental data to be retrieved in real time. The index data is inserted into the running memory to generate a second index node corresponding to the incremental data to be retrieved in the running memory, resulting in a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved using the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0187] In this embodiment, another retrieval method is disclosed, which can load a first index of existing data to be retrieved into the running memory, and generate corresponding index data based on the incremental data to be retrieved after receiving incremental data to be retrieved. The index data is continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, which has high retrieval timeliness for newly added data to be retrieved. Furthermore, only one index needs to be maintained in the running memory, which significantly reduces the memory usage of the retrieval service.

[0188] This disclosure provides a retrieval system, referring to... Figure 11 , Figure 11This is a schematic diagram of a retrieval system provided in an embodiment of this disclosure, such as... Figure 11 As shown, a retrieval system may include a retrieval server, a storage service server, and an index server.

[0189] The retrieval server is configured to load a first index corresponding to the existing data to be retrieved into the running memory. The first index contains multiple first index nodes. The existing data to be retrieved represents the data to be retrieved before the first index is loaded into the running memory. The first index nodes are generated based on the existing data to be retrieved. The server then generates index data corresponding to the incremental data to be retrieved in real time. The incremental data to be retrieved represents the data to be retrieved after the first index is loaded into the running memory. The server inserts the index data into the running memory to generate a second index node for the incremental data to be retrieved in the running memory, resulting in a second index containing the first index node and the second index node. Finally, the server retrieves the full amount of data to be retrieved using the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes both the existing data to be retrieved and the incremental data to be retrieved.

[0190] The indexing server is configured to retrieve existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved. The first index is stored in a distributed file system so that the retrieval server can retrieve the first index from the distributed file system and load the first index into the runtime memory to generate index data corresponding to the incremental data to be retrieved in real time. The index data is then inserted into the runtime memory to generate a second index node corresponding to the incremental data to be retrieved in the runtime memory, resulting in a second index containing the first index node and the second index node. The entire data to be retrieved is then retrieved using the second index stored in the runtime memory to obtain the retrieval results. The entire data to be retrieved includes both existing data to be retrieved and incremental data to be retrieved.

[0191] The storage service client is configured to write existing data to be retrieved into the write-ahead log, enabling the index server to retrieve the existing data from the write-ahead log and generate a first index corresponding to the existing data. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved. Real-time incremental data to be retrieved is also written into the write-ahead log, enabling the retrieval server to retrieve the incremental data from the write-ahead log, generate index data corresponding to the incremental data, and insert the index data into the runtime memory loaded with the first index. This generates a second index node corresponding to the incremental data in the runtime memory, resulting in a second index containing both the first and second index nodes. The entire set of data to be retrieved is then searched using the second index stored in the runtime memory to obtain the search results. The entire set of data to be retrieved includes both existing and incremental data.

[0192] like Figure 11 As shown, Figure 11 The arrows in the diagram indicate the data flow direction. The storage service server retrieves the full amount of data to be retrieved and stores the incremental data to be retrieved in the write-ahead log of the distributed file system. The storage service server then retrieves incremental data to be retrieved from the write-ahead log that does not yet have a corresponding index in the index server. Based on this incremental data, it generates raw data for indexing and stores it in the raw data storage space of the distributed file system. The index server periodically retrieves raw data from the raw data storage space of the distributed file system, generates index nodes corresponding to the incremental data to be retrieved, and supplements its own persistent first index with these index nodes. This continuously updates the first index stored in the index file storage space of the distributed file system so that the latest version of the first index can be directly retrieved from the index file storage space the next time the retrieval server runs. The index server can retrieve the latest version of the first index from the index file storage space during runtime (boot-up, initialization, etc.) and load the first index into the running memory. During runtime, it continuously retrieves the latest incremental data to be retrieved from the write-ahead log and generates corresponding second index nodes based on the incremental data to supplement the running memory, maintaining the integrity of the index in the running memory.

[0193] In summary, the retrieval system provided by this disclosure can load a first index of existing data to be retrieved into the running memory, and generate corresponding index data based on the incremental data to be retrieved after receiving incremental data to be retrieved. The index data is then continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, resulting in high retrieval timeliness for newly added data to be retrieved. Furthermore, only one index needs to be maintained in the running memory, significantly reducing the memory usage of the retrieval service.

[0194] Figure 12 This is a block diagram of a retrieval device provided in an embodiment of this disclosure, such as... Figure 12 As shown, it includes:

[0195] The index loading module 501 is configured to load the first index corresponding to the existing data to be retrieved in the running memory. The first index contains multiple first index nodes. The existing data to be retrieved represents the data to be retrieved generated before loading the first index in the running memory. The first index nodes are generated based on the existing data to be retrieved.

[0196] The index data module 502 is configured to generate index data corresponding to the incremental data to be retrieved in real time. The incremental data to be retrieved represents the data to be retrieved after the first index is loaded into the running memory.

[0197] The node insertion module 503 is configured to insert index data into the running memory to generate a second index node for incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node.

[0198] The data retrieval module 504 is configured to retrieve the full amount of data to be retrieved using a second index stored in the running memory, and obtain the retrieval results. The full amount of data to be retrieved includes existing data to be retrieved and incremental data to be retrieved.

[0199] In one alternative implementation, the index loading module includes:

[0200] The first index submodule is configured to obtain the first index corresponding to the existing data to be retrieved from the index server. The index server is used to generate the first index based on the existing data to be retrieved.

[0201] The index space submodule is configured to request an index storage space in the running memory with a resource capacity greater than the resource usage of the first index.

[0202] The index loading submodule is configured to load the first index into the index storage space.

[0203] In one alternative embodiment, the apparatus further includes:

[0204] The acquisition submodule is configured to acquire the estimated runtime of the retrieval server, the previous runtime, and the historical resource usage of the first historical index loaded during the previous runtime of the retrieval server; where the estimated runtime represents the time that the retrieval server is expected to continuously provide retrieval services.

[0205] The historical resource usage change rate submodule is configured to determine the historical resource usage change rate of the first index based on the previous run time, historical resource usage, and resource usage.

[0206] The space size submodule is configured to determine resource capacity based on estimated runtime, historical resource usage change rate, and resource usage.

[0207] In one alternative implementation, the index data module includes:

[0208] The adjacent node information submodule is configured to determine the adjacent node information corresponding to the incremental data to be retrieved from the index storage space.

[0209] The index data submodule is configured to generate index data corresponding to the incremental data to be retrieved based on the incremental data to be retrieved and the information of adjacent nodes.

[0210] In one optional implementation, the indexes in the index storage space are stored in the index storage space in the form of a skip list, and the apparatus further includes:

[0211] The storage mode setting module is configured to set the storage mode of the index storage space to lock-free programming mode, so that the index storage space is readable during the insertion of index data.

[0212] In one alternative implementation, the node insertion module includes:

[0213] The node insertion submodule is configured to insert index data into the free storage space of the index storage space, where the free storage space represents the storage space in the index storage space that does not contain any index data.

[0214] In one alternative implementation, the data retrieval module includes:

[0215] The third index node submodule is configured to retrieve search keywords and match third index nodes from the second index that meet preset similarity conditions with the search keywords.

[0216] The fourth index node submodule is configured to determine a preset number of fourth index nodes from the second index that satisfy a preset distance condition with respect to the third index node.

[0217] The search results submodule is configured to obtain the first data to be searched that generates the third index node and the second data to be searched that generates the fourth index node, and obtain the search results consisting of the first data to be searched and the second data to be searched, so as to complete the search operation on the full amount of data to be searched.

[0218] In one alternative embodiment, the apparatus further includes:

[0219] The incremental data to be retrieved module is configured to retrieve incremental data to be retrieved from the write-ahead log, which is stored by the storage service. The write-ahead log is used to delete incremental data to be retrieved when the number of reads of incremental data to be retrieved reaches a preset number.

[0220] In summary, the retrieval device provided in this embodiment can load a first index of existing data to be retrieved into the running memory, and generate corresponding index data based on the incremental data to be retrieved after receiving incremental data to be retrieved. The index data is then continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, resulting in high retrieval timeliness for newly added data to be retrieved. Furthermore, only one index needs to be maintained in the running memory, significantly reducing the memory usage of the retrieval service.

[0221] Figure 13 This is a block diagram of another retrieval device provided in an embodiment of this disclosure, such as... Figure 13 As shown, it includes:

[0222] The existing data writing module 511 is configured to write existing data to be retrieved into a write-ahead log, so that the index server can obtain the existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0223] The incremental data writing module 512 is configured to write the incremental data to be retrieved in real time into the write-ahead log, so that the retrieval server can obtain the incremental data to be retrieved from the write-ahead log, generate index data corresponding to the incremental data to be retrieved, and insert the index data into the running memory loaded with the first index to generate a second index node corresponding to the incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved through the second index stored in the running memory to obtain the retrieval result. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0224] In summary, the alternative retrieval device provided in this embodiment can load a first index of existing data to be retrieved into the running memory, and generate corresponding index data based on the incremental data to be retrieved after receiving incremental data to be retrieved. The index data is then continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, resulting in high retrieval timeliness for newly added data to be retrieved. Furthermore, it only requires maintaining one index in the running memory, significantly reducing the memory usage of the retrieval service.

[0225] Figure 14 This is a block diagram of another retrieval device provided in the embodiments of this disclosure, such as... Figure 14 As shown, it includes:

[0226] The existing data to be retrieved module 521 is configured to retrieve existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved.

[0227] The index storage module 522 is configured to store the first index in a distributed file system, so that the retrieval server obtains the first index from the distributed file system, loads the first index into the running memory, generates index data corresponding to the incremental data to be retrieved in real time, inserts the index data into the running memory to generate a second index node corresponding to the incremental data to be retrieved in the running memory, and obtains a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved through the second index stored in the running memory to obtain the retrieval result. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

[0228] In summary, the retrieval device provided in this disclosure can load a first index of existing data to be retrieved into the running memory, and generate corresponding index data based on the incremental data to be retrieved after receiving incremental data to be retrieved. The index data is then continuously inserted into the running memory as a second index node to obtain a second index. This allows each retrieval task to retrieve the latest data to be retrieved through the second index, resulting in high retrieval timeliness for newly added data to be retrieved. Furthermore, only one index needs to be maintained in the running memory, significantly reducing the memory usage of the retrieval service.

[0229] Figure 15 This is a block diagram illustrating an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0230] Reference Figure 15 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0231] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0232] Memory 604 is used to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, multimedia, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0233] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0234] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a multimedia mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0235] Audio component 610 is used to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) used to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0236] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0237] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0238] Communication component 616 facilitates wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0239] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a retrieval method provided in the embodiments of this disclosure.

[0240] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0241] Figure 16This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. (Refer to...) Figure 16 The electronic device 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by a memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform a retrieval method provided in embodiments of this disclosure.

[0242] Electronic device 700 may also include a power supply component 726 configured to perform power management of electronic device 700, a wired or wireless network interface 750 configured to connect electronic device 700 to a network, and an input / output (I / O) interface 758. Electronic device 700 may operate on an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0243] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the retrieval method.

[0244] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the applications disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0245] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A retrieval method, characterized in that, Applied to a retrieval server, the method includes: A first index corresponding to the existing data to be retrieved is loaded into the running memory. The first index contains multiple first index nodes. The existing data to be retrieved represents the data to be retrieved generated before the first index is loaded into the running memory. The first index nodes are generated based on the existing data to be retrieved. Generate index data corresponding to the incremental data to be retrieved in real time, wherein the incremental data to be retrieved represents the data to be retrieved generated after loading the first index into the running memory; The index data is inserted into the running memory to generate a second index node for the incremental data to be retrieved in the running memory, resulting in a second index containing the first index node and the second index node; The full set of data to be retrieved is retrieved by using the second index stored in the running memory, and the retrieval results are obtained. The full set of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

2. The method according to claim 1, characterized in that, The loading of the first index corresponding to the existing data to be retrieved into the running memory includes: The first index corresponding to the existing data to be retrieved is obtained from the index server, and the index server is used to generate the first index based on the existing data to be retrieved. Based on the resource usage of the first index, apply for index storage space with a resource capacity in the running memory, wherein the resource capacity is greater than the resource usage; The first index is loaded into the index storage space.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the estimated runtime of the retrieval server, the previous runtime, and the historical resource usage of the first historical index loaded during the previous runtime of the retrieval server; wherein, the estimated runtime represents the estimated duration for which the retrieval server will need to continuously provide retrieval services; Based on the previous running time, the historical resource usage, and the resource usage, determine the historical resource usage change rate of the first index; The resource capacity is determined based on the estimated running time, the historical rate of change in resource usage, and the resource usage.

4. The method according to claim 2, characterized in that, The generation of index data corresponding to the incremental data to be retrieved in real time includes: Determine the adjacent node information corresponding to the incremental data to be retrieved from the index storage space; Based on the incremental data to be retrieved and the information of the adjacent nodes, index data corresponding to the incremental data to be retrieved is generated.

5. The method according to claim 2, characterized in that, The indexes in the index storage space are stored in the index storage space in the form of skip lists, and the method further includes: The index storage space is set to a lock-free programming mode so that the index storage space is readable during the insertion of index data.

6. The method according to claim 2, characterized in that, The step of inserting the index data into the running memory includes: The index data is inserted into the free storage space of the index storage space, where the free storage space represents the storage space in the index storage space that does not contain any index data.

7. The method according to any one of claims 1 to 6, characterized in that, The process of retrieving the full set of data to be retrieved using a second index stored in the running memory, and obtaining the retrieval results, includes: Obtain the search keywords, and match the third index node from the second index that meets the preset similarity condition with the search keywords; From the second index, determine a preset number of fourth index nodes whose distance from the third index node satisfies a preset distance condition; The first data to be retrieved for generating the third index node and the second data to be retrieved for generating the fourth index node are obtained, and the retrieval results composed of the first data to be retrieved and the second data to be retrieved are obtained to complete the retrieval operation on the full amount of data to be retrieved.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The incremental data to be retrieved is obtained from the write-ahead log, which is stored by the storage service. The write-ahead log is used to delete the incremental data to be retrieved when the number of reads of the incremental data to be retrieved reaches a preset number.

9. A retrieval method, characterized in that, Applied to the storage service side, the method includes: The existing data to be retrieved is written to the write-ahead log so that the index server can obtain the existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved. The incremental data to be retrieved in real time is written to the write-ahead log so that the retrieval server can obtain the incremental data to be retrieved from the write-ahead log, generate index data corresponding to the incremental data to be retrieved, and insert the index data into the running memory loaded with the first index to generate a second index node corresponding to the incremental data to be retrieved in the running memory, thus obtaining a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved through the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

10. A retrieval method, characterized in that, Applied to the indexing server, the method includes: The existing data to be retrieved is obtained from the write-ahead log, and a first index corresponding to the existing data to be retrieved is generated. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved. The first index is stored in a distributed file system so that the retrieval server can obtain the first index from the distributed file system and load the first index into the running memory to generate index data corresponding to the incremental data to be retrieved in real time. The index data is then inserted into the running memory to generate a second index node corresponding to the incremental data to be retrieved in the running memory, resulting in a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved using the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

11. A retrieval device, characterized in that, The device, used in a retrieval server, includes: The index loading module is configured to load a first index corresponding to the existing data to be retrieved in the running memory. The first index contains multiple first index nodes. The existing data to be retrieved represents the data to be retrieved generated before the first index is loaded in the running memory. The first index nodes are generated based on the existing data to be retrieved. The index data module is configured to generate index data corresponding to the incremental data to be retrieved in real time, wherein the incremental data to be retrieved represents the data to be retrieved generated after loading the first index into the running memory; The node insertion module is configured to insert the index data into the running memory to generate a second index node for the incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node. The data retrieval module is configured to retrieve the full set of data to be retrieved using a second index stored in the running memory, and to obtain retrieval results. The full set of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

12. A retrieval device, characterized in that, The device, used in storage services, includes: The existing data writing module is configured to write existing data to be retrieved into a write-ahead log, so that the index server can obtain the existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved. The incremental data writing module is configured to write the incremental data to be retrieved in real time into the write-ahead log, so that the retrieval server can obtain the incremental data to be retrieved from the write-ahead log, generate index data corresponding to the incremental data to be retrieved, and insert the index data into the running memory loaded with the first index to generate a second index node corresponding to the incremental data to be retrieved in the running memory, thereby obtaining a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved through the second index stored in the running memory to obtain the retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

13. A retrieval device, characterized in that, The device, used in an indexing server, includes: The existing data to be retrieved module is configured to retrieve existing data to be retrieved from the write-ahead log and generate a first index corresponding to the existing data to be retrieved. The first index contains multiple first index nodes, which are generated based on the existing data to be retrieved. An index storage module is configured to store the first index in a distributed file system, enabling the retrieval server to retrieve the first index from the distributed file system, load the first index into the running memory, generate index data corresponding to the incremental data to be retrieved in real time, insert the index data into the running memory to generate a second index node corresponding to the incremental data to be retrieved in the running memory, and obtain a second index containing the first index node and the second index node. The full amount of data to be retrieved is retrieved using the second index stored in the running memory to obtain retrieval results. The full amount of data to be retrieved includes the existing data to be retrieved and the incremental data to be retrieved.

14. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 10.

Images