Index generation method, apparatus, and electronic device

Abstract

The disclosure provides an index generation method and device and electronic equipment, and relates to the technical field of computers, in particular to the artificial intelligence technical field such as big data and natural language processing. The specific scheme is: in the case that a search request for a first vector database is received and the first vector database has no available index, first, a first parameter group and a second parameter group are determined, and a first index of the first vector database is constructed based on the first parameter group, so as to search the first vector database based on the first index; then, in the case that a preset condition is met, a second index of the first vector database is constructed based on the second parameter group; in the case that it is determined that the second index is constructed, the first index is deleted, and the first vector database is searched based on the second index.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as big data and natural language processing, specifically to an index generation method, apparatus and electronic device. Background Technology

[0002] In existing technologies, vector databases typically use a hierarchical navigable small world (HNSW) approach to build indexes. However, vector databases using this method currently cannot simultaneously achieve good index building speed and search performance, thus impacting user experience. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] The first aspect of this disclosure provides an index generation method, including:

[0005] Upon receiving a search request for the first vector database, and when the first vector database has no available index, a first parameter group and a second parameter group are determined, wherein the parameter value in the second parameter group is greater than the corresponding parameter value in the first parameter group.

[0006] Based on the first parameter group, a first index is constructed for the first vector database, so as to search the first vector database based on the first index;

[0007] Under the condition that the preset conditions are met, a second index of the first vector database is constructed based on the second parameter group;

[0008] Once the second index is determined to be complete, the first index is deleted, and the first vector database is searched based on the second index.

[0009] A second aspect of this disclosure provides an index generation apparatus, comprising:

[0010] The determination module is used to determine a first parameter group and a second parameter group when a search request for a first vector database is received and the first vector database has no available index, wherein the parameter value in the second parameter group is greater than the corresponding parameter value in the first parameter group;

[0011] The first construction module is configured to construct a first index of the first vector database based on the first parameter group, so as to search the first vector database based on the first index;

[0012] The second construction module is used to construct a second index of the first vector database based on the second parameter group when preset conditions are met;

[0013] The processing module is configured to delete the first index and search the first vector database based on the second index when it is determined that the second index has been constructed.

[0014] A third aspect of this disclosure provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the index generation method as proposed in the first aspect of this disclosure.

[0015] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the index generation method as proposed in the first aspect of this disclosure.

[0016] A fifth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the index generation method as proposed in the first aspect of this disclosure.

[0017] The index generation method, apparatus, and electronic device provided in this disclosure have the following beneficial effects:

[0018] In this embodiment, upon receiving a search request for a first vector database and finding no available index for the first vector database, a first parameter set and a second parameter set are first determined. Based on the first parameter set, a first index for the first vector database is constructed to search the first vector database. Then, under preset conditions, a second index for the first vector database is constructed based on the second parameter set. Once the second index is complete, the first index is deleted, and the first vector database is searched based on the second index. Therefore, by constructing a first index for the first vector database based on the first parameter set, and then constructing a second index based on the second parameter set under preset conditions and deleting the first index, the efficiency and reliability of index generation are improved, while also enhancing the index construction speed and search performance.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a flowchart illustrating an index generation method provided in an embodiment of the present disclosure;

[0022] Figure 2 This is a flowchart illustrating an index generation method provided in an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of the first index and the second index constructed for embodiments of this disclosure;

[0024] Figure 4 This is a flowchart illustrating an index generation method provided in an embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of the index generation apparatus provided in the embodiments of this disclosure;

[0026] Figure 6 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] This disclosure relates to the fields of artificial intelligence technology, such as big data and natural language processing.

[0029] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0030] Big data refers to data sets that cannot be captured, managed, and processed within a certain time frame using conventional software tools. It is a massive, rapidly growing, and diverse information asset that requires new processing models to achieve stronger decision-making, insight discovery, and process optimization capabilities.

[0031] Natural Language Processing (NLP) is an interdisciplinary field combining computer science, artificial intelligence, and linguistics. It primarily studies how to enable computers to understand, process, generate, and simulate human language, thereby achieving the ability to engage in natural dialogue with humans.

[0032] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] The index generation method, apparatus, and electronic device of this disclosure are described below with reference to the accompanying drawings.

[0034] Figure 1 This is a flowchart illustrating an index generation method provided in an embodiment of the present disclosure.

[0035] like Figure 1 As shown, the index generation method may include the following steps:

[0036] Step 101: Upon receiving a search request for the first vector database, and finding that the first vector database has no available index, determine the first parameter group and the second parameter group, wherein the parameter values ​​in the second parameter group are greater than the corresponding parameter values ​​in the first parameter group.

[0037] It should be noted that the first vector database can be any pre-set vector database, and this disclosure does not impose any restrictions on it.

[0038] The first parameter group and the second parameter group can both be parameter groups containing parameters used to construct the index of the first vector database. For example, the first parameter group and the second parameter group can both contain the threshold number of neighbor nodes for each vector in the first vector database, and the number of candidate neighbors for each vector, etc. The threshold number of neighbor nodes can be a critical value for the number of neighbor nodes for each vector, which can be preset, and this disclosure does not limit it.

[0039] In some possible implementations, the first parameter group and the second parameter group can be determined based on the number of vectors contained in the first vector database, thereby ensuring the reliability of the determined first parameter group and the second parameter group.

[0040] It should be noted that the number of vectors contained in the first vector database may vary, and the determined first parameter group and second parameter group may be different. This disclosure does not limit this.

[0041] In some possible implementations, two sets of parameters in any set of preset parameter pairs can be designated as the first parameter set and the second parameter set, respectively, thereby improving the accuracy of the determined first parameter set and second parameter set.

[0042] The parameter pair can be a parameter pair that includes the threshold parameter for the number of neighboring nodes for each vector in the first vector database, and the parameter for the number of candidate neighbors for the corresponding vector.

[0043] In this disclosure, when a search request for a first vector database is received and no index is available for the first vector database, an index for the first vector database needs to be constructed in order to ensure user experience. At this time, the first parameter set and the second parameter set used to construct the index for the first vector database can be determined firstly by the number of vectors contained in the first vector database, or by two sets of parameters in any set of preset parameter pairs. This provides the conditions for constructing the index for the first vector database and improves the flexibility of determining the parameters used to construct the index for the first vector database.

[0044] Step 102: Based on the first parameter group, construct the first index of the first vector database, so as to search the first vector database based on the first index.

[0045] It should be noted that the first index may include vector data in the first vector database and the edge set associated with the vector data, wherein the edge set may include the set of neighboring nodes that have connecting edges with the nodes of the vector data.

[0046] In this disclosure, after determining the first parameter group and the second parameter group, in order to improve the index building speed of the first vector database and enhance the user search experience, a first index can be built in the first vector database based on the first parameter group, so that users can search the first vector database based on the first index which has a faster building speed.

[0047] Step 103: Under the condition that the preset conditions are met, construct the second index of the first vector database based on the second parameter group.

[0048] The preset condition can be a condition for determining whether a second index of the first vector database can be constructed based on the second parameter group. It can be any pre-set condition, and this disclosure does not limit it.

[0049] It should be noted that the range of vector data corresponding to the second index is the same as that of the first index, and the range of the edge set in the second index is greater than that in the first index.

[0050] In some possible implementations, the preset conditions may include at least one of the following: the remaining resources of the system are greater than the resource threshold; the duration for which the remaining resources of the system are greater than the resource threshold is greater than the duration threshold; an instruction to create a second index is received; the current runtime segment is within a preset time range.

[0051] The resource threshold can be a critical value of the remaining system resources used to determine whether a second index of the first vector database can be constructed based on the second parameter group. It can be preset, and this disclosure does not limit it.

[0052] The duration threshold can be a critical value used to determine whether the remaining resources of the system can be used to build a second index. It can be preset, and this disclosure does not limit it.

[0053] The time range can be the time range used to construct the second index based on the second parameter group. The range can be preset, and this disclosure does not limit it.

[0054] In this disclosure, when the remaining resources of the system are greater than the resource threshold, or when the duration of the remaining resources being greater than the resource threshold is greater than the duration threshold, it can be determined that the current remaining resources of the system meet the resource requirements for constructing the second index based on the second parameter set. At this time, the second index of the first vector database can be constructed based on the second parameter set. Alternatively, when an instruction to create the second index is received, or when the current runtime is within a preset time range, the second index can be constructed based on the second parameter set, thereby improving the success rate of constructing the second index.

[0055] In this disclosure, after constructing a first index and searching the first vector database based on the first index, in order to improve the search effect of the first vector database, a second index of the first vector database can be constructed based on a second parameter group.

[0056] Step 104: Once the second index is determined to be complete, delete the first index and search the first vector database based on the second index.

[0057] In this disclosure, once the construction of the second index is completed, since the data range corresponding to the second index is the same as the data range corresponding to the first index, in order to improve system resource utilization and search performance, the first index can be deleted, and the first vector database can be searched based on the second index.

[0058] In this embodiment, upon receiving a search request for a first vector database and finding no available index for the first vector database, a first parameter set and a second parameter set are first determined. Based on the first parameter set, a first index for the first vector database is constructed, and the database is searched using this first index. Then, under preset conditions, a second index for the first vector database is constructed based on the second parameter set. Once the second index is complete, the first index is deleted, and the database is searched again based on the second index. Thus, by constructing a first index for the first vector database based on the first parameter set, and then, under preset conditions, constructing a second index based on the second parameter set with larger parameter values, and deleting the first index, the efficiency and reliability of index generation are improved, while also enhancing the index construction speed and search performance.

[0059] Figure 2 This is a schematic flowchart of an index generation method provided in an embodiment of the present disclosure.

[0060] like Figure 2 As shown, the index generation method may include the following steps:

[0061] Step 201: Upon receiving a search request for the first vector database, and finding that the first vector database has no available index, determine the first parameter group and the second parameter group, wherein the parameter values ​​in the second parameter group are greater than the corresponding parameter values ​​in the first parameter group.

[0062] Step 202: Based on the first parameter group, construct the first index of the first vector database, so as to search the first vector database based on the first index.

[0063] The specific implementation of steps 201 to 202 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0064] Step 203: Store the distance set associated with the first edge set in the first index, wherein the distance set includes the first distance value of each connecting edge in the first edge set.

[0065] The first edge set can be a set of nodes that includes the connecting edges between the vector nodes in the first index and their neighboring nodes.

[0066] In this disclosure, after constructing the first index of the first vector database, the distance set associated with the first edge set in the first index can be stored, thereby providing a data foundation for improving the efficiency of constructing the second index.

[0067] In this disclosure, taking HNSW as an example, when generating the distance set associated with the first edge set, since the distance set is calculated when building the graph in HNSW, generating the distance set associated with the first edge set will not consume extra time. Moreover, the first vector database can usually control the number of nodes in HNSW through sharding, so that the space occupied by the generated distance set is small, thereby saving system resources.

[0068] HNSW is short for Hierarchical Navigable Small World.

[0069] Step 204: Under the condition that the preset conditions are met, construct the second edge set in the second index of the first vector database based on the second parameter set and the distance set.

[0070] In this disclosure, under the condition of satisfying preset conditions, in order to improve the user search effect and the efficiency of index construction, the second edge set in the second index of the first vector database can be constructed based on the second parameter set and the distance set.

[0071] The following is combined with Figure 3 The following example illustrates the construction of the first index and the second index using the index generation method provided in this disclosure. Figure 3 This is a schematic diagram of the first index and the second index constructed for embodiments of this disclosure. Wherein, Figure 3 The index diagram in the example uses the HNSW index diagram; level represents the layer; neighbors can be the neighboring nodes of node #1 or node #2 in each layer; distances can be the set of distance values ​​of the connecting edges between node #1 or node #2 and the neighboring nodes in each layer.

[0072] like Figure 3 As shown, the first index of node #1 or node #2 constructed through the first parameter combination can include the original data (floating-point vector) and the first edge set. Simultaneously, it can also generate the distance set associated with the first edge set of node #1 or node #2. Under certain conditions, based on the distance set associated with the first edge set and the second parameter set, the second index of node #1 or node #2 constructed includes the original data (floating-point vector) and the second edge set. Figure 3 It can be seen that the original data range corresponding to the first index of node #1 or node #2 is the same as the original data range corresponding to the second index.

[0073] Step 205: If the second edge set in the second index is determined to be completed, delete the first edge set and search the first vector database based on the second index.

[0074] In this disclosure, when it is determined that the second edge set in the second index has been constructed, in order to save system resources, the first edge set can be deleted, and the first vector data can be searched based on the second index, thereby improving the search effect and enhancing the user experience.

[0075] In this embodiment, upon receiving a search request for a first vector database and finding no available index for the first vector database, a first parameter set and a second parameter set are first determined. Based on the first parameter set, a first index for the first vector database is constructed to search the database. Then, the distance set associated with the first edge set in the first index is stored. Under preset conditions, a second edge set in the second index of the first vector database is constructed based on the second parameter set and the distance set. Finally, once the second index is determined to be complete, the first index is deleted, and the first vector database is searched based on the second index. Thus, after constructing the first index based on the first parameter set, and under preset conditions, constructing the second index of the first vector database based on the distance set associated with the first edge set and the second parameter set, the efficiency of index generation is improved, enhancing the user experience.

[0076] Figure 4 This is a flowchart illustrating an index generation method provided in an embodiment of the present disclosure.

[0077] like Figure 4 As shown, the index generation method may include the following steps:

[0078] Step 401: Upon receiving a search request for the first vector database, and finding that the first vector database has no available index, determine the first parameter group and the second parameter group, wherein the parameter values ​​in the second parameter group are greater than the corresponding parameter values ​​in the first parameter group.

[0079] Step 402: Based on the first parameter group, construct a first index for the first vector database, and search the first vector database based on the first index.

[0080] Step 403: Store the distance set associated with the first edge set in the first index, wherein the distance set includes the first distance value of each connecting edge in the first edge set.

[0081] The specific implementation of steps 401 to 403 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0082] Step 404: Under the condition that the preset conditions are met, the candidate neighbor set of the first vector in the first vector database is determined based on the exploration element EF construction parameters in the second parameter group.

[0083] EF can be an abbreviation for Exploration Factor.

[0084] The exploration element EF construction parameter can be used to determine the number of candidate neighbors for the first vector, and it can be any pre-set value. For example, when EF is 200, the number of candidate neighbors for the first vector can be determined to be 200, etc. This disclosure does not limit this.

[0085] It should be noted that the first vector can be any vector in the first vector database, and this disclosure does not impose any restrictions on it.

[0086] The candidate neighbor set can be a set of candidate neighbors that contain the first vector.

[0087] In this disclosure, under the condition that the preset conditions are met, the candidate neighbor set of the first vector can be determined first based on the exploration element EF construction parameters in the second parameter set, so as to provide conditions for constructing the second edge set in the second index.

[0088] It should be noted that the exploration element EF construction parameters in the second parameter group are different, and the candidate neighbor set of the determined first vector may be different. This disclosure does not limit this.

[0089] It should be noted that the values ​​of the exploration element EF construction parameters in the second parameter group are greater than the values ​​of the exploration element EF construction parameters in the first parameter group.

[0090] Step 405: Based on the identifier of each candidate neighbor in the candidate neighbor set, traverse the distance set corresponding to the first vector to obtain the first distance value of the connecting edge corresponding to the candidate neighbor.

[0091] The identifier of the candidate neighbor can be used to represent the candidate neighbor of the first vector. It can be any pre-set identifier, and this disclosure does not limit it.

[0092] In this disclosure, after determining the candidate neighbor set of the first vector, the distance set corresponding to the first vector can be traversed based on the identifier of each candidate neighbor in the candidate neighbor set to obtain the first distance value of the connecting edge corresponding to the candidate neighbor.

[0093] It should be noted that different first vectors may result in different distance sets, but this disclosure does not impose any restrictions on this.

[0094] Step 406: If the distance set does not contain the first distance value of the connecting edge corresponding to any candidate neighbor vector, calculate the second distance value between the first vector and any candidate neighbor vector.

[0095] In this disclosure, when traversing the distance set corresponding to the first vector based on the identifier of each candidate neighbor in the candidate neighbor set to obtain the first distance value of the connecting edge corresponding to the candidate neighbor, if the distance set does not contain the first distance value of the connecting edge corresponding to any candidate neighbor vector, the second distance value between the first vector and any candidate neighbor vector can be directly calculated, thereby improving the flexibility and reliability of determining the distance value between the first vector and the candidate neighbor vector.

[0096] Step 407: Determine the neighbor nodes of the first vector at each layer based on the first distance value or the second distance value corresponding to each candidate neighbor of the first vector.

[0097] In this disclosure, after determining the first distance value or the second distance value corresponding to each candidate neighbor of the first vector, the candidate neighbor nodes with smaller distance values ​​can be determined as the neighbor nodes of the first vector based on the threshold number of neighbor nodes of the first vector in the second parameter group and the distance values ​​of the candidate neighbor nodes of the first vector in each layer. This disclosure does not limit this step.

[0098] Step 408: Based on the neighbor nodes corresponding to each vector in the first vector database at each layer, construct the second edge set in the second index.

[0099] In this disclosure, after determining the neighbor nodes corresponding to the first vector at each layer, a second edge set in the second index can be constructed based on the neighbor nodes corresponding to each vector at each layer in the first vector database, thereby improving the accuracy and reliability of the constructed second edge set of the second index.

[0100] Step 409: If the second edge set in the second index is determined to be completed, delete the first edge set and search the first vector database based on the second index.

[0101] The specific implementation of step 409 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0102] In this embodiment of the disclosure, when a search request for a first vector database is received and no available index is found in the first vector database, a first parameter group and a second parameter group are first determined. Based on the first parameter group, a first index of the first vector database is constructed to search the first vector database. Then, the distance set associated with the first edge set in the first index is stored. Under the condition of satisfying a preset condition, the candidate neighbor set of the first vector in the first vector database is determined based on the exploration element EF construction parameters in the second parameter group. Based on the identifier of each candidate neighbor in the candidate neighbor set, the distance set corresponding to the first vector is traversed to obtain the first distance value of the connecting edge corresponding to the candidate neighbor. If the distance set does not contain the first distance value of the connecting edge corresponding to any candidate neighbor vector, the second distance value between the first vector and any candidate neighbor vector is calculated. Then, based on the first distance value or the second distance value corresponding to each candidate neighbor of the first vector, the neighbor node corresponding to the first vector in each layer is determined. Based on the neighbor node corresponding to each vector in each layer in the first vector database, a second edge set in the second index is constructed. When it is determined that the second edge set in the second index has been constructed, the first edge set is deleted, and the first vector database is searched based on the second index. Therefore, after constructing the first index of the first vector database, under the condition of satisfying the preset conditions, the candidate neighbors of the first vector are determined based on the exploration element EF construction parameters in the second parameter group. Based on the distance set associated with the first edge set in the first index, the distance value of the connecting edge corresponding to the candidate neighbor is determined. When the distance set associated with the first edge set does not contain the distance value of the connecting edge corresponding to any candidate neighbor, the distance value of the connecting edge of any candidate neighbor is determined by direct calculation. Based on the distance value of each candidate neighbor, the neighbor nodes of the first vector in each layer are determined. Based on the neighbor nodes of each vector in the first vector database, the second edge set in the second index is constructed, thereby improving the accuracy and reliability of index generation and improving the effect of index search.

[0103] To implement the above embodiments, this disclosure also proposes an index generation apparatus.

[0104] Figure 5 This is a schematic diagram of the structure of the index generation apparatus provided in the embodiments of this disclosure.

[0105] like Figure 5 As shown, the index generation device 500 includes: a determination module 501, a first construction module 502, a second construction module 503, and a processing module 504.

[0106] The determination module 501 is used to determine a first parameter group and a second parameter group when a search request for a first vector database is received and the first vector database has no available index, wherein the parameter value in the second parameter group is greater than the corresponding parameter value in the first parameter group.

[0107] The first construction module 502 is used to construct a first index of the first vector database based on the first parameter group, so as to search the first vector database based on the first index;

[0108] The second construction module 503 is used to construct a second index of the first vector database based on the second parameter group when the preset conditions are met.

[0109] The processing module 504 is used to delete the first index and search the first vector database based on the second index after determining that the second index has been constructed.

[0110] In one possible implementation of this disclosure, the first building module 502 is further configured to:

[0111] Store the distance set associated with the first edge set in the first index, wherein the distance set includes the first distance value of each connecting edge in the first edge set;

[0112] The second building module 503 mentioned above is also used for:

[0113] Based on the second parameter set and the distance set, construct the second edge set of the first vector database.

[0114] In one possible implementation of this disclosure, the second building module 503 is further configured to:

[0115] Based on the exploration element EF construction parameters in the second parameter group, determine the candidate neighbor set of the first vector in the first vector database;

[0116] Based on the identifier of each candidate neighbor in the candidate neighbor set, traverse the distance set corresponding to the first vector to obtain the first distance value of the connecting edge corresponding to the candidate neighbor;

[0117] If the distance set does not contain the first distance value of the connecting edge corresponding to any candidate neighbor vector, calculate the second distance value between the first vector and any candidate neighbor vector.

[0118] Based on the first distance value or the second distance value corresponding to each candidate neighbor of the first vector, determine the neighbor node corresponding to the first vector at each layer;

[0119] Based on the neighbor nodes corresponding to each vector in the first vector database at each layer, construct the second edge set.

[0120] In one possible implementation of this disclosure, the second building module 503 is further configured to:

[0121] Delete the first side set.

[0122] In one possible implementation of this disclosure, the determining module 501 is specifically used for:

[0123] The first parameter group and the second parameter group are determined based on the number of vectors contained in the first vector database.

[0124] In one possible implementation of this disclosure, the determining module 501 is specifically used for:

[0125] Define two sets of parameters from any set of preset parameters as the first parameter and the second parameter set, respectively.

[0126] In one possible implementation of this disclosure, the preset conditions include at least one of the following:

[0127] The system's remaining resources are greater than the resource threshold;

[0128] The duration for which the system's remaining resources exceed the resource threshold is greater than the duration threshold.

[0129] Received instruction to create a second index;

[0130] The current runtime segment is within the preset time range.

[0131] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0132] In this embodiment, upon receiving a search request for a first vector database and finding no available index for the first vector database, a first parameter set and a second parameter set are first determined. Based on the first parameter set, a first index for the first vector database is constructed to search the first vector database. Then, under preset conditions, a second index for the first vector database is constructed based on the second parameter set. Once the second index is complete, the first index is deleted, and the first vector database is searched based on the second index. Therefore, by constructing a first index for the first vector database based on the first parameter set, and then constructing a second index based on the second parameter set under preset conditions and deleting the first index, the efficiency and reliability of index generation are improved, while also enhancing the index construction speed and search performance.

[0133] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0134] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0135] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0136] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0137] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the index generation method. For example, in some embodiments, the index generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the index generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the index generation method by any other suitable means (e.g., by means of firmware).

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable indexing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0143] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0144] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this disclosure, the words "if" and "suppose" as used may be interpreted as "when," "when," "in response to determination," or "in the circumstances."

[0146] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An index generation method, comprising: Upon receiving a search request for the first vector database, and when the first vector database has no available index, a first parameter group and a second parameter group are determined, wherein the parameter value in the second parameter group is greater than the corresponding parameter value in the first parameter group. Based on the first parameter group, a first index is constructed for the first vector database, so as to search the first vector database based on the first index; Under the condition that the preset conditions are met, a second index of the first vector database is constructed based on the second parameter group; Once the second index is determined to be complete, the first index is deleted, and the first vector database is searched based on the second index.

2. The method as described in claim 1, wherein, After constructing the first index of the first vector database based on the first parameter group, the method further includes: Store the distance set associated with the first edge set in the first index, wherein the distance set includes the first distance value of each connecting edge in the first edge set; The construction of the second index of the first vector database based on the second parameter group includes: Based on the second parameter set and the distance set, a second edge set of the first vector database is constructed.

3. The method as described in claim 2, wherein, The step of constructing the second edge set of the first vector database based on the second parameter set and the distance set includes: Based on the exploration element EF construction parameters in the second parameter group, determine the candidate neighbor set of the first vector in the first vector database; Based on the identifier of each candidate neighbor in the candidate neighbor set, traverse the distance set corresponding to the first vector to obtain the first distance value of the connecting edge corresponding to the candidate neighbor; If the distance set does not contain the first distance value of the connecting edge corresponding to any candidate neighbor, calculate the second distance value between the first vector and any candidate neighbor; Based on the first distance value or the second distance value corresponding to each candidate neighbor of the first vector, determine the neighbor node corresponding to the first vector in each layer; The second edge set is constructed based on the neighbor nodes corresponding to each vector in the first vector database at each layer.

4. The method of claim 2, wherein, Deleting the first index includes: Delete the first edge set.

5. The method of claim 1, wherein, Determining the first parameter set and the second parameter set includes: The first parameter group and the second parameter group are determined based on the number of vectors contained in the first vector database.

6. The method of claim 1, wherein, Determining the first parameter set and the second parameter set includes: Two sets of parameters from any set of preset parameter pairs are respectively designated as the first parameter set and the second parameter set.

7. The method as described in any one of claims 1-6, wherein, The preset conditions include at least one of the following: The system's remaining resources are greater than the resource threshold; The duration for which the system's remaining resources exceed the resource threshold is greater than the duration threshold. Received instruction to create a second index; The current runtime segment is within the preset time range.

8. An index generation apparatus, wherein, The device includes: The determination module is used to determine a first parameter group and a second parameter group when a search request for a first vector database is received and the first vector database has no available index, wherein the parameter value in the second parameter group is greater than the corresponding parameter value in the first parameter group. The first construction module is configured to construct a first index of the first vector database based on the first parameter group, so as to search the first vector database based on the first index; The second construction module is used to construct a second index of the first vector database based on the second parameter group when preset conditions are met; The processing module is configured to delete the first index and search the first vector database based on the second index when it is determined that the second index has been constructed.

9. The apparatus of claim 8, wherein, The first building module is also used for: Store the distance set associated with the first edge set in the first index, wherein the distance set includes the first distance value of each connecting edge in the first edge set; The second building module is also used for: Based on the second parameter set and the distance set, a second edge set of the first vector database is constructed.

10. The apparatus of claim 9, wherein, The second building module is also used for: Based on the exploration element EF construction parameters in the second parameter group, determine the candidate neighbor set of the first vector in the first vector database; Based on the identifier of each candidate neighbor in the candidate neighbor set, traverse the distance set corresponding to the first vector to obtain the first distance value of the connecting edge corresponding to the candidate neighbor; If the distance set does not contain the first distance value of the connecting edge corresponding to any candidate neighbor, calculate the second distance value between the first vector and any candidate neighbor; Based on the first distance value or the second distance value corresponding to each candidate neighbor of the first vector, determine the neighbor node corresponding to the first vector in each layer; The second edge set is constructed based on the neighbor nodes corresponding to each vector in the first vector database at each layer.

11. The apparatus of claim 9, wherein, The second building module is also used for: Delete the first edge set.

12. The apparatus of claim 8, wherein, The determining module is specifically used for: The first parameter group and the second parameter group are determined based on the number of vectors contained in the first vector database.

13. The apparatus of claim 8, wherein, The determining module is specifically used for: Two sets of parameters from any set of preset parameters are respectively designated as the first parameter set and the second parameter set.

14. The apparatus according to any one of claims 8-13, wherein, The preset conditions include at least one of the following: The system's remaining resources are greater than the resource threshold; The duration for which the system's remaining resources exceed the resource threshold is greater than the duration threshold. Received instruction to create a second index; The current runtime segment is within the preset time range.

15. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that may be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images