Data query method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of big data, and provides a data query method and device, electronic equipment and a storage medium, which comprise the following steps: in response to a data query request, a target connected graph is constructed based on a plurality of original vector data included in different index objects that have been written, the original vector data comprises a query text vector and a corresponding query result vector, the data query request carries a target query text vector and a query result quantity requirement; based on the target connected graph, a plurality of candidate coordinate positions adjacent to the coordinate position of the target query text are found, and at least one target coordinate position is determined from the plurality of candidate coordinate positions based on the query result quantity requirement; and the query result vector of the query text vector corresponding to the at least one target coordinate position is determined as the query result of the data query request. The application combines the near neighbor text query and the graph algorithm, reduces the time complexity of data query, and improves the data query efficiency.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to data query methods, devices, electronic equipment and storage media. Background Technology

[0002] In the field of big data, Elasticsearch is a free and open-source distributed search and analytics engine suitable for all types of data, including text, numbers, geospatial data, and both structured and unstructured data. Elasticsearch's full-text search function segments the query text into tokens and calculates the relevance score for each word to find similar result text; essentially, it's a term-based search. However, when it's necessary to retrieve a wider range of result text based on synonyms / near-synonyms for each word, Elasticsearch's full-text search function is not suitable. In such cases, the vector search function provided by ElatisSearch 7.x can be used to retrieve a broader range of result text. Therefore, how to perform fast and accurate vector queries using ElatisSearch 7.x has become a crucial problem that urgently needs to be solved.

[0003] In related technologies, based on the implementation principle of the vector search function provided by the ElatisSearch 7.x component, the result text vector that matches the query text vector is first searched using script_score. Then, the vector similarity score is calculated for each of the multiple matched result text vectors. Based on the vector similarity score of each result text vector, the target result text is fed back to the user.

[0004] However, since the script_score query method uses a linear query approach, it needs to traverse all the currently stored vector data one by one without skipping any, until a result text vector matching the query text vector is found. This results in high time complexity and low query efficiency for linear queries. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention proposes a data query method that combines nearest-neighbor text query and graph algorithm. This eliminates the need to traverse and query all currently stored vector data one by one. Instead, it only needs to determine the final query result from a subset of original vector data that satisfies a preset nearest-neighbor relationship. This reduces the time complexity of data query and significantly improves the efficiency of data query.

[0006] The present invention also proposes a data query device.

[0007] The present invention also proposes an electronic device.

[0008] The present invention also proposes a non-transitory computer-readable storage medium.

[0009] A data query method according to a first aspect of the present invention includes:

[0010] In response to a data query request, a target connectivity graph is constructed based on the multiple raw vector data included in each of the different index objects that have been written. The raw vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connectivity graph are interconnected. The data query request carries the target query text vector and the query result quantity requirement.

[0011] Based on the target connectivity graph, find multiple candidate coordinate positions adjacent to the coordinate position of the target query text, and determine at least one target coordinate position from the multiple candidate coordinate positions based on the query result quantity requirement;

[0012] The query result vector of the query text vector corresponding to the at least one target coordinate position is determined as the query result of the data query request.

[0013] According to the data query method of this embodiment, when the ElatisSearch 7.x component responds to a data query request, it first constructs a target connectivity graph based on the multiple original vector data included in each of the different index objects that have been written. Given that the coordinate positions corresponding to different query text vectors in the target connectivity graph are interconnected, and the data query request carries the target query text vector and a query result quantity requirement, it searches for multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph. Based on the query result quantity requirement, it determines at least one target coordinate position from the multiple candidate coordinate positions. Then, it determines the query result vector of the query text vector corresponding to the at least one target coordinate position as the query result of the data query request. In this way, by combining nearest-neighbor text query and graph algorithms, it is not necessary to traverse and query all currently stored vector data one by one. Instead, it only needs to determine the final query result from the partial original vector data that satisfies the preset nearest-neighbor relationship, which reduces the time complexity of data query and significantly improves data query efficiency.

[0014] According to one embodiment of the present invention, constructing a target connectivity graph based on multiple original vector data included in each of the different index objects that have been written includes:

[0015] For each index object, a subgraph of the index object is constructed based on the multiple original vector data included in each index object, wherein the coordinate positions corresponding to the query text vectors of the multiple original vector data in the subgraph are interconnected.

[0016] Based on each of the subgraphs, the target connected graph is constructed, wherein the subgraphs in the target connected graph are interconnected.

[0017] According to an embodiment of the present invention, the writing process of the multiple original vector data included in the index object for each of the different index objects that have been written includes:

[0018] In response to a data write command, multiple raw data to be written are vectorized to obtain multiple raw vector data; each raw vector data includes the query result vector and the query text vector.

[0019] Based on the index object generation method of the multiple original vector data and the preset plugin library, the index object corresponding to the multiple original vector data is determined, and the index object includes the multiple original vector data;

[0020] The index object is stored in a Segment document under the Lucene file, and the cumulative time for responding to the data write command reaches a preset time. The Segment document is then flushed to the disk, and the Segment document stored on the disk is used to construct the target connectivity graph.

[0021] According to an embodiment of the present invention, the preset plugin library is an HNSW plugin library, and the method for generating index objects based on the plurality of original vector data and the preset plugin library, determining the index objects corresponding to the plurality of original vector data, includes:

[0022] The multiple original vector data are converted into data types supported by the HNSW plugin library to obtain multiple vector data conversion results;

[0023] The transformation results of the multiple vector data are added to the preset index of the preset ElatisSearch component;

[0024] Based on the completion flag, the index object generation method of the HNSW plugin library is called to determine the index object.

[0025] According to an embodiment of the present invention, the preset plugin library is an LSH plugin library, and the method for generating index objects based on the plurality of original vector data and the preset plugin library, determining the index objects corresponding to the plurality of original vector data, includes:

[0026] Obtain the target hash function corresponding to each of the multiple original data to be written;

[0027] For each piece of raw data to be written and each target hash function, based on the hash index object generation method of the LSH plugin library, the target hash function is used to hash the raw vector data to determine the hash value corresponding to the raw vector data;

[0028] For each hash value and each piece of raw data to be written, the raw data to be written is mapped to the hash bucket where the hash value is located, and the index object is determined.

[0029] According to an embodiment of the present invention, the preset plugin library is an IVSPQ plugin library, and the method for generating index objects based on the plurality of original vector data and the preset plugin library, determining the index objects corresponding to the plurality of original vector data, includes:

[0030] For each original vector data, based on the mapping relationship between vector data, cluster points, and encoding methods pre-stored in the IVSPQ plugin library, the target cluster point and target encoding method corresponding to the original vector data are determined.

[0031] For each original vector data corresponding to its target clustering point and target encoding method, based on the encoding index object generation method of the IVSPQ plugin library, the original data to be written is encoded and clustered using the target encoding method and the target clustering point to obtain the clustering encoding result;

[0032] For each clustering encoding result and each piece of raw data to be written, the raw data to be written is mapped to the encoding table where the clustering encoding result is located, and the index object is determined.

[0033] According to one embodiment of the present invention, the method further includes:

[0034] The original vector data is stored in a preset data type of the preset plugin library. The preset data type is used to store the original vector data without dimension limitation.

[0035] According to one embodiment of the present invention, finding multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph includes:

[0036] The target connected graph is loaded into memory, and the multiple candidate coordinate positions are found based on the target connected graph;

[0037] or,

[0038] Based on the result of loading the target connected graph and the target connected graph, the multiple candidate coordinate positions are searched.

[0039] According to one embodiment of the present invention, constructing a target connectivity graph based on multiple original vector data included in each of the different index objects that have been written includes:

[0040] Compare the different index objects that have been written with the index objects used in the previously constructed connected graph;

[0041] Determine that the different index objects that have been written are different from the index objects used in the previously constructed connected graph, and construct the target connected graph based on the multiple original vector data included in each of the different index objects that have been written.

[0042] A data query apparatus according to a second aspect of the present invention includes:

[0043] The connected graph construction module is used to respond to data query requests by constructing a target connected graph based on multiple original vector data included in different index objects that have been written. The original vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connected graph are interconnected. The data query request carries the target query text vector and the number of query results required.

[0044] The data query module is used to find multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph, and to determine at least one target coordinate position from the multiple candidate coordinate positions based on the query result quantity requirement; and to determine the query result vector of the query text vector corresponding to the at least one target coordinate position as the query result of the data query request.

[0045] According to the data query apparatus of the present invention, when responding to a data query request, it first constructs a target connectivity graph based on the multiple original vector data included in each of the different index objects that have been written, and connects the coordinate positions corresponding to different query text vectors in the target connectivity graph to each other. When the data query request carries the target query text vector and the number of query results required, it searches for multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph, and determines at least one target coordinate position from the multiple candidate coordinate positions based on the number of query results required. Then, it determines the query result vector of the query text vector corresponding to the at least one target coordinate position as the query result of the data query request.

[0046] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: When the ElatisSearch 7.x component responds to a data query request, it first constructs a target connectivity graph based on the multiple original vector data included in each of the different index objects that have been written. Then, it connects the coordinate positions corresponding to different query text vectors in the target connectivity graph. Given that the data query request carries the target query text vector and a requirement for the number of query results, it searches for multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph. Based on the requirement for the number of query results, it determines at least one target coordinate position from the multiple candidate coordinate positions. Finally, it determines the query result vector of the query text vector corresponding to at least one target coordinate position as the query result of the data query request. In this way, by combining nearest-neighbor text query and graph algorithms, it is not necessary to traverse and query all currently stored vector data one by one. Instead, it only needs to determine the final query result from a portion of the original vector data that satisfies the preset nearest-neighbor relationship, which reduces the time complexity of data query and significantly improves data query efficiency.

[0047] Furthermore, by first constructing subgraphs for each indexed object and then constructing the target connected graph based on each subgraph, the accuracy and reliability of the connected graph construction are improved.

[0048] Furthermore, the ElatisSearch 7.x component achieves data writing by first vectorizing the raw data to be written, then determining the index objects, and finally flushing the index objects stored in the Segment documents under the Lucene file to disk. This ensures that the target connectivity graph can be accurately and quickly constructed when responding to data query commands, providing reliable data support for improving data query efficiency. Moreover, when the cumulative time for responding to data write commands reaches a preset time, all index objects determined within the preset time are stored in the Segment documents under the Lucene file. This ensures that the storage operation for the corresponding index objects in different Segment documents is performed at preset time intervals, thereby improving the efficiency of subsequently flushing the Segment documents to disk.

[0049] Furthermore, by storing the raw vector data in the preset data types of the preset plugin library, the drawback of the existing dense_vector type in Elasticsearch 7.x components being unable to store vector data with millions or even tens of millions of dimensions is solved. This improves the applicability of using the preset data types of the preset plugin library for vector data storage and provides a reliable guarantee for the richness and comprehensiveness of subsequent query results.

[0050] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the data query method provided by the present invention;

[0053] Figure 2 This is a data writing flowchart when the preset plugin library provided by this invention is the HNSW plugin library;

[0054] Figure 3 This is a data writing flowchart when the preset plugin library provided by this invention is the LSH plugin library;

[0055] Figure 4 This is a data writing flowchart when the preset plugin library provided by this invention is the IVSPQ plugin library;

[0056] Figure 5 This is a schematic diagram of the data query device provided by the present invention;

[0057] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0059] In embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. In the textual description of the present invention, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, it should be noted that the serial numbers assigned to the described objects in the present invention, such as "first," "second," etc., are only used to distinguish the described objects and do not have any sequential or technical meaning.

[0060] In the field of big data, Elasticsearch is a free and open-source distributed search and analytics engine suitable for all types of data, including text, numbers, geospatial data, and both structured and unstructured data. Elasticsearch's full-text search function segments the query text into tokens and calculates the relevance score for each word to find similar results; essentially, it's a term-basic search. However, when a broader range of results needs to be retrieved based on synonyms / near-synonyms for each word, Elasticsearch's full-text search is unsuitable. In such cases, the vector search functionality provided by ElatisSearch 7.x can be used to retrieve a wider range of results. Therefore, how to perform fast and accurate vector queries using ElatisSearch 7.x has become a crucial issue that urgently needs to be addressed.

[0061] In related technologies, based on the implementation principle of the vector search function provided by the ElatisSearch 7.x component, the result text vector that matches the query text vector is first searched using script_score, then the vector similarity score is calculated for each of the multiple matched result text vectors, and finally the target result text is fed back to the user based on the vector similarity score of each result text vector.

[0062] However, since the script_score query method uses a linear query approach, it needs to traverse all the currently stored vector data one by one without skipping any, until a result text vector matching the query text vector is found. This results in high time complexity and low query efficiency for linear queries.

[0063] To address the aforementioned technical problems, this invention provides a data query method, apparatus, electronic device, and storage medium. The following is a detailed description... Figures 1-6This invention describes a data query method, apparatus, electronic device, and storage medium. The data query method can be executed by an ElatisSearch 7.x component. This ElatisSearch 7.x component possesses software extensibility, allowing the implementation of a pre-defined plugin library with nearest-neighbor search capabilities. This pre-defined plugin library is a third-party plugin written into the ElatisSearch 7.x component in software. The ElatisSearch 7.x component possesses at least nearest-neighbor search capabilities and target connectivity graph construction capabilities. The ElatisSearch 7.x component is version 7.x of the ElatisSearch component.

[0064] It should be noted that the following method embodiments are illustrated using an ElatisSearch7.x component containing the preset plugin library as the execution subject.

[0065] To facilitate understanding of the data query method provided by this invention, the following examples will provide a detailed description of the method. It is understood that these examples can be combined with each other, and similar concepts or processes may not be repeated in some embodiments.

[0066] Reference Figure 1 This is a flowchart illustrating the data query method provided by the present invention, as shown below. Figure 1 As shown, the data writing method includes the following steps 110 to 130.

[0067] Step 110: In response to the data query request, construct a target connectivity graph based on the multiple original vector data included in each of the different index objects that have been written. Each original vector data includes a query text vector and a corresponding query result vector. The coordinate positions corresponding to different query text vectors in the target connectivity graph are interconnected. The data query request carries the requirement for the number of target query text vectors and query result vectors.

[0068] Each index object corresponds to a physical query index. Each raw vector data set includes not only the query result vector and the corresponding query text vector, but may also include a query text ID vector and a business primary key field vector. Furthermore, the query text vector in each raw vector data set participates in the construction of the target connectivity graph and subsequent data queries, while other information such as the query result vector, query text ID vector, and business primary key field vector in each raw vector data set is only stored. The target query text vector is used to locate the starting query position in the target connectivity graph, and the query result vector data is used to determine the final number of returned query results.

[0069] Specifically, when the ElatisSearch 7.x component receives a data query request from a user, it immediately responds to the request by constructing a target connectivity graph based on the multiple raw vector data included in each of the different index objects that have already been written. For example, if three index objects have been written when responding to a data query request, and the first index object includes four raw vector data, the second index object includes five raw vector data, and the third index object includes three raw vector data, then the coordinate positions corresponding to the four raw vector data, the five raw vector data, and the three raw vector data in the target connectivity graph are all interconnected.

[0070] It should be noted that the type of the query result vector can be determined based on the type of the query text vector. For example, when the type of the query text vector is a question type, such as "How is the weather today?", the type of the query result vector is an answer type, such as "The weather is sunny today".

[0071] Step 120: Based on the target connectivity graph, find multiple candidate coordinate positions adjacent to the coordinate position of the target query text, and determine at least one target coordinate position from the multiple candidate coordinate positions based on the query result quantity requirements.

[0072] The required number of query results can be a specific threshold or a recall rate. No specific limitation is made here. When the required number of query results is a recall rate, the product of the total number of candidate coordinate positions and the recall rate can be used as the threshold for the number of query results.

[0073] Specifically, to find multiple candidate coordinate positions adjacent to the coordinate position of the target query text in the target connected graph, we can first take the coordinate position of the target query text in the target connected graph as the starting position, and then determine all coordinate positions in the target connected graph that satisfy a preset nearest neighbor relationship with the starting position as candidate coordinate positions. This preset nearest neighbor relationship can be determined based on a preset nearest neighbor query algorithm. That is, the preset nearest neighbor relationship can be that the distance from the starting position meets a preset distance threshold, in which case all coordinate positions in the target connected graph that meet the preset distance threshold can be taken as candidate coordinate positions; or, the preset nearest neighbor relationship can be a circle drawn with the starting position as the center and a preset radius, in which case all coordinate positions in the circle other than the starting position are candidate coordinate positions.

[0074] At this point, if the total number of candidate coordinate positions matches the required number of query results (i.e., the total number of candidate coordinate positions is the same as the threshold determined based on the number of query results), then multiple candidate coordinate positions can be used as target coordinate positions. Conversely, if the total number of candidate coordinate positions does not match the required number of query results (i.e., the threshold determined based on the number of query results is less than the total number of the second coordinate positions), then a relevance scoring algorithm can be used to calculate the relevance score corresponding to each candidate coordinate position, and the target coordinate position with the highest relevance score is selected. The relevance scoring algorithm here can be either the Best Matching (BM)25 algorithm or the Term Frequency-Inverse Document Frequency (TF-IDE) algorithm. The 25 in the BM25 algorithm refers to the 25th iteration, and the BM25 algorithm is currently the most mainstream algorithm in the field of information retrieval for calculating question and document similarity scores. The TF-IDE algorithm is a statistical method used to evaluate the importance of a "word" to a document in a document set or corpus.

[0075] Step 130: Determine the query result vector of the query text vector corresponding to at least one target coordinate position as the query result of the data query request.

[0076] Specifically, since each original vector data includes a query text vector and a corresponding query result vector, when determining the coordinate position of each target, the query result vector of the query text vector corresponding to each target coordinate position can be determined as the query result of the data query request.

[0077] The data query method provided by this invention, when the ElatisSearch 7.x component responds to a data query request, firstly constructs a target connectivity graph based on the multiple original vector data included in each of the different index objects already written. Then, it connects the coordinate positions corresponding to different query text vectors in the target connectivity graph. Given that the data query request carries the target query text vector and a requirement for the number of query results, it searches for multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph. Based on the requirement for the number of query results, it determines at least one target coordinate position from the multiple candidate coordinate positions. Finally, it determines the query result vector of the query text vector corresponding to at least one target coordinate position as the query result of the data query request. In this way, by combining nearest-neighbor text query and graph algorithms, it eliminates the need to traverse and query all currently stored vector data one by one. Instead, it only needs to determine the final query result from a subset of original vector data that satisfies the preset nearest-neighbor relationships, thus reducing the time complexity of data queries and significantly improving data query efficiency.

[0078] Understandably, based on the above Figure 1 The data query method shown, in one example embodiment, involves first constructing a subgraph based on the multiple original vector data included in each index object when building a connected graph, and then constructing a connected graph based on each subgraph. Therefore, the specific process of constructing the target connected graph based on the multiple original vector data included in each of the different index objects in step 110 may include:

[0079] First, for each index object, a subgraph of the index object is constructed based on the multiple original vector data included in each index object. The coordinate positions corresponding to the query text vectors of the multiple original vector data in the subgraph are interconnected. Then, based on each subgraph, a target connectivity graph is constructed, in which the subgraphs are interconnected.

[0080] Specifically, each coordinate position in each subgraph includes a vertex and the coordinate position of that vertex.

[0081] For example, if three index objects have been written in response to a data query request, and the first index object includes four raw vector data, the second index object includes five raw vector data, and the third index object includes three raw vector data in response, then the constructed target connectivity graph includes subgraph p1 of the first index object, subgraph p2 of the second index object, and subgraph p3 of the third index object. Subgraph p1 includes four coordinate positions that are interconnected, subgraph p2 includes five coordinate positions that are interconnected, and subgraph p3 includes three coordinate positions that are interconnected. The four, five, and three coordinate positions are also interconnected. Furthermore, the construction of the target connectivity graph can specifically be based on the multiple raw vector data included in each of the written index objects, constructing a target connectivity graph in D-dimensional space where the subgraphs of each index object are interconnected, and D is an integer greater than 500. The reason why D > 500 is that the existing ElatisSearch 7.x components can only perform data queries on vector datasets with ≤ 500 dimensions and the recall effect is generally poor. The method of this invention constructs a target connectivity graph in the D-dimensional space when responding to data query commands, so that the constructed target connectivity graph can be used for data query operations on vector datasets of millions or even tens of millions.

[0082] The data query method provided by this invention improves the accuracy and reliability of constructing a connected graph by first constructing subgraphs for each index object and then constructing a target connected graph based on each subgraph.

[0083] Understandably, based on the above Figure 1In one example embodiment of the data query method shown, step 110 involves writing multiple raw vector data for each of the different index objects that have been written. Specifically, the writing process for the multiple raw vector data included in one of the index objects may include:

[0084] First, in response to the data write command, multiple raw data to be written are vectorized to obtain multiple raw vector data. Each raw vector data includes a query result vector and a query text vector. Further, based on the multiple raw vector data and the index object generation method of the preset plugin library, the index object corresponding to the multiple raw vector data is determined. The index object includes multiple raw vector data. Then, the index object is stored in the Segment document under the Lucene file. After determining that the cumulative time for responding to the data write command has reached the preset time, the Segment document is flushed to the disk. The Segment document stored on the disk is used to construct the target connectivity graph.

[0085] Specifically, when the ElatisSearch 7.x component receives a data query request from a user, it immediately responds to the request by calling a vector transformation model to vectorize each piece of raw data to be written in the query request, resulting in multiple raw vector data. Each piece of raw data to be written includes query result data, query text number data, and business primary key field data, etc. The corresponding converted raw vector data can be an N-dimensional vector array of length N, where N is a positive integer, and the original data can be [0.12, 0.31, 0.33, ...].

[0086] At this point, the index objects corresponding to multiple raw vector data can be determined by calling the index object generation method of the preset plugin library. The index objects are then stored in the Segment document under the Lucene file. When the Segment document meets the disk refresh conditions, the Segment document is flushed to the disk. The disk refresh conditions here are the cumulative duration of responding to the data write command reaching the preset duration. The preset duration can be set manually, for example, the preset duration can be 60 seconds.

[0087] Based on this, when the cumulative duration of the response data write command reaches a preset duration, all the raw vector data determined within the preset duration can be stored in the Segment document under the Lucene file. That is, during the data write operation, a disk flush operation is performed every preset interval, and the corresponding Segment document is different each time a disk input operation is performed. The disk flush operation can be an operation that flushes all the raw vector data determined within the preset duration into the Segment document under the Lucene file. For example, if 8 raw vector data are determined after 60 seconds, all 8 raw vector data can be stored in the Segment document S1 under the Lucene file; if 3 raw vector data are determined after another 60-second interval, all 3 raw vector data can be stored in the Segment document S2 under the Lucene file.

[0088] The data writing method provided by this invention, using the ElatisSearch 7.x component, achieves data writing by first vectorizing the raw data to be written, then determining the index objects, and finally flushing the index objects stored in the Segment documents under the Lucene file to the disk. This ensures that the target connectivity graph can be accurately and quickly constructed when responding to data query commands, providing reliable data support for improving data query efficiency. Furthermore, when the cumulative duration of responding to data write commands reaches a preset duration, all index objects determined within the preset duration are stored in the Segment documents under the Lucene file. This ensures that the storage operation of storing the corresponding index objects in different Segment documents is performed at preset intervals, thereby improving the efficiency of flushing the Segment documents to the disk.

[0089] Understandably, based on the above Figure 1 The data query method shown, in one example embodiment, when the preset plugin library is the HNSW plugin library, determines the index objects corresponding to the multiple raw vector data based on the index object generation method of the preset plugin library. Its specific implementation process may include:

[0090] First, multiple raw vector data are converted into data types supported by the HNSW plugin library to obtain multiple vector data conversion results; then, these multiple vector data conversion results are added to the preset index of the preset ElatisSearch component; then, based on the completion flag, the index object generation method of the HNSW plugin library is called to determine the index object corresponding to the raw vector data.

[0091] The preset ElatisSearch component can specifically be an ElatisSearch 7.x component, and the preset index can be an index of the ElatisSearch 7.x component. HNSW stands for Hierarchical Navigable SmallWorld, and can specifically refer to a hierarchical navigable small-world network. The HNSW plugin library is a graph-based algorithm library in the KNN search domain (or ANN search domain). Furthermore, KNN specifically refers to K-Nearest Neighbor, i.e., K nearest neighbors. When the index objects stored in the preset plugin library constitute a large-scale dataset, the computational cost of using KNN becomes too high. In this case, only A approximate nearest neighbors can be considered, i.e., KNN can be replaced with ANN. Here, ANN specifically refers to Approximate Nearest Neighbor and is an algorithm used to find the nearest neighbors in large-scale datasets; A < K and A and K are both positive integers.

[0092] Specifically, refer to Figure 2 The following is a flowchart illustrating the data writing process when the preset plugin library provided by this invention is the HNSW plugin library. Figure 2 As shown, each piece of raw data to be written is vectorized through a vector transformation model to obtain the corresponding raw vector data. Then, each piece of raw vector data is converted into a data type supported by the HNSW plugin library and written into the index of the ElatisSearch 7.x component to instruct the index to execute the process of writing vector data to the Lucene file. At this time, the vector data conversion results obtained from each piece of raw vector data can be added to the index of the ElatisSearch 7.x component. After the addition is completed, the index object generation method of the HNSW plugin library can be called through the application program interface (API) of the HNSW plugin library. Specifically, the build method of the HNSW plugin library can be called to obtain the index object returned by the build method. This index object is the index object corresponding to multiple pieces of raw vector data.

[0093] The data query method provided by this invention first converts multiple raw vector data into data types supported by the HNSW plugin library, then adds the conversion results to the preset index of the preset ElatisSearch component, and finally calls the index object generation method of the HNSW plugin library to determine the index objects corresponding to the multiple raw vector data. In this way, the pre-set algorithm library in the HNSW plugin library can accurately and quickly determine the index objects corresponding to multiple raw vector data, improving the convenience and reliability of data writing.

[0094] Understandably, based on the above Figure 1 The data query method shown, in one example embodiment, when the preset plugin library is the LSH plugin library, determines the index objects corresponding to the multiple original vector data based on the index object generation method of the preset plugin library. Its specific implementation process includes:

[0095] First, obtain the target hash function corresponding to each of the multiple raw data to be written; then, for each raw data to be written and each target hash function, based on the hash index object generation method of the LSH plugin library, use the target hash function to hash the raw vector data to determine the hash value corresponding to the raw vector data; then, for each hash value and each raw data to be written, map the raw data to be written to the hash bucket where the hash value is located to determine the index object corresponding to the multiple raw vector data.

[0096] LSH stands for Locality Sensitive Hashing, and it can specifically refer to the Locality Sensitive Hashing algorithm, which is an approximate nearest neighbor search algorithm based on a hash function.

[0097] Specifically, refer to Figure 3 The following is a flowchart illustrating the data writing process when the preset plugin library provided by this invention is the LSH plugin library. Figure 3 As shown, each piece of raw data to be written is vectorized through a vector transformation model to obtain the corresponding raw vector data. Then, each piece of raw vector data is converted into a data type supported by the HNSW plugin library and written into the index of the ElatisSearch 7.x component to instruct the index to execute the process of writing vector data to the Lucene file. At this time, the target hash function corresponding to each piece of raw data to be written can be obtained, that is, the appropriate hash function selected for each piece of raw data to be written. Then, through the API of the LSH plugin library, the target hash function is called to hash each piece of raw vector data separately to obtain the hash value corresponding to each piece of raw vector data. Each piece of raw vector data is then mapped to the hash bucket where the corresponding hash value is located, thereby constructing a hash table for multiple pieces of raw vector data. The constructed hash table is the index object corresponding to multiple pieces of raw vector data.

[0098] The data query method provided by this invention, based on the hash index object generation method of the LSH plugin library, first uses the target hash function corresponding to each piece of raw data to be written to hash the corresponding raw multidimensional data. Then, each piece of raw data to be written is mapped to the hash bucket containing the corresponding hash value, thereby determining the index object of multiple raw vector data. In this way, the index object corresponding to the raw vector data can be accurately and quickly determined by the approximate nearest neighbor search algorithm based on the hash function in the LSH plugin library, which also improves the convenience and reliability of data writing.

[0099] Understandably, based on the above Figure 1 The data query method shown, in one example embodiment, when the preset plugin library is the IVSPQ plugin library, determines the index objects corresponding to the multiple raw vector data based on the index object generation method of the preset plugin library. Its specific implementation process may include:

[0100] First, for each original vector data, based on the pre-stored mapping relationship between vector data, cluster points, and encoding methods in the IVSPQ plugin library, the target cluster points and target encoding methods corresponding to the original vector data are determined. Next, for each original vector data, based on the target cluster points and target encoding methods corresponding to each original vector data, the encoding index object generation method in the IVSPQ plugin library is used to encode and cluster the original data to be written, obtaining clustering encoding results. Then, for each clustering encoding result and each original data to be written, the original data to be written is mapped to the encoding table where the clustering encoding results are located, determining the index object corresponding to the original vector data.

[0101] The IVSPQ plugin library includes an approximate nearest neighbor search algorithm based on vector quantization, which can be used for fast retrieval of high-dimensional data. The full English name of IVSPQ is Inverted Vector Quantization with Sequential Projections and Query-adaptive Refinement, which is a vector quantization approximate nearest neighbor search algorithm based on inverted indexes and using sequential projection and query-adaptive refinement.

[0102] Specifically, refer to Figure 4 The following is a flowchart illustrating the data writing process when the preset plugin library provided by this invention is the IVSPQ plugin library. Figure 4As shown, each piece of raw data to be written is vectorized through a vector transformation model to obtain corresponding raw vector data. Then, each piece of raw vector data is converted into a data type supported by the HNSW plugin library and written into the index of the ElatisSearch 7.x component to instruct the index to execute the process of writing vector data to the Lucene file. At this time, the mapping relationship between vector data, cluster points, and encoding methods pre-stored in the IVSPQ plugin library can be called through the API of the IVSPQ plugin library to determine the target cluster point and target encoding method corresponding to each piece of raw vector data. Furthermore, for the target cluster point and target encoding method corresponding to each piece of raw vector data, based on the encoding index object generation method of the IVSPQ plugin library, the target encoding method and target cluster point are used to encode and cluster the corresponding raw data to be written, and each piece of raw data to be written is mapped to the encoding table where the corresponding clustering encoding result is located, thus constructing multiple encoding tables for raw vector data. The constructed encoding tables are the index objects corresponding to multiple pieces of raw vector data.

[0103] The data query method provided by this invention, based on the approximate nearest neighbor search algorithm of the IVSPQ plugin library, first obtains the target cluster points and target encoding methods corresponding to multiple original vector data. Then, it uses the target encoding methods and target cluster points to encode and cluster the corresponding original data to be written. Finally, it maps each piece of original data to be written to the encoding table containing the corresponding clustering encoding results, thereby determining the index object corresponding to multiple original vector data. In this way, the index object corresponding to multiple original vector data can be accurately and quickly determined by the approximate nearest neighbor search algorithm based on vector quantization in the IVSPQ plugin library, thereby improving the convenience and reliability of data writing.

[0104] Understandably, based on the above Figure 1 The data query method shown, in one example embodiment, is unsuitable for storing vector datasets of millions or even tens of millions of dimensions because the existing dense_vector type in Elasticsearch 7.x components can only store vector data of no more than 500 dimensions. Therefore, the preset plugin library provided by this invention not only includes vector search algorithms but also data types for storing millions or even tens of millions of vector data. Based on this, the data query method provided by this invention may further include:

[0105] The raw vector data is stored in the preset data type of the preset plugin library. The preset data type is used to store raw vector data without any dimension limitation.

[0106] The preset data type can be specifically knn_vector, which is used to store vector data. The knn_vector type does not have a specific limit on the storage dimension, that is, it can store vector data with more than 500 dimensions.

[0107] Specifically, each original vector data contains a DocValue data, which is the corresponding original multidimensional vector array. Once an original multidimensional vector array is obtained, it can be stored as a DocValue data in a preset data type.

[0108] It should be noted that the preset data type is a new data type implemented through a third-party plugin called the preset plugin library, so there is no dimensional limitation for storing vector data.

[0109] The data query method provided by this invention solves the drawback that the existing dense_vector type in Elasticsearch 7.x components cannot store vector data with millions or even tens of millions of dimensions by storing the raw vector data in a preset data type of a preset plugin library. This improves the applicability of using the preset data type of the preset plugin library for vector data storage and provides a reliable guarantee for the richness and comprehensiveness of subsequent query results.

[0110] Understandably, based on the above Figure 1 In one example embodiment of the data query method shown, to improve query speed, the constructed target connected graph can be loaded into memory; alternatively, the target connected graph can be loaded directly if query speed is not a requirement. Based on this, step 120, which involves finding multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connected graph, may include:

[0111] Load the target connected graph into memory, and based on the target connected graph, find multiple candidate coordinate positions that are adjacent to the coordinate position of the target query text;

[0112] or,

[0113] Based on the result of loading the target connectivity graph and the target connectivity graph, find multiple candidate coordinate positions that are adjacent to the coordinate position of the target query text.

[0114] Specifically, to improve query speed, the target connectivity graph can be loaded into memory periodically or sequentially using the Guava memory framework, and then multiple candidate coordinate positions adjacent to the target query text's coordinate position can be found. Alternatively, if query speed is not a priority, the target connectivity graph can be loaded directly, and multiple candidate coordinate positions adjacent to the target query text's coordinate position can be found. For example, when multiple raw vector data included in different index objects are stored on disk, the target connectivity graph can be constructed using the raw vector data included in each index object. The constructed target connectivity graph can be loaded directly or loaded into memory.

[0115] The data query method provided by this invention improves the flexibility and reliability of quantity queries by first loading the constructed target connectivity graph directly or loading it into memory, and then performing a search for multiple candidate coordinate positions.

[0116] Understandably, based on the above Figure 1 In one example embodiment of the data query method shown, upon receiving a data query request, it can first determine whether the previously constructed connected graph is still applicable, and if not, then construct the target connected graph required for this data query. Based on this, the specific implementation process of step 110 may include:

[0117] First, the different index objects that have been written are compared with the index objects used in the previously constructed connected graph. Then, it is determined that the different index objects that have been written are different from the index objects used in the previously constructed connected graph. Based on the multiple original vector data included in each of the different index objects that have been written, the target connected graph is constructed.

[0118] Specifically, when a data query request is received, it can first be determined whether the previously constructed connected graph is applicable to this data query. That is, it can be determined whether the index object used in the previously constructed connected graph is the same as the currently written different index objects. When it is determined that the index object used in the previously constructed connected graph is not the same as the currently written different index objects, it can be confirmed that the previously constructed connected graph is not applicable to this data query. At this time, the target connected graph can be constructed based on the multiple original vector data included in each of the different index objects that have been written.

[0119] The data query method provided by this invention, upon receiving a data query request, first determines whether the previously constructed connected graph is applicable to the current data query. If it is determined that the previously constructed connected graph is not applicable to the current data query, a target connected graph applicable to the current data query is constructed. This improves the rationality and necessity of constructing the target connected graph, providing a more reliable guarantee for subsequent data queries.

[0120] It should be noted that, in conjunction with the above... Figures 2-4 After performing data queries, the HNSW plugin library showed the highest query performance with a recall rate exceeding 95%. The LSH plugin library offered a simple implementation and high query performance. The IVSPQ plugin library also demonstrated high query performance with a recall rate exceeding 90%. Furthermore, by using the KNN algorithm from the HNSW plugin library, the time complexity for data queries decreased from O(n) to O(log(n)), and the recall effect was significantly improved; where n represents the total dimension of the original vector data included in each of the different indexed objects.

[0121] Reference Figure 5 This is a schematic diagram of the structure of the data query device provided by the present invention, as shown below. Figure 5 As shown, the data query device 500 includes a connected graph construction module 510 and a data query module 520.

[0122] The connected graph construction module 510 is used to respond to data query requests by constructing a target connected graph based on the multiple original vector data included in each of the different index objects that have been written. The original vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connected graph are interconnected. The data query request carries the target query text vector and the number of query results required.

[0123] The data query module 520 is used to find multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph, and to determine at least one target coordinate position from the multiple candidate coordinate positions based on the query result quantity requirements; and to determine the query result vector of the query text vector corresponding to at least one target coordinate position as the query result of the data query request.

[0124] Optionally, the connected graph construction module 510 can be used to construct a subgraph of each index object based on the multiple original vector data included in each index object, wherein the coordinate positions corresponding to the query text vectors of the multiple original vector data in the subgraph are interconnected; and to construct a target connected graph based on each subgraph, wherein the subgraphs in the target connected graph are interconnected.

[0125] Optionally, the data query device provided by the present invention may further include a data writing module, used to vectorize multiple raw data to be written in response to a data writing command, to obtain multiple raw vector data; each raw vector data includes a query result vector and a query text vector; based on the multiple raw vector data and an index object generation method of a preset plugin library, an index object corresponding to the multiple raw vector data is determined, the index object including the multiple raw vector data; the index object is stored in a Segment document under the Lucene file, and when the cumulative duration of responding to the data writing command reaches a preset duration, the Segment document is flushed to the disk, the Segment document stored on the disk is used to construct the target connectivity graph.

[0126] Optionally, if the default plugin library is the HNSW plugin library, the data writing module is also used to convert multiple raw vector data into data types supported by the HNSW plugin library to obtain multiple vector data conversion results; add the multiple vector data conversion results to the default index of the default ElatisSearch component; and, based on the completion flag, call the index object generation method of the HNSW plugin library to determine the index object corresponding to the multiple raw vector data.

[0127] Optionally, if the default plugin library is the LSH plugin library, the data writing module is also used to obtain the target hash function corresponding to each of the multiple raw data to be written; for each raw data to be written and each target hash function, based on the hash index object generation method of the LSH plugin library, the target hash function is used to hash the raw vector data to determine the hash value corresponding to the raw vector data; for each hash value and each raw data to be written, the raw data to be written is mapped to the hash bucket where the hash value is located to determine the index object corresponding to the multiple raw vector data.

[0128] Optionally, if the default plugin library is the IVSPQ plugin library, the data writing module is also used to determine the target clustering point and target encoding method corresponding to each original vector data based on the mapping relationship between vector data, clustering points, and encoding methods pre-stored in the IVSPQ plugin library; for each original vector data corresponding to its own target clustering point and target encoding method, based on the encoding index object generation method of the IVSPQ plugin library, the original data to be written is encoded and clustered using the target encoding method and target clustering point to obtain the clustering encoding result; for each clustering encoding result and each original data to be written, the original data to be written is mapped to the encoding table where the clustering encoding result is located to determine the index object corresponding to multiple original vector data.

[0129] Optionally, the data writing module is also used to store the raw vector data into a preset data type in the preset plugin library. The preset data type is used to store the raw vector data without any dimension limitation.

[0130] Optionally, the data query module provided by the present invention may further include a data loading module, used to load the target connected graph into memory and search for multiple candidate coordinate positions based on the target connected graph; or, based on the result of loading the target connected graph and the target connected graph, search for multiple candidate coordinate positions.

[0131] Optionally, the connected graph construction module 510 is specifically used to compare the different index objects that have been written with the index objects used in the previously constructed connected graph; determine that the different index objects that have been written are different from the index objects used in the previously constructed connected graph; and construct the target connected graph based on the multiple original vector data included in each of the different index objects that have been written.

[0132] The data query device 500 provided by the present invention can execute the technical solution of the data query method in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the data query method. Please refer to the implementation principle and beneficial effects of the data query method. It will not be repeated here.

[0133] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device 600 may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following methods:

[0134] In response to a data query request, a target connectivity graph is constructed based on the multiple raw vector data included in each of the different index objects that have been written. The raw vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connectivity graph are interconnected. The data query request carries the target query text vector and the query result quantity requirement. Based on the target connectivity graph, multiple candidate coordinate positions adjacent to the coordinate position of the target query text are found, and based on the query result quantity requirement, at least one target coordinate position is determined from the multiple candidate coordinate positions. The query result vector of the query text vector corresponding to at least one target coordinate position is determined as the query result of the data query request.

[0135] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] On the other hand, embodiments of the present invention disclose a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as including:

[0137] In response to a data query request, a target connectivity graph is constructed based on the multiple raw vector data included in each of the different index objects that have been written. The raw vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connectivity graph are interconnected. The data query request carries the target query text vector and the query result quantity requirement. Based on the target connectivity graph, multiple candidate coordinate positions adjacent to the coordinate position of the target query text are found, and based on the query result quantity requirement, at least one target coordinate position is determined from the multiple candidate coordinate positions. The query result vector of the query text vector corresponding to at least one target coordinate position is determined as the query result of the data query request.

[0138] In another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the transmission methods provided in the above embodiments, including, for example:

[0139] In response to a data query request, a target connectivity graph is constructed based on the multiple raw vector data included in each of the different index objects that have been written. The raw vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connectivity graph are interconnected. The data query request carries the target query text vector and the query result quantity requirement. Based on the target connectivity graph, multiple candidate coordinate positions adjacent to the coordinate position of the target query text are found, and based on the query result quantity requirement, at least one target coordinate position is determined from the multiple candidate coordinate positions. The query result vector of the query text vector corresponding to at least one target coordinate position is determined as the query result of the data query request.

[0140] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only for illustrating the present invention and not for limiting the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and should be covered within the scope of the claims of the present invention.

Claims

1. A data query method, characterized in that, include: In response to a data query request, a target connectivity graph is constructed based on the multiple raw vector data included in each of the different index objects that have been written. The raw vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connectivity graph are interconnected. The data query request carries the target query text vector and the query result quantity requirement. Based on the target connectivity graph, find multiple candidate coordinate positions adjacent to the coordinate position of the target query text, and determine at least one target coordinate position from the multiple candidate coordinate positions based on the query result quantity requirement. The query result vector of the query text vector corresponding to the at least one target coordinate position is determined as the query result of the data query request; Specifically, the writing process for the multiple original vector data of each of the different index objects that have been written includes: In response to a data write command, multiple raw data to be written are vectorized to obtain multiple raw vector data; each raw vector data includes the query result vector and the query text vector. Based on the index object generation method of the multiple original vector data and the preset plugin library, the index object corresponding to the multiple original vector data is determined, and the index object includes the multiple original vector data; The index object is stored in a Segment document under the Lucene file, and the cumulative time for responding to the data write command reaches a preset time. The Segment document is then flushed to the disk, and the Segment document stored on the disk is used to construct the target connectivity graph.

2. The data query method according to claim 1, characterized in that, The construction of the target connected graph based on the multiple original vector data included in each of the different index objects that have been written includes: For each index object, a subgraph of the index object is constructed based on the multiple original vector data included in each index object, wherein the coordinate positions corresponding to the query text vectors of the multiple original vector data in the subgraph are interconnected. Based on each of the subgraphs, the target connected graph is constructed, wherein the subgraphs in the target connected graph are interconnected.

3. The data query method according to claim 1, characterized in that, The preset plugin library is the HNSW plugin library. The method for generating index objects based on the multiple raw vector data and the preset plugin library, which determines the index objects corresponding to the multiple raw vector data, includes: The multiple original vector data are converted into data types supported by the HNSW plugin library to obtain multiple vector data conversion results; The transformation results of the multiple vector data are added to the preset index of the preset ElatisSearch component; Based on the completion flag, the index object generation method of the HNSW plugin library is called to determine the index object.

4. The data query method according to claim 1, characterized in that, The preset plugin library is an LSH plugin library. The method for generating index objects based on the multiple raw vector data and the preset plugin library, which determines the index objects corresponding to the multiple raw vector data, includes: Obtain the target hash function corresponding to each of the multiple original data to be written; For each piece of raw data to be written and each target hash function, based on the hash index object generation method of the LSH plugin library, the target hash function is used to hash the raw vector data to determine the hash value corresponding to the raw vector data; For each hash value and each piece of raw data to be written, the raw data to be written is mapped to the hash bucket where the hash value is located, and the index object is determined.

5. The data query method according to claim 1, characterized in that, The preset plugin library is an IVSPQ plugin library. The method for generating index objects based on the multiple raw vector data and the preset plugin library, which determines the index objects corresponding to the multiple raw vector data, includes: For each original vector data, based on the mapping relationship between vector data, cluster points, and encoding methods pre-stored in the IVSPQ plugin library, the target cluster point and target encoding method corresponding to the original vector data are determined. For each original vector data corresponding to its target clustering point and target encoding method, based on the encoding index object generation method of the IVSPQ plugin library, the original data to be written is encoded and clustered using the target encoding method and the target clustering point to obtain the clustering encoding result; For each clustering encoding result and each piece of raw data to be written, the raw data to be written is mapped to the encoding table where the clustering encoding result is located, and the index object is determined.

6. The data query method according to any one of claims 1 to 5, characterized in that, The method further includes: The original vector data is stored in a preset data type of the preset plugin library. The preset data type is used to store the original vector data without dimension limitation.

7. The data query method according to any one of claims 1 to 5, characterized in that, The step of finding multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connected graph includes: The target connected graph is loaded into memory, and the multiple candidate coordinate positions are found based on the target connected graph; or, The target connected graph is loaded directly to find the multiple candidate coordinate positions.

8. The data query method according to any one of claims 1 to 5, characterized in that, The construction of the target connected graph based on the multiple original vector data included in each of the different index objects that have been written includes: Compare the different index objects that have been written with the index objects used in the previously constructed connected graph; Determine that the different index objects that have been written are different from the index objects used in the previously constructed connected graph, and construct the target connected graph based on the multiple original vector data included in each of the different index objects that have been written.

9. A data query device, characterized in that, include: The connected graph construction module is used to respond to data query requests by constructing a target connected graph based on multiple original vector data included in different index objects that have been written. The original vector data includes query text vectors and corresponding query result vectors. The coordinate positions corresponding to different query text vectors in the target connected graph are interconnected. The data query request carries the target query text vector and the number of query results required. The data query module is used to find multiple candidate coordinate positions adjacent to the coordinate position of the target query text based on the target connectivity graph, and to determine at least one target coordinate position from the multiple candidate coordinate positions based on the query result quantity requirement. The query result vector of the query text vector corresponding to the at least one target coordinate position is determined as the query result of the data query request; The data writing module is used to vectorize multiple raw data to be written in response to a data writing command, thereby obtaining the multiple raw vector data; each raw vector data includes the query result vector and the query text vector; Based on the index object generation method of the multiple original vector data and the preset plugin library, the index object corresponding to the multiple original vector data is determined, and the index object includes the multiple original vector data; The index object is stored in a Segment document under the Lucene file, and the cumulative time for responding to the data write command reaches a preset time. The Segment document is then flushed to the disk, and the Segment document stored on the disk is used to construct the target connectivity graph.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data query method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data query method as described in any one of claims 1 to 8.

Images