Data Processing Method, Device, Medium and Product Based on a Vector Set

Abstract

The present application discloses a data processing method, device, medium, and product based on a vector set, relating to the technical field of data processing, including: receiving a query instruction, where the composite index data includes a composite shape graph, and the composite shape graph is constructed with vector indices in the union of the vector indices of the initial vectors in a group of initial vector sets as shape nodes, and the index indication data of each initial vector set is used to indicate, in the composite shape graph, the vector indices of the initial vectors in each initial vector set; determining the index indication data corresponding to the query instruction, performing vector index retrieval in the composite shape graph according to the query vector and the index indication data corresponding to the query instruction, and generating a query result based on a group of retrieved vector indices, solving the problem of large storage overhead in the related art and achieving the technical effect of improving the utilization rate of storage resources.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a data processing method, device, medium, and product based on a vector set. Background Art

[0002] In today's data-intensive and artificial intelligence-driven application environment, vector data retrieval has become a key capability, especially when dealing with high-dimensional and massive vector sets. In the data processing methods of related technologies, vector indexes are constructed using techniques such as KD (K-Dimensional) trees, LSH (Locality-Sensitive Hashing), PQ (Product Quantization), and HNSW (Hierarchical Navigable Small World). For each vector data set, a vector index data is constructed independently, generating an independent index structure for each vector set. As the number of vector data sets grows, this independent index construction method poses a severe challenge to hardware resources. For example, on a high-performance computing device FPGA (Field Programmable Gate Array), the capacity of on-chip storage (such as SRAM (Static Random Access Memory)) is limited, and the space required to store multiple independent vector indexes will increase significantly with the increase in the number of vector sets, resulting in a large storage overhead.

[0003] Therefore, there is a problem of large storage overhead in related technologies. Summary of the Invention

[0004] This application provides a data processing method, device, medium, and product based on a vector set to at least solve the problem of large storage overhead in related technologies.

[0005] This application provides a data processing method based on a vector set, including:

[0006] Receiving a query instruction, where the query instruction is used to query an initial vector in a group of initial vector sets. The vector index data of the group of initial vector sets includes composite index data and index indication data for each initial vector set in the group of initial vector sets. The composite index data includes a composite shape graph, and the composite shape graph is constructed with the vector indexes in the union of the vector indexes of the initial vectors in the group of initial vector sets as shape nodes. The index indication data for each initial vector set is used to indicate the vector indexes of the initial vectors in each initial vector set in the composite shape graph;

[0007] Determine the index indication data corresponding to the query instruction, where the index indication data corresponding to the query instruction is the index indication data of the initial vector set associated with the query vector indicated by the query instruction;

[0008] According to the query vector and the index indication data corresponding to the query instruction, perform vector index retrieval in the composite shape graph, and generate a query result based on a set of retrieved vector indexes, where the query result includes the initial vectors corresponding to the vector indexes in the set of vector indexes.

[0009] This application also provides a data processing device based on a vector set, including:

[0010] A receiving module, configured to receive a query instruction, where the query instruction is used to query an initial vector in a set of initial vector sets, the vector index data of the set of initial vector sets includes composite index data and the index indication data of each initial vector set in the set of initial vector sets, the composite index data includes a composite shape graph, the composite shape graph is constructed with the vector indexes in the union of the vector indexes of the initial vectors in the set of initial vector sets as shape nodes, and the index indication data of each initial vector set is used to indicate the vector indexes of the initial vectors in each initial vector set in the composite shape graph;

[0011] A determining module, configured to determine the index indication data corresponding to the query instruction, where the index indication data corresponding to the query instruction is the index indication data of the initial vector set associated with the query vector indicated by the query instruction;

[0012] A retrieval module, configured to perform vector index retrieval in the composite shape graph according to the query vector and the index indication data corresponding to the query instruction, and generate a query result based on a set of retrieved vector indexes, where the query result includes the initial vectors corresponding to the vector indexes in the set of vector indexes.

[0013] This application also provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any of the above data processing methods based on a vector set when executing the computer program.

[0014] This application also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above data processing methods based on a vector set are implemented.

[0015] This application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any of the above data processing methods based on a vector set are implemented.

[0016] Through this application, a query instruction is received. The query instruction is used to query an initial vector in a set of initial vector sets. The vector index data of the set of initial vector sets includes composite index data and index indication data for each initial vector set in the set of initial vector sets. The composite index data includes a composite shape graph, which is constructed with vector indices in the union of the vector indices of the initial vectors in the set of initial vector sets as shape nodes. The index indication data for each initial vector set is used to indicate, in the composite shape graph, the vector indices of the initial vectors in each initial vector set; through the composite shape graph and the index indication data, without recording the index structure of each initial vector set, the waste of storage resources is reduced, the storage overhead is reduced, the problem of large storage overhead in the related art is solved, and the utilization rate of storage resources is improved. Moreover, according to the query vector and the index indication data corresponding to the query instruction, vector index retrieval is performed in the composite shape graph, and a query result is generated based on the retrieved set of vector indices, which improves the efficiency of vector retrieval to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 FIG. 10 is a schematic diagram of an application scenario of a data processing method based on a vector set provided by an embodiment of the present application.

[0019] Figure 2 FIG. 14 is a schematic flowchart of a data processing method based on a vector set provided by an embodiment of the present application.

[0020] Figure 3 FIG. 18 is a schematic diagram of a data processing method based on a vector set provided by an embodiment of the present application.

[0021] Figure 4 FIG. 22 is a schematic flowchart of another data processing method based on a vector set provided by an embodiment of the present application.

[0022] Figure 5 FIG. 26 is a schematic diagram of another data processing method based on a vector set provided by an embodiment of the present application.

[0023] Figure 6 FIG. 30 is a schematic flowchart of yet another data processing method based on a vector set provided by an embodiment of the present application.

[0024] Figure 7 FIG. 34 is a schematic diagram of yet another data processing method based on a vector set provided by an embodiment of the present application.

[0025] Figure 8 This is a schematic structural diagram of a data processing method based on a vector set provided by an embodiment of the present application. Detailed implementation manners

[0026] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0027] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0028] To enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0029] In the data processing methods of the related art, vector indexes are constructed using, for example, KD trees, LSH (Locality-Sensitive Hashing), PQ (Product Quantization), and HNSW (Hierarchical Navigable Small World Graph). For each vector data set, a vector index data is independently constructed, and an independent index structure is generated for each vector set. As the number of vector data sets grows, this independent index construction method poses a severe challenge to hardware resources. For example, on a high-performance computing device FPGA, the capacity of on-chip storage (such as SRAM) is limited, and the space required to store multiple independent vector indexes will increase significantly as the number of vector sets increases, resulting in a large storage overhead in the related art.

[0030] To solve the above problems, an embodiment of the present application provides a data processing method based on a vector set, which receives a query instruction for querying initial vectors in a set of initial vector sets. The vector index data of a set of initial vector sets includes composite index data and index indication data for each initial vector set in the set of initial vector sets. The composite index data includes a composite shape graph, which is constructed with vector indices in the union of vector indices of initial vectors in the set of initial vector sets as shape nodes. The index indication data for each initial vector set is used to indicate, in the composite shape graph, the vector indices of the initial vectors in each initial vector set. Through the composite shape graph and the index indication data, there is no need to record the index structure of each initial vector set, reducing the waste of storage resources, lowering the storage overhead, solving the problem of large storage overhead in the related art, and improving the utilization rate of storage resources. Moreover, according to the query vector and the index indication data corresponding to the query instruction, vector index retrieval is performed in the composite shape graph, and a query result is generated based on the retrieved set of vector indices, improving the efficiency of vector retrieval to a certain extent.

[0031] According to one aspect of an embodiment of the present application, a data processing method based on a vector set is provided. Optionally, in this embodiment, the above data processing method based on a vector set may be but is not limited to being applied to a hardware environment including a terminal device 102 and a server 104 as shown in Figure 1 Figure. The server 104 can be connected to the terminal device 102 through a network and can be used to provide services (such as application services, etc.) for the terminal device 102 or the client installed on the terminal device 102. A database can be set up on the server 104 or independently of the server 104 to provide data storage services for the server 104.

[0032] The above network may include but is not limited to at least one of the following: a wired network, a wireless network. The above wired network may include but is not limited to at least one of the following: a wide area network, a metropolitan area network, a local area network. The above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC (Personal Computer), a mobile phone, a tablet computer, etc. The server 104 may be but is not limited to a cloud server, a server cluster, or other server types.

[0033] The data processing method based on a vector set in an embodiment of the present application may be executed by the server 104, or may be executed by the terminal device 102, or may be jointly executed by the server 104 and the terminal device 102. Among them, when the terminal device 102 executes the data processing method based on a vector set in an embodiment of the present application, it may also be executed by the client installed on it.

[0034] Taking the execution of the data processing method based on the vector set in this embodiment by the terminal device 102 as an example, Figure 2 FIG. is a schematic flowchart of an optional data processing method based on a vector set according to an embodiment of the present application. As Figure 2 shown, the process of this method may include the following steps:

[0035] Step S202, receive a query instruction, where the query instruction is used to query an initial vector in a set of initial vector sets. The vector index data of a set of initial vector sets includes composite index data and index indication data of each initial vector set in the set of initial vector sets. The composite index data includes a composite shape graph, and the composite shape graph is constructed with vector indexes in the union of the vector indexes of the initial vectors in a set of initial vector sets as shape nodes. The index indication data of each initial vector set is used to indicate the vector indexes of the initial vectors in each initial vector set in the composite shape graph.

[0036] It should be noted that the initial vector set may refer to a collection of original vector data used to construct vector index data. The initial vector refers to the original vector. Each initial vector set may represent a type of data or a vector data set in a specific application scenario. These initial vector sets may come from various application scenarios, such as image recognition, natural language processing, bioinformatics analysis, etc., and usually have high-dimensional features, and it is necessary to optimize the retrieval performance by constructing an index data structure.

[0037] The composite index data may be a data structure constructed by merging the vector indexes of multiple initial vector sets. The design of the composite index data can reduce the storage overhead to a certain extent. By sharing part of the structure of the vector index, multiple vector sets can share the same index structure, improving the retrieval efficiency.

[0038] The composite shape graph is a graphical data structure transformed from the union of the vector indexes of multiple vector sets. Among them, the shape nodes in the composite shape graph may contain vector index information selected from different initial vector sets. The composite shape graph forms a comprehensive retrieval path network through the connection and jump relationships between the shape nodes, and can support the data retrieval of multiple vector sets at the same time. Optionally, the composite shape graph may cover the shape nodes of all the initial vector sets in a set of initial vector sets. The composite shape graph can support retrieval operations across multiple initial vector sets, reducing the space and time resources required for repeated index construction.

[0039] The index indication data of the initial vector set can be used to indicate the vector index of the initial vectors in the initial vector set, that is, the metadata that can be provided for each initial vector set to indicate the position information of each initial vector set in the composite shape graph, such as the input parameters of the initial vector set at the entrance of the composite shape graph. The index indication data can be used during the query process to guide the retrieval process and locate the vector index of the initial vector set related to the query vector. The vector index can be the range information of the shape node corresponding to the initial vector set in the composite shape graph. Optionally, the index indication data of each initial vector set can be used to indicate the vector index of the initial vectors in the initial vector set. Specifically, the index indication data can be used to indicate the retrieval range of the initial vector set in the shared shape graph. For example, the retrieval starting point.

[0040] In the composite shape graph, a node can represent a region composed of a specific vector index or a combination of vector indexes.

[0041] For example, there is an initial vector set A, and the index indication data of the initial vector set A can include information pointing to the relevant shape nodes of A in the composite shape graph. Such as, the entrance information of the initial vector set in the composite shape graph, that is, the starting position and height. In this way, during the vector query, the index indication data corresponding to the query vector can be obtained through the query vector for the query.

[0042] Optionally, in the composite shape graph, a shape node can be used to associate with one or more initial vector sets.

[0043] Optionally, the composite shape graph can be formed by merging the shape graphs of multiple initial vector sets, including the union of all shape nodes. The shape graph can be regarded as a special graphical representation, which can include various types of shape nodes and edges to represent complex data structures or models.

[0044] In the composite shape graph, the same shape nodes can be shared by multiple initial vector sets, and the unique shape nodes of each initial vector set are added to form a larger index network, which can support the efficient retrieval of multiple vector sets.

[0045] Optionally, on the FPGA, the composite shape graph can be stored in the on-chip memory of the FPGA (such as SRAM). The index indication data of each initial vector set in a group of initial vector sets can be stored in solid-state storage (such as a hard disk). Optionally, the composite index data can be public index data, and the index indication data can be private index data for each initial vector set to be used to indicate how to query the initial vector (i.e., the original vector) associated with the query vector in each initial vector set.

[0046] Optionally, upon receiving a query instruction, the query instruction can be parsed, and the current composite shape graph can be retrieved from a preset storage area. Optionally, upon receiving a storage instruction, index construction processing can be performed on the set of vectors to be stored indicated by the storage instruction to update the composite shape graph. The updated composite shape graph can include the vector indices of the set of vectors to be stored, i.e., the shape nodes of the set of vectors to be stored in the composite shape graph.

[0047] Step S204, determine the index indication data corresponding to the query instruction, where the index indication data corresponding to the query instruction is the index indication data of the initial vector set associated with the query vector indicated by the query instruction.

[0048] Optionally, the query instruction is used to query an initial vector in a set of initial vector sets. Specifically, the query instruction is used to instruct the vector index query engine to find the initial vector most similar to a specific query vector in the stored vector set. The query instruction can carry specific information of the query vector, including its dimension, eigenvalue, etc., and the identification of the vector set to be retrieved. According to the identification of the vector set to be retrieved, it is determined which initial vector set the query vector is associated with. Optionally, the query vector can be associated with at least one initial vector set.

[0049] Optionally, the initial vector set associated with the query vector can be determined according to the vector characteristics of the query vector indicated by the query instruction. It can also be determined according to the identification of the vector set indicated by the query instruction which initial vector set the query vector indicated by the query instruction is associated with.

[0050] Step S206, perform vector index retrieval in the composite shape graph according to the query vector and the index indication data corresponding to the query instruction, and generate a query result based on the retrieved set of vector indices, where the query result includes the initial vectors corresponding to the vector indices in the set of vector indices.

[0051] Optionally, before performing vector index retrieval, the query vector can be preliminarily processed using a preset processing method according to the preliminarily processed query vector. Specifically, the preset processing method can be the method when the initial vector set associated with the query vector is stored. The index indication data corresponding to the query instruction, i.e., the index indication data corresponding to the initial vector set associated with the query vector.

[0052] Optionally, according to the index indication data corresponding to the initial vector set associated with the query vector and the preliminarily processed query vector, locate and access the composite shape graph to effectively find the vector indices related to the target query vector in the composite shape graph.

[0053] Optionally, a retrieved set of vector indices can be the preliminary retrieval result. It is necessary to process the initial vectors corresponding to the set of vector indices and the query vector to obtain the query result. It should be noted that the query result includes the initial vectors corresponding to the vector indices in the set of vector indices. Optionally, the query result includes the initial vectors corresponding to all or part of the vector indices in the set of vector indices.

[0054] Through the embodiments of the present application, a query instruction is received. The query instruction is used to query the initial vectors in a set of initial vector sets. The vector index data of the set of initial vector sets includes composite index data and index indication data for each initial vector set in the set of initial vector sets. The composite index data includes a composite shape graph. The composite shape graph is constructed with the vector indices in the union of the vector indices of the initial vectors in the set of initial vector sets as shape nodes. The index indication data for each initial vector set is used to indicate the vector indices of the initial vectors in each initial vector set in the composite shape graph. Through the composite shape graph and the index indication data, there is no need to record the index structure of each initial vector set, reducing the waste of storage resources, lowering the storage overhead, solving the problem of large storage overhead in the related art, and improving the utilization rate of storage resources. Moreover, according to the query vector and the index indication data corresponding to the query instruction, vector index retrieval is performed in the composite shape graph, and a query result is generated based on the retrieved set of vector indices, which improves the efficiency of vector retrieval to a certain extent.

[0055] In an exemplary embodiment, in order to further improve the utilization rate of storage resources, before receiving the query instruction, the above method further includes: performing dimensionality reduction processing on each initial vector set respectively to obtain each dimensionality-reduced initial vector set; performing quantization encoding on each dimensionality-reduced initial vector set respectively to obtain the quantization result of each initial vector set; constructing composite index data and index indication data for each initial vector set according to the quantization result of each initial vector set.

[0056] It should be noted that multiple vector dimensionality reduction and quantization encoding modules can be used to perform dimensionality reduction processing and quantization encoding on each initial vector set respectively. Dimensionality reduction parameters and quantization encoding parameters can be pre-configured for the multiple vector dimensionality reduction and quantization encoding modules to perform dimensionality reduction processing and quantization encoding on the initial vector sets that need to be dimensionally reduced and encoded to different degrees.

[0057] Specifically, as Figure 3 shown, in Figure 3Among them, there are m initial vector sets and m vector dimensionality reduction and quantization encoding modules. For each of the m initial vector sets, vector dimensionality reduction and quantization encoding are performed in parallel. The quantization results of each initial vector set are input into the index construction engine to construct composite index data and index indication data for each initial vector set, and the composite index data and index indication data for each initial vector set are stored in a preset storage area, that is, in the FPGA. The query vector is preprocessed before querying, and the preprocessed result is input into the FPGA-based index query engine to obtain an index query result. The index query result is post-processed to obtain a query result. For example, a group of the most similar initial vectors.

[0058] Optionally, dimensionality reduction processing can use algorithms such as PCA (Principal Component Analysis) and t-SNE (t-distributed Stochastic Neighbor Embedding) to reduce the dimension of the vectors, reduce storage requirements and computational complexity. Each initial vector set after dimensionality reduction is more compact and convenient for storage.

[0059] After dimensionality reduction processing, the dimensionality-reduced data set can be quantized and encoded to convert the continuous values of the vectors into discrete codewords or codebooks, thereby reducing the complexity of vector identification. Optionally, the quantization encoding method can be product quantization (PQ), scalar quantization or other quantization methods to encode each dimension or subspace of the vector and generate quantization results that are easy to store and retrieve.

[0060] Optionally, according to the quantization results of each initial vector set, composite index data and corresponding index indication data are constructed for multiple initial vector sets. Optionally, according to the quantization results of the initial vector sets and preset mapping rules, the shape nodes of each initial vector set are determined, and the shape nodes of each initial vector set are used to construct a composite shape graph covering all initial vector sets. In the composite shape graph, the shared shape nodes can be referenced by multiple initial vector sets, forming a network structure, which greatly reduces the repeated storage of shape nodes, thereby saving storage space. The construction of the composite shape graph is the core of the composite index data, ensuring the retrieval of multiple vector sets in a single index structure. For each initial vector set, its exclusive index indication data can be generated. The index indication data can include entry information pointing to this initial vector set in the composite shape graph. The index indication data can enable the query vector to quickly locate the vector index related to it, thereby improving the retrieval efficiency.

[0061] Through this embodiment, through dimensionality reduction and quantization coding, the original high-dimensional vector data is converted into a form that occupies less space. Further, by constructing a composite shape graph and recording index indication data, efficient storage and reuse of vector index data are achieved, reducing waste of storage resources and lowering storage costs. The construction of the composite shape graph allows for flexible switching between different vector sets without reconstructing the index structure. Moreover, with the construction of the composite shape graph and the guidance of the index indication data, the query vector can quickly find its matching vector in the composite shape graph, reducing ineffective path exploration and enhancing the retrieval efficiency and response speed.

[0062] In an exemplary embodiment, quantization coding is respectively performed on each initial vector set after dimensionality reduction to obtain the quantization result of each initial vector set, including: calculating the statistical values of each initial vector set after dimensionality reduction under different vector dimensions in at least two vector dimensions, where the statistical value is one of the average value and the median; according to the statistical values of each initial vector set after dimensionality reduction under different vector dimensions, performing quantization coding on the vector values of the initial vectors in each initial vector set after dimensionality reduction under different vector dimensions to obtain the quantization result of each initial vector set.

[0063] Optionally, for each initial vector set after dimensionality reduction, first, calculate the statistical values of each initial vector set in at least two vector dimensions. The selection of the statistical value includes the average value or the median, and which specific statistical value to use can depend on the characteristics of the vector set and the scenario requirements. The average value reflects the mean value of the numerical values in the vector set, while the median can better reflect the central tendency of the data, especially when the data distribution has skewness or outliers.

[0064] Specifically, after dimensionality reduction processing, data preprocessing is performed on the initial vector set after dimensionality reduction processing to remove or process outliers. Select at least two vector dimensions, and for the selected vector dimensions, calculate the statistical values for each dimension. When the statistical value is the average value, calculate the sum of the numerical values in this dimension and then divide by the size of the vector set; when the statistical value is the median, first sort the numerical values in this dimension and then select the numerical value at the middle position as the median. Specifically, the selected vector dimensions can be the dimensions with relatively high correlation with the vector features, or can also be the dimensions that have a key impact on the retrieval performance.

[0065] After calculating the statistical value, the vector values of the initial vector can be quantized and encoded according to the statistical value. Optionally, the process of quantizing and encoding the vector values of the initial vector may include: based on the calculated statistical value (average or median), determining one or more quantization thresholds for each dimension. These thresholds are used to divide the continuous vector values into different quantization intervals. The numerical range of each vector dimension is divided into multiple quantization intervals, and the specific division method depends on the setting of the thresholds. For example, if the median is used as the threshold, the values can be divided into two intervals: less than the median and greater than or equal to the median. Assign a code to each quantization interval, and these codes are usually in binary form (such as 0 or 1) to simplify the representation of the initial vector. The numerical value of each dimension of the initial vector will be encoded as the corresponding number according to the quantization interval it belongs to, thereby generating the quantized vector.

[0066] Optionally, during the quantization and encoding process, a series of quantization parameters can also be generated, which may include quantization thresholds, the division method of quantization intervals, and encoding rules. These quantization parameters can be used for subsequent vector retrieval.

[0067] Optionally, through the statistical value, the distribution characteristics of the vector set in each dimension can be better understood, and then a more appropriate quantization strategy can be selected. For example, if the variation in a certain dimension is large, uniform quantization may be adopted; if the median is relatively stable, non-uniform quantization can be considered. Such a quantization strategy can more accurately reflect the similarity between vectors and improve the retrieval accuracy.

[0068] Through this embodiment, the quantization and encoding can significantly reduce the storage requirements of vector data, enabling the system to more efficiently manage and retrieve large-scale vector data sets. By selecting different statistical values (average or median) for quantization in different dimensions, it can flexibly adapt to the characteristics of different vector sets and ensure the rationality of the quantization and encoding.

[0069] In an exemplary embodiment, the composite index data further includes a composite mapping dictionary, which is used to record the mapping relationship between the node identifiers of the shape nodes in the composite shape graph and the information represented by the shape nodes; according to the quantization results of each initial vector set, constructing the composite index data and the index indication data of each initial vector set, including: based on the quantization result of the first initial vector set, constructing an initial shape graph and an initial mapping dictionary, where the first initial vector set is one of a set of initial vector sets; taking the other initial vector sets in the set of initial vector sets except the first initial vector set as the second initial vector set in turn and performing the following construction operations to obtain the composite index data and the index indication data of the initial vector set: according to the quantization result of the second initial vector set, constructing a binary tree of the second initial vector set; according to the initial mapping dictionary, performing mapping processing on the binary tree of the second initial vector set to obtain the shape graph of the second initial vector set; updating the initial shape graph and the initial mapping dictionary according to the shape graph of the second initial vector set to obtain the updated initial shape graph and the updated initial mapping dictionary, and recording the index indication data of the second initial vector set; where the composite shape graph is the initial shape graph after the last update, and the composite mapping dictionary is the initial mapping dictionary after the last update.

[0070] It should be noted that the composite shape graph may include multiple shape nodes and the jump relationships corresponding to the multiple shape nodes. The composite shape graph can be constructed from the shape graphs of a set of initial vector sets. The shape graph of each initial vector set can be a partial area of the composite shape graph, that is, the shape graph of each initial vector set can be a subset of the composite shape graph. Of course, it can also be understood as the union of the shape graphs of a set of initial vector sets, which is the composite shape graph. The composite mapping dictionary can be a mapping dictionary between shape numbers and shape nodes.

[0071] Optionally, based on the quantization result of the first initial vector set, constructing the initial shape graph and the initial mapping dictionary. Specifically, the initial shape graph can be constructed through a preset mapping rule and the quantization result of the first initial vector set, and the initial mapping dictionary can be obtained by recording the mapping relationship of the initial shape graph. The first initial vector set can be a randomly selected one from a set of initial vector sets. The initial mapping dictionary records the mapping relationship between the node identifier of the shape node and the information it represents. For example, the source of the vector set, the vector type, and the relevance to the quantization result.

[0072] For the other initial vector sets in the set of initial vector sets except the first initial vector set, the other initial vector sets can be taken as the second initial vector set in turn and perform the construction operation to obtain the composite index data and the index indication data of the initial vector set.

[0073] Specifically, for the second set of initial vectors being currently processed, based on the quantization results of the second set of initial vectors, a binary tree representation is constructed. Each node of the binary tree corresponds to a decision point in the quantization encoding process and is used for subsequent construction of the shape graph and update of the mapping dictionary. The binary tree of the second set of initial vectors is mapped using the previously constructed initial mapping dictionary, converting the nodes in the binary tree into shape nodes, thereby generating the shape graph of the second set of initial vectors, ensuring that the newly added vector set can find corresponding descriptions in the composite shape graph, and at the same time starting to form a shared set of shape nodes. According to the shape graph of the second set of initial vectors, the initial shape graph and the initial mapping dictionary are updated to reflect the characteristics of the newly added vector set. This update process may involve addition and deletion of shape nodes, adjustment of the shape graph structure, and update of the mapping relationships in the mapping dictionary. Specifically, during the update process, if a shape node not recorded in the initial mapping dictionary is encountered, the initial shape nodes need to be extended to allocate identifiers for the new shape nodes.

[0074] After the update of the shape graph and the mapping dictionary is completed, the index indication data of the second set of initial vectors is recorded, such as: information such as the shape entry position, the height of the shape graph, etc., for positioning during subsequent retrieval. Of course, the index indication data can also record the leaf node array corresponding to this initial vector set, as well as the vector list. Specifically, the leaf node array can be used to store a data set of all the leaf node information corresponding to this initial vector set. Leaf nodes are terminal nodes without child nodes and can be used to associate with specific vectors in the initial vector set. The vector list is a data structure closely associated with the leaf node array and can record all the vector data belonging to a certain or a group of leaf nodes. The vector data can be directly stored in the vector list. Of course, it can also store vector IDs (i.e., pointers to the original vectors). Each vector or vector ID in the vector list has an index corresponding to its position in the list.

[0075] When the processing of all the initial vector sets is completed, the initially updated shape graph becomes the composite shape graph. The composite shape graph contains the shape node information of all the initial vector sets, forming a comprehensive vector index data structure. Similarly, the initially updated initial mapping dictionary becomes the composite mapping dictionary, which can completely record the mapping relationship between the node identifiers of all the shape nodes in the composite shape graph and the information they represent.

[0076] In one example, as Figure 4 shown, the steps of the index construction process for the newly added vector set are as follows:

[0077] Step 1, load the initial mapping dictionary M1 and the initial shape graph G1;

[0078] Step 2: Perform dimensionality reduction and quantization on the input vector dataset D2 to obtain dataset D`2;

[0079] Step 3: According to the initial mapping dictionary M1, construct a binary tree T2 for the vector dataset D2;

[0080] Step 4: According to the binary tree T2 and the initial mapping dictionary M1, construct a shape graph G2 for the vector dataset D2;

[0081] Step 5: According to the shape graph G2 of the vector dataset D2, perform an update process on the initial mapping dictionary M1 and the initial shape graph G1. The updated shape graph is G = G1 U G2, and the updated mapping dictionary is M2;

[0082] Step 6: Store the updated data in the shape graph and the updated mapping dictionary.

[0083] Specifically, write the newly added shape nodes of the shape graph G2 into the SRAM and solid-state storage of the FPGA, and write the updated mapping dictionary M2 into the DRAM (Dynamic Random-Access Memory) and solid-state storage of the FPGA.

[0084] Through the above example, it can be obtained that there will be a considerable number of overlapping parts between the shape graph G1 and the shape graph G2, and the updated shape graph is the union of G1 and G2.

[0085] Of course, it is also possible to construct the shape graphs of each initial vector set in a parallel manner and merge the shape graphs of each initial vector set to obtain a composite shape graph. Specifically, each initial vector set is regarded as a separate task and assigned to different processing units in the computing architecture (such as different stream processors in the FPGA or CPU multi-cores). In this way, the construction work of the shape graph of each initial vector set can be carried out simultaneously without waiting for the processing of the previous initial vector set to complete. Once the construction of the shape graphs of all initial vector sets is completed, the next step is to merge these independent shape graphs to form a composite shape graph. During the merging process, the shape nodes in each shape graph will be compared to identify the same shape nodes, that is, those nodes that exist in multiple shape graphs. The same shape nodes are marked as shared nodes and are only recorded once in the composite shape graph, thus avoiding duplicate storage and saving storage resources. For different shape nodes, they will each occupy a position in the composite shape graph and maintain their respective characteristics. At the same time as merging the shape graphs, the initial mapping dictionary also needs to be updated to become a composite mapping dictionary.

[0086] Through the above embodiments, the shape graphs of each initial vector set are constructed in a parallel processing manner and merged, which can make full use of the parallel computing power of modern computing architectures. Especially in large FPGAs and multi-core processors, the speed of index construction can be significantly improved.

[0087] Optionally, the composite mapping dictionary enables quick location of specific vectors in the composite shape graph. Even when the vector set is growing and changing, the efficient retrieval performance can be maintained by updating the mapping dictionary. This method is not only applicable to static vector sets but can also flexibly handle dynamically changing vector sets, including but not limited to image feature vectors, text semantic vectors, audio waveform vectors, etc., greatly expanding the application scenarios.

[0088] Optionally, for the construction mechanism of composite index data, when there is a new initial vector set to be stored, according to the composite mapping dictionary, the shape graph of the new initial vector set can be constructed to update the composite shape graph and the composite mapping dictionary.

[0089] Through the above embodiments, the construction mechanism of composite index data allows the system to dynamically integrate new vector sets without reconstructing the entire index structure, enhancing the adaptability and flexibility of the system.

[0090] Through this embodiment, by sharing the shape nodes in the composite shape graph and the composite mapping dictionary, the independent index data storage requirements for each vector set are significantly reduced, the overall storage overhead is lowered, and the utilization rate of storage resources is improved.

[0091] In an exemplary embodiment, based on the quantization result of the first initial vector set, constructing an initial shape graph and an initial mapping dictionary includes: according to the quantization result of the first initial vector set, constructing a binary tree of the first initial vector set; mapping the binary tree of the first initial vector set into shape nodes in the initial shape graph to obtain the initial shape graph; recording the mapping relationship between the path encoding of the binary tree of the first initial vector set and the shape nodes in the initial shape graph during the process of mapping the binary tree of the first initial vector set into the shape nodes in the initial shape graph to obtain the initial mapping dictionary.

[0092] Optionally, quantization encoding converts the numerical values of the initial vectors in each initial vector set into binary identifiers, that is, an encoding sequence composed of '0' and '1'. According to the quantization result of the first initial vector, a binary tree of the first initial vector set can be constructed. In the binary tree, each non-leaf node represents a quantization encoding dimension comparison decision point, and through this decision point, the data set can be further subdivided. If the value of the quantization encoding in a certain dimension is '0', then go down along the left subtree; if it is '1', then go down along the right subtree. The leaf nodes contain the vector information of all quantization encodings that are the same under the subtree.

[0093] After the binary tree of the first initial vector set is constructed, the binary tree of the first initial vector set can be mapped into a preset shape graph (which can be a blank shape graph template). In this mapping process, each node of the binary tree is converted into a shape node in the shape graph, and an initial shape graph is obtained. Among them, non-leaf nodes represent the branch points of vector indices, and leaf nodes contain specific vector information or links pointing to vectors.

[0094] Optionally, during the process of constructing the shape graph according to the path encoding, each node (especially non-leaf nodes) in the binary tree represents a path decision in the vector space, and the shape nodes are the abstract representations of these decisions. When several non-leaf nodes have the same path decision (i.e., they exhibit similar vector features in the vector space after dimensionality reduction quantization), then these non-leaf nodes can be mapped to the same shape node, thereby reducing the number of shape nodes in the shape graph to save storage space.

[0095] During the mapping operation, the mapping relationship between the path encoding of the binary tree of the first initial vector set and the shape nodes in the initial shape graph is recorded, that is, the mapping relationship between the path encoding from the root node to any leaf node in the binary tree of the first initial vector set and the corresponding shape nodes in the initial shape graph. The path encoding is concatenated by the quantization encodings of each decision point along the path from the root node to the leaf node in the binary tree of the first initial vector set, and the path encoding is used to identify a path from the root node to the leaf node.

[0096] Similarly, the construction process of the shape graph of the second initial vector set can refer to the construction process of the shape graph of the first initial vector above, and this application will not elaborate here.

[0097] Based on the mapping relationship between the path encoding and the shape nodes, an initial mapping dictionary can be constructed. Optionally, the initial mapping dictionary can be a mapping dictionary from the shape numbers of the first initial vector set to the shape nodes.

[0098] Through this embodiment, by mapping the binary tree of the first initial vector set into shape nodes and recording them in the initial mapping dictionary, not only the integrity of the vector index information is maintained, but also the storage space requirement is reduced through the sharing of shape nodes. The construction of the initial shape graph and the initial mapping dictionary provides a basis for the further integration and management of composite index data.

[0099] In an exemplary embodiment, according to the quantization result of the second initial vector set, the second initial vector set is segmented to obtain a first sub-vector set and a second sub-vector set; according to the first sub-vector set, the left subtree of the binary tree of the second initial vector set is constructed, and according to the second sub-vector set, the right subtree of the binary tree of the second initial vector set is constructed to obtain the binary tree of the second initial vector set.

[0100] It should be noted that the quantization result of the second initial vector set includes a series of binary codes composed of '0' and '1', representing the coding values of the vector in each dimension after dimensionality reduction. The quantization result of the second initial vector set is analyzed to segment the initial vectors in the second initial vector set according to different dimension values, obtaining a first sub-vector set and a second sub-vector set. For example, for a certain vector dimension, the initial vector with a coding value of "0" is classified into the first sub-vector set, and the initial vector with a coding value of "1" is classified into the second sub-vector set.

[0101] Based on the first sub-vector set, the left subtree of the binary tree of the second initial vector set is constructed, and according to the second sub-vector set, the right subtree of the binary tree of the second initial vector set is constructed. Optionally, dimensionality reduction quantization coding is performed on the first sub-vector set and mapped to a binary tree structure to obtain the left subtree of the binary tree of the second initial vector set, where non-leaf nodes represent the coding decision points of subsequent dimensions, and leaf nodes contain vector information with exactly the same quantization coding. Similarly, the right subtree of the binary tree is constructed using the second sub-vector set.

[0102] After completing the construction of the left subtree and the right subtree, these two subtrees are merged with the existing binary tree structure to form a complete binary tree of the second initial vector set.

[0103] Subsequently, similar to the processing method of the first initial vector set, the complete binary tree structure is mapped into a shape graph to generate a set of shape nodes of the second initial vector set to construct the shape graph of the second initial vector set. In this process, the mapping relationship between the path coding and the shape nodes in the composite mapping dictionary is recorded for subsequent index data query.

[0104] Similarly, for the construction process of the binary tree of the first initial vector set, reference can be made to the construction process of the above-mentioned second initial vector, which will not be elaborated in this application.

[0105] For the segmentation processing of the second initial vector set, a segmentation strategy can be adopted. For example, mean segmentation, median segmentation or other statistic segmentation can be used to adapt to vector sets with different characteristics, including but not limited to word vectors in natural language processing, image feature vectors in computer vision, etc.

[0106] Through this embodiment, the second initial vector set is segmented to obtain a first sub-vector set and a second sub-vector set, and the left sub-tree and the right sub-tree are respectively constructed to obtain a complete binary tree, which can ensure that the characteristics of each initial vector set are fully considered in the construction process of the composite shape graph, so that the required vector can be more accurately matched during retrieval.

[0107] In an exemplary embodiment, according to the initial mapping dictionary, the binary tree of the second initial vector set is mapped to obtain the shape graph of the second initial vector set, including: traversing the binary tree of the second initial vector set according to the path encoding of the binary tree of the second initial vector set; during the traversal of the binary tree of the second initial vector set, determining whether there is a specified shape node in the initial shape graph, where the specified shape node is a shape node in the initial shape graph that matches the non-leaf node of the binary tree of the second initial vector set; in the case where there is a specified shape node, performing an update process on the specified shape node; in the case where there is no specified shape node, creating a new shape node and updating the created shape node to the initial shape graph to obtain the shape graph of the second initial vector set.

[0108] It should be noted that the path encoding can be on the path from the root node of the binary tree to the non-leaf node of the binary tree. Optionally, the acquisition method of the path encoding of the binary tree can be starting from the binary tree of the second initial vector set (i.e., the root node of the binary tree), and obtaining the path encoding of each non-leaf node in the binary tree. It consists of the encoding (0 or 1) of each decision point, representing a specific path to reach that node.

[0109] Based on the path encoding, starting from the root node of the binary tree, traverse layer by layer to each non-leaf node. During the traversal, the path encoding can be used to track the path to each non-leaf node. During the traversal of the binary tree of the second initial vector set, for each encountered non-leaf node, the following operations can be performed:

[0110] Query whether there is a specified shape node that matches in the initial shape graph according to the path encoding of the current node. The specified shape node refers to a shape node in the initial shape graph that has the same path encoding feature as the non-leaf node of the current binary tree; if a specified shape node is found in the initial shape graph, it indicates that the current node in the binary tree has the same decision characteristic as a certain existing node in the initial shape graph. At this time, the specified shape node will be updated, which may include adding a vector set index pointing to the node, or updating the associated statistical information, etc., to keep the shape graph up-to-date. If no specified shape node that matches the non-leaf node in the binary tree is found in the initial shape graph, a new shape node can be created and added to the initial shape graph to add a new decision point in the initial shape graph, and the new decision point is used to represent the decision characteristic of the current non-leaf node.

[0111] After each creation or update of a shape node, the current modified status information can be reflected in the initial shape graph and the initial mapping dictionary. Specifically, the update process of the shape node may include modifying the attributes of the shape node, such as adjusting the weight of the shape node, updating the associated statistical information, or expanding the connection range of the node, to ensure that the characteristics of the node in the associated binary tree can be accurately reflected.

[0112] After creating or updating a shape node, the structure of the initial shape graph can be adjusted, including updating the connection relationship between shape nodes, or optimizing the mapping of path encoding, to ensure that all shape nodes can be correctly and efficiently associated with each other in the graph.

[0113] As the processing of the second initial vector set is completed, the initial shape graph has undergone a series of updates and expansions, and finally forms a shape graph of the second initial vector set that contains the shape node information of the second initial vector set, that is, the composite shape graph. This composite shape graph retains all shape nodes corresponding to the decision paths of the second initial vector, and also takes into account the sharing with the previously existing shape nodes. And during the entire mapping process, the mapping relationship between all path encodings and shape nodes is recorded, and finally a composite mapping dictionary is formed.

[0114] It should be noted that for each initial vector set, the shape node mapped by the root node of the binary tree of each initial vector set can be recorded as the starting node of the initial vector set in the shared shape graph; the number of nodes passed from the root node to the farthest leaf node in the binary tree of the initial vector set is recorded as the number of nodes in the longest path of the initial vector set in the shared shape graph.

[0115] Optionally, in the case of storing a new vector set, the composite shape graph can be dynamically updated according to the vector set to be stored, so as to enhance the adaptability and flexibility of the shape graph.

[0116] Through this embodiment, by updating existing shape nodes or creating new nodes, the shape graph sharing between different vector sets is ensured, the storage space occupation is reduced, and the retrieval efficiency is improved.

[0117] In an exemplary embodiment, step S206 includes: performing dimensionality reduction processing and quantization coding on a query vector to obtain a quantization result of the query vector; performing node matching processing in a composite shape graph according to the quantization result of the query vector and index indication data corresponding to a query instruction; and determining a set of vector indexes according to the node type of the shape node matched in the composite shape graph.

[0118] It should be noted that the query vector indicated by the received query instruction is subjected to dimensionality reduction processing to reduce the dimension of the query vector. The dimensionality reduction process usually projects a high-dimensional vector into a low-dimensional space by means of techniques such as linear algebra transformation or feature selection, while trying to keep the relative distance between vectors unchanged.

[0119] Next, the dimensionality-reduced query vector is subjected to quantization coding to convert continuous numerical features into binary coding. Specifically, the numerical value of each dimension is compared with a preset threshold or statistical value (such as the median or mean), and then the coding value is determined. The quantization result of the quantization coding constitutes the quantization result of the query vector, which is used to guide the subsequent index query process.

[0120] Based on the quantization result of the query vector and the index indication data, node matching processing is performed in the composite shape graph. Specifically, according to the index indication data, the shape entry of the query vector in the composite shape graph is determined, and starting from the shape entry, matching processing is performed along the path indicated by the quantization result of the query vector until a leaf node is matched or the matching fails. According to the node type of the shape node matched as indicated by the matching result, a set of vector indexes corresponding to the query vector is determined.

[0121] Through this embodiment, the amount of computation is reduced by dimensionality reduction and quantization coding, the matching speed of the query vector in the composite shape graph is accelerated, and thus the retrieval efficiency is significantly improved. The quantization coding result corresponds closely to the shape nodes in the composite shape graph, which can ensure to a certain extent that the query vector can be accurately matched to a vector index similar to it, improving the retrieval accuracy. The use of the composite shape graph and the composite mapping dictionary can minimize the occupation of storage resources, especially in the scenario of simultaneously processing multiple vector sets, realizing the efficient utilization of storage and computing resources.

[0122] In an exemplary embodiment, dimensionality reduction processing and quantization encoding are performed on a query vector to obtain a quantization result of the query vector, including: obtaining a specified dimensionality reduction parameter corresponding to the query vector and a specified quantization parameter corresponding to the query vector, where the specified dimensionality reduction parameter is a parameter used for dimensionality reduction processing on an initial vector set associated with the query vector, and the specified quantization parameter is a parameter used for quantization encoding on the initial vector set associated with the query vector; performing dimensionality reduction processing on the query vector according to the specified dimensionality reduction parameter to obtain a dimensionality-reduced query vector; and performing quantization encoding on the dimensionality-reduced query vector according to the specified quantization parameter to obtain a quantization result of the query vector.

[0123] It should be noted that in order to achieve efficient and accurate processing of the query vector, the dimensionality reduction parameter and quantization parameter during dimensionality reduction and quantization encoding for each initial vector set can be pre-recorded, so that when a query instruction is received, according to the initial vector set associated with the query vector indicated by the query instruction, the specified dimensionality reduction parameter and specified quantization parameter used in the index construction stage can be obtained.

[0124] Optionally, for each query vector, it is first necessary to determine the initial vector set to which it belongs or is associated. Because when constructing the index, different initial vector sets may use different dimensionality reduction and quantization parameters, and these parameters are customized according to the characteristics of their respective initial vector sets to optimize the retrieval efficiency and storage cost.

[0125] Obtain the dimensionality reduction parameter used by the initial vector set associated with the query vector in the index construction stage. These parameters may include the eigenvector matrix of principal component analysis (PCA) for dimensionality reduction, the result matrix of SVD (singular value decomposition), or the parameters of other dimensionality reduction algorithms. Subsequently, the quantization parameter used by the initial vector set during quantization processing can be obtained. The quantization parameter may cover the quantization bit number, quantization threshold (such as median, mean, etc.), or other statistical information for quantization encoding.

[0126] Perform dimensionality reduction processing on the query vector using the specified dimensionality reduction parameter to transform the high-dimensional query vector into a low-dimensional space. This process usually involves matrix operations, such as multiplying by the eigenmatrix of PCA or applying the compression matrix of SVD, so as to reduce the dimension and maintain the main similarity features between vectors.

[0127] Perform quantization encoding on the dimensionality-reduced query vector based on the specified quantization parameter. Specifically, compare the numerical value of each dimension of the query vector with the quantization threshold, and encode it as '0' or '1' according to the comparison result. Through quantization encoding, the query vector is transformed into a vector composed of binary values, that is, the quantization result of the query vector, which is convenient for efficient matching in the composite shape diagram.

[0128] Through this embodiment, processing the query vector using the specified dimensionality reduction and quantization parameters during the construction of the initial vector set index can ensure that the transformation of the query vector during dimensionality reduction and quantization maintains the similarity relationship with the vectors in the initial vector set, thereby improving the accuracy of retrieval.

[0129] In an exemplary embodiment, the index indication data includes start indication data and range indication data. The start indication data is used to indicate the starting node of the initial vector set in the composite shape graph; the range indication data is used to indicate the number of nodes in the longest path of the initial vector set in the composite shape graph.

[0130] Perform node matching processing in the composite shape graph according to the quantization result of the query vector and the index indication data corresponding to the query instruction, including:

[0131] According to the quantization result of the query vector, the start indication data corresponding to the query instruction, and the range indication data corresponding to the query instruction, determine the indication path corresponding to the query vector. Among them, the indication path includes the shape nodes sequentially passed in the composite shape graph starting from the starting node indicated by the start indication data corresponding to the query instruction according to the quantization result of the query vector; sequentially use the shape nodes indicated by the indication path as the current shape nodes to perform the following node matching processing: when the value of the dimension corresponding to the current shape node is the same as the number of nodes in the longest path, determine that the current shape node is a leaf node; when there is no child node pointer for the current shape node, determine that the current shape node is an empty node; when the current shape node is neither an empty node nor a leaf node, jump to the next shape node of the current shape node.

[0132] It should be noted that the start indication data can indicate the starting node of the initial vector set in the composite shape graph, that is, the position information of the starting node of the initial vector set in the composite shape graph, which is the entrance of the shape graph. The starting node can be the first decision point in constructing the shape graph based on the quantized and encoded initial vector set.

[0133] The range indication data can be used to indicate the number of nodes in the longest path of the initial vector set in the composite shape graph. The number of nodes in the longest path is the maximum number of shape nodes that can be passed from the starting node in the composite shape graph by the initial vector set, which can be used to reflect the scale of the initial vector set in the composite shape graph and is crucial for limiting the query range, preventing out-of-bounds access, and resource management and optimization.

[0134] During the process of mapping the binary tree of the initial vector set to the shape graph, the shape node mapped by the root node in the binary tree of the initial vector set can be used as the starting node of the initial vector set in the composite shape graph, and the path of the shape node farthest from the starting node during the mapping process can be used as the longest path.

[0135] Optionally, according to the quantization result of the query vector, the starting point indication data, and the range indication data, determine the indication path of the query vector. The indication path of the query vector can be determined by 0 and 1 in the quantization result of the query vector, and each value corresponds to a specific shape node decision. Starting from the starting node indicated by the starting point indication data, traverse in the composite shape graph along this path.

[0136] Specifically, the process of node matching processing includes: sequentially taking the shape nodes indicated by the indication path as the current shape nodes. When the number of nodes in the current shape node corresponding dimension value reaches the number of nodes of the longest path specified by the range indication data, it is determined that the bottom layer of the initial vector set in the composite shape graph is reached. At this time, the current shape node is confirmed as a leaf node. The leaf node can include a vector index or a directly corresponding vector list.

[0137] During the traversal process, if the current shape node does not have a child node pointer (that is, it is not connected to any child nodes), then it is determined that the current shape node is an empty node. The appearance of an empty node indicates that there is no further decision path for matching in a specific dimension. At this time, remedial measures can be taken, such as retreating to the nearest parent node of the current shape node or returning the longest matching path. When the current shape node is neither an empty node nor a leaf node, according to the quantization value ('0' or '1') of the query vector in the current dimension, jump to the left child node or the right child node of the current shape node, and continue to search deeper along the indication path to the next layer.

[0138] Specifically, as Figure 5 shown, for each initial vector set in the m initial vector sets, the construction of the vector index can be performed inside the CPU. When the CPU receives the externally input initial vector set, the input initial vector set can be subjected to vector dimensionality reduction and quantization encoding, and the dimensionality reduction and quantization parameters are recorded.

[0139] According to the quantization result of each input initial vector set, construct a composite shape graph and a composite mapping dictionary. And during the process of constructing the composite shape graph and the composite mapping dictionary, record the leaf node array corresponding to the initial vector set in the composite shape graph and the vector list corresponding to each leaf node in the leaf node array. Write the composite mapping dictionary into the off-chip storage of the FPGA corresponding to the FPGA vector index query engine; write the composite shape graph into the on-chip storage of the FPGA in the FPGA vector index query engine.

[0140] That is, during the query process, input the dimension-reduced and quantized query vector into the multi-stage pipelined processing unit. In the multi-stage pipelined processing unit, output the matching vector list according to the dimension-reduced and quantized query vector.

[0141] Specifically, a multi-stage pipelined processing unit can be deployed in the FPGA vector index query engine. For example, a set of 32-stage pipelined processing units is designed on the FPGA, which means that the maximum dimension that can be processed is 32 (after dimensionality reduction and quantization), and the corresponding maximum shape height is 32 + 1. Taking two vector sets with dimensions of 16 and 20 after dimensionality reduction and quantization as an example, after the index construction of these two vector sets is completed, their heights in the composite shape graph are 16 + 1 (i.e., the number of nodes in the longest path is 17) and 20 + 1 (i.e., the number of nodes in the longest path is 21), respectively. Then, these two vector sets need to complete 16 steps and 20 steps of matching jump calculations respectively when performing index queries. Different pipelined processing units in the multi-stage pipelined processing unit can process concurrently. Specifically, the processing process of the multi-stage pipelined processing unit can be as follows Figure 6 as shown.

[0142] Step 1, enter the next-stage pipelined processing unit, and the corresponding processing height is current_height;

[0143] Specifically, current_height is the height of the current pipelined processing unit.

[0144] Step 2, obtain the value val (i.e., 0 or 1) on the current dimension of the input vector, the current shape node and its height h;

[0145] Specifically, the height h is the number of nodes in the longest path passed during index query.

[0146] Step 3, judge whether current_height > h;

[0147] Specifically, in the case of current_height > h, it is determined that the number of nodes in the longest path is exceeded, that is, jump to Step 8; when current_height is not greater than h, jump to Step 4.

[0148] Step 4, whether the shape node is an empty node;

[0149] Specifically, in the case that the shape node is an empty node, jump to Step 7; in the case that the shape node is not an empty node, jump to Step 5.

[0150] It should be noted that in the pipeline running program, if an empty node is matched, a no-operation needs to be used to complete each step of the program until the pipeline running program for the query instruction ends.

[0151] Step 5, jump from the current shape to the next shape node according to the matching relationship between val and 0 or 1, and h = h - 1;

[0152] Step 6, determine whether current_height == 2 (to determine whether the next shape node is a leaf node);

[0153] Specifically, in the case of a leaf node, jump to Step 9; in the case of not being a leaf node, jump to Step 10.

[0154] Step 7, determine whether current_height > 2;

[0155] Specifically, in the case of current_height > 2, it is necessary to continue to perform a no-operation subsequently to complete the operations in all dimensions, so jump to Step 8; in the case of current_height not being greater than 2, it means that each operation for this query instruction has been completed, that is, it is determined that the longest path is the path before reaching this no-node, and jump to Step 9;

[0156] Step 8, perform a no-operation (keep the current dimension of the input vector unchanged), and jump to Step 1 to enter the next-level pipelined processing unit; it should be noted that in the case where the height of the current pipeline unit is greater than the h of the current shape node, performing a no-operation can be used to align the height; in the case of matching a no-node, a no-operation is required to complete each step of the program until the pipeline running program for the query instruction ends.

[0157] Step 9, output the identifier of the matched leaf node;

[0158] Specifically, the node identifier of the leaf node can be the subscript information of the leaf node.

[0159] Step 10, increment the current dimension of the input vector by 1.

[0160] Specifically, after incrementing the current dimension of the input vector by 1, jump to Step 1.

[0161] Through the above example, the detailed operation process of the multi-level pipelined processing unit in the FPGA vector index query engine, including dimension matching, node type judgment, vector index determination, and the processing of the final result, is described. This process ensures the high speed, accuracy, and efficiency of the vector retrieval process through hardware acceleration, especially outstanding in dealing with large-scale high-dimensional vector data sets.

[0162] Through this embodiment, in the context of high-dimensional vector data sets. The utilization of the starting point indication data and the range indication data ensures that the query vector can be accurately located to the vector index most relevant to it, thereby improving the accuracy of the retrieval. And by restricting the search range, that is, determining the starting node and the number of nodes in the longest path, traversal of irrelevant nodes is avoided, significantly improving the retrieval efficiency, especially in the context of large-scale, high-dimensional vector data sets.

[0163] In an exemplary embodiment, a set of vector indices is determined according to the node type of the shape nodes matched in the composite shape graph, including: when the shape nodes matched in the composite shape graph include leaf nodes, the node identifier of the matched leaf nodes is determined as the vector index in the set of vector indices; when the shape nodes matched in the composite shape graph include empty nodes, the node identifier of the leaf nodes connected to the last shape node before the matched empty node is determined as the vector index in the set of vector indices.

[0164] Optionally, when the shape node matched in the composite shape graph is a leaf node, it means that the query vector has been accurately located to the vector list that best matches it. In this case, the node identifier of the leaf node is directly used as part of the vector index to obtain the vector list corresponding to the leaf node.

[0165] It should be noted that the node identifier of each leaf node can correspond to one or more vector indices, and these vector indices are used to indicate the vector lists stored on the FPGA or external memory.

[0166] Optionally, in the composite shape graph, when the matched node is an empty node, this means that the decision path of the query vector is interrupted at the current node and cannot be further located to a specific leaf node. Specifically, it can be traced back to the shape node before encountering the empty node, which is a non-empty but also non-leaf node and has the function of pointing to multiple leaf nodes. By obtaining the node identifiers of all the leaf nodes connected to this non-leaf node and also non-empty node, these identifiers form a list of vector indices.

[0167] Optionally, the index indication data may further include an array of leaf nodes corresponding to each set of initial vectors, and a list of vectors corresponding to each leaf node in the array of leaf nodes. Specifically, the array of leaf nodes corresponding to each set of initial vectors can be used to store a data set of all leaf node information corresponding to each set of initial vectors. A leaf node is a terminal node without child nodes and can be used to associate with specific vectors in the set of initial vectors. Among them, through the node identifier corresponding to the leaf node (a unique identifier used to uniquely identify the leaf node in the index structure), a reference to a series of vectors or vector IDs associated with the leaf node stored in the vector list can be pointed to. This means that when a leaf node is found through the path of the composite shape graph, all vector data mapped to the leaf node during the index construction process can be quickly accessed through the array of leaf nodes of the leaf node. The vector list is a data structure closely associated with the array of leaf nodes and can record all vector data belonging to a certain or a group of leaf nodes. Vector data can be directly stored in the vector list. Of course, vector IDs (i.e., pointers to the original vectors) can also be stored. Each vector or vector ID in the vector list has an index corresponding to its position in the list.

[0168] In one example, when a leaf node cannot be directly matched, that is, a null node is turned to on the path, the vector set corresponding to the longest match is returned. Assume that the subscript value calculated when reaching the longest match node (i.e., the shape node where the next step will turn to a null node) is i, and the total number of leaf nodes on the longest match node is r. Then, the leaf nodes with subscripts from i to i + r in the array of leaf nodes are within the matching range to form a set of vector indices.

[0169] Through this embodiment, when a leaf node cannot be directly matched, a reasonable retrieval result can also be obtained by backtracking to the nearest valid node, avoiding retrieval failures caused by data loss and improving the robustness of the system. Moreover, by directly locating the leaf node or collecting the indices of connected leaf nodes, the query request can be quickly responded to, avoiding a lengthy traversal process and maintaining the efficiency of the retrieval.

[0170] In an exemplary embodiment, generating a query result based on a retrieved set of vector indices includes: determining the initial vectors indicated by the vector indices in the set of vector indices as candidate vectors to obtain a list of candidate vectors; calculating the similarity between the candidate vectors in the list of candidate vectors and the query vector to obtain the similarities corresponding to the candidate vectors in the list of candidate vectors; sorting the candidate vectors in the list of candidate vectors in descending order according to the similarities corresponding to the candidate vectors in the list of candidate vectors, and screening out the first preset number of candidate vectors in the list of candidate vectors to obtain the query result.

[0171] Optionally, each vector index in a set of vector indexes is used as an indicator to locate in the stored initial vector set. Specifically, each vector index can locate a set of vector lists recorded by the leaf node, and the vector list can include some or all of the initial vectors in the initial vector set associated with the query vector.

[0172] Based on the initial vectors obtained from the vector indexes, a candidate vector list including a set of candidate vectors is obtained. For each candidate vector in the candidate vector list, the similarity between it and the query vector is calculated. Optionally, the similarity calculation process can be based on the measurement of vector distance, such as Euclidean distance, cosine similarity, etc., specifically depending on the application field and data characteristics.

[0173] Based on the similarity between the candidate vectors and the query vector, the candidate vector list is sorted. Generally, the vector with the highest similarity is ranked first in the list, while the vectors with lower similarity are located at the end of the list. Based on the sorted candidate vector list, the top pre-set number of candidate vectors are filtered out, and these vectors are considered to be the closest to the query vector. The pre-set number can be set according to the application scenario and requirements, such as "Top-K" similar images in image search. Based on the set of the most similar vectors, that is, the top pre-set number of candidate vectors, the query result is constituted.

[0174] In one example, to better understand the process of vector query processing where multiple data sets can reuse one index data in the present application, as Figure 7 shown, Figure 7 focusing on the processing flow of vector similarity query, which covers the CPU processing part and the FPGA processing part. It shows how to process the query vector through dimensionality reduction and quantization. Specifically, the query vector is processed for dimensionality reduction and quantization by obtaining the dimensionality reduction and quantization coding module corresponding to the query vector to obtain the quantization result of the query vector. Then, based on the index indication data corresponding to the vector set and the quantization result of the query vector, the composite shape graph is read through a multi-stage pipelined processing unit to obtain a matching vector list (candidate vector list), and the similarity in the matching vector list is calculated, so as to output the top pre-set number of vectors with the largest similarity.

[0175] Optionally, the selection of the pre-set number can be flexibly adjusted according to specific application requirements. For example, in a recommendation system, a smaller pre-set number may be set to provide personalized recommendation results; while in a search engine, a larger pre-set number may be set to cover more search results.

[0176] Through this embodiment, by calculating similarities and sorting, the quality of the retrieval results can be ensured, that is, the vectors most similar to the query vector are returned. This method is particularly applicable to scenarios with vector representations, such as image retrieval, document retrieval, audio retrieval, etc., and can effectively find the items that best match the query object, greatly facilitating users to find the required information in massive data.

[0177] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0178] The embodiment of the present application also provides a data processing device based on a vector set, as Figure 8 shown, the device includes:

[0179] A receiving module 802, configured to receive a query instruction, where the query instruction is used to query an initial vector in a set of initial vector sets. The vector index data of a set of initial vector sets includes composite index data and index indication data for each initial vector set in the set of initial vector sets. The composite index data includes a composite shape graph, and the composite shape graph is constructed with the vector indexes in the union of the vector indexes of the initial vectors in a set of initial vector sets as shape nodes. The index indication data for each initial vector set is used to indicate, in the composite shape graph, the vector indexes of the initial vectors in each initial vector set;

[0180] A determining module 804, configured to determine the index indication data corresponding to the query instruction, where the index indication data corresponding to the query instruction is the index indication data of the initial vector set associated with the query vector indicated by the query instruction;

[0181] A retrieval module 806, configured to perform vector index retrieval in the composite shape graph according to the query vector and the index indication data corresponding to the query instruction, and generate a query result based on the retrieved set of vector indexes, where the query result includes the initial vectors corresponding to the vector indexes in the set of vector indexes.

[0182] In an exemplary embodiment, the data processing device based on a vector set further includes: a first processing module, configured to perform dimensionality reduction processing on each initial vector set respectively to obtain each initial vector set after dimensionality reduction; a second processing module, configured to perform quantization encoding on each initial vector set after dimensionality reduction respectively before receiving the query instruction to obtain the quantization result of each initial vector set; a construction module, configured to construct the composite index data and the index indication data for each initial vector set according to the quantization result of each initial vector set.

[0183] In an exemplary embodiment, the second processing module includes: a first calculation unit configured to calculate statistical values of each initial vector set after dimensionality reduction in different vector dimensions of at least two vector dimensions, where the statistical value is one of an average value and a median value; and a first encoding unit configured to perform quantization encoding on the vector values of the initial vectors in each initial vector set after dimensionality reduction in different vector dimensions according to the statistical values of each initial vector set after dimensionality reduction in different vector dimensions, so as to obtain a quantization result of each initial vector set.

[0184] In an exemplary embodiment, the composite index data further includes a composite mapping dictionary, and the composite mapping dictionary is used to record the mapping relationship between the node identifier of the shape node in the composite shape graph and the information represented by the shape node.

[0185] The construction module includes: a first construction unit configured to construct an initial shape graph and an initial mapping dictionary based on the quantization result of the first initial vector set, where the first initial vector set is one initial vector set in a set of initial vector sets; and a second construction unit configured to sequentially use the other initial vector sets in the set of initial vector sets except the first initial vector set as the second initial vector set to perform the following construction operations, so as to obtain composite index data and index indication data of the initial vector sets: constructing a binary tree of the second initial vector set according to the quantization result of the second initial vector set; performing mapping processing on the binary tree of the second initial vector set according to the initial mapping dictionary to obtain a shape graph of the second initial vector set; updating the initial shape graph and the initial mapping dictionary according to the shape graph of the second initial vector set to obtain an updated initial shape graph and an updated initial mapping dictionary, and recording the index indication data of the second initial vector set; where the composite shape graph is the initial shape graph after the last update, and the composite mapping dictionary is the initial mapping dictionary after the last update.

[0186] In an exemplary embodiment, the first construction unit includes: a construction sub-module configured to construct a binary tree of the first initial vector set according to the quantization result of the first initial vector set; a mapping sub-module configured to map the binary tree of the first initial vector set into a shape node in the initial shape graph to obtain the initial shape graph; and a recording sub-module configured to record the mapping relationship between the path encoding of the binary tree of the first initial vector set and the shape node in the initial shape graph during the process of mapping the binary tree of the first initial vector set into the shape node in the initial shape graph, so as to obtain the initial mapping dictionary.

[0187] In an exemplary embodiment, the second construction unit includes: a splitting sub-module, configured to split the second initial vector set according to the quantization result of the second initial vector set to obtain a first sub-vector set and a second sub-vector set; a first construction sub-module, configured to construct a left subtree of the binary tree of the second initial vector set according to the first sub-vector set, and construct a right subtree of the binary tree of the second initial vector set according to the second sub-vector set, to obtain the binary tree of the second initial vector set.

[0188] In an exemplary embodiment, the second construction unit includes: a traversing sub-module, configured to traverse the binary tree of the second initial vector set according to the path encoding of the binary tree of the second initial vector set; a judging sub-module, configured to judge whether there is a specified shape node in the initial shape graph during the process of traversing the binary tree of the second initial vector set, where the specified shape node is a shape node in the initial shape graph that matches a non-leaf node of the binary tree of the second initial vector set; a first updating sub-module, configured to perform an updating process on the specified shape node when there is a specified shape node; a second updating sub-module, configured to create a new shape node and update the created shape node to the initial shape graph when there is no specified shape node, to obtain the shape graph of the second initial vector set.

[0189] In an exemplary embodiment, the retrieval module 806 includes: a retrieval processing unit, configured to perform dimensionality reduction processing and quantization encoding on a query vector to obtain a quantization result of the query vector; a node matching unit, configured to perform node matching processing in a composite shape graph according to the quantization result of the query vector and index indication data corresponding to a query instruction; an index determination unit, configured to determine a set of vector indexes according to the node type of the shape node matched in the composite shape graph.

[0190] In an exemplary embodiment, the retrieval processing unit includes: a retrieval acquisition sub-module, configured to acquire specified dimensionality reduction parameters corresponding to the query vector and specified quantization parameters corresponding to the query vector, where the specified dimensionality reduction parameters are parameters used for performing dimensionality reduction processing on an initial vector set associated with the query vector, and the specified quantization parameters are parameters used for performing quantization encoding on the initial vector set associated with the query vector; a dimensionality reduction processing sub-module, configured to perform dimensionality reduction processing on the query vector according to the specified dimensionality reduction parameters to obtain a dimensionality-reduced query vector; a quantization processing sub-module, configured to perform quantization encoding on the dimensionality-reduced query vector according to the specified quantization parameters to obtain the quantization result of the query vector.

[0191] In an exemplary embodiment, the index indication data includes start point indication data and range indication data. The start point indication data is used to indicate the starting node of the initial vector set in the composite shape graph; the range indication data is used to indicate the number of nodes in the longest path of the initial vector set in the composite shape graph.

[0192] The node matching unit includes: a path determination sub-module, configured to determine an indication path corresponding to a query vector according to a quantization result of the query vector, start indication data corresponding to a query instruction, and range indication data corresponding to the query instruction, where the indication path includes shape nodes sequentially passed through in a composite shape graph starting from a start node indicated by the start indication data corresponding to the query instruction according to the quantization result of the query vector;

[0193] a node matching sub-module, configured to sequentially use the shape nodes indicated by the indication path as the current shape node to perform the following node matching processing: when the value of the dimension corresponding to the current shape node is the same as the number of nodes in the longest path, determine that the current shape node is a leaf node; when there is no child node pointer for the current shape node, determine that the current shape node is an empty node; when the current shape node is neither an empty node nor a leaf node, jump to the next shape node of the current shape node.

[0194] In an exemplary embodiment, the index determination unit includes: a first determination sub-module, configured to, when the shape nodes matched in the composite shape graph include leaf nodes, determine the node identifier of the matched leaf node as a vector index in a set of vector indexes; a second determination sub-module, configured to, when the shape nodes matched in the composite shape graph include empty nodes, determine the node identifier of the leaf node connected to the last shape node before the matched empty node as a vector index in a set of vector indexes.

[0195] In an exemplary embodiment, the retrieval module 806 includes a candidate determination unit, configured to determine the initial vectors indicated by the vector indexes in a set of vector indexes as candidate vectors to obtain a candidate vector list; a similarity calculation unit, configured to calculate the similarity between the candidate vectors in the candidate vector list and the query vector to obtain the similarities corresponding to the candidate vectors in the candidate vector list; a screening unit, configured to sort the candidate vectors in the candidate vector list in descending order according to the similarities corresponding to the candidate vectors in the candidate vector list, and screen out the first preset number of candidate vectors in the candidate vector list to obtain a query result.

[0196] For the description of the features in the embodiments corresponding to the vector set-based data processing device, reference may be made to the relevant description in the embodiments corresponding to the vector set-based data processing method, which will not be elaborated here one by one.

[0197] An embodiment of the present application further provides an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above embodiments of the vector set-based data processing method.

[0198] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any one of the above embodiments of the data processing method based on a vector set when running.

[0199] In an exemplary embodiment, the above computer-readable storage medium may include but is not limited to: various media that can store computer programs such as USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0200] An embodiment of the present application further provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any one of the above embodiments of the data processing method based on a vector set.

[0201] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any one of the above embodiments of the data processing method based on a vector set.

[0202] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0203] The above has introduced in detail a data processing method, device, medium, and product based on a vector set provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data processing method based on a vector set, characterized in that: include: Receive a query instruction, wherein the query instruction is used to query initial vectors in a set of initial vector sets, the vector index data of the set of initial vector sets includes composite index data and index indication data of each initial vector set in the set of initial vector sets, the composite index data includes a composite shape graph, the composite shape graph is constructed using vector indices in a union of vector indices of the initial vectors in the set of initial vector sets as shape nodes, and the index indication data of each initial vector set is used to indicate the vector index of the initial vector in each initial vector set in the composite shape graph; Determining index indication data corresponding to the query instruction, wherein the index indication data corresponding to the query instruction is index indication data of an initial vector set associated with the query vector indicated by the query instruction; According to the query vector and the index indication data corresponding to the query instruction, a vector index retrieval is performed in the composite shape graph, and a query result is generated based on the retrieved set of vector indexes, wherein the query result includes an initial vector corresponding to a vector index in the set of vector indexes.

2. The method according to claim 1, characterized in that Before receiving the query instruction, the method further includes: Performing dimensionality reduction processing on each of the initial vector sets to obtain each of the initial vector sets after dimensionality reduction; Performing quantization encoding on each of the initial vector sets after dimensionality reduction to obtain a quantization result of each of the initial vector sets; The composite index data and the index indication data of each initial vector set are constructed according to the quantization result of each initial vector set.

3. The method according to claim 2, characterized in that The step of performing quantization encoding on each of the initial vector sets after dimensionality reduction to obtain a quantization result of each of the initial vector sets comprises: Calculating statistical values of each of the initial vector sets after dimensionality reduction in different vector dimensions of the at least two vector dimensions, wherein the statistical value is one of an average and a median; According to the statistical values of each initial vector set after dimensionality reduction at different vector dimensions, the vector numerical values of the initial vectors in each initial vector set after dimensionality reduction at different vector dimensions are quantized and encoded to obtain a quantization result of each initial vector set.

4. The method according to claim 2, characterized in that The composite index data further includes a composite mapping dictionary, wherein the composite mapping dictionary is used to record a mapping relationship between node identifiers of shape nodes in the composite shape graph and information represented by the shape nodes; The constructing the composite index data and the index indication data of each initial vector set according to the quantization result of each initial vector set includes: constructing an initial shape map and an initial mapping dictionary based on a quantization result of a first initial vector set, wherein the first initial vector set is an initial vector set in the group of initial vector sets; The following construction operations are performed on the other initial vector sets in the group of initial vector sets except the first initial vector set as the second initial vector set to obtain the composite index data and the index indication data of the initial vector set: constructing a binary tree of the second initial vector set according to the quantization result of the second initial vector set; performing mapping processing on the binary tree of the second initial vector set according to the initial mapping dictionary to obtain a shape graph of the second initial vector set; updating the initial shape map and the initial mapping dictionary according to the shape map of the second initial vector set to obtain an updated initial shape map and an updated initial mapping dictionary, and recording index indication data of the second initial vector set; The composite shape graph is the initial shape graph after the last update, and the composite mapping dictionary is the initial mapping dictionary after the last update.

5. The method according to claim 4, characterized in that The step of constructing an initial shape graph and an initial mapping dictionary based on the quantization result of the first initial vector set includes: constructing a binary tree of the first initial vector set according to a quantization result of the first initial vector set; Mapping the binary tree of the first initial vector set into shape nodes in the initial shape graph to obtain the initial shape graph; In the process of mapping the binary tree of the first initial vector set into shape nodes in the initial shape graph, the mapping relationship between the path encoding of the binary tree of the first initial vector set and the shape nodes in the initial shape graph is recorded to obtain the initial mapping dictionary.

6. The method according to claim 4, characterized in that The step of constructing a binary tree of the second initial vector set according to the quantization result of the second initial vector set includes: Segmenting the second initial vector set according to a quantization result of the second initial vector set to obtain a first sub-vector set and a second sub-vector set; A left subtree of the binary tree of the second initial vector set is constructed based on the first subvector set, and a right subtree of the binary tree of the second initial vector set is constructed based on the second subvector set to obtain the binary tree of the second initial vector set.

7. The method according to claim 4, characterized in that The step of mapping the binary tree of the second initial vector set according to the initial mapping dictionary to obtain a shape graph of the second initial vector set includes: traversing the binary tree of the second initial vector set according to the path code of the binary tree of the second initial vector set; In the process of traversing the binary tree of the second initial vector set, determining whether there is a specified shape node in the initial shape graph, wherein the specified shape node is a shape node in the initial shape graph that matches a non-leaf node in the binary tree of the second initial vector set; If the specified shape node exists, update the specified shape node; In the case that the designated shape node does not exist, a new shape node is created and the created shape node is updated to the initial shape graph to obtain the shape graph of the second initial vector set.

8. The method according to claim 1, characterized in that The performing vector index retrieval in the composite shape graph according to the query vector and the index indication data corresponding to the query instruction includes: Performing dimensionality reduction processing and quantization encoding on the query vector to obtain a quantization result of the query vector; performing node matching processing in the composite shape graph according to the quantization result of the query vector and the index indication data corresponding to the query instruction; The set of vector indexes is determined according to the node type of the matched shape node in the composite shape graph.

9. The method according to claim 8, characterized in that The performing dimensionality reduction processing and quantization encoding on the query vector to obtain a quantization result of the query vector includes: Obtaining a specified dimensionality reduction parameter corresponding to the query vector and a specified quantization parameter corresponding to the query vector, wherein the specified dimensionality reduction parameter is a parameter used for performing dimensionality reduction processing on an initial vector set associated with the query vector, and the specified quantization parameter is a parameter used for quantization encoding of the initial vector set associated with the query vector; Performing dimensionality reduction processing on the query vector according to the specified dimensionality reduction parameter to obtain the query vector after dimensionality reduction; The query vector after dimension reduction is quantized and encoded according to the specified quantization parameter to obtain a quantization result of the query vector.

10. The method according to claim 8, characterized in that The index indication data includes start point indication data and range indication data, wherein the start point indication data is used to indicate a starting node of the initial vector set in the composite shape graph; The range indication data is used to indicate the number of nodes of the longest path of the initial vector set in the composite shape graph; The performing node matching processing in the composite shape graph according to the quantization result of the query vector and the index indication data corresponding to the query instruction includes: determining an indication path corresponding to the query vector based on a quantization result of the query vector, start indication data corresponding to the query instruction, and range indication data corresponding to the query instruction, wherein the indication path includes shape nodes sequentially passed through in the composite shape graph starting from the start node indicated by the start indication data corresponding to the query instruction according to the quantization result of the query vector; The shape nodes indicated by the indicated path are sequentially used as current shape nodes to perform the following node matching process: When the value of the dimension corresponding to the current shape node is the same as the number of nodes in the longest path, determining that the current shape node is a leaf node; In the case that the current shape node does not have a child node pointer, determining that the current shape node is an empty node; When the current shape node is neither the empty node nor the leaf node, jump to the next shape node of the current shape node.

11. The method according to claim 8, characterized in that The determining the set of vector indexes according to the node types of the shape nodes matched in the composite shape graph includes: In a case where the shape node matched in the composite shape graph includes a leaf node, determining the node identifier of the matched leaf node as a vector index in the set of vector indices; When the shape nodes matched in the composite shape graph include a null node, the node identifier of the leaf node connected to the last shape node before the null node matched is determined as the vector index in the set of vector indices.

12. The method according to any one of claims 1 to 11, characterized in that Generating query results based on the retrieved set of vector indexes includes: Determine an initial vector indicated by a vector index in the set of vector indices as a candidate vector, and obtain a candidate vector list; Calculating similarities between candidate vectors in the candidate vector list and the query vector to obtain similarities corresponding to the candidate vectors in the candidate vector list; The candidate vectors in the candidate vector list are sorted in descending order according to the similarities corresponding to the candidate vectors in the candidate vector list, and a first preset number of candidate vectors in the candidate vector list are screened out to obtain the query result.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the vector set-based data processing method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the vector set-based data processing method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the vector set-based data processing method according to any one of claims 1 to 12 are implemented.

Images