Vector similarity determination method and vector search method

Abstract

A vector similarity determination method and a vector search method. The vector similarity determination method comprises: acquiring a first dense-sparse vector and a second dense-sparse vector (S101); calculating a first similarity and a second similarity on the basis of the first dense-sparse vector and the second dense-sparse vector (S102); and determining the similarity between the first dense-sparse vector and the second dense-sparse vector on the basis of the first similarity and the second similarity (S103).

Description

Vector similarity determination method and vector search method

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 28, 2023, with application number 202311607446.6, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data processing technology, for example, to a vector similarity determination method and a vector search method. Background Art

[0003] To facilitate the processing of data such as text and images, related technologies typically vectorize the data and represent it using vectors. Vectors are typically one-dimensional arrays used to store data of the same type, such as feature vectors in machine learning. Each value in the vector represents a numerical feature of the corresponding dimension. Vectors come in two types: dense vectors and sparse vectors. Dense vectors refer to vectors in which all or most of the values ​​in a one-dimensional array representing a vector are non-zero. Most machine learning models generate dense feature vectors, such as OpenAI's text-embedding-ada-002 model and the commonly used open source word2vec model. Dense vectors generally have dimensions below 2000. Sparse vectors refer to vectors in which the one-dimensional array representing a vector is very high-dimensional, but most of the values ​​are zero. The dimensions of sparse vectors are generally hundreds of thousands or even millions.

[0004] When searching with sparse and dense vectors, there are two classic approaches in the industry: 1. Use sparse vectors for coarse ranking. For example, to search for the top 10, calculate similarity using sparse vectors and find 100 data points as a candidate set. Then, use dense vectors to calculate similarity for this candidate set and perform fine ranking, with the 10 most similar data points as the final result set. 2. Use sparse and dense vectors to calculate similarity separately, find the top 10 result sets, and then calculate the reciprocal sorting and fusion RRF scores of the two top 10 result sets. The 10 data points with the highest RRF scores are selected as the final result set.

[0005] Related technologies calculate similarity by treating sparse and dense vectors as two separate pieces of data. This fails to consider the distribution of both data in the vector space, resulting in inaccurate similarity calculations, which in turn affects search results and leads to low search recall. Improving the accuracy of vector similarity calculations remains an unresolved issue.

[0006] Summary of the Invention

[0007] The present application provides a vector similarity determination method and a vector search method to solve the problem of low accuracy of similarity calculation results between dense vectors and sparse vectors.

[0008] According to one aspect of the present application, a method for determining vector similarity is provided, comprising:

[0009] Obtain a first dense sparse vector and a second dense sparse vector, where the first dense sparse vector is obtained by concatenating the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by concatenating the second dense vector and the second sparse vector;

[0010] Calculating a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector;

[0011] The similarity between the first dense sparse vector and the second dense sparse vector is determined according to the first similarity and the second similarity.

[0012] According to another aspect of the present application, a vector search method is provided, comprising:

[0013] Get the dense and sparse vector to be searched;

[0014] Searching a search graph for a hierarchical navigable small world based on a graph algorithm, reading a dense sparse vector to be matched in the search graph for the hierarchical navigable small world, and calculating a similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, where the similarity is calculated according to the vector similarity determination method described in any embodiment of the present application;

[0015] A target dense sparse vector that matches the dense sparse vector to be searched is determined according to the similarity.

[0016] According to another aspect of the present application, a vector similarity determination device is provided, comprising:

[0017] a vector acquisition module configured to acquire a first dense sparse vector and a second dense sparse vector, wherein the first dense sparse vector is obtained by concatenating the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by concatenating the second dense vector and the second sparse vector;

[0018] a similarity calculation module, configured to calculate a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector;

[0019] The similarity determination module is configured to determine the similarity between the first dense sparse vector and the second dense sparse vector according to the first similarity and the second similarity.

[0020] According to another aspect of the present application, a vector search device is provided, comprising:

[0021] A module for obtaining a vector to be searched is configured to obtain a dense or sparse vector to be searched;

[0022] a vector search module configured to search a search graph of a hierarchical navigable small world based on a graph algorithm, read a dense sparse vector to be matched in the search graph of the hierarchical navigable small world, and calculate a similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, where the similarity is calculated according to the vector similarity determination method described in any embodiment of the present application;

[0023] The target vector determination module is configured to determine a target dense sparse vector that matches the dense sparse vector to be searched according to the similarity.

[0024] According to another aspect of the present application, an electronic device is provided, comprising:

[0025] at least one processor; and

[0026] a memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vector similarity determination method described in any embodiment of the present application or the vector search method described in any embodiment of the present application.

[0028] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the vector similarity determination method described in any embodiment of the present application or the vector search method described in any embodiment of the present application when executed.

[0029] The technical solution of an embodiment of the present invention obtains a first dense sparse vector and a second dense sparse vector, wherein the first dense sparse vector is obtained by splicing the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by splicing the second dense vector and the second sparse vector; calculates a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector; determines the similarity between the first dense sparse vector and the second dense sparse vector based on the first similarity and the second similarity, thereby solving the problem of low accuracy of similarity calculation results between dense vectors and sparse vectors, and obtains a new vector, namely, a dense sparse vector, by splicing the dense vector and the sparse vector. The final similarity of the first dense sparse vector and the second dense sparse vector is determined by calculating the first similarity and the second similarity of the first dense sparse vector and the second dense sparse vector. Compared with the prior art that calculates the similarity from the perspectives of dense vectors and sparse vectors respectively, the present application considers both dense vectors and sparse vectors at the same time to calculate the similarity of the vectors, and the result is more accurate; the implementation is simple and effectively ensures data consistency; the data is represented by dense sparse vectors to better reflect the actual data space distribution, thereby ensuring the accuracy of the similarity result and improving the recall rate of the vector search. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The following will introduce the drawings required for the description of the embodiments. The drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] FIG1 is a flowchart of a method for determining vector similarity according to Embodiment 1 of the present application;

[0032] FIG2 is a diagram showing an example of a dense sparse vector representation provided according to the first embodiment of the present application;

[0033] FIG3 is a flow chart of a method for determining vector similarity according to the second embodiment of the present application;

[0034] FIG4 is a flow chart of a vector search method provided according to Embodiment 3 of the present application;

[0035] FIG5 is a flowchart of a vector search method provided according to a fourth embodiment of the present application;

[0036] FIG6 is an example diagram of dense vector storage in the related art;

[0037] FIG7 is an example diagram of storing a vector in a pre-allocated memory space according to a fourth embodiment of the present application;

[0038] FIG8 is an example diagram of storing vectors in another pre-allocated memory space according to the fourth embodiment of the present application;

[0039] FIG9 is an example diagram of storing a dense sparse vector to be stored according to the fourth embodiment of the present application;

[0040] FIG10 is an example diagram of storing a dense sparse vector to be stored according to the fourth embodiment of the present application;

[0041] FIG11 is a schematic structural diagram of a vector similarity determination device provided according to Embodiment 5 of the present application;

[0042] FIG12 is a schematic structural diagram of a vector search device according to a sixth embodiment of the present application;

[0043] FIG13 is a schematic structural diagram of an electronic device provided according to Embodiment 7 of the present application. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units listed, but may include other steps or units that are not listed or inherent to these processes, methods, products or devices.

[0046] Example 1

[0047] Figure 1 is a flow chart of a vector similarity determination method provided in Example 1 of the present application. This embodiment is applicable to the case of calculating the similarity of two vectors. The method can be performed by a vector similarity determination device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. As shown in Figure 1, the method includes:

[0048] S101. Obtain a first dense sparse vector and a second dense sparse vector, where the first dense sparse vector is obtained by concatenating the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by concatenating the second dense vector and the second sparse vector.

[0049] In this embodiment, the first dense sparse vector can be understood as a vector obtained by splicing a dense vector and a sparse vector; the first dense vector can be understood as a dense vector, and the first sparse vector can be understood as a sparse vector; the first dense sparse vector is obtained by splicing the first dense vector and the first sparse vector. When splicing the first dense vector and the first sparse vector, the first sparse vector can be spliced ​​after the first dense vector, or the first dense vector can be spliced ​​after the first sparse vector. Since the dimension of the first dense vector is fixed, the embodiment of the present application takes the splicing of the first sparse vector after the first dense vector as an example.

[0050] In scenarios such as machine learning, data is vectorized to obtain its corresponding dense vectors and sparse vectors. The dense vectors and sparse vectors corresponding to the data are concatenated to obtain the dense sparse vectors corresponding to the data. The embodiment of the present application newly defines a vector form to represent the data simultaneously through dense vectors and sparse vectors.

[0051] When it is necessary to calculate the similarity of vectors, a first dense sparse vector and a second dense sparse vector are obtained. The first dense sparse vector and the second dense sparse vector can be pre-stored in a storage space and read from the corresponding storage space when calculating the similarity, or the first dense sparse vector and the second dense sparse vector can be received by user input or sent by an external device.

[0052] Exemplarily, an embodiment of the present application provides a method for representing a dense sparse vector in memory. Taking a 32-bit floating-point vector as an example, in memory, the dense vector is stored in continuous memory with a length of dimension d times 32 bits; the sparse vector is stored in a compressed manner, including a 32-bit fixed-length header (head) to store the number of non-zero values ​​of the sparse vector. Each non-zero value in the sparse vector is stored using 64 bits, the first 32 bits store the subscript, and the second 32 bits store the vector value. The non-zero values ​​of the sparse vector are arranged in ascending order according to the subscript; after the sparse vector is directly spliced ​​into the dense vector, it is the memory representation of the dense sparse vector. Figure 2 provides an example diagram of the representation of a dense sparse vector. As shown in the figure, the dense sparse vector is obtained by splicing the dense vector with the sparse vector.

[0053] S102: Calculate a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector.

[0054] In this embodiment, the first similarity can be understood as a similarity calculated based on the first dense sparse vector and the second dense sparse vector; the second similarity can be understood as another similarity calculated based on the first dense sparse vector and the second dense sparse vector, wherein the first similarity and the second similarity are calculated in different ways or using different data.

[0055] Determine the first dense vector and the first sparse vector included in the first dense sparse vector, determine the second dense vector and the second sparse vector included in the second dense sparse vector, process the first dense vector and the first sparse vector, process the second dense vector and the second sparse vector, align the first dense sparse vector and the second dense sparse vector, calculate the similarity of the aligned dense sparse vectors using different calculation methods to obtain a first similarity and a second similarity; or directly calculate the similarity of the first dense sparse vector and the second dense sparse vector using different methods to obtain the first similarity and the second similarity, etc.

[0056] S103 : Determine the similarity between the first dense sparse vector and the second dense sparse vector according to the first similarity and the second similarity.

[0057] The first similarity and the second similarity can represent the similarity between the first dense sparse vector and the second dense sparse vector from different dimensions. After obtaining the first similarity and the second similarity, the first similarity and the second similarity are comprehensively used to determine the final similarity. For example, the maximum value, the minimum value, the average value, etc. between the first similarity and the second similarity are determined, and the final determined similarity is used as the similarity between the first dense sparse vector and the second dense sparse vector.

[0058] It should be noted that the method for determining vector similarity provided in this embodiment can be applied to different vector search algorithms. During the vector search process, the similarity is calculated using the method for determining vector similarity provided in the embodiment of this application to achieve vector search.

[0059] The embodiment of the present application provides a method for determining vector similarity, which solves the problem of low accuracy of similarity calculation results between dense vectors and sparse vectors. By splicing dense vectors and sparse vectors, a new vector, namely a dense sparse vector, is obtained. The final similarity between the first dense sparse vector and the second dense sparse vector is determined by calculating the first similarity and the second similarity of the first dense sparse vector and the second dense sparse vector. Compared with the related art that calculates the similarity from the perspectives of dense vectors and sparse vectors respectively, the present application considers both dense vectors and sparse vectors at the same time to calculate the similarity of vectors, and the result is more accurate; it is simple to implement and effectively ensures data consistency; by representing data through dense sparse vectors, the real data space distribution is better reflected, thereby ensuring the accuracy of the similarity result, and thus improving the recall rate of vector search.

[0060] Example 2

[0061] FIG3 is a flow chart of a method for determining vector similarity provided in Example 2 of the present application. This embodiment is a refinement of the above embodiment. As shown in FIG3 , the method includes:

[0062] S201. Obtain a first dense sparse vector and a second dense sparse vector, where the first dense sparse vector is obtained by concatenating the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by concatenating the second dense vector and the second sparse vector.

[0063] S202: Calculate the similarity between the first dense vector and the second dense vector to obtain a first similarity.

[0064] The similarity between the first dense vector and the second dense vector can be calculated directly according to a predetermined similarity calculation method. The dimensions of the first dense vector and the second dense vector are the same, and the vector values ​​corresponding to each dimension can also be directly determined. The first similarity is directly calculated. The calculation method of the first similarity can be cosine similarity, Euclidean distance, etc.

[0065] S203 : Calculate the similarity between the first sparse vector and the second sparse vector to obtain a second similarity.

[0066] The similarity between the first sparse vector and the second sparse vector can also be calculated directly according to a predetermined similarity calculation method. Since the values ​​of a large number of dimensions in the first sparse vector and the second sparse vector are 0, when calculating the vector values, the second similarity can be calculated according to the subscripts (i.e., dimensions) recorded in the sparse vectors to obtain the second similarity. The second similarity can be calculated using a cosine similarity, Euclidean distance, or the like. The calculation methods for the first and second similarities can be the same or different.

[0067] As an optional embodiment of this embodiment, this optional embodiment calculates the similarity between the first sparse vector and the second sparse vector, and obtains the second similarity optimized as:

[0068] A1. Determine a first dimension identifier in a first sparse vector and a first vector value corresponding to the first dimension identifier.

[0069] In this embodiment, the first dimension identifier can be understood as information for identifying different dimensions in the first sparse vector; the first vector value can be understood as a vector value that is not 0 in the first sparse vector.

[0070] The first sparse vector includes a large number of zero-valued values, so only non-zero values ​​are stored. Referring to Figure 2 , the subscript in Figure 2 is the first dimension identifier, which identifies the dimension containing non-zero values, for example, 10. The value stored after the first dimension identifier is the first vector value, i.e., the value after the subscript, for example, 0.1. All first dimension identifiers in the first sparse vector are determined, and the first vector value corresponding to each first dimension identifier is determined.

[0071] A2. Determine a second dimension identifier in the second sparse vector and a second vector value corresponding to the second dimension identifier.

[0072] In this embodiment, the second dimension identifier can be understood as information for identifying different dimensions in the second sparse vector; the second vector value can be understood as a vector value that is not 0 in the second sparse vector.

[0073] The second sparse vector also includes a large number of values ​​that are 0. When storing, only non-zero values ​​are stored. Referring to Figure 2 , the subscripts in Figure 2 are second dimension identifiers, and the values ​​after the subscripts are the second vector values. All second dimension identifiers in the second sparse vector are determined, and the second vector value corresponding to each second dimension identifier is determined.

[0074] A3. Compare each first dimension identifier and each second dimension identifier, and determine the same first dimension identifier and second dimension identifier as the target dimension identifier.

[0075] In this embodiment, the target dimension identifier can be understood as identification information of a dimension whose value is not 0 in both the first sparse vector and the second sparse vector.

[0076] Compare all first dimension identifiers and second dimension identifiers to determine whether there are identical first dimension identifiers and second dimension identifiers. If there are identical first dimension identifiers and second dimension identifiers, determine this identifier as the target dimension identifier. There can be multiple target dimension identifiers.

[0077] A4. Calculate similarity based on the first vector value and the second vector value corresponding to the target dimension identifier to determine a second similarity.

[0078] Determine the first vector value corresponding to the target dimension identifier in the first sparse vector, and determine the second vector value corresponding to the target dimension identifier in the second sparse vector; calculate the similarity based on the first vector values ​​and the second vector corresponding to all target dimension identifiers whose values ​​are not 0. The similarity calculation method can be pre-set, and the second similarity is obtained by performing the similarity calculation according to the calculation formula.

[0079] Optionally, the first similarity and the second similarity are inner product similarities.

[0080] The first similarity and the second similarity are calculated using the inner product similarity calculation method. The first similarity is calculated as follows: multiply the values ​​of the same dimension of the first dense vector and the second dense vector, and add the products of each dimension; the first similarity is calculated as follows: multiply the first vector value and the second vector value corresponding to the target dimension identifier, and add the products corresponding to all target dimension identifiers.

[0081] For example, the present embodiment provides a calculation formula for inner product similarity:

[0082] Among them, p(A,B) is the inner product similarity between vector A and vector B, a i is the value of the i-th dimension in vector A; b i is the value of the i-th dimension in vector B, and n is the dimension of vector A and vector B.

[0083] As can be seen from the above formula, when calculating the inner product similarity, we directly calculate the product of the values ​​in the same dimension and sum them to obtain the inner product similarity. Since any value multiplied by 0 is 0, when calculating the second similarity, we can count the target dimension identifiers whose values ​​are not 0, calculate the product of the first vector value and the second vector value corresponding to the target dimension identifier, and then sum them to obtain the second similarity.

[0084] S204 : Perform weighted summation on the first similarity and the second similarity to obtain the similarity between the first dense sparse vector and the second dense sparse vector.

[0085] The first similarity and the second similarity are weighted and summed. The weights can be set according to different scenarios, the data types corresponding to the vectors, etc., or can be set by the user. For example, the user specifies the weights of dense vectors and sparse vectors in the final score, such as setting the weight of dense vectors to 0.7 and the weight of sparse vectors to 0.3. According to the definition of dense-sparse vector inner product similarity, the dense part and the sparse part of the vector are multiplied by the corresponding weights respectively, that is, the first similarity and the second similarity are multiplied by the corresponding weights respectively. The similarity obtained after the weighted sum is the similarity of the first dense sparse vector and the second dense sparse vector.

[0086] The embodiment of the present application provides a method for determining vector similarity, which solves the problem of low accuracy of similarity calculation results between dense vectors and sparse vectors. By splicing dense vectors and sparse vectors, a new vector, namely a dense sparse vector, is obtained. The first similarity between the first dense vector and the second dense vector, as well as the second similarity between the first sparse vector and the second sparse vector are calculated, and the first similarity and the second similarity are weightedly summed to obtain the final similarity between the first dense sparse vector and the second dense sparse vector, thereby improving the accuracy of similarity calculation. Compared with the related art that calculates similarity from the perspectives of dense vectors and sparse vectors respectively, the present application considers both dense vectors and sparse vectors at the same time, calculates the similarity of vectors, and the result is more accurate, thereby improving the recall rate of vector search. The implementation is simple and effectively ensures data consistency. The data is represented by dense sparse vectors to better reflect the real data space distribution, thereby avoiding the inaccurate similarity calculation results caused by calculating the similarity of dense and sparse vectors separately in the traditional solution, thereby causing performance and recall rate losses.

[0087] Example 3

[0088] FIG4 is a flow chart of a vector search method provided in Example 3 of the present application. This embodiment is applicable to the case of searching for vectors. The method can be performed by a vector search device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. As shown in FIG4 , the method includes:

[0089] S301: Obtain a dense or sparse vector to be searched.

[0090] In this embodiment, the dense sparse vector to be searched can be understood as the dense sparse vector with the search requirement. The dense sparse vector to be searched can be input by the user, obtained by vectorizing data, or determined by other devices or business modules and transmitted to the execution device before the search. After obtaining the dense sparse vector to be searched, similarity is calculated based on the vector similarity calculation method provided in any embodiment of this application to determine the dense sparse vector that matches the dense sparse vector to be searched.

[0091] S302: Search the search graph of the hierarchical navigable small world based on a graph algorithm, read the dense sparse vector to be matched in the search graph of the hierarchical navigable small world, and calculate the similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, where the similarity is calculated according to the vector similarity determination method of any embodiment of the present application.

[0092] In this embodiment, the hierarchical navigable small world (HNSW) stores different vectors and the relationships between them through a search graph. The dense sparse vector to be matched can be understood as a dense sparse vector that needs to calculate similarity with the dense sparse vector to determine whether it matches.

[0093] A search graph for a hierarchical, navigable small world is pre-built. When constructing the search graph, the similarities between different dense and sparse vectors need to be calculated. Each dense and sparse vector is then added to the layers to form the search graph. The similarities between different dense and sparse vectors during the search graph construction process are calculated using the vector similarity determination method provided in any of the embodiments of this application.

[0094] The search graph for hierarchical navigable small worlds is searched using the HNSW graph algorithm. First, the identifiers of all dense sparse vectors in the top layer of the search graph for hierarchical navigable small worlds are determined. The identifiers of the dense sparse vectors are used as the identifiers of the dense sparse vectors to be matched. Based on the identifiers, the dense sparse vectors to be matched are read from the bottom layer storing the vector values. The similarity between the read dense sparse vectors to be searched and the dense sparse vectors to be matched is calculated. One or more dense sparse vectors to be matched with a high similarity are selected and mapped to the next layer. For each selected dense sparse vector to be matched, the nearest neighbors of the dense sparse vector to be matched in the next layer are determined. The vectors corresponding to the nearest neighbors are used as the dense sparse vectors to be matched. The similarity between each dense sparse vector to be matched and the dense sparse vector to be searched is continuously calculated. The dense sparse vectors to be matched with a high similarity are continuously selected and mapped to the next layer until the bottom layer is mapped, thereby obtaining the similarity saved during the search process. During the search process, the number of dense sparse vectors participating in the similarity calculation in each layer may be uncertain, which is related to the number of neighbors of each dense sparse vector. After completing the similarity calculation, a preset number of dense sparse vectors to be matched can be saved, that is, a preset number of dense sparse vectors to be matched with higher similarity are selected for saving.

[0095] Related technology When performing a joint search of sparse vectors and dense vectors, sparse vectors are first used for coarse sorting and then dense vectors are used for fine sorting. If the accuracy of the similarity result of the sparse vector is low, the quality of the candidate set will be poor, the recall rate will be seriously affected, and two sets of retrieval systems need to be maintained to calculate the similarity. The engineering complexity is high, and data consistency is not easy to ensure. When using sparse vectors and dense vectors to calculate the similarity and then calculate the RRF score, since the sparse and dense vectors are two independent data, the distribution of the two data in the vector space cannot be considered at the same time when creating the index, which affects the recall rate. The fusion algorithm that selects the final result from the two independent result sets depends on the scenario and cannot guarantee the recall rate in general scenarios. In the process of vector search, the embodiment of the present application redefines a vector, namely a dense sparse vector, and defines a similarity calculation method for dense sparse vectors. The similarity is calculated according to this similarity calculation method and then the vector search is performed, which can effectively improve the recall rate and is not dependent on a specific scenario.

[0096] S303 : Determine a target dense sparse vector that matches the dense sparse vector to be searched according to the similarity.

[0097] In this embodiment, the target dense sparse vector can be understood as a dense sparse vector that matches the dense sparse vector to be searched. All similarities saved during the calculation process are compared, and the value with the higher similarity is determined. The dense sparse vector corresponding to the higher similarity value is determined as the target dense sparse vector that matches the dense sparse vector to be searched. The number of target dense sparse vectors can be one or more. The target dense sparse vector can be fed back to the user as a search result to obtain other data processing layers for data processing.

[0098] It should be noted that in the vector search method provided in the embodiments of the present application, the similarity calculation between the two vectors involved in the vector search process is calculated using the vector similarity determination method provided in the first or second embodiment of the present application.

[0099] The embodiment of the present application provides a vector search method, which solves the problem of low recall rate caused by low accuracy of similarity calculation results when dense vectors and sparse vectors are jointly searched. By splicing dense vectors and sparse vectors, a new vector, namely a dense sparse vector, is obtained. In the vector search process, the final similarity is determined by calculating the first similarity and the second similarity of the first dense sparse vector and the second dense sparse vector, thereby improving the accuracy of the similarity calculation and thus improving the accuracy of the search; compared with the related art in which similarity is calculated from two perspectives, dense vector and sparse vector respectively, the present application considers both dense vector and sparse vector at the same time, calculates the similarity of vectors, and the result is more accurate, thereby improving the recall rate during vector search; the implementation is simple, and only a set of retrieval systems needs to be maintained, effectively ensuring data consistency; the data is represented by dense sparse vectors to better reflect the real data space distribution, thereby avoiding the inaccurate similarity calculation results caused by calculating the similarity of dense and sparse vectors separately in the traditional solution, thereby avoiding the performance and recall rate loss caused by the inaccurate similarity calculation results; it does not rely on specific scenarios and guarantees the recall rate in general scenarios.

[0100] Example 4

[0101] FIG5 is a flow chart of a vector search method provided in Example 4 of the present application. This embodiment is a refinement of the above embodiment. As shown in FIG5 , the method includes:

[0102] S401: Obtain a dense sparse vector to be stored.

[0103] In this embodiment, the dense sparse vectors to be stored can be understood as dense sparse vectors with storage requirements, and the dense sparse vectors to be stored can form a search graph of a hierarchical navigable small world.

[0104] The dense sparse vector to be stored can be input by a user, or can be pre-saved in a file or transmitted to the execution device by another device or terminal. The dense sparse vector to be stored can be obtained. The number of dense sparse vectors to be stored can be one or more. That is, when obtaining the dense sparse vector to be stored, one dense vector to be stored can be obtained, or multiple dense sparse vectors to be stored can be obtained in batches, so as to construct a hierarchical navigable small-world search graph based on the dense sparse vectors to be stored.

[0105] Taking the graph algorithm hnswlib as an example, hnswlib is an hnsw implementation that currently only supports dense vectors. When constructing an hnswlib graph object, a maximum number of elements ( max_elements ) is specified. This maximum number represents the maximum number of vectors required to form a hierarchical, navigable, small-world search graph. The primary data structures used to represent the graph include linklists and data_level0_memory_ . Linklists stores the M neighbor IDs for a point in layers 1 through n; data_level0_memory_ stores the 2M neighbor IDs for a point in layer 0, as well as the vector data itself.

[0106] Among them, a point in the memory of layer0 is represented as:

[0107] listsize, integer, represents the number of valid neighbors, the maximum is 2M;

[0108] 2M integers, storing the neighbor ID set;

[0109] d floating-point types, storing vector data;

[0110] A long integer, storing the external label of the point;

[0111] A point in layer0 is represented as a fixed-length memory. To ensure performance, space is pre-allocated when the hnswlib object is initialized. The memory size allocated by data_level0_memory_ is the maximum number of points (max_elements) multiplied by the fixed memory size of each point. Figure 6 provides an example diagram of dense vector storage in related technology, where each point is a vector.

[0112] This embodiment of the application extends hnswlib to support dense sparse vectors, requiring two key steps: 1. Using a dense sparse vector distance calculation method to replace the existing dense vector distance calculation method; 2. Unlike dense vectors, dense sparse vectors are variable-length data, requiring support for variable-length vector data storage while ensuring the performance of the two addressing operations mentioned above. Therefore, this embodiment of the application provides a new dense sparse vector storage method and similarity calculation method when constructing a hierarchical navigable small-world search graph.

[0113] S402: Storing the dense and sparse vectors to be stored in pre-allocated memory space.

[0114] Pre-allocate memory space for storing all dense sparse vectors to be stored. The memory space can be divided into different small spaces to store the corresponding dense sparse vectors to be stored. After obtaining the dense sparse vectors to be stored, the dense sparse vectors to be stored are stored in the pre-allocated memory space. The memory space can be pre-allocated based on the number of dense sparse vectors to be stored, as well as a maximum length or a preset length, where the preset length is determined based on the typical range of lengths of different dense sparse vectors to be stored.

[0115] Optionally, the pre-allocated memory space includes a first memory space and a second memory space;

[0116] As an optional embodiment of this embodiment, this optional embodiment stores the dense sparse vector to be stored through pre-allocated memory space, and is optimized as follows: determining the length of the dense sparse vector to be stored and storing it in the corresponding first memory space, judging whether the length of the dense sparse vector to be stored is greater than the length of the second memory space, and if so, storing the dense sparse vector to be stored in the corresponding virtual address space; otherwise, storing the dense sparse vector to be stored in the corresponding second memory space.

[0117] In some embodiments, the length of the second memory space is determined based on the dense vector dimension in the dense sparse vector to be stored and the maximum number of non-zero values ​​of the constrained sparse vector; the size of the virtual address space is determined by rounding the maximum value of the length of the dense sparse vector to be stored.

[0118] In this embodiment, the first memory space and the second memory space respectively store two different data, the first memory space is used to store the length of the dense sparse vector, and the second memory space is used to store the value of the dense sparse vector. The pre-allocated memory space is divided into a first memory space and a second memory space, one first memory space corresponds to one second memory space, and together store a dense sparse vector; the memory space includes multiple groups of first memory spaces and second memory spaces, and the number of groups of the first memory space and the second memory space is the maximum number of points max_elements. Since the second memory space stores the value of the dense sparse vector to be stored, which includes dense vectors and sparse vectors, the size of the second memory space is determined according to the dense vector dimension in the dense sparse vector to be stored and the maximum number of non-zero values ​​of the constrained sparse vector. The dense vector dimension in the dense sparse vector to be stored is usually predetermined, and the maximum number of non-zero values ​​of the constrained sparse vector can also be set in advance according to the business scenario.

[0119] Exemplarily, an embodiment of the present application provides a method for determining the size of a second memory space. Taking the maximum number of non-zero values ​​of a constrained sparse vector equal to N as an example, the maximum length of a floating-point sparse vector is sizeof(uint32_t)+N*sizeof(uint32_t)+N*sizeof(float), and the maximum length of the entire floating-point dense sparse vector is max_vector_size=d*sizeof(float)+sizeof(uint32_t)+N*sizeof(uint32_t)+N*sizeof(float), that is, the maximum length of a dense sparse vector is max_vector_size=4+8N+4d, where N can be determined based on the maximum number of non-zero values ​​of most sparse vectors in each dense sparse vector to be stored, or based on the maximum number of non-zero values ​​of all sparse vectors in each dense sparse vector to be stored. The value of N can be obtained by implementing an analysis data set, or it can be estimated based on the business of generating sparse vectors.

[0120] In most application scenarios, the lengths of generated sparse vectors do not vary significantly, so preallocating space based on the maximum size during graph index construction is feasible. However, in some scenarios, the lengths of sparse vectors may vary significantly. For example, most sparse vectors may have only 20 non-zero values, but a very small number may have 10,000 non-zero values. If N is calculated based on the maximum non-zero value among all sparse vectors, then N needs to be 10,000. However, since most sparse vectors do not have 10,000 values ​​to store, preallocating space based on the maximum size in this case will cause excessive memory pressure. Therefore, N can be set to 20, which is the maximum number of non-zero values ​​in most sparse vectors.

[0121] For example, Figure 7 provides an example diagram of storing vectors in pre-allocated memory space. As shown in the figure, the memory space variable_l0_memory is divided into two parts: the first memory space head and the second memory space body. When the variable_l0_memory object is initialized, a total memory size of max_elements times the maximum number of points multiplied by a long integer is allocated for the head, and a total memory size of max_elements times max_vector_size is allocated for the body. The head stores the actual length of the dense sparse vector. Addressing vector data only requires calculating the relative offset of the body using the identifier of the dense sparse vector. When writing data, it is only necessary to copy the data to the corresponding offset in the body and record the actual length in the head. The entire read and write process does not require any lock protection. Figure 7 exemplifies the storage of three dense sparse vectors. Figure 8 provides another example diagram of storing vectors in pre-allocated memory space. Compared with Figure 7, a new dense sparse vector is stored, and a total of four dense sparse vectors are stored.

[0122] After obtaining the dense sparse vector to be stored, the length of the dense sparse vector to be stored can be determined accordingly, and the length can be stored in the corresponding first memory space. Determine whether the length of the dense sparse vector to be stored is greater than the length of the second memory space. If so, it can be determined that the first memory space cannot store all vector values ​​of the dense sparse vector to be stored. Use mmap to map out a virtual address space, and store the dense sparse vector to be stored in the corresponding virtual address space. The size of the virtual address space is obtained by rounding the maximum value of the length of the dense sparse vector to be stored to the integer of the data page, that is, rounding up to the size of a data page of 4KB. The maximum value of the length of the dense sparse vector to be stored can be pre-set, or it can be determined by comparing the lengths of each dense sparse vector to be stored when obtaining the dense sparse vectors to be stored in batches; the size of the virtual address space can also be determined based on the maximum number of non-zero values ​​of all the dense sparse vectors to be stored, that is, the maximum value of the length of the dense sparse vector to be stored can be determined based on the maximum number of non-zero values ​​and the dimension of the dense vector. If the length of the dense sparse vector to be stored is not greater than the length of the second memory space, the dense sparse vector to be stored is directly stored in the corresponding second memory space.

[0123] Exemplarily, the maximum number of non-zero values ​​of a constrained sparse vector is N, representing the maximum number of non-zero values ​​in most sparse vectors, for example, N = 32. Based on N, the length of the second memory space can be calculated. The length of the second memory space is normal_sparse_size, and the maximum length of a dense sparse vector is max_vector_size. Therefore, a total of max_elements multiplied by normal_sparse_size is allocated to the second memory space body. Max_vector_size is rounded up to 4KB, and mmap is used to map a virtual address space huge_body. The length of huge_body is the value obtained by multiplying max_elements by max_vector_size, rounded up to 4KB. When a dense sparse vector is written, if the length of the dense sparse vector is less than or equal to normal_sparse_size, the dense sparse vector is stored in body; if the length of the dense sparse vector data is greater than normal_sparse_size, the dense sparse vector is stored in huge_body.

[0124] Assuming that the maximum value max_vector_size of the length of all dense sparse vectors to be stored is 16380, the maximum number of non-zero values ​​N of the constrained sparse vector is 32, the dense vector dimension is 64, and max_elements is 1024, then the length of the second memory space normal_sparse_size = 64×4+4+8×32 = 516, and max_vector_size is rounded to 16384 after the data page is rounded; assuming that the length of the dense sparse vector 4 to be stored is 13386, and the non-zero values ​​of the sparse parts of the other vectors are all less than 32. Figure 9 provides an example diagram for storing dense sparse vectors to be stored. When the length of the dense sparse vector to be stored is not greater than the length of the second memory space, the dense sparse vector to be stored is stored in the corresponding second memory space body. When the length of the dense sparse vector to be stored is greater than the length of the second memory space, the dense sparse vector to be stored is stored in the corresponding virtual address space huge_body. The memory overhead with huge_page optimization is: head, 8192 bytes; body, 528384 bytes; huge_body, 16384 bytes; total memory overhead is 552960 bytes. Without huge_page optimization, the memory overhead is: head, 8192 bytes; body, 16777216 bytes; total memory overhead is 16785408 bytes.

[0125] S403: Construct a hierarchical navigable small-world search graph according to the dense and sparse vectors to be stored.

[0126] The similarity calculated when constructing a hierarchical navigable small-world search graph based on the dense and sparse vectors to be stored is calculated using the vector similarity determination method provided in any embodiment of the present application.

[0127] The dense sparse vectors to be stored are placed in the corresponding layers to form a search graph for a hierarchical navigable small world. When placing the dense sparse vectors to be stored in the corresponding layers, they can be placed in the layer0 layer first, and then the similarity is calculated. Based on the similarity, the neighbors of the dense sparse vectors are determined, and appropriate dense sparse vectors are selected to map to the upper layers, or layers are added, etc. In the process of constructing a search graph for a hierarchical navigable small world, it is necessary to calculate the similarity in order to determine the neighbors of each dense sparse vector and the position of each dense sparse vector. The similarity at this time is calculated using the vector similarity determination method of any embodiment of the present application. When there are multiple dense sparse vectors to be stored, the dense sparse vectors to be stored are placed in the search graph in turn to complete the construction of the search graph for a hierarchical navigable small world.

[0128] It is important to note that, in the process of constructing a search graph for a hierarchical navigable small world, if the similarity between a newly added dense sparse vector to be stored and a dense sparse vector already existing in the search graph for the hierarchical navigable small world is calculated, since the dense sparse vector is already stored in the memory space at this time, it is necessary to read it from the memory space. The dense sparse vector to be read is used as the dense sparse vector to be read, and the length of the dense sparse vector to be read is read from the corresponding first memory space according to the identifier of the dense sparse vector to be read; it is determined whether the length of the dense sparse vector to be read is greater than the length of the second memory space; if so, it is determined that the dense sparse vector to be read is stored in the virtual address space, and then the dense sparse vector to be read is read from the corresponding virtual memory space; otherwise, it is determined that the dense sparse vector to be read is stored in the second memory space, and then the dense sparse vector to be read is read from the corresponding second memory space.

[0129] S404: When the memory merging condition is met, the pre-allocated memory space is merged to obtain a merged storage space.

[0130] In this embodiment, the memory merging condition can be understood as a judgment condition for merging memory space and releasing excess space. The memory merging condition can be detecting that all dense and sparse vectors have been stored, or receiving a user-triggered memory merging operation, or the number of currently stored dense and sparse vectors to be stored reaches a set threshold.

[0131] Pre-set memory merge conditions. When the conditions are met, the search graph for the hierarchical navigable small world is considered complete and no new data will be added. Since some of the pre-allocated memory space is not fully populated, the pre-allocated memory space is merged to avoid memory waste, save memory overhead, and improve data storage and query performance. This merging of the pre-allocated memory space frees up excess space, resulting in a merged storage space. After the memory merge is executed, the search graph for the hierarchical navigable small world refuses to add new data, allowing vector searches to be performed based on the search graph.

[0132] As an optional embodiment of this embodiment, this optional embodiment merges the pre-allocated memory space to obtain a merged storage space, which is optimized as follows:

[0133] B1. Determine the total length of the storage space and the offset of each dense sparse vector to be stored according to the length stored in the first memory space of the memory space.

[0134] Read the lengths stored in all first memory spaces in the memory space, calculate the sum of the lengths to obtain the total length of the storage space required to store the vector data, and at the same time, for each dense sparse vector to be stored, determine the offset of the dense sparse vector to be stored according to the length of the dense sparse vector to be stored stored before the dense sparse vector to be stored.

[0135] B2. Allocate a new memory space according to the total length of the storage space, where the new memory space includes a third memory space and a fourth memory space.

[0136] A new memory space is allocated according to the total length of the storage space so as to store the vector through the new memory space, where the new memory space includes a third memory space and a fourth memory space.

[0137] B3. Storing the offset of each to-be-stored dense sparse vector into the corresponding third memory space, and storing each to-be-stored dense sparse vector into the corresponding fourth memory space according to the offset, to obtain a merged storage space.

[0138] The third memory space is used to store the offset, and the offset of each dense sparse vector to be stored is stored in the corresponding third memory space; the fourth memory space is used to store the dense sparse vector, and each dense sparse vector to be stored is stored in the corresponding fourth memory space according to the offset. When storing each dense sparse vector to be stored in the fourth memory space, space allocation is performed according to the size of the vector, which will not cause memory waste, thereby completing the merger of the memory space and obtaining the merged storage space.

[0139] Exemplarily, FIG10 provides an example diagram for storing a dense sparse vector to be stored. In this case, the storage space obtained is the storage space after the merge process, where the offset is stored in the head and the dense sparse vector to be stored is stored in the body.

[0140] In the process of constructing a search graph, the embodiment of the present application first allocates a fixed-length memory space to store dense sparse vectors, that is, pre-allocates a second memory space and a virtual address space to store the dense sparse vectors to be stored. The advantage is that it avoids the need to introduce locks to protect the insertion and search processes of data during the map construction process, but it will cause a certain amount of memory waste. For this reason, a memory merging process is introduced to avoid memory waste. After the memory is merged, the head stores the offset of the dense sparse vector to be stored in the body. To address the dense sparse vector, it is only necessary to add the offset stored in the head to the body. After the memory is merged, new points are no longer allowed to be inserted, and the entire addressing process is still lock-free.

[0141] After the hierarchical navigable small-world search graph is constructed, the search graph usually does not change. During the vector search process, only the search graph needs to be queried.

[0142] S405: Obtain the dense and sparse vector to be searched.

[0143] S406: Search the search graph of the hierarchical navigable small world based on a graph algorithm, read the dense sparse vector to be matched in the search graph of the hierarchical navigable small world, and calculate the similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, where the similarity is calculated according to the vector similarity determination method of any embodiment of the present application.

[0144] As an optional embodiment of this embodiment, this optional embodiment reads the dense sparse vector to be matched in the search graph of the hierarchical navigable small world, including:

[0145] C1. Read the length from the corresponding first memory space according to the identifier of the dense sparse vector to be matched;

[0146] C2. Determine whether the length of the to-be-matched dense sparse vector is greater than the length of the second memory space. If so, read the to-be-matched dense sparse vector from the corresponding virtual memory space; otherwise, read the to-be-matched dense sparse vector from the corresponding second memory space.

[0147] The hierarchical navigable small-world search graph can be used for practical search applications after construction is complete, or during the construction process. If memory merging is not performed, during vector search, the identifier of the dense sparse vector to be matched is determined, and the length of the dense sparse vector to be matched is read from the corresponding first memory space based on the identifier of the dense sparse vector to be matched; a determination is made as to whether the length of the dense sparse vector to be matched is greater than the length of the second memory space; if so, it is determined that the dense sparse vector to be matched is stored in the virtual address space, and the dense sparse vector to be matched is read from the corresponding virtual memory space; otherwise, it is determined that the dense sparse vector to be matched is stored in the second memory space, and the dense sparse vector to be matched is read from the corresponding second memory space.

[0148] It should be noted that when memory merging is not performed, if a vector search is performed or a new dense sparse vector to be stored is stored in the memory space and the search graph is updated, similarity calculation may be involved. The similarity calculation at this time needs to read the dense sparse vectors already contained in the search graph from the stored memory space. The reading method is the same, and the length is used to determine whether the vector is stored in the second memory space or in the virtual address space, and then read the data from the corresponding space.

[0149] As an alternative embodiment of this embodiment, this alternative embodiment optimizes reading the dense sparse vector to be matched in the search graph of the hierarchical navigable small world as follows:

[0150] D1. Determine the offset of the dense sparse vector to be matched according to the identifier of the dense sparse vector to be matched.

[0151] During the search process, if memory merging has been performed, all dense sparse vectors have been stored in the memory space, so the value of the dense sparse vector needs to be read from the memory space. The offset of the dense sparse vector to be matched is read from the head according to the identifier of the dense sparse vector to be matched.

[0152] D2. Read the to-be-matched dense sparse vector from the storage space corresponding to the search graph of the hierarchical navigable small world according to the offset of the to-be-matched dense sparse vector.

[0153] According to the offset, the dense and sparse vector to be matched is directly read from the storage space corresponding to the search graph of the hierarchical navigable small world, that is, the body.

[0154] The embodiments of the present application provide two methods for reading the dense sparse vectors to be matched in the search graph of the hierarchical navigable small world, which can realize the reading of the dense sparse vectors to be matched under different circumstances. A vector search is performed before the memory is merged. At this time, the dense sparse vectors to be matched can be read using the steps C1-C2. This method can be applied to the scenario where a vector search is performed in the process of constructing a search graph; a vector search is performed after the memory is merged. At this time, the dense sparse vectors to be matched can be read using the steps D1-D2. This method can be applied to the scenario where the search graph is completed and no new vectors are added. In actual application, the appropriate method can be selected to read the dense sparse vectors to be matched by judging whether the memory merge has been performed.

[0155] S407 : Determine a target dense sparse vector that matches the dense sparse vector to be searched according to the similarity.

[0156] When feeding back the target dense sparse vector, the external label Label of the target dense sparse vector can be fed back at the same time to facilitate the use of the target dense sparse vector.

[0157] The embodiment of the present application provides a vector search method, which solves the problem of low recall rate caused by low accuracy of similarity calculation results when jointly searching dense vectors and sparse vectors. By splicing dense vectors and sparse vectors, a new vector, namely a dense sparse vector, is obtained. In the process of searching for vectors based on a hierarchical navigable small world, the final similarity is determined by calculating the first similarity and the second similarity of the first dense sparse vector and the second dense sparse vector, thereby improving the accuracy of the similarity calculation and thus improving the accuracy of the search. Compared with the related art that calculates similarity from the perspectives of dense vectors and sparse vectors respectively, the present application considers both dense vectors and sparse vectors at the same time, calculates the similarity of vectors, and the result is more accurate, thereby improving the recall rate during vector search. The implementation is simple, and only one retrieval system needs to be maintained, effectively ensuring data consistency. Furthermore, the present application expands the data storage of the hierarchical navigable small-world search graph, and hierarchically stores dense sparse vectors by pre-allocating fixed-length memory space and virtual address space. No lock protection is required during the entire process, and storing larger dense sparse vectors in the virtual address space can effectively save memory space. After storing all dense sparse vectors, a merge operation is performed on the memory space to release excess space, and dense sparse vectors are stored in a variable-length manner. Before merging, the dense sparse vectors use fixed-length addressing, and after merging, the dense sparse vectors use variable-length addressing. By extending the graph algorithm HNSW, graph algorithms can be used to directly perform mixed retrieval on dense sparse vectors, and data can be represented by dense sparse vectors to better reflect the actual data space distribution, avoiding the performance and recall rate losses caused by searching dense and sparse vectors separately in traditional solutions. The recall rate is guaranteed in general scenarios without relying on specific scenarios.

[0158] Example 5

[0159] FIG11 is a schematic diagram of the structure of a vector similarity determination device provided in Example 5 of the present application. As shown in FIG11 , the device includes: a vector acquisition module 51 , a similarity calculation module 52 , and a similarity determination module 53 .

[0160] The vector acquisition module 51 is configured to acquire a first dense sparse vector and a second dense sparse vector, where the first dense sparse vector is obtained by concatenating the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by concatenating the second dense vector and the second sparse vector;

[0161] A similarity calculation module 52 is configured to calculate a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector;

[0162] The similarity determination module 53 is configured to determine the similarity between the first dense sparse vector and the second dense sparse vector according to the first similarity and the second similarity.

[0163] The embodiment of the present application provides a vector similarity determination device, which solves the problem of low accuracy of similarity calculation results between dense vectors and sparse vectors. By splicing dense vectors and sparse vectors, a new vector, namely a dense sparse vector, is obtained. The final similarity of the first dense sparse vector and the second dense sparse vector is determined by calculating the first similarity and the second similarity of the first dense sparse vector and the second dense sparse vector. Compared with the related art that calculates the similarity from the perspectives of dense vectors and sparse vectors respectively, the present application considers both dense vectors and sparse vectors at the same time to calculate the similarity of vectors, and the result is more accurate; the implementation is simple and data consistency is effectively guaranteed; the data is represented by dense sparse vectors to better reflect the real data space distribution, thereby improving the recall rate of vector search.

[0164] Optionally, the similarity calculation module 52 includes:

[0165] a first similarity calculation unit, configured to calculate the similarity between the first dense vector and the second dense vector to obtain a first similarity;

[0166] The second similarity calculation unit is configured to calculate the similarity between the first sparse vector and the second sparse vector to obtain a second similarity.

[0167] Optionally, the second similarity calculation unit is configured to determine the first dimension identifier in the first sparse vector and the first vector value corresponding to the first dimension identifier; determine the second dimension identifier in the second sparse vector and the second vector value corresponding to the second dimension identifier; compare each first dimension identifier and each second dimension identifier, and determine the same first dimension identifier and second dimension identifier as the target dimension identifier; calculate the similarity based on the first vector value and the second vector value corresponding to the target dimension identifier, and determine the second similarity.

[0168] Optionally, the first similarity and the second similarity are inner product similarities.

[0169] Optionally, the similarity determination module 53 is configured to: perform weighted summation on the first similarity and the second similarity to obtain the similarity between the first dense sparse vector and the second dense sparse vector.

[0170] The vector similarity determination device provided in the embodiment of the present application can execute the vector similarity determination method provided in the first or second embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0171] Example 6

[0172] FIG12 is a schematic diagram of the structure of a vector search device provided in Example 6 of the present application. As shown in FIG12 , the device includes: a vector acquisition module 61 to be searched, a vector search module 62 and a target vector determination module 63 .

[0173] A vector acquisition module 61 to be searched is configured to acquire a dense or sparse vector to be searched;

[0174] a vector search module 62 configured to search a search graph of a hierarchical navigable small world based on a graph algorithm, read a dense sparse vector to be matched in the search graph of the hierarchical navigable small world, and calculate a similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, where the similarity is calculated according to the vector similarity determination method described in any embodiment of the present application;

[0175] The target vector determination module 63 is configured to determine a target dense sparse vector that matches the dense sparse vector to be searched according to the similarity.

[0176] An embodiment of the present application provides a vector similarity determination device, which solves the problem of low recall rate caused by low accuracy of similarity calculation results when jointly searching dense vectors and sparse vectors. By splicing dense vectors and sparse vectors, a new vector, namely a dense sparse vector, is obtained. The first similarity between the first dense vector and the second dense vector, as well as the second similarity between the first sparse vector and the second sparse vector, is calculated, and the first similarity and the second similarity are weightedly summed to obtain the final similarity between the first dense sparse vector and the second dense sparse vector, thereby improving the accuracy of the similarity calculation. Compared with the related art in which the similarity is calculated from the perspectives of dense vectors and sparse vectors respectively, the present application considers both dense vectors and sparse vectors at the same time to calculate the similarity of the vectors, resulting in more accurate results and higher recall rate. The implementation is simple and effectively ensures data consistency. The data is represented by dense sparse vectors to better reflect the actual data space distribution, thereby avoiding the inaccurate similarity calculation results caused by calculating the similarity of dense and sparse vectors separately in the traditional solution, thereby avoiding the performance and recall rate loss caused by the inaccurate similarity calculation results. The recall rate is guaranteed in general scenarios without relying on specific scenarios.

[0177] Optionally, the device further includes:

[0178] A module for obtaining vectors to be stored, configured to obtain dense and sparse vectors to be stored;

[0179] A vector storage module, configured to store the dense and sparse vectors to be stored in a pre-allocated memory space;

[0180] A search graph construction module is configured to construct a hierarchical navigable small-world search graph according to the dense and sparse vectors to be stored.

[0181] Optionally, the pre-allocated memory space includes a first memory space and a second memory space;

[0182] a vector storage module configured to: determine the length of the dense sparse vector to be stored and store it in the corresponding first memory space; determine whether the length of the dense sparse vector to be stored is greater than the length of the second memory space; if so, store the dense sparse vector to be stored in the corresponding virtual address space; otherwise, store the dense sparse vector to be stored in the corresponding second memory space;

[0183] The length of the second memory space is determined according to the dense vector dimension in the dense sparse vector to be stored and the maximum number of non-zero values ​​of the constrained sparse vector; the size of the virtual address space is determined by rounding the maximum value of the length of the dense sparse vector to be stored.

[0184] Optionally, the device further includes:

[0185] The memory merging module is configured to merge the pre-allocated memory space to obtain a merged storage space when a memory merging condition is met.

[0186] Optional, memory merge module, including:

[0187] a total length determining unit, configured to determine the total length of the storage space and the offset of each dense sparse vector to be stored according to the length stored in the first memory space of the memory space;

[0188] a space allocation unit, configured to allocate a new memory space according to the total length of the storage space, wherein the new memory space includes a third memory space and a fourth memory space;

[0189] The merging unit is configured to store the offset of each of the dense sparse vectors to be stored in the corresponding third memory space, and store each of the dense sparse vectors to be stored in the corresponding fourth memory space according to the offset, to obtain a merged storage space.

[0190] Optionally, the vector search module 62 includes:

[0191] a length reading unit, configured to read the length from the corresponding first memory space according to the identifier of the dense sparse vector to be matched;

[0192] The first vector reading unit is configured to determine whether the length of the to-be-matched dense sparse vector is greater than the length of the second memory space, and if so, read the to-be-matched dense sparse vector from the corresponding virtual memory space; otherwise, read the to-be-matched dense sparse vector from the corresponding second memory space.

[0193] Optionally, the vector search module 62 includes:

[0194] an offset determining unit, configured to determine an offset of the dense sparse vector to be matched according to an identifier of the dense sparse vector to be matched;

[0195] The second vector reading unit is configured to read the to-be-matched dense sparse vector from a storage space corresponding to a search graph of the hierarchical navigable small world according to an offset of the to-be-matched dense sparse vector.

[0196] The vector search device provided in the embodiment of the present application can execute the vector search method provided in the third or fourth embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0197] Example 7

[0198] FIG13 shows a block diagram of an electronic device 70 that can be used to implement an embodiment of the present application. The electronic device can represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples.

[0199] As shown in FIG13 , the electronic device 70 includes at least one processor 71 and a memory, such as a read-only memory (ROM) 72, a random access memory (RAM) 73, etc., which is communicatively connected to the at least one processor 71. The memory stores a computer program that can be executed by the at least one processor, and the processor 71 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 72 or the computer program loaded from the storage unit 78 into the random access memory (RAM) 73. Various programs and data required for the operation of the electronic device 70 can also be stored in the RAM 73. The processor 71, ROM 72, and RAM 73 are connected to each other via a bus 74. An input / output (I / O) interface 75 is also connected to the bus 74.

[0200] Multiple components in the electronic device 70 are connected to the I / O interface 75, including an input unit 76, such as a keyboard, a mouse, etc.; an output unit 77, such as various types of displays, speakers, etc.; a storage unit 78, such as a magnetic disk, an optical disk, etc.; and a communication unit 79, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 79 allows the electronic device 70 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0201] The processor 71 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 71 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 71 executes the various methods and processes described above, such as the vector similarity determination method or the vector search method.

[0202] In some embodiments, the vector similarity determination method or the vector search method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 78. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 70 via the ROM 72 and / or the communication unit 79. When the computer program is loaded into the RAM 73 and executed by the processor 71, one or more steps of the vector similarity determination method or the vector search method described above may be performed. Alternatively, in other embodiments, the processor 71 may be configured to perform the vector similarity determination method or the vector search method in any other appropriate manner (e.g., by means of firmware).

[0203] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0204] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0205] An embodiment of the present application provides a storage medium containing computer instructions, which, when executed by a computer processor, are used to perform a vector similarity determination method, the method including: obtaining a first dense sparse vector and a second dense sparse vector, where the first dense sparse vector is obtained by concatenating a first dense vector and a first sparse vector, and the second dense sparse vector is obtained by concatenating a second dense vector and a second sparse vector; calculating a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector; and determining the similarity between the first dense sparse vector and the second dense sparse vector based on the first similarity and the second similarity.

[0206] Embodiments of the present application provide a storage medium containing computer instructions. When executed by a computer processor, the computer instructions are used to perform a vector search method, the method comprising: obtaining a dense sparse vector to be searched; searching a search graph of a hierarchical navigable small world based on a graph algorithm, reading a dense sparse vector to be matched from the search graph of the hierarchical navigable small world, calculating a similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, the similarity being calculated according to the vector similarity determination method described in any embodiment of the present application; and determining a target dense sparse vector that matches the dense sparse vector to be searched based on the similarity.

[0207] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. Examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0208] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0209] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0210] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0211] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this application can be achieved.

Claims

1. A method for determining vector similarity, comprising: Obtain a first dense sparse vector and a second dense sparse vector, where the first dense sparse vector is obtained by concatenating the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by concatenating the second dense vector and the second sparse vector; Calculating a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector; The similarity between the first dense sparse vector and the second dense sparse vector is determined according to the first similarity and the second similarity.

2. The method according to claim 1, wherein: The calculating the first similarity and the second similarity based on the first dense sparse vector and the second dense sparse vector includes: Calculating a similarity between the first dense vector and the second dense vector to obtain a first similarity; The similarity between the first sparse vector and the second sparse vector is calculated to obtain a second similarity.

3. The method according to claim 2, wherein: The calculating the similarity between the first sparse vector and the second sparse vector to obtain a second similarity includes: Determine a first dimension identifier in the first sparse vector and a first vector value corresponding to the first dimension identifier; Determine a second dimension identifier in the second sparse vector and a second vector value corresponding to the second dimension identifier; Comparing each of the first dimension identifiers and each of the second dimension identifiers, and determining the same first dimension identifier and second dimension identifier as a target dimension identifier; The similarity is calculated based on the first vector value and the second vector value corresponding to the target dimension identifier to determine a second similarity.

4. The method according to claim 2, wherein: The first similarity and the second similarity are inner product similarities.

5. The method according to claim 1, wherein: The determining the similarity between the first dense sparse vector and the second dense sparse vector according to the first similarity and the second similarity includes: A weighted sum is performed on the first similarity and the second similarity to obtain a similarity between the first dense sparse vector and the second dense sparse vector.

6. A vector search method, comprising: Get the dense and sparse vector to be searched; Based on the graph algorithm, the search graph of the hierarchical navigable small world is searched, and the hierarchical navigable small world is read. a dense sparse vector to be matched in a search graph of a small world, calculating a similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, wherein the similarity is calculated according to the method for determining vector similarity according to any one of claims 1 to 5; A target dense sparse vector matching the dense sparse vector to be searched is determined according to the similarity.

7. The method according to claim 6, wherein: Constructing a search graph of the hierarchical navigable small world, including: Get the dense sparse vector to be stored; Storing the dense sparse vector to be stored in a pre-allocated memory space; Constructing a hierarchical navigable small-world search graph according to the dense sparse vector to be stored; Wherein, the similarity calculated when constructing a hierarchical navigable small-world search graph according to the dense sparse vectors to be stored is calculated using the vector similarity determination method described in any one of claims 1 to 5.

8. The method according to claim 7, wherein: The pre-allocated memory space includes a first memory space and a second memory space; The storing the dense sparse vector to be stored in the pre-allocated memory space includes: Determine the length of the dense sparse vector to be stored and store it in the corresponding first memory space, determine whether the length of the dense sparse vector to be stored is greater than the length of the second memory space, and if so, store the dense sparse vector to be stored in the corresponding virtual address space; otherwise, store the dense sparse vector to be stored in the corresponding second memory space; Among them, the length of the second memory space is determined according to the dense vector dimension in the dense sparse vector to be stored and the maximum number of non-zero values ​​of the constrained sparse vector; the size of the virtual address space is determined by rounding the maximum value of the length of the dense sparse vector to be stored.

9. The method according to claim 8, wherein: The step of reading the dense and sparse vector to be matched in the search graph of the hierarchical navigable small world comprises: Reading the length from the corresponding first memory space according to the identifier of the dense sparse vector to be matched; Determine whether the length of the to-be-matched dense sparse vector is greater than the length of the second memory space; if so, read the to-be-matched dense sparse vector from the corresponding virtual memory space; otherwise, read the to-be-matched dense sparse vector from the corresponding second memory space.

10. The method according to claim 7, further comprising: When the memory merging condition is met, the pre-allocated memory space is merged to obtain a merged storage space.

11. The method according to claim 10, wherein: The merging of the pre-allocated memory spaces to obtain the merged storage space includes: Determine the total length of the storage space and the offset of each dense sparse vector to be stored according to the length stored in the first memory space of the memory space; Allocate a new memory space according to the total length of the storage space, the new memory space comprising a third memory space and a fourth memory space; The offset of each of the to-be-stored dense sparse vectors is stored in the corresponding third memory space, and each of the to-be-stored dense sparse vectors is stored in the corresponding fourth memory space according to the offset, to obtain a merged storage space.

12. The method according to claim 11, wherein: The step of reading the dense and sparse vector to be matched in the search graph of the hierarchical navigable small world comprises: Determining an offset of the to-be-matched dense sparse vector according to an identifier of the to-be-matched dense sparse vector; The to-be-matched dense sparse vector is read from a storage space corresponding to a search graph of a hierarchical navigable small world according to an offset of the to-be-matched dense sparse vector.

13. A vector similarity determination device, comprising: A vector acquisition module, configured to acquire a first dense sparse vector and a second dense sparse vector, wherein the first dense sparse vector is obtained by concatenating the first dense vector and the first sparse vector, and the second dense sparse vector is obtained by concatenating the second dense vector and the second sparse vector; a similarity calculation module, configured to calculate a first similarity and a second similarity based on the first dense sparse vector and the second dense sparse vector; The similarity determination module is configured to determine the similarity between the first dense sparse vector and the second dense sparse vector according to the first similarity and the second similarity.

14. A vector search device, comprising: A module for obtaining a vector to be searched, configured to obtain a dense or sparse vector to be searched; a vector search module, configured to search a search graph of a hierarchical navigable small world based on a graph algorithm, read a dense sparse vector to be matched in the search graph of the hierarchical navigable small world, and calculate a similarity between the dense sparse vector to be searched and the dense sparse vector to be matched, wherein the similarity is calculated according to the vector similarity determination method according to any one of claims 1 to 5; The target vector determination module is configured to determine a target dense sparse vector that matches the dense sparse vector to be searched according to the similarity.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vector similarity determination method described in any one of claims 1 to 5 or the vector search method described in any one of claims 6 to 12.

16. A computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the vector similarity determination method described in any one of claims 1 to 5 or the vector search method described in any one of claims 6 to 12 when executed.

Images