A method and system for retrieving similarity of hundreds of millions of vectors in a single machine environment

Abstract

The application relates to the technical field of information retrieval, in particular to a method and system for retrieving the similarity of a hundred-billion-level vector in a single-machine environment. The method comprises the following steps: in response to the reception of a retrieval request, performing feature extraction on the retrieval request to obtain a feature set; converting the feature set into a sequence and loading the sequence into a pre-trained word vector model to obtain a to-be-fragmented vector and a first to-be-processed vector; performing fragmented storage and streaming single-block loading on the to-be-fragmented vector to obtain a second to-be-processed vector; taking the block data volume of the second to-be-processed vector as a decision selection criterion; selecting a corresponding similarity calculation model by using a preset adaptive selection mechanism to calculate the vector similarity between the second to-be-processed vector and the first to-be-processed vector; inputting the vector similarity into a preset sorting algorithm to intercept a local retrieval result from multiple second to-be-processed vectors; and finally querying and returning a global retrieval result from a preset single-machine storage medium according to the local retrieval result.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology. More specifically, this invention relates to a method and system for retrieving vector similarity at the level of tens of billions in a single-machine environment. Background Technology

[0002] With the development of artificial intelligence and big data technologies, the data processing dimensions of systems such as advertising information retrieval, recommendation systems, and user profiling have increased significantly. To avoid irrelevant or weakly related information affecting the output of these systems, it is usually necessary to rely on a separate search module to perform targeted retrieval of relevant information. Currently, these systems commonly use vector similarity-based retrieval methods. When the scale of user or item vectors reaches tens of billions, distributed clusters or multi-machine, multi-GPU configurations are typically required to support index building and retrieval tasks.

[0003] However, in resource-constrained scenarios, such as edge node deployments, small-scale SSP advertising platforms, or single-machine environments used for early / maintenance testing, sufficient memory and disk resources are generally unavailable to load all vector data simultaneously. Under such single-machine memory-constrained conditions, vector retrieval struggles to guarantee the simultaneous loading of the complete index, and retrieval latency is significantly affected by memory I / O, making it difficult to guarantee real-time response to retrieval commands and the return of retrieval results. Furthermore, existing retrieval solutions often employ a single indexing strategy, lacking adaptive adjustment mechanisms for different data volumes, leading to wasted computing power or low retrieval efficiency.

[0004] Therefore, existing technologies struggle to achieve large-scale vector recall under limited memory conditions. Summary of the Invention

[0005] To address the aforementioned technical problem of difficulty in achieving large-scale vector retrieval under limited memory conditions, this invention discloses a method and system for retrieving vector similarity at the level of hundreds of billions in a single-machine environment.

[0006] In a first aspect, this invention discloses a method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment, including: In response to the receipt of a retrieval request, feature extraction is performed on the retrieval request to obtain a feature set; After converting the feature set into a sequence, it is loaded into a pre-trained word vector model to obtain the vector to be segmented and the first vector to be processed. The vector to be fragmented is stored in fragments and loaded in a streaming single block to obtain the second vector to be processed. Using the block data size of the second vector to be processed as the decision selection criterion, a preset adaptive selection mechanism is used to select the corresponding similarity calculation model to calculate the vector similarity between the second vector to be processed and the first vector to be processed. The vector similarity is input into a preset sorting algorithm to extract local retrieval results from multiple second vectors to be processed; Based on the local search results, query and return the global search results from the preset single-machine storage medium.

[0007] Beneficial Effects: The method of this invention first maps multi-dimensional business features into a first vector to be processed and a vector to be segmented in a unified space using a pre-trained word vector model, laying the mathematical foundation for high-dimensional feature computation. Next, addressing the inherent contradiction between the massive data volume (tens of billions) and the limited physical memory of a single machine, this invention introduces a segmented storage and streaming single-block loading mechanism. Through a "time-for-space" scheduling strategy, it breaks through the memory overflow (OOM) and disk I / O latency bottlenecks inevitably caused by traditional global indexing schemes. At the single-block execution level, this invention further uses the current data size of the segment as a decision criterion, adaptively matching the optimal similarity calculation model, dynamically eliminating the computational waste of a single retrieval algorithm for small-scale data, balancing retrieval efficiency and matching accuracy. Finally, it queries the single-machine storage medium using local retrieval results and returns the global retrieval results. Compared to existing technologies, this method can still quickly complete the recall of large-scale vectors even in a single-machine environment with severely limited hardware resources.

[0008] Preferably, the vector to be fragmented is fragmented and stored, and then loaded in a streaming single-block manner to obtain a second vector to be processed, including: The vector to be sharded is divided into multiple independent data blocks and stored in the external storage medium of a single-machine environment; A streaming iterative mechanism is adopted. Each data block is selected and loaded from the external storage medium into the system memory of the single-machine environment. The vector corresponding to the data block loaded into the system memory is used as the second vector to be processed.

[0009] Preferably, after the partial search results are obtained, a memory reclamation instruction is triggered to release the resources occupied by the second vector to be processed in the system memory.

[0010] Preferably, before selecting the corresponding similarity calculation model using a preset adaptive selection mechanism, the method of the present invention further includes: If the block data size of the second vector to be processed is greater than or equal to a preset threshold, a quantizer is trained independently for each second vector to be processed and an index is established.

[0011] Preferably, using the block data size of the second vector to be processed as the decision-making criterion, a preset adaptive selection mechanism is used to select the corresponding similarity calculation model to calculate the vector similarity between the second vector to be processed and the first vector to be processed, including: Determine whether the block data size of the second vector to be processed is less than a preset threshold; If so, calculate the Euclidean distance between the first vector to be processed and the second vector to be processed as the vector similarity; If not, use a quantizer to calculate vector similarity.

[0012] Preferably, feature extraction is performed on the retrieval request to obtain a feature set, including: Extract user characteristics, advertising characteristics, and platform characteristics from advertising business scenarios; By combining user characteristics, advertising characteristics, and platform characteristics, a feature set is obtained.

[0013] Preferably, after converting the feature set into a sequence, it is loaded into a pre-trained word vector model to obtain the vector to be segmented and the first vector to be processed, including: After converting the feature set into a sequence, it is input into a pre-trained word vector model to obtain basic dense vectors for users and advertisements respectively. The first weighted summation is performed on the basic dense vector of the advertisement to obtain the first vector to be processed. The user's basic dense vector is then subjected to a second weighted summation to obtain the vector to be sliced.

[0014] Preferably, the TopK algorithm is used for sorting.

[0015] Preferably, the above method is used in edge nodes, SSP advertising platforms, or test standalone environments.

[0016] Secondly, the present invention also discloses a single-machine environment vector similarity retrieval system for hundreds of billions of vectors, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the single-machine environment vector similarity retrieval method for hundreds of billions of vectors described in the first aspect is implemented.

[0017] The beneficial effects of this invention are as follows: (1) Compared with the prior art, the method of the present invention is able to quickly complete the large-scale vector recall even in a single machine environment with severely limited hardware resources for the task of searching for hundreds of billions of requests.

[0018] (2) Compared with the prior art, the method of the present invention dynamically eliminates the waste of computing power of a single retrieval algorithm for small-scale data, and achieves a balance between retrieval efficiency and matching accuracy. Attached Figure Description

[0019] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein: Figure 1 This is a flowchart of the method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to Embodiment 1 of the present invention; Figure 2 This is an information flow diagram of the billion-level vector similarity retrieval method in a single-machine environment in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the structure of a single-machine environment-based vector similarity retrieval system with a scale of tens of billions in Embodiment 2 of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Example 1 like Figure 1 and Figure 2 As shown, the present invention discloses a method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment, including: S10: In response to receiving a retrieval request, perform feature extraction on the retrieval request to obtain a feature set.

[0023] In this embodiment, the above-mentioned search request mainly refers to search tasks targeting hundreds of billions of users, advertisements, and platforms, such as "intended customers to purchase the top brand lipstick on the top e-commerce platform".

[0024] More specifically, step S10 above includes: S11: Extract user characteristics, advertising characteristics, and platform characteristics from advertising business scenarios.

[0025] It should be noted that retrieval requests are generally represented in natural language. Semantic understanding and feature aggregation can be performed in step S11 in conjunction with the current semantic parsing model, thereby obtaining user features, advertising features, and platform features in the advertising business scenario.

[0026] S12: Combine user features, advertising features, and platform features to obtain a feature set.

[0027] It should be noted that before performing step S10 above, the built-in features of each platform (such as advertised products or user groups) need to be sequenced to train a word vector model. In this embodiment, the word vector model can be the currently open-source word2vec or deepwalk model. For feature quantization, each feature can be defined as a corresponding vector, and these vectors can be concatenated to obtain a feature set.

[0028] For example, the definition of the above feature set can be:

[0029] in, Represents a set of discrete features in the global dimension. This represents the first set of features. This represents the second set of features. Indicates the first A set of features. In this embodiment, the first set of features can refer to the user feature set, the second set of features can refer to the advertising feature set, and if there are only three features to be processed, then... that is It refers to the set of platform features.

[0030] S20: After converting the feature set into a sequence, load it into the pre-trained word vector model to obtain the vector to be segmented and the first vector to be processed.

[0031] To transform the aforementioned unordered discrete feature set into a feature sequence suitable for model processing, it is necessary to convert the feature set into a feature sequence according to business logic based on a preset logical mapping relationship:

[0032] in, The above subscripts represent a feature sequence arranged according to a preset business logic. , ... Characterizes the logical order or time step sequence. The total length of the feature sequence is represented.

[0033] In this embodiment, a sequence-trained word vector model, such as word2vec or deepwalk, is used to obtain the word vector for each feature. Its general mathematical representation is as follows:

[0034] in, This represents a pre-trained word vector model. Characterize a having A continuous real-valued vector of dimension, the physical meaning of the above mathematical representation is to convert natural language or discrete labels into vectors of fixed length. A multidimensional floating-point array is used to facilitate subsequent calculations by the computer.

[0035] More specifically, adapting to the advertising business logic, the above step S20 includes: First, the feature set is input into the pre-trained word vector model to obtain the basic dense vectors for users and advertisements respectively. Then, the basic dense vectors for advertisements are subjected to a first weighted summation to obtain the first vector to be processed. At the same time, the basic dense vectors for users are subjected to a second weighted summation to obtain the vector to be segmented.

[0036] In this embodiment, the above-mentioned fragmented vectors are applied to the current advertising business scenario, namely a large number of user vectors.

[0037] S30: The vector to be fragmented is fragmented and stored, and then loaded in a streaming single block to obtain the second vector to be processed.

[0038] Step S30 above includes: S31: Divide the vector to be fragmented into multiple independent data blocks and store them in the external storage medium of the single-machine environment.

[0039] Specifically, the user vector is divided into several independent data blocks based on the actual scenario (e.g., region or platform):

[0040] in, This represents the full dataset containing tens of billions of candidate user vectors; Indicates the first The data is divided into blocks. It should be noted that the application scenarios in this embodiment are mainly edge nodes, SSP advertising platforms, or single-machine test environments, with supporting hardware or communication characteristics of a single machine and memory attributes of less than or equal to 64GB RAM. In this scenario, only one data block is loaded into the RAM (i.e., external storage medium) at a time. During this process, memory resources are dynamically reclaimed after each loading; this effect can be achieved using the Free operator.

[0041] S32: A streaming iteration mechanism is adopted. Each data block is selected and loaded from the external storage medium into the system memory of the single-machine environment. The vector corresponding to the data block loaded into the system memory is used as the second vector to be processed.

[0042] By using steps S31-S32 above, a scheduling strategy that trades time for space can be implemented, thereby breaking the memory overflow (OOM) and disk I / O latency bottlenecks caused by traditional global indexing schemes.

[0043] S40: Using the block data volume of the second vector to be processed as the decision selection benchmark, a preset adaptive selection mechanism is used to select the corresponding similarity calculation model to calculate the vector similarity between the second vector to be processed and the first vector to be processed.

[0044] Further, step S40 above includes: S41: Determine whether the block data volume of the second vector to be processed is less than the preset threshold.

[0045] S42: If so, calculate the Euclidean distance between the first vector to be processed and the second vector to be processed as the vector similarity.

[0046] It should be further explained that this implementation method provides an automatic degradation mechanism to reduce training costs. When the amount of data in a block is less than a preset threshold, the Euclidean distance algorithm is used to calculate the similarity between the two. The specific algorithm details are as follows:

[0047] In the formula, This represents a function for calculating spatial distance. This represents the second vector to be processed, which exists in the same multidimensional space. This represents the first vector to be processed within the same multidimensional space.

[0048] S43: If not, proceed with the following steps: First, train a quantizer independently for each second vector to be processed and build an index. The specific quantizer quantization formula is as follows:

[0049] In the formula, This represents the original high-dimensional continuous feature vector; This means that the original high-dimensional continuous feature vectors are replaced by the nearest cluster centers after quantization mapping. The approximate vector obtained later; This indicates that it is specifically for the first A quantizer function that is trained independently for each data block; This represents a quantized codebook, which is a pre-calculated set of discrete cluster centers.

[0050] Next, a quantizer is used to calculate vector similarity. Specifically, the quantizer is trained once, and then the similarity calculation is performed directly using the quantizer. The specific formula for calculating the similarity between the ad vector and the second vector to be processed in the block is as follows:

[0051] In the formula, Indicates the first Users in the block Final similarity response, ad query vector With the Users in the block User vectors , This represents the distance calculation function. The smaller the value, the more similar the two numbers are.

[0052] Preferably, if a linear scan is used, the formula for calculating the similarity is:

[0053] Through the above technical solution, at the execution level of single-block flow, the method of this embodiment takes the data scale of the current block as the decision benchmark, adaptively matches the optimal similarity calculation model, and dynamically eliminates the waste of computing power of a single retrieval algorithm for small-scale data, thereby taking into account both retrieval efficiency and matching accuracy.

[0054] S50: Input the vector similarity into the preset sorting algorithm to extract local retrieval results from multiple second vectors to be processed.

[0055] In this embodiment, the sorting algorithm described above can be the TopK algorithm. Its specific algorithm expression is expanded as follows:

[0056] In the formula, Indicates the first Local search results corresponding to each block (the second vector to be processed); This represents the preset global target recall total; This indicates the total number of data blocks. This represents a local truncation operator. The physical meaning of the above formula is based on similarity scoring. Sort by highest to lowest, only the top-ranked items are included. A number of candidate users.

[0057] More intuitively, when applied to business processing logic, the above algorithm is essentially allocating query quotas. For example, if you need to find 10,000 people globally (corresponding to the K value), you divide the entire Earth into multiple regions. To be fair, you rank each region internally and only select the top 1,000 people.

[0058] Preferably, after the partial search results are obtained, a memory reclamation instruction is triggered to release the resources occupied by the second vector to be processed in the system memory.

[0059] S60: Based on the local search results, query and return the global search results from the preset single-machine storage medium.

[0060] Specifically, by merging all local search results, the global seed users (i.e., the global search results) can be obtained. The mathematical expression for the global search results is:

[0061] In the formula, This indicates the global search results.

[0062] Through the above steps S10-S60, the method of the present invention has at least the following advantages: Firstly, it supports the location retrieval of billions of customers globally, even under the constraints of a single-machine hardware environment.

[0063] Secondly, even in a single-machine environment with severely limited hardware resources, it can still quickly complete the recall of large-scale vectors.

[0064] Third, the method in this embodiment can dynamically eliminate the waste of computing power of a single retrieval algorithm on small-scale data, achieving a balance between retrieval efficiency and matching accuracy.

[0065] More specifically, compared with similar products, the method in this embodiment performs multi-dimensional characteristic comparisons:

[0066] It is evident that the product developed based on the method of this embodiment can achieve both retrieval efficiency and matching accuracy in retrieval scenarios with data scales of tens of billions, even with limited memory.

[0067] Example 2 like Figure 3 As shown, the present invention also discloses a single-machine environment vector similarity retrieval system with a capacity of tens of billions of vectors, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the single-machine environment vector similarity retrieval method with a capacity of tens of billions of vectors described in Embodiment 1 is implemented.

[0068] The system in this embodiment also includes other components well known to those skilled in the art, such as communication interfaces. Their settings and functions are known in the art, and therefore will not be described in detail here.

[0069] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.

[0070] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise expressly and specifically defined.

[0071] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment, characterized in that, include: In response to receiving a retrieval request, feature extraction is performed on the retrieval request to obtain a feature set; The feature set is converted into a sequence and then loaded into a pre-trained word vector model to obtain the vector to be segmented and the first vector to be processed. The vector to be fragmented is stored in fragments and loaded in a streaming single block to obtain a second vector to be processed. Using the block data volume of the second vector to be processed as the decision selection benchmark, a preset adaptive selection mechanism is used to select the corresponding similarity calculation model to calculate the vector similarity between the second vector to be processed and the first vector to be processed. The vector similarity is input into a preset sorting algorithm to extract local retrieval results from multiple second vectors to be processed; Based on the local search results, the corresponding global search results are queried from the preset single-machine storage medium and returned.

2. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to claim 1, characterized in that, The vector to be fragmented is fragmented and stored, and then loaded in a streaming single-block manner to obtain a second vector to be processed, including: The vector to be fragmented is divided into multiple independent data blocks and stored in the external storage medium of a single-machine environment; A streaming iterative mechanism is adopted. Each data block is selected and loaded from the external storage medium into the system memory of the single-machine environment. The vector corresponding to the data block loaded into the system memory is used as the second vector to be processed.

3. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to claim 2, characterized in that, After the local search results are obtained, a memory reclamation instruction is triggered to release the resources occupied by the second vector to be processed in the system memory.

4. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to claim 1, characterized in that, Before selecting the corresponding similarity calculation model using a preset adaptive selection mechanism, the method further includes: If the block data size of the second vector to be processed is greater than or equal to a preset threshold, a quantizer is trained independently for each second vector to be processed and an index is established.

5. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to claim 4, characterized in that, Using the block data size of the second vector to be processed as the decision-making criterion, a preset adaptive selection mechanism is used to select the corresponding similarity calculation model to calculate the vector similarity between the second vector to be processed and the first vector to be processed, including: Determine whether the block data size of the second vector to be processed is less than a preset threshold; If so, calculate the Euclidean distance between the first vector to be processed and the second vector to be processed as the vector similarity; If not, the vector similarity is calculated using the quantizer.

6. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to claim 1, characterized in that, Feature extraction is performed on the search request to obtain a feature set, including: Extract user characteristics, advertising characteristics, and platform characteristics from advertising business scenarios; The user features, the advertising features, and the platform features are concatenated to obtain the feature set.

7. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to claim 1, characterized in that, After converting the feature set into a sequence, it is loaded into a pre-trained word vector model to obtain the vector to be segmented and the first vector to be processed, including: The feature set is converted into a sequence and then input into a pre-trained word vector model to obtain basic dense vectors for users and advertisements respectively. The first weighted summation is performed on the basic dense vector of the advertisement to obtain the first vector to be processed. The user's basic dense vector is then subjected to a second weighted summation to obtain the vector to be sliced.

8. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to claim 1, characterized in that, The sorting algorithm used is the TopK algorithm.

9. The method for retrieving vector similarity at the level of hundreds of billions in a single-machine environment according to any one of claims 1-8, characterized in that, Used for edge nodes, SSP advertising platforms, or testing single-machine environments.

10. A vector similarity retrieval system with a capacity of tens of billions of vectors in a single-machine environment, characterized in that, It includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the method for retrieving vector similarity of hundreds of billions in a single-machine environment as described in any one of claims 1-9 is implemented.

Images