Method, device and electronic equipment for fast lookup of multidimensional vectors in a database

Abstract

The application discloses a method and device for quickly searching a multi-dimensional vector in a database and electronic equipment. The method comprises: performing sparse and bucket processing on each multi-dimensional vector in the database, wherein the multi-dimensional vector comprises X groups of vectors, and the spatial dimensions of at least two groups of vectors are different; when a to-be-queried multi-dimensional vector is received, performing sparse and bucket processing on the to-be-queried multi-dimensional vector to obtain M segments of DxX to-be-queried signature matrices; selecting D Xx1 to-be-queried cluster center vectors in at least one segment of the to-be-queried signature matrices, and determining one or more buckets, to which the to-be-queried cluster center vectors are mapped in a corresponding bucket set, as hit buckets; and quickly searching one or more target multi-dimensional vectors based on the cluster center vectors in the hit buckets and the to-be-queried multi-dimensional vector. The application can shorten the searching time and improve the searching efficiency on the basis of ensuring the searching accuracy.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to video processing technology, and more particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for quickly searching multidimensional vectors in a database. Background Technology

[0002] Multidimensional vectors can be used to represent information about the same entity across multiple dimensions. A multidimensional vector consists of multiple sets of vectors, with at least two sets having different spatial dimensions. Multidimensional vectors effectively capture the structure of complex data and express the relationships between different dimensions. In databases, multidimensional vectors are frequently used to represent unstructured data, such as text, audio, images, and video.

[0003] When multiple vectors in a multidimensional vector represent different types of information, the spatial dimensions (or vector lengths) of these vectors may differ. When multiple vectors of varying lengths exist within a multidimensional vector, they cannot be treated as a single vector for searching; each dimension must be searched separately. This hinders fast searching of multidimensional vectors in a database. Previous search methods standardized multidimensional vectors of different lengths to the same length, for example, by truncating or padding with zeros. However, these methods often result in the loss of information from the original multidimensional vectors, especially when there are significant differences in vector lengths in the original data, leading to inaccurate vectors found. Inaccurate vectors hinder subsequent analysis and statistical work. For example, when a multidimensional vector represents text, audio, or image information from a video, inaccurate searching will prevent the rapid discovery of identical or similar videos, making functions such as deduplication impossible. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method for quickly searching multidimensional vectors in a database, comprising: performing sparse and bucketing processing on each multidimensional vector in the database, wherein the multidimensional vector includes X groups of vectors and at least two groups of vectors have different spatial dimensions; the sparse and bucketing processing includes: dividing each group of vectors in the multidimensional vector into M equal segments to obtain M segments of multidimensional sub-vectors; clustering each segment of multidimensional sub-vectors into N clusters, and obtaining an M×N×X sparse matrix based on each segment of multidimensional sub-vectors and the N clusters; mapping the M×N×X sparse matrix using a hash cluster numbering function to obtain an M×D×X signature matrix; slicing the M×D×X signature matrix into M segments of D×X signature matrices; mapping and bucketing the D X×1 cluster center vectors in each signature matrix to obtain M bucket sets, each bucket set including 2 XThe system uses three buckets: When a multidimensional vector to be queried is received, it is sparsely and bucketed to obtain M segments of D×X signature matrices to be queried; at least D X×1 cluster center vectors from one segment of the M segments of the D×X signature matrix are selected, and one or more buckets mapped to the corresponding bucket set by the cluster center vectors are identified as the hit buckets; one or more target multidimensional vectors are quickly found based on the cluster center vectors in the hit buckets and the multidimensional vector to be queried.

[0005] Preferably, the distance between the multidimensional sub-vector and the corresponding N cluster center vectors is calculated as follows: if the distance between them is less than or equal to the distance threshold, it is counted as 1; otherwise, it is counted as 0, resulting in a sparse matrix of M×N×X.

[0006] Preferably, the method of quickly finding one or more target multidimensional vectors based on the cluster center vectors in the hit buckets and the multidimensional vector to be queried includes: obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; calculating the similarity between the multidimensional vector to be queried and each original multidimensional vector in the corresponding dimension, calculating the similarity of each dimension, and calculating the average similarity; sorting the multiple original multidimensional vectors in descending order according to their average similarity; and selecting one or more original multidimensional vectors with the highest sorted value as the target multidimensional vectors.

[0007] Preferably, the method of quickly finding one or more target multidimensional vectors based on the cluster center vectors in the hit buckets and the multidimensional vector to be queried includes: obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; calculating the similarity between the multidimensional vector to be queried and each original multidimensional vector in the corresponding dimension, assigning different weights to each dimension similarity to calculate a weighted average similarity; sorting the multiple original multidimensional vectors in descending order according to the weighted average similarity; and selecting one or more original multidimensional vectors with the highest sorted value as the target multidimensional vector.

[0008] Preferably, the method of quickly finding one or more target multidimensional vectors based on the cluster center vectors in the hit buckets and the multidimensional vector to be queried includes: determining the hit dimension according to the numbering information of the hit buckets; when the hit dimension meets the filtering conditions of the target multidimensional vector, obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; and determining one or more target multidimensional vectors based on the similarity between the multidimensional vector to be queried and the corresponding dimension of each original multidimensional vector.

[0009] Preferably, the method of quickly finding one or more target multidimensional vectors based on the cluster center vectors in the hit buckets and the multidimensional vector to be queried includes: obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; calculating the similarity between the multidimensional vector to be queried and the corresponding dimension of each original multidimensional vector; if the similarity between the multidimensional vector to be queried and the corresponding dimension of the original multidimensional vector is greater than the similarity threshold, it is considered as a hit of the corresponding dimension vector; and finding one or more original multidimensional vectors as target multidimensional vectors based on the hit status of one or more target dimensions of the multidimensional vector to be queried and the corresponding dimension of each original multidimensional vector.

[0010] Preferably, 0 ≤ X ≤ 6, and X is a positive integer.

[0011] According to another aspect of this application, an apparatus for fast searching of multi-dimensional vectors is proposed, comprising: a first processing module for performing sparse and bucketing processing on each multi-dimensional vector in a database, wherein the multi-dimensional vector includes X groups of vectors and at least two groups of vectors have different dimensions; the sparse and bucketing processing includes: dividing each group of vectors in the multi-dimensional vector into M equal segments to obtain M segments of multi-dimensional sub-vectors; clustering each segment of multi-dimensional sub-vectors into N clusters, and obtaining an M×N×X sparse matrix based on each segment of multi-dimensional sub-vectors and the N clusters; mapping and processing the X groups of vectors in the M×N×X sparse matrix using a hash cluster numbering function to obtain an M×D×X signature matrix; slicing the M×D×X signature matrix into M segments of D×X signature matrices; mapping and bucketing the D X×1 cluster center vectors in each signature matrix to obtain M bucket sets, each bucket set including 2 X The system comprises: a first processing module, a second processing module, and a third processing module. The first processing module performs sparse and bucketing processing on the multidimensional vector to be queried when a multidimensional vector to be queried is received, to obtain M segments of D×X signature matrices to be queried. The second processing module determines at least D X×1 cluster center vectors from the M segments of the signature matrix to be queried, and determines one or more buckets mapped to the corresponding bucket set by the cluster center vectors to be queried as the hit buckets. The third processing module searches for one or more target multidimensional vectors based on the cluster center vectors in the hit buckets and the multidimensional vector to be queried.

[0012] This application provides an electronic device, which includes a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the steps of the method described above.

[0013] This application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the method described above.

[0014] Using embodiments of this application, both the multidimensional vector to be queried and the multidimensional vectors in the database are subjected to sparsification and bucketing to obtain a signature matrix. The signature matrix has advantages such as having the same length, data sparsity, and no loss of original vector similarity. When multiple vectors in the signature matrix are mapped to the same bucket, it indicates that the multiple vectors have a high degree of similarity. By determining the hitting bucket through multiple vectors in the signature matrix to be queried, similar multidimensional vectors can be quickly found, and high similarity in the query can be ensured. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings of the embodiments of this application will be briefly described below.

[0016] Figure 1 This is a schematic diagram of the system architecture of an embodiment of this application.

[0017] Figure 2 This is a flowchart illustrating a method for quickly searching multidimensional vectors in a database according to an embodiment of this application.

[0018] Figure 3 This is a schematic diagram illustrating sparse processing of a multidimensional vector according to an embodiment of this application.

[0019] Figure 4 This is a schematic diagram illustrating the bucketing process of a multidimensional vector according to an embodiment of this application.

[0020] Figure 5 This is a structural block diagram of a device for quickly searching multidimensional vectors in a database, according to an embodiment of this application.

[0021] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0022] The principles and spirit of this application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided to make the principles and spirit of this application clearer and more thorough, enabling those skilled in the art to better understand and implement the principles and spirit of this application. The exemplary embodiments provided herein are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments described herein without inventive effort are within the scope of protection of this application.

[0023] Embodiments of this application relate to terminal devices and / or servers. Those skilled in the art will understand that embodiments of this application can be implemented as a system, apparatus, device, method, computer-readable storage medium, or computer program product. Therefore, this application can be specifically implemented in at least one of the following forms: entirely hardware, entirely software, or a combination of hardware and software. According to embodiments of this application, this application claims protection for a method, apparatus, electronic device, computer-readable storage medium, and computer program product for rapidly searching multidimensional vectors in a database. Figure 1 A schematic diagram of a system architecture according to an embodiment of this application is shown. For example... Figure 1 As shown, the system includes a terminal device 102 and a server 104. The terminal device 102 can include at least one of the following: a smartphone, tablet, laptop, desktop computer, smart TV, various wearable devices, augmented reality (AR) devices, virtual reality (VR) devices, etc. A client can be installed on the terminal device 102. For example, the client can be a client specifically designed to perform a particular function (such as an application app), or a client with multiple embedded application applets (with different functions), or a client logged in via a browser. Users can operate on the terminal device 102. For example, a user can open the client installed on the terminal device 102 and input commands through the client, or a user can open the browser installed on the terminal device 102 and input commands through the browser. After receiving the user's input command, the terminal device 102 sends a request message containing the command to the server 104. After receiving the request message, the server 104 performs the corresponding processing and then returns the processing result information to the terminal device 102. The user command is completed through a series of data processing and information interaction.

[0024] In this document, terms such as first, second, and third are used only to distinguish one entity (or operation) from another entity (or operation), and are not intended to require or imply any order or relationship between these entities (or operations).

[0025] Figure 2 This is a flowchart illustrating a method for quickly searching multidimensional vectors in a database according to an embodiment of this application. The method includes:

[0026] S101, perform sparse and bucketing processing on each multidimensional vector in the database. Each multidimensional vector contains X groups of vectors, with at least two groups having different spatial dimensions. The sparse and bucketing processing includes: dividing each group of vectors in the multidimensional vector into M equal segments to obtain M segments of multidimensional sub-vectors; clustering each segment of multidimensional sub-vectors into N clusters, and obtaining an M×N×X sparse matrix based on each segment of multidimensional sub-vectors and the N clusters; mapping the M×N×X sparse matrix using a hash cluster numbering function to obtain an M×D×X signature matrix; slicing the M×D×X signature matrix into M segments of D×X signature matrices; mapping and bucketing the D X×1 cluster center vectors in each signature matrix to obtain M bucket sets, each bucket set containing 2... X Individual buckets;

[0027] S102, when the multidimensional vector to be queried is received, the multidimensional vector to be queried is subjected to sparse and bucket processing to obtain M segments of D×X signature matrix to be queried;

[0028] S103, select at least D X×1 cluster center vectors from the M segments of the D×X signature matrix to be queried, and determine one or more buckets that the cluster center vectors to be queried are mapped to in the corresponding bucket set as the hit buckets.

[0029] S104 quickly finds one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried.

[0030] First, the multidimensional vectors in the database need to undergo sparsification and bucketing. The purpose of sparsification is to transform multidimensional vectors of different lengths into vectors of equal length without losing the similarity to the original vectors. Specifically, clustering algorithms (such as K-means clustering, hierarchical clustering, etc.) are used to divide each multidimensional sub-vector into N clusters. An M×N×X sparse matrix is ​​generated based on the similarity between each multidimensional sub-vector and the N clusters, where 0 ≤ X ≤ 6, and X is a positive integer. The information in the M×N×X sparse matrix only expresses the positional relationship with similar clusters, thus preserving the characteristic structure of the original vector representation while reducing storage requirements and computational overhead. A hash cluster numbering function is used to map the M×N×X sparse matrix to a smaller range, resulting in a smaller M×D×X signature matrix, where the value of D is much smaller than the value of N. Using the hash cluster numbering function allows for a more compact representation of a high-dimensional, multi-row sparse matrix while retaining sufficient information to represent the data characteristics of the sparse matrix.

[0031] In each signature matrix, the D X×1 cluster center vectors are mapped to one or more buckets based on similarity. Mapping to the same bucket indicates high similarity, facilitating subsequent searching for similar vectors. This application determines the number of buckets to be 2 based on the X dimensions of the multidimensional vector. X This method can cover all cases where one or more dimensions of dimension X are selected. By numbering the buckets (corresponding to the selected dimensions), it is possible to quickly determine which dimensions the query vector is similar to the vectors within that bucket. For example, all vectors in bucket [000001] are similar to the vectors in the first dimension. When the query vector matches the bucket numbered [000001], it can be quickly determined that the query vector is similar to the vector in the first dimension of the vectors within bucket [000001], facilitating subsequent analysis and statistical analysis of similarity.

[0032] Similarly, the multidimensional vector to be queried is also subjected to sparse and bucketing processing to obtain M segments of D×X signature matrices to be queried. Then, at least one segment of the signature matrix is ​​used to determine one or more buckets that map to the D X×1 cluster center vectors of the query cluster in the corresponding bucket set as the hitting buckets, thereby quickly finding one or more target multidimensional vectors. By performing sparse and bucketing processing on both the multidimensional vectors in the database and the query vectors, similar target multidimensional vectors can be found, which can ensure the accuracy of the query, shorten the search time, and improve the query efficiency.

[0033] According to an embodiment of this application, the distance between each multidimensional sub-vector and the corresponding N cluster center vectors is calculated. If the distance between them is less than or equal to a distance threshold, it is recorded as 1; otherwise, it is recorded as 0, resulting in an M×N×X sparse matrix. The threshold method can be used to determine which specific cluster among the N clusters the multidimensional sub-vector is similar to. If the multidimensional sub-vector is similar to a certain cluster, the corresponding position is filled with 1; otherwise, it is filled with 0. In this way, the M×N×X sparse matrix records the position information of similar clusters.

[0034] According to embodiments of this application, quickly finding one or more target multidimensional vectors based on cluster center vectors in the hit buckets and the multidimensional vector to be queried includes: obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; calculating the similarity between the multidimensional vector to be queried and each original multidimensional vector in corresponding dimensions, calculating the similarity for each dimension, and calculating the average similarity; sorting the multiple original multidimensional vectors in descending order according to their average similarity; and selecting the one or more original multidimensional vectors with the highest sorted values ​​as target multidimensional vectors. By first determining the hit buckets, the search range can be narrowed, and one or more cluster center vectors that are relatively similar can be filtered out. Then, by calculating the similarity with the cluster center vectors in each dimension, the one or more most similar target multidimensional vectors can be determined by calculating the average similarity.

[0035] According to embodiments of this application, quickly finding one or more target multidimensional vectors based on cluster center vectors in hit buckets and query multidimensional vectors includes: obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; calculating the similarity between the query multidimensional vector and each original multidimensional vector in corresponding dimensions, assigning different weights to each calculated dimension similarity to calculate a weighted average similarity; sorting the multiple original multidimensional vectors in descending order according to the weighted average similarity; and selecting one or more original multidimensional vectors with the highest sorted values ​​as target multidimensional vectors. Since the importance of each dimension vector in a multidimensional vector is different, assigning different weights to different dimension vectors ensures fair calculation across dimensions. This also avoids the problem of reduced search accuracy due to excessively high similarity in one dimension affecting the overall similarity.

[0036] According to embodiments of this application, quickly finding one or more target multidimensional vectors based on cluster center vectors in hit buckets and the multidimensional vector to be queried includes: determining the hit dimension based on the numbering information of the hit buckets; when the hit dimension meets the filtering conditions of the target multidimensional vectors, obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; and determining one or more target multidimensional vectors based on the similarity between the multidimensional vector to be queried and the corresponding dimension of each original multidimensional vector. The filtering conditions include: whether there exists a target multidimensional vector similar to the vector to be queried in one or more target dimensions.

[0037] The number of buckets in this application is 2. X At that time, 2 X Each bucket can cover all cases where one or more dimensions are distributed within the same bucket. The bucket number indicates which specific dimensions (one or more) the vector within that bucket is similar to. For example, if the current filtering condition is: does a vector exist that is similar to the query vector in dimensions 1 and 4? Then, we can search the bucket numbers for information containing vectors in dimensions 1 and 4. If so, we search for one or more target multidimensional vectors in the corresponding bucket. Using bucket numbers to match filtering conditions allows for quick determination of whether a matching target multidimensional vector exists, thus shortening the search time.

[0038] According to embodiments of this application, quickly finding one or more target multidimensional vectors based on the cluster center vectors in the hit buckets and the multidimensional vector to be queried includes: obtaining the original multidimensional vector corresponding to each cluster center vector in the hit buckets; calculating the similarity between the multidimensional vector to be queried and the corresponding dimension of each original multidimensional vector; if the similarity between the multidimensional vector to be queried and the corresponding dimension of the original multidimensional vector is greater than a similarity threshold, it is considered a hit vector of the corresponding dimension; finding one or more original multidimensional vectors as target multidimensional vectors based on the hit situation of one or more target dimensions of the multidimensional vector to be queried and the corresponding dimension of each original multidimensional vector. By statistically analyzing the hit situation of the multidimensional vector to be queried and the corresponding dimension of the original multidimensional vector, it can be applied to multiple aspects of the search function, such as determining the type of multidimensional vector by targeting the similarity of one or more target dimensions of the multidimensional vector to be queried.

[0039] The implementation methods and advantages of the embodiments of this application have been described above through multiple examples. The specific processing procedures of the embodiments of this application are described in detail below with reference to specific examples.

[0040] A sample multidimensional vector is a group of vectors with not all of the same length. For example, a multidimensional vector can be represented as follows:

[0041] S=[b k1 b k2 b k3 b k4 b k5 b k6 ]6

[0042] Where b is usually an even number, such as 2. 6 k1, k2, k3, k4, k5, k6 are not completely identical, and each sample has b. k1 b k2 b k3 b k4 b k5 b k6 A vector of dimension b, defined as a dense vector. k1 b k2 b k3 b k4 b k5 b k6 A vector that can be used to represent a student's scores in Chinese, Mathematics, English, Physical Education, Art, and Music, or b k1 b k2 b k3 b k4 b k5 b k6This is a data vector that can be used to represent the title, audio, person image, face image, and background image of a video. It vectorizes the multi-dimensional information of a sample and forms a multi-dimensional vector, facilitating fast lookup using vector similarity. The method for fast multi-dimensional vector lookup requires sparse and bucketed processing of the multi-dimensional vector of each sample in the database. The specific processing steps are as follows:

[0043] (1) Sparsity treatment

[0044] Figure 3 This is a schematic diagram illustrating sparse processing of multidimensional vectors according to embodiments of this application. For example... Figure 3 As shown, the sample multidimensional vector 301 is a vector group comprising X (e.g., X is 6) groups, all of which are dense vectors. First, the vectors of each group are divided into M segments (e.g., M is 4) to obtain segmented vectors 302. Each segmented vector 302 is then clustered into N (e.g., N is 256) clusters 303. Each cluster 303 has one cluster center vector, resulting in M×N (e.g., 4×256) cluster center vectors. Therefore, the X (e.g., X is 6) groups have a total of M×N×X (4×256×6) clusters and M×N×X (4×256×6) cluster center vectors.

[0045] Each sample in the training set has an M×N×X (4×256×6) cluster center. The original vector of each sample is compared with the cluster center vector (the distance between the original vector and the cluster center vector is calculated). If the distance is less than or equal to a neighborhood threshold, it is considered to occupy the cluster (value 1); if the distance is greater than the neighborhood threshold, it is considered not to occupy the cluster (value 0). The original vector group then becomes an M×N×X (4×256×6) sparse vector cube 304, where all elements are 0 and 1, forming a sparse matrix. Through sparsity processing, the original 6-dimensional multi-dimensional vector is transformed into a uniform-length, sparse 4×256×6 vector cube, preserving the feature structure of the original vector representation while reducing storage requirements and computational overhead.

[0046] (2) Dividing into separate containers

[0047] Figure 4 This is a schematic diagram illustrating the bucketing process for multidimensional vectors according to an embodiment of this application. For example... Figure 4 As shown, the N (256) dimensions of the M×N×X (4×256×6) sparse vector cube 304 are denoted as row numbers, with row numbers ranging from 0 to 255. The X (6) dimension vectors are hashed using the hash row number method, resulting in D (e.g., 20) function row numbers, which in turn yields the M×D×X (4×20×6) signature matrix 305. This mapping does not change the similarity of the original vectors.

[0048] The M×D×X (4×20×6) signature matrix 305 is processed by extracting 6 dimensions along the X-axis (Z-axis direction) to form a new X×1 sample cluster center vector 306, resulting in a total of M×D (4×20) new vectors. These new vectors are then sliced ​​to obtain the D×X signature matrices for segments d1, d2, d3, and d4. Each sample in the database undergoes this process to obtain the D×X signature matrices for segments d1, d2, d3, and d4. The corresponding segments of the D×X signature matrices for all samples are then bucketed to determine the bucketing. Bucket numbers can be assigned using MD5 or other hash methods. If the X×1 sample cluster center vectors of two samples in the same segment d1 belong to the same bucket, the two vectors are similar. If they belong to different buckets, the corresponding parts of the d1 segment are dissimilar.

[0049] This application extends the above bucketing to multiple bucketing methods, selecting one or more groups of vectors from X groups for bucketing, one of which has 2... X There are 2 such cases X Each bucket. Taking X as 6 as an example, the specific details of the bucket expansion are as follows:

[0050] If you choose one dimension for bucketing... =6 scenarios, choose 2 dimensions for bucketing. =15 scenarios, choosing 3 dimensions for bucketing =20 scenarios, choosing 4 dimensions for bucketing =15 cases, choosing 6 dimensions for bucketing =6 scenarios, choose 6 dimensions for bucketing. =1 case, choose 0 dimensions for bucketing. =1 case (this type is a special bucket, generally with no value). A total of 2 6 Each bin is subdivided. The calculation process is as follows:

[0051]

[0052] The 64 buckets mentioned above are numbered, and the numbers are associated with the selected dimensions. For example:

[0053] For example, bucket0=[000000] means that the first dimension to the sixth degree is 0, or it is considered that... =() of the buckets.

[0054] bucket1=[000001] indicates that the first dimension is 1 and the rest are 0, or that... =(1) buckets.

[0055] Bucket2=[000010] indicates that the second dimension is 1, and the rest are 0, or it is considered that... =(2) buckets.

[0056] Bucket3=[000011] indicates that the first and second dimensions are 1, and the rest are 0, or it is considered that... = (1,2) buckets.

[0057] ...

[0058] Bucket63=[111111] means that all dimensions from the first to the sixth are 1, or that... = (1,2,3,4,5,6) buckets.

[0059] The bucket number 1 indicates that the multidimensional vector within the bucket has selected a vector corresponding to the specified position dimension, while 0 indicates that no vector has been selected for the specified position dimension. Each segment has 64 types of extended buckets. When divided into 4 segments (M = 4), there are a total of 64 × 4 = 256 buckets. For each type of bucket, a segment of the sample will fall into a region within that type of bucket. Therefore, generally, only one segment of the query sample is needed to find similar vectors.

[0060] (3) Finding multidimensional vectors

[0061] Taking the representation of six dimensions of video information using multidimensional vectors as an example, the database stores multidimensional vectors representing these six dimensions. Each multidimensional vector in the database undergoes sparsity and bucketing processing. Similarly, for a video sample to be queried, a group of vectors in the six-dimensional space is extracted. After being divided into blocks, the distance difference between each block's vector and the cluster center vector is calculated. A threshold of 1 is set if the threshold is met, and 0 otherwise, resulting in a sparse vector group of "0101..." to be queried. At this point, the sample vectors become a 256×4×6 sparse vector cube (256 clusters, 4 blocks, 6-dimensional space). For each sparse vector group to be queried, the row numbers are hashed d times (e.g., d = 20), transforming each block into a 20-row vector group. The query signature matrix is ​​20×4×6. Let D = 20, band = 4, and each band have 20 rows.

[0062] Extract 20 6×1 six-dimensional vectors from one of the four segments of the signature matrix to be queried, and search for the hit status of the 20 6×1 six-dimensional vectors in the corresponding 64 buckets. When the 20 rows of six-dimensional vectors of a segment fall into any of the 64 buckets, search for the hit bucket region, and quickly find one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried.

[0063] Based on the hit bucket, the corresponding bucket number can be obtained. For example, if Bucket[001001]=Bucket9 and Bucket[111100]=Bucket9 are hit, the bucket number can be obtained. 60 Bucket[011110]=Bucket 30 Bucket[001101]=Bucket 13 Bucket[001000]=Bucket 8。

[0064] A total of 5 buckets with the correct numbers were matched. The statistics of the 5 matched buckets and their corresponding dimension numbers are shown in Table 1.

[0065] Table 1

[0066]

[0067] In Table 1, a value of 1 for the corresponding dimension indicates that the corresponding dimension is matched, meaning the corresponding dimension is similar; 0 indicates that the corresponding dimension is dissimilar. `Sum Of Bucket` counts the number of similar elements in the corresponding dimension across all matched buckets, and `SumOfBucket_filter` filters the matched buckets based on specific conditions. This allows for the rapid retrieval of one or more target multidimensional vectors based on the cluster center vector within the matched bucket and the multidimensional vector to be queried. Depending on the filtering conditions, there are various search methods; please refer to the following for details:

[0068] Method 1:

[0069] The search is performed using average similarity. The original multidimensional vector corresponding to the cluster center vector in each hit bucket is obtained. Then, the similarity between the query multidimensional vector and each original multidimensional vector in corresponding dimensions is calculated. The average similarity is calculated for each dimension, and then one or more original multidimensional vectors are found as the target multidimensional vector based on the magnitude of the average similarity.

[0070] Method 2:

[0071] The search is performed using weighted average similarity. Weighted average similarity is used to find one or more target multi-dimensional vectors. For example, a student has six subjects (Chinese, Math, English, Physical Education, Art, and Music). Most students score 100 in Physical Education. If the scores are calculated with equal weights, Physical Education scores will inflate the overall student score. Therefore, Physical Education scores can be weighted less before calculation. Specifically, as shown in Table 1 (Sum of Bucket), the coefficients 'a' for the corresponding dimensions (first to sixth dimensions) are 2, 1, 3, 5, 2, and 1, respectively. The weight of the corresponding dimension vector can be calculated using the following formula based on the coefficients 'a' for each dimension.

[0072] (1)

[0073] (2)

[0074] Among them, w i (a) represents the weight of the vector in the i-th dimension, c is a statistical value, and a is the coefficient corresponding to the i-th dimension.

[0075] By using the coefficients of each dimension obtained from statistics, the weights of each dimension vector can be calculated through formulas (1) and (2). The weight value of each dimension is set, and the overall similarity ranking score is calculated again to achieve fair calculation of each dimension, so that the target multidimensional vector found is more accurate.

[0076] Method 3:

[0077] Searching is done using bin number information. For example, the filter condition is: to find multidimensional vectors that are similar in all dimensions 1-6. If the bin number containing dimensions 1-6 is 63, then according to Table 1, no bin number 63 is matched, meaning there are no multidimensional vectors in the database that meet the filter condition. Using filter conditions allows for quick searching of multidimensional vectors that are similar in the target dimension, enabling targeted searches for specific multidimensional vectors to achieve various functions. For example, multidimensional vectors are used to represent video titles, audio, images of people, faces, and backgrounds. Figure 6 A vector with six dimensions can be used to quickly identify a video as a pirated video if a vector with all six dimensions is identical in the video library. This allows for the prohibition of the video's publication, protecting the legitimate rights of the original creator and encouraging original videos.

[0078] Method 4:

[0079] Search by setting filtering criteria. For example, a search can be conducted if the third dimension is different, while the others are similar. 60 Bucket 30 Bucket 13 After removing the remainder and summing, we can see that the fourth dimension still has the most similar samples, while the rest are from the first dimension, as shown in Table 1, SumOfBucket_filter.

[0080] Figure 5 This is a structural block diagram of a device for quickly searching multidimensional vectors in a database, according to an embodiment of this application. Figure 5 As shown, the method for quickly searching for multidimensional vector devices in a database includes:

[0081] The first processing module 110 is used to perform sparse and bucketing processing on each multidimensional vector in the database. The multidimensional vector includes X groups of vectors, and at least two groups of vectors have different dimensions. The sparse and bucketing processing includes: dividing each group of vectors in the multidimensional vector into M equal segments to obtain M segments of multidimensional sub-vectors; clustering each segment of multidimensional sub-vectors into N clusters, and obtaining an M×N×X sparse matrix based on each segment of multidimensional sub-vectors and the N clusters; mapping and processing the X groups of vectors in the M×N×X sparse matrix using a hash cluster numbering function to obtain an M×D×X signature matrix; slicing the M×D×X signature matrix into M segments of D×X signature matrices; mapping and bucketing the D X×1 cluster center vectors in each signature matrix to obtain M bucket sets, each bucket set including 2 X Individual buckets;

[0082] The second processing module 120 is used to perform sparse and bucketing processing on the multidimensional vector to be queried when it receives the multidimensional vector to be queried to obtain an M-segment D×X signature matrix to be queried.

[0083] The determination module 130 is used to select at least D X×1 cluster center vectors from at least one segment of the query signature matrix in the M segment D×X query signature matrix, and determine one or more buckets that the query cluster center vectors are mapped to in the corresponding bucket set as the hit buckets.

[0084] The search module 140 is used to quickly search for one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried.

[0085] The electronic device in this application embodiment may be a user terminal device, a server, other computing devices, or a cloud server. Figure 6 The diagram illustrates the hardware structure of an electronic device according to an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing computer program instructions. When the processor 601 executes the computer program instructions, it implements the process or function of any of the methods described above.

[0086] Specifically, processor 601 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. Memory 602 may include a mass storage device for data or instructions. For example, memory 602 may be at least one of the following: a hard disk drive (HDD), read-only memory (ROM), random access memory (RAM), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, universal serial bus (USB) drive, or other physical / tangible memory storage device. Alternatively, memory 602 may include removable or non-removable (or fixed) media. Furthermore, memory 602 may be internal or external to the integrated gateway disaster recovery device. Memory 602 may be non-volatile solid-state memory. In other words, typically memory 602 includes a tangible (non-transitory) computer-readable storage medium (such as a memory device) encoded with computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in the methods of the embodiments of this application. The processor 601 implements the process or function of any of the methods described in the above embodiments by reading and executing computer program instructions stored in the memory 602.

[0087] In one example Figure 6 The illustrated electronic device may also include a communication interface 603 and a bus 610. The processor 601, memory 602, and communication interface 603 are connected via bus 610 and communicate with each other. Communication interface 603 is primarily used to enable communication between modules, devices, units, and / or equipment in the embodiments of this application. Bus 610 may include hardware, software, or both, and can couple components of the online data traffic billing device together. For example, the bus may include at least one of the following: Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), HyperTransport (HT) Interconnect, Industry Standard Architecture (ISA) bus, Infinite Bandwidth Interconnect, Low Pin Count (LPC) bus, memory bus, Microchannel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus, or other suitable buses. Bus 610 may include one or more buses. Although specific buses are described or illustrated in the embodiments of this application, any suitable bus or interconnection method may be considered in the embodiments of this application.

[0088] In conjunction with the methods in the above embodiments, this application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the process or function of any of the methods in the above embodiments.

[0089] In addition, this application also provides a computer program product that stores computer program instructions, which, when executed by a processor, implement the process or function of any of the methods described above.

[0090] The flowcharts and / or block diagrams of methods, apparatuses, systems, and computer program products according to embodiments of this application have been exemplarily described above, and related aspects have been described. It should be understood that each block or combination thereof in the flowcharts and / or block diagrams may be implemented by computer program instructions, by dedicated hardware performing a specified function or action, or by a combination of dedicated hardware and computer instructions. For example, these computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to form a machine that enables the implementation of the function / action specified in each block or combination thereof in the flowcharts and / or block diagrams, executable via such processor. Such a processor may be a general-purpose processor, a dedicated processor, a special-purpose application processor, or a field-programmable logic circuit.

[0091] The functional blocks shown in the structural block diagrams of this application can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc.; when implemented in software, they are programs or code segments used to perform the required tasks. Programs or code segments can be stored in memory or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. Code segments can be downloaded via computer networks such as the Internet or intranets.

[0092] It should be noted that this application is not limited to the specific configurations and processes described above or shown in the figures. The above descriptions are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the described systems, devices, modules, or units can be referred to the corresponding processes in the method embodiments, and need not be repeated here. It should be understood that the scope of protection of this application is not limited thereto. Any person skilled in the art can conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application.

Claims

1. A method for quickly searching multidimensional vectors in a database, characterized in that, include: Each multidimensional vector in the database is subjected to sparse and bucket processing, wherein the multidimensional vector includes X groups of vectors and at least two groups of vectors have different spatial dimensions. The sparse and bucketing process includes: Divide each group of vectors in the multidimensional vector into M equal segments to obtain M multidimensional sub-vectors; Each multidimensional sub-vector is clustered into N clusters, and an M×N×X sparse matrix is ​​obtained based on each multidimensional sub-vector and the N clusters. The M×N×X sparse matrix is ​​mapped using a hash clustering numbering function to obtain an M×D×X signature matrix. Slice the M×D×X signature matrix into M segments of the D×X signature matrix; Each signature matrix's D X×1 cluster center vectors are mapped and bucketed, resulting in M ​​bucket sets. Each bucket set contains 2... X Individual buckets; When a multidimensional vector to be queried is received, the multidimensional vector to be queried is subjected to sparse and bucket processing to obtain an M-segment D×X signature matrix to be queried. In the M segments of the D×X signature matrix to be queried, select at least D X×1 cluster center vectors from the signature matrix to be queried, and determine one or more buckets that the cluster center vectors to be queried are mapped to in the corresponding bucket set as the hit buckets. Quickly find one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried.

2. The method according to claim 1, characterized in that, Further includes: Calculate the distance between each multidimensional subvector and its corresponding N cluster center vectors. If the distance between them is less than or equal to the distance threshold, it is counted as 1; otherwise, it is counted as 0. This yields an M×N×X sparse matrix.

3. The method according to claim 1, characterized in that, The method of quickly finding one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried includes: Obtain the original multidimensional vector corresponding to the cluster center vector of each hit bucket; Calculate the similarity between the query multidimensional vector and each original multidimensional vector in the corresponding dimension, calculate the similarity for each dimension, and calculate the average similarity. Sort the multiple original multidimensional vectors in descending order based on their average similarity; and Use one or more of the top-ranked original multidimensional vectors as the target multidimensional vector.

4. The method according to claim 1, characterized in that, The method of quickly finding one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried includes: Obtain the original multidimensional vector corresponding to the cluster center vector of each hit bucket; Calculate the similarity between the query multidimensional vector and each original multidimensional vector in the corresponding dimension, and assign different weights to the calculated similarity of each dimension to calculate the weighted average similarity. Sort the multiple original multidimensional vectors in descending order according to their weighted average similarity; and Use one or more of the top-ranked original multidimensional vectors as the target multidimensional vector.

5. The method according to claim 1, characterized in that, The method of quickly finding one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried includes: The hit dimension is determined based on the numbering information of the hit buckets; When the hit dimension meets the filtering conditions of the target multidimensional vector, the original multidimensional vector corresponding to each cluster center vector in the hit bucket is obtained; One or more target multidimensional vectors are determined based on the similarity between the query multidimensional vector and the corresponding dimension of each original multidimensional vector.

6. The method according to claim 1, characterized in that, The method of quickly finding one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried includes: Obtain the original multidimensional vector corresponding to the cluster center vector of each hit bucket; Calculate the similarity between the query multidimensional vector and the corresponding dimension of each original multidimensional vector; If the similarity between the multidimensional vector to be queried and the corresponding dimension of the original multidimensional vector is greater than the similarity threshold, it is considered that the corresponding dimension vector has been matched. One or more original multidimensional vectors are found as target multidimensional vectors based on the match between one or more target dimensions of the query multidimensional vector and the corresponding dimensions of each original multidimensional vector.

7. The method according to claim 1, characterized in that, in, 0 ≤ X ≤ 6, and X is a positive integer.

8. A device for quickly finding multi-dimensional vectors, characterized in that, include: The first processing module is used to perform sparse and bucketing processing on each multidimensional vector in the database. The multidimensional vector includes X groups of vectors, with at least two groups having different dimensions. The sparse and bucketing processing includes: dividing each group of vectors in the multidimensional vector into M equal segments to obtain M segments of multidimensional sub-vectors; clustering each segment of multidimensional sub-vectors into N clusters, and obtaining an M×N×X sparse matrix based on each segment of multidimensional sub-vectors and the N clusters; mapping the X groups of vectors in the M×N×X sparse matrix using a hash cluster numbering function to obtain an M×D×X signature matrix; slicing the M×D×X signature matrix into M segments of D×X signature matrices; mapping and bucketing the D X×1 cluster center vectors in each signature matrix to obtain M bucket sets, each bucket set including 2... X Individual buckets; The second processing module is used to perform sparse and bucketing processing on the multidimensional vector to be queried when it receives the multidimensional vector to be queried to obtain an M-segment D×X signature matrix to be queried. The determination module is used to select at least D X×1 cluster center vectors from at least one segment of the query signature matrix in the M segment D×X query signature matrix, and determine one or more buckets that the query cluster center vectors are mapped to in the corresponding bucket set as the hit buckets. The search module is used to quickly find one or more target multidimensional vectors based on the cluster center vector in the hit bucket and the multidimensional vector to be queried.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory storing computer program instructions; when the electronic device executes the computer program instructions, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images