Method and device for large-scale parallel processing of database

Abstract

Embodiments of this specification provide a method and apparatus for large-scale parallel database processing, wherein the method comprises: receiving a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched; using an approximate nearest neighbor search method to determine at least one search feature vector corresponding to the vector to be searched in a pre-generated feature vector index table, and obtaining a search computation node corresponding to each search feature vector, wherein the feature vector index table records multiple feature vectors and the computation node corresponding to each feature vector; searching for an approximate vector corresponding to the vector to be searched in each search computation node; and determining a target vector corresponding to the vector to be searched from at least one approximate vector. This method significantly reduces computational overhead while ensuring search accuracy.

Description

Technical Field

[0001] Embodiments of this specification relate to the field of computer technology, and more particularly to a method for large-scale parallel database processing. One or more embodiments of this specification also relate to a method for large-scale parallel database processing, two apparatuses for large-scale parallel database processing, two computing devices, and a computer-readable storage medium. Background Art

[0002] With the development of computer technology, distributed systems have also developed rapidly. The MPP architecture distributes tasks in parallel to multiple servers and computing nodes. After the calculation is completed on each computing node, the results of each part are aggregated to obtain the final result. The database that adopts the MPP architecture is called an MPP database.

[0003] Due to the rapid growth of current technology and the order of magnitude, traditional MPP databases are unable to meet the low-latency and high-efficiency requirements of current projects. For example, in face recognition projects, the project system will extract feature vectors from facial images captured by cameras, and then perform approximate nearest neighbor searches in the vector database stored in the MPP database. However, the database stores billions of data, and the search process is very long, which greatly consumes computer resources and is very inconvenient.

[0004] Therefore, how to solve the above problems has become a problem that needs to be solved urgently by technical personnel. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a method for large-scale parallel database processing. One or more embodiments of this specification also relate to a method for large-scale parallel database processing, two apparatuses for large-scale parallel database processing, two computing devices, and a computer-readable storage medium to address technical deficiencies in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a method for large-scale parallel processing of a database is provided, comprising:

[0007] Receiving a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched;

[0008] Determine at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval computation node corresponding to each retrieval feature vector, wherein the feature vector index table records a plurality of feature vectors and a computation node corresponding to each feature vector;

[0009] Retrieving an approximate vector corresponding to the vector to be retrieved in each of the retrieval calculation nodes;

[0010] A target vector corresponding to the vector to be retrieved is determined from at least one of the approximate vectors.

[0011] Optionally, determining at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method includes:

[0012] Calculating the difference between the vector to be searched and each feature vector in the feature vector index table in sequence;

[0013] The feature vector whose difference value is less than a preset threshold is determined as a search feature vector corresponding to the vector to be searched.

[0014] Optionally, determining a target vector corresponding to the to-be-searched vector from at least one of the approximate vectors includes:

[0015] Calculating the difference between each of the approximate vectors and the vector to be retrieved;

[0016] The approximate vector with the smallest difference value is selected as the target vector corresponding to the vector to be retrieved.

[0017] Optionally, after determining the target vector corresponding to the vector to be searched from at least one of the approximate vectors, the method further includes:

[0018] Acquire attribute information corresponding to the target vector according to the target vector;

[0019] The attribute information corresponding to the target vector is used as the search result corresponding to the item data to be retrieved.

[0020] Optionally, the method further includes:

[0021] Receive deletion instructions;

[0022] The delete instruction is executed in each computing node of the massively parallel processing database.

[0023] Optionally, the feature vector index table is obtained by training through the following steps, including:

[0024] Acquire a training sample, wherein the training sample includes a plurality of training vectors corresponding to the project data;

[0025] Clustering multiple training vectors in the training sample;

[0026] Determine at least two eigenvectors according to the clustering result, and determine a computing node corresponding to each of the eigenvectors;

[0027] A feature vector index table is generated according to each feature vector and a computing node corresponding to each feature vector.

[0028] Optionally, determining a computing node corresponding to each of the eigenvectors includes:

[0029] The computing node corresponding to each of the feature vectors is determined according to the number of computing nodes in the large-scale parallel processing database.

[0030] According to a second aspect of an embodiment of this specification, a method for large-scale parallel processing of a database is provided, comprising:

[0031] receiving a write instruction, wherein the write instruction includes a vector to be written corresponding to the project data to be written;

[0032] Determine a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor search method, and obtain a write computing node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and a computing node corresponding to each feature vector;

[0033] The vector to be written is written into the write computing node.

[0034] Optionally, determining the write feature vector corresponding to the to-be-written vector in a pre-generated feature vector index table using an approximate nearest neighbor search method includes:

[0035] Calculating the difference between the vector to be written and each eigenvector in the eigenvector index table in sequence;

[0036] The eigenvector with the smallest difference value is determined as the written eigenvector corresponding to the vector to be written.

[0037] Optionally, the method further includes:

[0038] Receive deletion instructions;

[0039] The delete instruction is executed in each computing node of the massively parallel processing database.

[0040] Optionally, the method further includes:

[0041] Receiving a search instruction, wherein the search instruction includes a to-be-searched vector corresponding to the to-be-searched item data;

[0042] Determine at least one retrieval feature vector corresponding to the vector to be retrieved in the feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval calculation node corresponding to each retrieval feature vector;

[0043] Retrieving an approximate vector corresponding to the vector to be retrieved in each of the retrieval calculation nodes;

[0044] A target vector corresponding to the vector to be retrieved is determined from at least one of the approximate vectors.

[0045] Optionally, determining at least one retrieval feature vector corresponding to the vector to be retrieved in the feature vector index table using an approximate nearest neighbor retrieval method includes:

[0046] Calculating the difference between the vector to be searched and each feature vector in the feature vector index table in sequence;

[0047] The feature vector whose difference value is less than a preset threshold is determined as a search feature vector corresponding to the vector to be searched.

[0048] Optionally, determining a target vector corresponding to the to-be-searched vector from at least one of the approximate vectors includes:

[0049] Calculating the difference between each of the approximate vectors and the vector to be retrieved;

[0050] The approximate vector with the smallest difference value is selected as the target vector corresponding to the vector to be retrieved.

[0051] Optionally, after determining the target vector corresponding to the vector to be searched from at least one of the approximate vectors, the method further includes:

[0052] Acquire attribute information corresponding to the target vector according to the target vector;

[0053] The attribute information corresponding to the target vector is used as the search result corresponding to the item data to be retrieved.

[0054] Optionally, the feature vector index table is obtained by training through the following steps, including:

[0055] Acquire a training sample, wherein the training sample includes a plurality of training vectors corresponding to the project data;

[0056] Clustering multiple training vectors in the training sample;

[0057] Determine at least two eigenvectors according to the clustering result, and determine a computing node corresponding to each of the eigenvectors;

[0058] A feature vector index table is generated according to each feature vector and a computing node corresponding to each feature vector.

[0059] Optionally, determining a computing node corresponding to each of the eigenvectors includes:

[0060] The computing node corresponding to each of the feature vectors is determined according to the number of computing nodes in the large-scale parallel processing database.

[0061] According to a third aspect of an embodiment of this specification, there is provided an apparatus for large-scale parallel processing of a database, comprising:

[0062] A first receiving module is configured to receive a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched;

[0063] A first node determination module is configured to determine at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval computation node corresponding to each retrieval feature vector, wherein the feature vector index table records a plurality of feature vectors and a computation node corresponding to each feature vector;

[0064] A first retrieval vector module is configured to retrieve an approximate vector corresponding to the vector to be retrieved in each retrieval calculation node;

[0065] The first vector determination module is configured to determine a target vector corresponding to the vector to be retrieved from at least one of the approximate vectors.

[0066] According to a fourth aspect of the embodiments of this specification, there is provided an apparatus for large-scale parallel processing of a database, comprising:

[0067] A third receiving module is configured to receive a write instruction, wherein the write instruction includes a to-be-written vector corresponding to the to-be-written project data;

[0068] a second node determination module configured to determine a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor search method, and obtain a write calculation node corresponding to the write feature vector, wherein the feature vector index table records a plurality of feature vectors and a calculation node corresponding to each feature vector;

[0069] The writing module is configured to write the vector to be written into the writing computing node.

[0070] According to a fifth aspect of the embodiments of this specification, there is provided a computing device, including:

[0071] memory and processor;

[0072] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions:

[0073] Receiving a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched;

[0074] Determine at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval computation node corresponding to each retrieval feature vector, wherein the feature vector index table records a plurality of feature vectors and a computation node corresponding to each feature vector;

[0075] Retrieving an approximate vector corresponding to the vector to be retrieved in each of the retrieval calculation nodes;

[0076] A target vector corresponding to the vector to be retrieved is determined from at least one of the approximate vectors.

[0077] According to a sixth aspect of the embodiments of this specification, there is provided a computing device, including:

[0078] memory and processor;

[0079] The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions:

[0080] receiving a write instruction, wherein the write instruction includes a vector to be written corresponding to the project data to be written;

[0081] Determine a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor search method, and obtain a write computing node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and a computing node corresponding to each feature vector;

[0082] The vector to be written is written into the write computing node.

[0083] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the method for large-scale parallel processing of databases.

[0084] The method for large-scale parallel processing of databases provided in the embodiments of this specification, when performing a search in a large-scale parallel processing database, determines the search computing node that needs to be searched through a preset feature vector index table, and performs the search in the search computing node, and determines the target vector corresponding to the vector to be searched in at least one obtained approximate vector, which can ensure the accuracy of the target vector. Through the method provided in this specification, it is only necessary to search in the search computing node without searching the vector to be searched in all computing nodes. While ensuring the accuracy of the search, the time for each search is greatly reduced, and useless computing consumption is also greatly avoided, thereby improving the search capability and thus enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 is a processing flow chart of a method for large-scale parallel processing of a database provided in the first embodiment of this specification;

[0086] Figure 2 This is a flow chart of a training process of a feature vector index table provided in the second embodiment of this specification;

[0087] Figure 3 is a process flow chart of a method for large-scale parallel database processing provided in the third embodiment of this specification;

[0088] Figure 4a A schematic diagram of the structure of a large-scale parallel processing database provided in the third embodiment of this specification is shown;

[0089] Figure 4b FIG2 shows a schematic diagram of vector writing provided by the third embodiment of this specification;

[0090] Figure 4c A schematic diagram of vector retrieval provided in the third embodiment of this specification is shown;

[0091] Figure 4d A schematic diagram of vector deletion provided by the third embodiment of this specification is shown;

[0092] Figure 5 A schematic diagram of the structure of an apparatus for large-scale parallel processing of a database provided in an embodiment of this specification is shown;

[0093] Figure 6 A schematic diagram of the structure of an apparatus for large-scale parallel processing of a database provided by another embodiment of this specification is shown;

[0094] Figure 7 This is a structural block diagram of a computing device provided in an embodiment of this specification;

[0095] Figure 8 This is a structural block diagram of a computing device provided in another embodiment of this specification. DETAILED DESCRIPTION

[0096] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0097] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0098] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0099] First, the terms involved in one or more embodiments of this specification are explained.

[0100] Massively Parallel Processing (MPP) databases: Databases that use the MPP architecture are called MPP databases. The MPP architecture distributes tasks in parallel across multiple servers and computing nodes. After the calculation is completed on each computing node, the results of each part are aggregated to obtain the final result.

[0101] Partition pruning: Partition pruning (partition pruning) means that for partitioned tables or partitioned indexes, the optimizer can automatically extract the partitions to be accessed from the from and where columns based on the partition key, thereby avoiding accessing all partitions, reducing I / O requests, and lowering server CPU pressure.

[0102] Approximate Nearest Neighbor Search (ANN) quickly retrieves the nearest N adjacent vectors in a high-dimensional space through a pre-built index and can return approximately accurate results.

[0103] Clustering: The process of grouping a collection of physical or abstract objects into clusters of similar objects.

[0104] In this specification, a method for large-scale parallel processing of databases is provided. This specification also relates to a method for large-scale parallel processing of databases, two apparatuses for large-scale parallel processing of databases, two computing devices, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0105] Figure 1 A processing flow chart of a method for large-scale parallel processing of a database according to an embodiment of the present specification is shown, including steps 102 to 108.

[0106] Step 102: Receive a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched.

[0107] A search instruction is an instruction used to search in a database, and generally includes the data content to be searched, the search database, the search conditions, etc.

[0108] The project data to be retrieved is the project data used by the retrieval instruction. In different projects, the project data to be retrieved is also different. For example, in a face recognition project, the project data to be retrieved is the face image to be retrieved, and in a voice recognition project, the project data to be retrieved is the voice information to be retrieved.

[0109] The vector to be retrieved is a vector obtained after the feature vectorization processing of the project data to be retrieved. When the project data is a face image, the vector to be retrieved is a face feature vector obtained after the feature vectorization processing of the face image. When the project data is voice data, the vector to be retrieved is a voice feature vector obtained after the feature vectorization processing of the voice data.

[0110] In the first embodiment provided in this specification, taking speech recognition as an example, the project data to be retrieved is the speech information to be retrieved, and the character attribute information corresponding to the speech information is retrieved. The character attribute information can be a person's name, age, place of origin, education level, etc. The speech information is processed by feature vectorization to obtain a vector to be retrieved V0. The retrieval instruction is to retrieve the speech vector corresponding to the vector to be retrieved in a massively parallel processing database (MPP database) to obtain the retrieval instruction.

[0111] Step 104: Determine at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval computing node corresponding to each retrieval feature vector, wherein the feature vector index table records multiple feature vectors and a computing node corresponding to each feature vector.

[0112] Approximate Nearest Neighbor Search (ANN search) is a method for quickly searching for the N nearest neighboring vectors in a high-dimensional space using a pre-built index. A pre-generated feature vector index table is a representation of the pre-built index in ANN search. The feature vector index table records multiple feature vectors and the computational node corresponding to each feature vector. A feature vector is a vector used for retrieval in the feature vector index table. The ANN search method is used to retrieve N retrieval feature vectors corresponding to the vector to be retrieved from the feature vector index table. The retrieval computation node corresponding to each retrieval feature vector is then determined based on the correspondence between the feature vectors and the computational nodes. The retrieval feature vector is the feature vector in the feature vector index table that corresponds to the vector to be retrieved during the vector retrieval process, and the retrieval computation node is the computational node corresponding to the retrieval feature vector.

[0113] Optionally, an approximate nearest neighbor retrieval method is used to determine at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table, including: sequentially calculating the difference value between the vector to be retrieved and each feature vector in the feature vector index table; and determining that the feature vector whose difference value is less than a preset threshold is the retrieval feature vector corresponding to the vector to be retrieved.

[0114] The difference between the vector to be retrieved and each eigenvector in the eigenvector index table is calculated respectively. The difference value is used to identify the difference between the two vectors. The smaller the difference value, the more similar the two vectors are. The eigenvector with a difference value less than a preset threshold is used as the retrieval eigenvector corresponding to the vector to be retrieved.

[0115] When calculating the difference between two vectors, a distance algorithm such as Euclidean distance, Manhattan distance, Minkowski distance, etc. can be used, or a similarity algorithm such as cosine similarity, Pearson correlation coefficient, log-likelihood similarity, etc. can be used. This application does not limit the specific algorithm for calculating the difference between vectors.

[0116] In practical applications, the difference values ​​may be sorted to obtain a preset number N of retrieval feature vectors, and the first N feature vectors with the smallest difference values ​​may be obtained as retrieval feature vectors corresponding to the vector to be retrieved.

[0117] In the first embodiment provided in this specification, following the above example, the pre-generated feature vector index table is shown in Table 1 below.

[0118] Serial number Eigenvector Compute nodes 1 <![CDATA[V1]]> Compute node 1 2 <![CDATA[V2]]> Compute node 5 3 <![CDATA[V3]]> Compute node 11 …… …… …… N <![CDATA[V N ]]> Compute node 7

[0119] Table 1

[0120] Calculate the vector to be retrieved V0 and the eigenvectors V1, V2, V3...V in the eigenvector index table respectively NThe cosine similarity of the vectors is calculated. The larger the cosine value, the smaller the difference value of the vectors. Feature vectors V1, V2, and V3 with cosine values ​​greater than the preset threshold are selected as the retrieval feature vectors corresponding to the search vector V0. This can be determined through the feature vector index table. The retrieval calculation nodes corresponding to the search vector V0 are calculation nodes 1, 5, and 11 respectively.

[0121] Step 106: Retrieve an approximate vector corresponding to the vector to be retrieved in each of the retrieval calculation nodes.

[0122] The approximate vector is the vector stored in the computing node that has the smallest difference value from the vector to be searched. A search instruction is issued to each retrieval computing node, and each retrieval computing node executes the search instruction. The approximate vector corresponding to the vector to be searched is searched from the vectors stored in the current retrieval computing node. The vector with the smallest difference value from the vector to be searched is obtained in each retrieval computing node as the approximate vector corresponding to the vector to be searched in the current retrieval computing node.

[0123] In the first embodiment provided in this specification, following the above example, the approximate vector of the vector to be searched V0 in the computing node No. 1 is V 11 , the approximate vector of the search vector V0 in the computing node No. 5 is V 21 , the approximate vector of the search vector V0 in the computing node No. 11 is V 31 .

[0124] Step 108: Determine a target vector corresponding to the vector to be retrieved from at least one of the approximate vectors.

[0125] Optionally, determining the target vector corresponding to the vector to be retrieved from at least one of the approximate vectors includes: calculating the difference value between each of the approximate vectors and the vector to be retrieved; and selecting the approximate vector with the smallest difference value as the target vector corresponding to the vector to be retrieved.

[0126] The target vector is the vector in the massively parallel processing database that has the smallest difference value with the vector to be searched. After obtaining at least one approximate vector, the difference value between each approximate vector and the vector to be searched is calculated. Then, from the at least one approximate vector, the one with the smallest difference value with the vector to be searched is determined as the target vector corresponding to the vector to be searched.

[0127] In the first embodiment provided in this specification, the above example is used to calculate the approximate vector V 11 、V 21 、V 31 The cosine similarity with the vector to be retrieved V0, from V 11 、V 21 、V 31Determine the approximate vector V with the largest cosine similarity 21 is the target vector corresponding to the vector V0 to be retrieved.

[0128] Optionally, after determining the target vector corresponding to the vector to be retrieved from at least one of the approximate vectors, it also includes: obtaining attribute information corresponding to the target vector based on the target vector; and using the attribute information corresponding to the target vector as the retrieval result corresponding to the item data to be retrieved.

[0129] In practical applications, after determining the target vector, it is also necessary to obtain attribute information corresponding to the target vector, and use the attribute information corresponding to the target vector as the search result corresponding to the item data to be searched in the search instruction.

[0130] In the first embodiment provided in this specification, the target vector V is obtained by following the above example. 21 The corresponding attribute information such as name, age, place of origin, education background, etc. is used as the attribute information of the person corresponding to the vector V0 to be retrieved.

[0131] Optionally, the method further includes: receiving a deletion instruction; and executing the deletion instruction in each computing node of the massively parallel processing database.

[0132] In actual applications, when it is necessary to delete information in the MPP database, the received deletion instruction is sent to each computing node of the MPP database, and the deletion instruction is executed in each computing node.

[0133] The first embodiment of this specification provides a method for large-scale parallel processing of databases. When searching in a large-scale parallel processing database, the retrieval computing node that needs to be searched is determined through a preset feature vector index table, and there is no need to search in all computing nodes, thereby avoiding useless computing consumption and improving retrieval efficiency. Searching in the retrieval computing node and determining the target vector corresponding to the vector to be searched from at least one obtained approximate vector can ensure the accuracy of the target vector. Through the method provided in this specification, while ensuring the retrieval accuracy, the retrieval time is shortened, the retrieval efficiency is improved, and the user experience is enhanced.

[0134] The following combination Figure 2 , explaining the steps of training and obtaining the feature vector index table in the embodiment of this specification, Figure 2 A flow chart of a training process of a feature vector index table provided in one embodiment of this specification is shown, and specific steps include 202 to 208.

[0135] Step 202: Acquire a training sample, wherein the training sample includes a plurality of training vectors corresponding to the project data.

[0136] The training sample is the sample data for training to generate the feature vector index table, wherein the training sample includes multiple training vectors corresponding to the project data, and the project data is data related to the project. For example, in a face recognition project, the project data is a face image, and the multiple training vectors corresponding to the project data are the feature vectors corresponding to the multiple face images. In a speech recognition project, the project data is speech data, and the multiple training vectors corresponding to the project data are the feature vectors corresponding to the speech data.

[0137] In the second embodiment provided in this specification, taking speech recognition as an example, training data is obtained, wherein the training data includes training vectors corresponding to 100 pieces of speech data.

[0138] Step 204: Clustering the multiple training vectors in the training sample.

[0139] Clustering is the process of dividing a collection of physical or abstract objects into multiple classes consisting of similar objects. Clustering operations are performed on multiple training vectors using a clustering algorithm. There are many clustering algorithms, such as the k-means clustering algorithm (k-means) and the mean shift method. Clustering algorithms are not limited in this specification.

[0140] In the second embodiment provided in this specification, following the above example, 100 training vectors in the training sample are clustered using the k-means clustering algorithm.

[0141] Step 206: Determine at least two eigenvectors according to the clustering result, and determine the computing node corresponding to each of the eigenvectors.

[0142] Determine K sets based on the clustering results, take the cluster center vector in each set as the feature vector of the current set, and determine the computing node corresponding to each feature vector based on the number of computing nodes in the large-scale parallel processing database. There are many methods for determining the computing node corresponding to the feature vector, such as by encoding each feature vector and generating it modulo the encoding of the computing node, or by obtaining the optimization result through the expectation-maximization algorithm (EM algorithm). The method for determining the computing node corresponding to each feature vector is not limited in this specification.

[0143] In the second embodiment provided in this specification, following the above example, the clustering results determine 9 sets, and the cluster center vectors of each set are B1, B2, B3, B4, B5, B6, B7, B8, and B9 respectively. These 9 vectors are eigenvectors. In this embodiment, the number of computing nodes of the large-scale parallel processing database is 3. The computing node corresponding to the eigenvectors B1, B4, and B7 is determined to be computing node No. 1 by taking the modulo method, the computing node corresponding to the eigenvectors B2, B5, and B8 is computing node No. 2, and the computing node corresponding to the eigenvectors B3, B6, and B9 is computing node No. 3.

[0144] Step 208: Generate a feature vector index table according to each feature vector and the computing node corresponding to each feature vector.

[0145] Each eigenvector and the computing node corresponding to each eigenvector are stored in a one-to-one correspondence to generate a eigenvector index table.

[0146] In the second embodiment provided in this specification, the above example is used as is. Referring to Table 2 below, Table 2 shows a feature vector index table generated in this embodiment.

[0147]

[0148]

[0149] Table 2

[0150] The training step of the feature vector index table provided in the second embodiment of this specification can generate a feature vector index table through a small amount of training data, thereby realizing partitioning and pruning of a large-scale parallel processing database through vectors. This facilitates the determination of feature vectors with small difference values ​​based on the vectors to be processed during subsequent database operations, and processing them in the computing nodes corresponding to the feature vectors without accessing each computing node in the large-scale parallel processing database, thereby reducing IO requests, reducing the pressure on the processor, and improving processing efficiency.

[0151] The following combined Figure 3 , taking the application of a method for large-scale parallel processing of databases provided in this specification in the field of face recognition as an example, a method for large-scale parallel processing of databases is described. Figure 3 A flowchart of a processing process of a method for large-scale parallel processing of a database provided in the third embodiment of this specification is shown, and the specific steps include steps 302 to 306.

[0152] Step 302: Receive a write instruction, wherein the write instruction includes a vector to be written corresponding to the item data to be written.

[0153] The write instruction is an instruction for writing data in the database. The project data to be written is the project data corresponding to the write instruction. In the field of face recognition, the project data to be written is the face image to be written into the database, and the vector to be written is the vector obtained after the face image is feature vectorized.

[0154] In the third embodiment provided in this specification, the project data to be written is a face image, and the vector C0 to be written is a vector obtained after the face image is characterized.

[0155] Step 304: Use the approximate nearest neighbor retrieval method to determine the write feature vector corresponding to the vector to be written in the pre-generated feature vector index table, and obtain the write calculation node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and the calculation node corresponding to each feature vector.

[0156] The write feature vector is a feature vector in the feature vector index table corresponding to the vector to be written during the vector writing process. The write computing node is a computing node corresponding to the write feature vector.

[0157] Optionally, an approximate nearest neighbor retrieval method is used to determine a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table, including: calculating the difference value between the vector to be written and each feature vector in the feature vector index table in turn; and determining the feature vector with the smallest difference value as the write feature vector corresponding to the vector to be written.

[0158] Calculate the difference between the vector to be written and each eigenvector in the eigenvector index table respectively. The difference value is used to identify the difference between the two vectors. The smaller the difference value, the more similar the two vectors are. The eigenvector with the smallest difference value with the vector to be written is used as the write eigenvector corresponding to the vector to be written.

[0159] In the third embodiment provided in this specification, the pre-generated feature vector index table is shown in Table 3 below.

[0160]

[0161]

[0162] Table 3

[0163] Calculate the vector C0 to be written and the eigenvectors V1, V2, V3...V in the eigenvector index table respectively N The smaller the Euclidean distance, the smaller the difference value of the vector. The eigenvector V2 with the smallest difference value is selected as the write eigenvector corresponding to the vector C0 to be written, and the write calculation node corresponding to the vector C0 to be written can be determined to be calculation node 3 through the eigenvector index table.

[0164] Step 306: Write the vector to be written into the write computing node.

[0165] In the third embodiment provided in this specification, the vector C0 to be written is written into computing node No. 3, completing the process of writing the new vector into the massively parallel processing database.

[0166] The method for large-scale parallel processing of databases provided in the embodiments of this specification, in the process of writing a vector to be written into the large-scale parallel processing database, obtains the feature vector with the smallest difference value from the vector to be written through the approximate nearest neighbor retrieval method, and then determines the write computing node corresponding to the vector to be written. The pre-generated feature vector index table plays the role of partitioning and pruning the large-scale parallel processing database, determines the corresponding write computing node for the vector to be written, and writes vectors with small difference values ​​to the same computing node to facilitate subsequent query use.

[0167] Optionally, the method further includes: receiving a deletion instruction; and executing the deletion instruction in each computing node of the massively parallel processing database.

[0168] After receiving the deletion instruction, the deletion instruction is directly sent to each computing node of the large-scale processing database, and the deletion instruction is executed in each computing node.

[0169] Optionally, the method further includes the following steps S3080 to S3088.

[0170] Step S3080: Receive a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched.

[0171] Step S3080 is consistent with the method of the above-mentioned step 102. For a specific explanation of step S3080, please refer to the detailed content of step 102 in the above-mentioned embodiment, which will not be repeated here.

[0172] In the third embodiment provided in this specification, a search instruction is received, and the search instruction includes a vector to be searched C1 corresponding to the face image to be searched.

[0173] Step S3082: using an approximate nearest neighbor search method, determining at least one search feature vector corresponding to the vector to be searched in the feature vector index table, and obtaining a search calculation node corresponding to each search feature vector.

[0174] Step S3082 is consistent with the method of the above-mentioned step 104. For a specific explanation of step S3082, please refer to the detailed content of step 104 in the above-mentioned embodiment, which will not be repeated here.

[0175] In the third embodiment provided in this specification, the vector to be searched C1 and the feature vectors V1, V2, V3, ... V in the feature vector index table shown in Table 3 are calculated respectively. N The smaller the Euclidean distance, the smaller the difference value of the vector. The feature vectors V1 and V2 whose Euclidean distance is less than the preset threshold are determined as the retrieval feature vectors corresponding to the vector to be retrieved C1, and the retrieval calculation nodes corresponding to the vector to be retrieved C1 are determined to be calculation nodes 1 and 3 through the feature vector index table.

[0176] Step S3084: searching for an approximate vector corresponding to the vector to be searched in each of the search calculation nodes.

[0177] Step S3084 is consistent with the method of the above-mentioned step 106. For a specific explanation of step S3084, please refer to the detailed content of step 106 in the above-mentioned embodiment, which will not be repeated here.

[0178] In the third embodiment provided in this specification, the approximate vector of the vector to be searched C1 retrieved in the computing node No. 1 is C 11 The approximate vector of the search vector C1 in the computing node 3 is C 31 .

[0179] Step S3086: Determine the target vector corresponding to the vector to be retrieved from at least one of the approximate vectors.

[0180] Step S3086 is consistent with the method of the above-mentioned step 108. For a specific explanation of step S3086, please refer to the detailed content of step 108 in the above-mentioned embodiment, which will not be repeated here.

[0181] In the third embodiment provided in this specification, the approximate vectors C are calculated respectively. 11 、C 31 The Euclidean distance from the vector to be retrieved C1, from C 11 、C 31 Determine the approximate vector C with the smallest Euclidean distance 31 is the target vector corresponding to the vector C1 to be retrieved.

[0182] Step S3088: Acquire attribute information corresponding to the target vector according to the target vector, and use the attribute information corresponding to the target vector as the search result corresponding to the item data to be retrieved.

[0183] In practical applications, after determining the target vector, it is also necessary to obtain attribute information corresponding to the target vector, and use the attribute information corresponding to the target vector as the search result corresponding to the item data to be searched in the search instruction.

[0184] In the third embodiment provided in this specification, the target vector C is obtained.31 The corresponding attribute information such as name, age, home address, workplace, etc. is used as the attribute information of the person corresponding to the vector C1 to be retrieved.

[0185] For ease of understanding, combined Figures 4a to 4d For further explanation of the face recognition scenario in the third embodiment, see Figures 4a to 4d , Figure 4a FIG. 2 shows a schematic diagram of the structure of a large-scale parallel processing database provided by the third embodiment of this specification. Figure 4b FIG. 4 shows a schematic diagram of vector writing provided by the third embodiment of this specification, Figure 4c FIG. 4 shows a schematic diagram of vector retrieval provided in the third embodiment of this specification. Figure 4d A schematic diagram of vector deletion provided by the third embodiment of this specification is shown.

[0186] See also Figure 4a , shows a structural diagram of a large-scale parallel processing database, where the client is connected to the front-end node of the large-scale parallel database, and the front-end node is connected to computing nodes 1-6 in the large-scale parallel database respectively.

[0187] Acquire 1000 face images in advance, extract the feature vector of each face image as a training vector, and generate training samples of face image feature vectors. Cluster the 1000 training vectors using the k-means clustering algorithm to obtain N cluster sets. The cluster center vector in each set is used as the feature vector of the current set, thereby obtaining N feature vectors V1, V2, V3...V N There are 6 computing nodes in the large-scale parallel processing database. The computing node corresponding to each eigenvector is determined by taking the modulus, and each eigenvector and the computing node corresponding to each eigenvector are stored in a one-to-one correspondence to generate a eigenvector index table as described in Table 3.

[0188] See also Figure 4b The camera device captures a face image, extracts a feature vector from the face image through feature extraction as the vector C0 to be written, and writes the vector C0 to be written into the front-end node of the large-scale parallel processing database through a write instruction.

[0189] The front-end node partitions and routes the vector C0 to be written through the feature vector index table, and respectively calculates the feature vectors V1, V2, V3, ... V in the feature vector index table and the vector C0 to be written. N The Euclidean distance between them is used to determine the eigenvector V2 with the smallest Euclidean distance as the write eigenvector corresponding to the vector C0 to be written, and the write computing node corresponding to the vector C0 to be written can be determined to be computing node No. 3 through the eigenvector index table.

[0190] The front-end node writes the vector C0 to be written into computing node 3 through partition routing, completing the writing process of the feature vector corresponding to the face image.

[0191] See also Figure 4c When performing face recognition, it is necessary to search the face image captured by the surveillance camera equipment to obtain the corresponding person attribute information such as name, age, place of origin, etc.

[0192] The facial image captured by the surveillance camera is processed into a feature vector to obtain a vector to be retrieved C1, and the vector to be retrieved C1 is input to the front-end node.

[0193] The front-end node performs vector query and partition pruning on the vector to be retrieved C1 through the feature vector index table, calculates the Euclidean distance between the vector to be retrieved C1 and each feature vector in the feature vector index table respectively, selects the feature vectors V1 and V2 whose Euclidean distance is less than the preset threshold as the retrieval feature vectors corresponding to the vector to be retrieved C1, and determines through the feature vector index table that the retrieval calculation nodes corresponding to the vector to be retrieved are calculation nodes 1 and 3 respectively.

[0194] Retrieve the approximate vector C of the vector to be retrieved C1 in computing node 1 11 , retrieve the approximate vector C of the vector to be retrieved C1 in computing node 3 31 , respectively calculate the approximate vector C 11 、C 31 The Euclidean distance from the vector to be retrieved C1 determines the approximate vector C in the calculation node 3 31 is the target vector of the vector C1 to be retrieved.

[0195] The name, age, place of origin and other information of the person stored corresponding to the target vector is used as the search result corresponding to the vector to be searched C1.

[0196] See also Figure 4d , the client delete instruction is sent to the front-end node, and the front-end node forwards the delete instruction to each computing node through partition routing, and executes the delete instruction in each computing node.

[0197] The method for large-scale parallel processing of databases provided in the embodiments of this specification, in the process of writing a vector to be written into the large-scale parallel processing database, obtains the feature vector with the smallest difference value from the vector to be written through the approximate nearest neighbor retrieval method, and then determines the write computing node corresponding to the vector to be written. The pre-generated feature vector index table plays the role of partitioning and pruning the large-scale parallel processing database, determines the corresponding write computing node for the vector to be written, and writes vectors with small difference values ​​to the same computing node to facilitate subsequent query use.

[0198] Secondly, when searching in a large-scale parallel processing database, the retrieval computing node that needs to be searched is determined through a preset feature vector index table, and there is no need to search in all computing nodes, which avoids useless computing consumption and improves retrieval efficiency. Searching in the retrieval computing node and determining the target vector corresponding to the vector to be retrieved in at least one obtained approximate vector can ensure the accuracy of the target vector. Through the method provided in this specification, while ensuring the retrieval accuracy, the retrieval time is shortened, the retrieval efficiency is improved, and the user experience is improved.

[0199] Corresponding to the above method embodiment, this specification also provides an apparatus embodiment for large-scale parallel processing of a database. Figure 5 FIG1 shows a schematic diagram of a structure of a device for large-scale parallel processing of a database provided by an embodiment of this specification. Figure 5 As shown, the device includes:

[0200] The first receiving module 502 is configured to receive a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched.

[0201] The first node determination module 504 is configured to use an approximate nearest neighbor retrieval method to determine at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table, and obtain a retrieval calculation node corresponding to each of the retrieval feature vectors, wherein the feature vector index table records multiple feature vectors and the calculation node corresponding to each feature vector.

[0202] The first vector retrieval module 506 is configured to retrieve an approximate vector corresponding to the vector to be retrieved in each of the retrieval calculation nodes.

[0203] The first vector determination module 508 is configured to determine a target vector corresponding to the vector to be retrieved from at least one of the approximate vectors.

[0204] Optionally, the first determination node module 504 is further configured to calculate the difference value between the vector to be retrieved and each feature vector in the feature vector index table in turn; and determine that the feature vector whose difference value is less than a preset threshold is the retrieval feature vector corresponding to the vector to be retrieved.

[0205] Optionally, the first vector determination module 508 is further configured to calculate the difference between each of the approximate vectors and the vector to be retrieved; and select the approximate vector with the smallest difference as the target vector corresponding to the vector to be retrieved.

[0206] Optionally, the device further includes:

[0207] The first matching result module is configured to obtain attribute information corresponding to the target vector according to the target vector; and use the attribute information corresponding to the target vector as a search result corresponding to the item data to be retrieved.

[0208] Optionally, the device further includes:

[0209] a second receiving module, configured to receive a deletion instruction;

[0210] The first execution instruction module is configured to execute the deletion instruction in each computing node of the massively parallel processing database.

[0211] Optionally, the apparatus further includes a training module, the training module being used to train and generate the feature vector index table, the training module including:

[0212] A sample acquisition submodule is configured to acquire a training sample, wherein the training sample includes a plurality of training vectors corresponding to the project data;

[0213] a clustering submodule, configured to cluster a plurality of training vectors in the training sample;

[0214] a determination submodule, configured to determine at least two eigenvectors according to the clustering result, and determine a computing node corresponding to each of the eigenvectors;

[0215] The generating submodule is configured to generate a feature vector index table according to each feature vector and the computing node corresponding to each feature vector.

[0216] Optionally, the determination submodule is configured to determine the computing node corresponding to each of the feature vectors according to the number of computing nodes in the large-scale parallel processing database.

[0217] The device for large-scale parallel processing of databases provided in the embodiments of this specification, when performing a search in the large-scale parallel processing database, determines the search computing node that needs to be searched through a preset feature vector index table, and performs the search in the search computing node, and determines the target vector corresponding to the vector to be searched in at least one obtained approximate vector, which can ensure the accuracy of the target vector. The method provided in this specification eliminates the need to search the vector to be searched in all computing nodes, while ensuring the accuracy of the search, greatly reducing the time for each search, and also greatly reducing useless computing consumption, thereby improving the search capability and thus enhancing the user experience.

[0218] The above is a schematic diagram of an apparatus for massively parallel database processing according to this embodiment. It should be noted that the technical solution of the apparatus for massively parallel database processing shares the same concept as the technical solution of the method for massively parallel database processing described above. For details not described in detail in the technical solution of the apparatus for massively parallel database processing, please refer to the description of the technical solution of the method for massively parallel database processing described above.

[0219] Corresponding to the above method embodiment, this specification also provides an apparatus embodiment for large-scale parallel processing of a database. Figure 6 FIG1 shows a schematic diagram of a structure of a device for large-scale parallel processing of a database provided by an embodiment of this specification. Figure 6 As shown, the device includes:

[0220] The third receiving module 602 is configured to receive a write instruction, wherein the write instruction includes a vector to be written corresponding to the item data to be written.

[0221] The second node determination module 604 is configured to use an approximate nearest neighbor retrieval method to determine the write feature vector corresponding to the vector to be written in a pre-generated feature vector index table, and obtain the write calculation node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and the calculation node corresponding to each feature vector.

[0222] The writing module 606 is configured to write the vector to be written into the writing computing node.

[0223] Optionally, the second determination node module 604 is further configured to use the approximate nearest neighbor retrieval method to calculate the approximate values ​​of the vector to be written and each feature vector in the feature vector index table in sequence; and determine the feature vector with the largest approximate value as the write feature vector corresponding to the vector to be written.

[0224] Optionally, the device further includes:

[0225] a fourth receiving module, configured to receive a deletion instruction;

[0226] The second execution instruction module is configured to execute the deletion instruction in each computing node of the massively parallel processing database.

[0227] Optionally, the device further includes:

[0228] a fifth receiving module configured to receive a search instruction, wherein the search instruction includes a to-be-searched vector corresponding to the to-be-searched item data;

[0229] A third node determination module is configured to determine at least one retrieval feature vector corresponding to the vector to be retrieved in the feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval calculation node corresponding to each retrieval feature vector;

[0230] A second retrieval vector module is configured to retrieve an approximate vector corresponding to the vector to be retrieved in each retrieval calculation node;

[0231] The second vector determination module is configured to determine a target vector corresponding to the vector to be retrieved from at least one of the approximate vectors.

[0232] Optionally, the third determination node module is further configured to calculate the difference value between the vector to be retrieved and each feature vector in the feature vector index table in sequence; and determine that the feature vector whose difference value is less than a preset threshold is the retrieval feature vector corresponding to the vector to be retrieved.

[0233] Optionally, the second vector determination module is further configured to calculate the difference value between each of the approximate vectors and the vector to be retrieved; and select the approximate vector with the smallest difference value as the target vector corresponding to the vector to be retrieved.

[0234] Optionally, the device further includes:

[0235] The second matching result module is configured to obtain attribute information corresponding to the target vector according to the target vector; and use the attribute information corresponding to the target vector as a search result corresponding to the item data to be retrieved.

[0236] Optionally, the apparatus further includes a training module, the training module being used to train and generate the feature vector index table, the training module including:

[0237] A sample acquisition submodule is configured to acquire a training sample, wherein the training sample includes a plurality of training vectors corresponding to the project data;

[0238] a clustering submodule, configured to cluster a plurality of training vectors in the training sample;

[0239] a determination submodule, configured to determine at least two eigenvectors according to the clustering result, and determine a computing node corresponding to each of the eigenvectors;

[0240] The generating submodule is configured to generate a feature vector index table according to each feature vector and the computing node corresponding to each feature vector.

[0241] Optionally, the determination submodule is configured to determine the computing node corresponding to each of the feature vectors according to the number of computing nodes in the large-scale parallel processing database.

[0242] The embodiment of this specification provides a device for large-scale parallel processing of databases. In the large-scale parallel processing database, the approximate nearest neighbor retrieval method is used to obtain the feature vector with the smallest difference value from the vector to be written, and then the write computing node corresponding to the vector to be written is determined. The pre-generated feature vector index table plays the role of partitioning and cutting for the large-scale parallel processing database, determining the corresponding write computing node for the vector to be written, and writing vectors with small difference values ​​into the same computing node to facilitate subsequent query use. When searching in the large-scale parallel processing database, the preset feature vector index table is used to determine the retrieval computing node to be searched, and the search is performed in the retrieval computing node. The target vector corresponding to the vector to be searched is determined in at least one approximate vector obtained, which can ensure the accuracy of the target vector. The method provided in this specification does not require the search for the vector to be searched in all computing nodes. While ensuring the accuracy of the retrieval, the time for each retrieval is greatly reduced, and useless computing consumption is also greatly reduced, thereby improving the retrieval capability and thus enhancing the user experience.

[0243] The above is a schematic diagram of an apparatus for massively parallel database processing according to this embodiment. It should be noted that the technical solution of the apparatus for massively parallel database processing shares the same concept as the technical solution of the method for massively parallel database processing described above. For details not described in detail in the technical solution of the apparatus for massively parallel database processing, please refer to the description of the technical solution of the method for massively parallel database processing described above.

[0244] Figure 7 7 shows a block diagram of a computing device 700 according to one embodiment of the present disclosure. Components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0245] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0246] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 7 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0247] Computing device 700 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 700 can also be a mobile or stationary server.

[0248] The memory 710 is used to store computer-executable instructions, and the processor 720 is used to execute the following computer-executable instructions:

[0249] Receiving a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched;

[0250] Determine at least one retrieval feature vector corresponding to the vector to be retrieved in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval computation node corresponding to each retrieval feature vector, wherein the feature vector index table records a plurality of feature vectors and a computation node corresponding to each feature vector;

[0251] Retrieving an approximate vector corresponding to the vector to be retrieved in each of the retrieval calculation nodes;

[0252] A target vector corresponding to the vector to be retrieved is determined from at least one of the approximate vectors.

[0253] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the aforementioned method for large-scale parallel database processing are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned method for large-scale parallel database processing.

[0254] Figure 88 shows a block diagram of a computing device 800 according to an embodiment of the present application. Components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0255] Computing device 800 also includes an access device 840 that enables computing device 800 to communicate via one or more networks 860 .

[0256] The access method of the computing device 800 is the same as that of the computing device 700 described above, and will not be repeated here.

[0257] The processor 820 is configured to execute the following computer-executable instructions:

[0258] receiving a write instruction, wherein the write instruction includes a vector to be written corresponding to the project data to be written;

[0259] Determine a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor search method, and obtain a write computing node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and a computing node corresponding to each feature vector;

[0260] The vector to be written is written into the write computing node.

[0261] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the aforementioned method for large-scale parallel database processing are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned method for large-scale parallel database processing.

[0262] An embodiment of the present specification further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the steps of the method for large-scale parallel processing of a database.

[0263] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the method for large-scale parallel database processing described above share the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the method for large-scale parallel database processing described above.

[0264] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0265] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0266] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0267] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0268] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for massively parallel processing a database, wherein the massively parallel processing database uses vectors for partition pruning, the method comprising: receiving a write instruction, wherein the write instruction includes a vector to be written corresponding to the project data to be written; Determining a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtaining a write calculation node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and the calculation node corresponding to each feature vector; determining a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method includes: sequentially calculating the difference between the vector to be written and each feature vector in the feature vector index table; determining the feature vector with the smallest difference value as the write feature vector corresponding to the vector to be written; The vector to be written is written into the write computing node.

2. The method for massively parallel processing of a database according to claim 1 , further comprising: Receive deletion instructions; The delete instruction is executed in each computing node of the massively parallel processing database.

3. The method for massively parallel processing of a database according to claim 1 or 2, further comprising: Receiving a search instruction, wherein the search instruction includes a vector to be searched corresponding to the item data to be searched; Determine at least one retrieval feature vector corresponding to the vector to be retrieved in the feature vector index table using an approximate nearest neighbor retrieval method, and obtain a retrieval calculation node corresponding to each retrieval feature vector; Retrieving an approximate vector corresponding to the vector to be retrieved in each of the retrieval calculation nodes; A target vector corresponding to the vector to be retrieved is determined from at least one of the approximate vectors.

4. The method for massively parallel processing of a database according to claim 3, wherein determining at least one search feature vector corresponding to the vector to be searched in the feature vector index table using an approximate nearest neighbor search method comprises: Calculating the difference between the vector to be searched and each feature vector in the feature vector index table in sequence; The feature vector whose difference value is less than a preset threshold is determined as a search feature vector corresponding to the vector to be searched.

5. The method for massively parallel processing of a database according to claim 3, wherein determining a target vector corresponding to the vector to be searched from at least one of the approximate vectors comprises: Calculating the difference between each of the approximate vectors and the vector to be retrieved; The approximate vector with the smallest difference value is selected as the target vector corresponding to the vector to be retrieved.

6. The method for massively parallel processing of a database according to claim 3, further comprising: after determining a target vector corresponding to the vector to be searched from at least one of the approximate vectors; Acquire attribute information corresponding to the target vector according to the target vector; The attribute information corresponding to the target vector is used as the search result corresponding to the item data to be retrieved.

7. The method for large-scale parallel database processing according to claim 1, wherein the feature vector index table is obtained by training through the following steps, including: Acquire a training sample, wherein the training sample includes a plurality of training vectors corresponding to the project data; Clustering multiple training vectors in the training sample; Determine at least two eigenvectors according to the clustering result, and determine a computing node corresponding to each of the eigenvectors; A feature vector index table is generated according to each feature vector and a computing node corresponding to each feature vector.

8. The method for large-scale parallel database processing according to claim 7, wherein determining the computing node corresponding to each of the feature vectors comprises: The computing node corresponding to each of the feature vectors is determined according to the number of computing nodes in the large-scale parallel processing database.

9. A device for massively parallel processing a database, wherein the massively parallel processing database uses vectors for partition pruning, the device comprising: A third receiving module is configured to receive a write instruction, wherein the write instruction includes a to-be-written vector corresponding to the to-be-written project data; The second node determination module is configured to determine the write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtain the write calculation node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and the calculation node corresponding to each feature vector; the method of determining the write feature vector corresponding to the vector to be written in the pre-generated feature vector index table using the approximate nearest neighbor retrieval method includes: sequentially calculating the difference between the vector to be written and each feature vector in the feature vector index table; and determining the feature vector with the smallest difference value as the write feature vector corresponding to the vector to be written; The writing module is configured to write the vector to be written into the writing computing node.

10. A computing device comprising: memory and processor; The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions: receiving a write instruction, wherein the write instruction includes a vector to be written corresponding to the project data to be written; Determining a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method, and obtaining a write calculation node corresponding to the write feature vector, wherein the feature vector index table records multiple feature vectors and the calculation node corresponding to each feature vector; determining a write feature vector corresponding to the vector to be written in a pre-generated feature vector index table using an approximate nearest neighbor retrieval method includes: sequentially calculating the difference between the vector to be written and each feature vector in the feature vector index table; determining the feature vector with the smallest difference value as the write feature vector corresponding to the vector to be written; The vector to be written is written into the write computing node.

11. A computer-readable storage medium storing computer instructions, wherein when the instructions are executed by a processor, the steps of the method for large-scale parallel processing of a database according to any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method for large-scale parallel processing of a database as described in any one of claims 1 to 8.

Images