Data processing method and device for vector database, medium and product

Abstract

The invention provides a data processing method and device for a vector database, a medium and a product. The data processing method of the vector database comprises the following steps: selecting a preset quantity of vector data from all vector data of the vector database as a clustering center; obtaining the distance between each piece of to-be-classified vector data and all the clustering centers, wherein the to-be-classified vector data is vector data which is not used as the clustering center in all the vector data; and recording the to-be-classified vector data into at least one class group including the class group corresponding to the cluster center closest to the to-be-classified vector data according to the distance between each to-be-classified vector data and all the cluster centers. When the to-be-queried vector is located at the edge of the class group corresponding to the nearest clustering center, only several pieces of vector data most similar to the to-be-queried vector are searched from the class group corresponding to the nearest clustering center; and vector data which are classified according to the prior art and only belong to the edges of other class groups and are relatively similar to the class groups can also be obtained.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data processing method, device, medium and product for a vector database. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, the demand for processing and analyzing massive high-dimensional data is growing. Although traditional relational databases perform well in processing structured data, they face many challenges in processing unstructured or semi-structured data (such as images, text, audio, etc.). In particular, for tasks that require similarity search, such as image retrieval, speech recognition, and recommendation systems, traditional databases often have difficulty in efficiently processing data in high-dimensional space.

[0003] Therefore, vector databases came into being. As the name implies, vector databases are used to store and query vector data, so that similarity searches can be performed efficiently. In order to improve retrieval efficiency, a common practice is to classify and store vector data. Each cluster has a cluster center. When performing queries, the distance between the query vector and all cluster centers is obtained, the nearest cluster center is found, and then vector data similar to the query vector is searched in the cluster corresponding to the cluster center. However, this data processing method may lose vector data in other clusters that are more similar to the query vector, and further improvement is needed. Summary of the invention

[0004] An object of the present invention is to provide a vector database data processing method, device, medium and product that can solve any of the above problems.

[0005] In particular, the present invention provides a data processing method for a vector database, comprising:

[0006] Selecting a preset number of vector data from all vector data in the vector database as cluster centers;

[0007] Obtain the distance between each vector data to be classified and all cluster centers, where the vector data to be classified is the vector data that is not used as the cluster center among all the vector data;

[0008] According to the distance between each vector data to be classified and all the cluster centers, the vector data to be classified is recorded into at least one cluster including the cluster corresponding to the cluster center closest to the vector data.

[0009] Optionally, the step of recording the vector data to be classified into at least one group including the group corresponding to the cluster center closest to the vector data to be classified according to the distance between each vector data to be classified and all cluster centers includes:

[0010] The vector data to be classified is recorded in the cluster corresponding to all cluster centers whose distance is less than or equal to the reference distance. The reference distance is obtained by multiplying the distance between the vector data to be classified and the nearest cluster center by a preset correction value. The preset correction value is greater than 1 and less than or equal to 1.5.

[0011] Optionally, the preset correction value is greater than or equal to 1.15 and less than or equal to 1.25.

[0012] Optionally, the step of recording the vector data to be classified into the clusters corresponding to all cluster centers whose distances are less than or equal to the reference distance includes:

[0013] Sort the distances between the classified vector data and all cluster centers by size;

[0014] The vector data to be classified is recorded into the cluster corresponding to the cluster center closest to it;

[0015] Determine whether the distance between the cluster center to be determined and the vector data to be classified is less than or equal to the reference distance in order from small to large. If so, the vector data to be classified is recorded in the group corresponding to the cluster center to be determined. If not, the classification of the vector data to be classified is determined to be completed.

[0016] The cluster centers to be determined are the remaining cluster centers except the cluster center closest to the vector data to be classified.

[0017] Optionally, after the step of recording the vector data to be classified into at least one group including the group corresponding to the cluster center closest to the vector data to be classified according to the distance between each vector data to be classified and all the cluster centers, the following steps are included:

[0018] Take the average of all vector data in each cluster as the new cluster center;

[0019] Repeat the steps of obtaining the distance between each vector data to be classified and all cluster centers and taking the average value of all vector data in each cluster as the new cluster center until the cluster center of each cluster no longer changes.

[0020] Optionally, the step of selecting a preset number of vector data as cluster centers from all vector data in the vector database includes:

[0021] Randomly select a vector data from all the vector data in the vector database as the cluster center;

[0022] Get the shortest distance between the vector data that is not used as the cluster center and the existing cluster center;

[0023] The vector data with the largest shortest distance to the existing cluster center is determined as the next cluster center;

[0024] Repeat the steps of obtaining the shortest distance between the vector data that is not used as the cluster center and the existing cluster center and determining the vector data with the largest shortest distance to the existing cluster center as the next cluster center until the number of cluster centers reaches a preset number.

[0025] Optionally, after completing the cluster construction, the data processing method of the vector database further includes:

[0026] Get the vector to be queried;

[0027] Get the distance between the vector data to be queried and all cluster centers, and find the cluster center with the closest distance;

[0028] Get the distance between the query vector and all vector data in the cluster corresponding to the nearest cluster center;

[0029] Take out a set amount of vector data according to the distance from small to large.

[0030] According to another aspect of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer executable program stored in the memory and running on the processor, and when the processor executes the computer executable program, the data processing method of the vector database described in any one of the above items is implemented.

[0031] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer executable program is stored. When the computer executable program is executed by a processor, the data processing method of the vector database according to any one of the above items is implemented.

[0032] According to another aspect of the present invention, a computer program product is provided, including a computer executable program, which implements the vector database data processing method according to any one of the above items when executed by a processor.

[0033] The data processing method of the vector database of the present invention selects a preset number of vector data as cluster centers from all vector data in the vector database, obtains the distance between each vector data to be classified and all cluster centers, the vector data to be classified is the vector data that is not used as the cluster center among all vector data, and records the vector data to be classified into at least one group including the group corresponding to the cluster center closest to it according to the distance between each vector data to be classified and all cluster centers. Specifically, based on the distance from the cluster center, a vector data to be classified is at least classified into the group corresponding to the cluster center closest to it, and may also be classified into the group corresponding to other cluster centers whose distance is close to it and its distance to the nearest cluster center, and when a vector data to be classified is classified into multiple groups, the vector data must be located at the edge of the group in multiple groups. In this way, when the vector to be queried is at the edge of the cluster corresponding to the nearest cluster center, even if only a few vector data most similar to the vector to be queried are searched from the cluster corresponding to the nearest cluster center, vector data that is relatively similar to the vector to be queried can be obtained, which should only belong to the edges of other clusters according to the existing technology classification, effectively reducing the occurrence of missing vector data that is more similar to the vector to be queried, and helping to make the query results better.

[0034] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0036] Figure 1 It is a simple schematic diagram of the data clustering form in the prior art;

[0037] Figure 2 is a schematic flow chart of a data processing method for a vector database according to an embodiment of the present invention;

[0038] Figure 3 is a schematic flow chart of a step of selecting a cluster center in a data processing method for a vector database according to an embodiment of the present invention;

[0039] Figure 4 is a simple schematic diagram of a clustering form formed by a data processing method for a vector database according to an embodiment of the present invention;

[0040] Figure 5is a schematic flow chart of the step of recording the vector data to be classified into the corresponding category group in the data processing method of the vector database according to one embodiment of the present invention;

[0041] Figure 6 is a schematic flow chart of a data processing method of a vector database according to another embodiment of the present invention;

[0042] Figure 7 is a schematic flow chart of a data processing method of a vector database according to yet another embodiment of the present invention;

[0043] Figure 8 is a schematic diagram of a computer device according to an embodiment of the present invention;

[0044] Fig. 9 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention;

[0045] Fig.10 is a schematic diagram of a computer program product according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] It should be understood by those skilled in the art that the embodiments described below are only some embodiments of the present invention, rather than all embodiments of the present invention, and these embodiments are intended to explain the technical principles of the present invention, rather than to limit the protection scope of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present invention.

[0047] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or used in combination with these instruction execution systems, devices or equipment.

[0048] The flow chart provided by the present invention is not intended to indicate that the operation of the method will be performed in any particular order, or all operations of the method are included in all every case. In addition, the method may include additional operations. Within the scope of the technical thinking provided by the present embodiment method, additional changes can be made to the above method.

[0049] Reference Figure 1 As shown, the prior art is first described with a simple example. Figure 1In the figure, the range enclosed by the dotted circle represents the two groups divided by the vector database. The square block in each dotted circle represents the cluster center of the corresponding group, and the circular block in each dotted circle represents the vector data divided into the group represented by the dotted circle. The five-pointed star block in the figure represents the query vector, which is closer to the cluster center of the group represented by the dotted circle on the left, so the vector data most similar to the query vector will be searched from the group represented by the dotted circle on the left, that is, the vector data closest to the query vector will be searched according to the distance between the query vector and the vector data in the group represented by the dotted circle on the left as the query result.

[0050] However, it can be seen that when the vector to be queried is closer to the edge of the cluster, there is actually vector data that is closer to the vector to be queried, that is, more similar, in the edge area of ​​the cluster represented by the dotted circle on the right. But because the search is only conducted in the cluster represented by the dotted circle on the left, the vector data that is more similar to the vector to be queried and is located at the edge of the cluster represented by the dotted circle on the right will be missed.

[0051] like Figure 2 As shown, in one embodiment, the data processing method of the vector database generally includes:

[0052] Step S101, select a preset number of vector data from all vector data in the vector database as cluster centers. Specifically, the preset number is set according to the amount of data in the vector database.

[0053] Reference Figure 3 As shown, in one embodiment, this step includes:

[0054] Step S201, randomly select a vector data from all the vector data in the vector database as a cluster center. Specifically, randomly select a vector data from all the vector data as the first generated cluster center.

[0055] Step S202, obtaining the shortest distance between the vector data that is not used as the cluster center and the existing cluster center. That is, for each vector data that is not used as the cluster center, obtain the shortest distance between it and all the cluster centers. The above distance can be the distance between vectors such as Euclidean distance and Manhattan distance. It is preferred to use Euclidean distance, which is simpler to calculate.

[0056] For example, if there is only one cluster center at present, the vector data that is not used as the cluster center has only the distance to one cluster center, and this distance is the shortest distance. If there are already two cluster centers, respectively recorded as cluster center A and cluster center B, then the vector data that is not used as the cluster center has the distance to both cluster centers, that is, the distance to cluster center A and the distance to cluster center B, and the shorter distance is the shortest distance between the vector data that is not used as the cluster center and the existing cluster center. If there are more cluster centers at present, the shortest distance between the vector data that is not used as the cluster center and the existing cluster center can be referred to as described above.

[0057] Step S203, determine the vector data with the largest shortest distance to the existing cluster center as the next cluster center. Specifically, after obtaining the shortest distances between all vector data not used as cluster centers and the existing cluster centers, determine the vector data with the largest shortest distance to the existing cluster centers as the next cluster center.

[0058] For example, if there are three vector data that are not used as cluster centers, and there is already a cluster center, the shortest distances between the three vector data that are not used as cluster centers and the existing cluster center are D1, D2, and D3, respectively, among which D2 is the largest, then the vector data with the shortest distance D2 to the existing cluster center will be used as the next cluster center, that is, the second cluster center.

[0059] Step S204, determine whether the number of cluster centers reaches the preset number, if yes, execute step S205. If no, return to execute step S202. That is, determine whether the number of cluster centers that have been determined meets the required preset number, if not, continue to select new cluster centers.

[0060] Step S205, ending the cluster center selection process. Specifically, after the number of cluster centers reaches a preset number, no new cluster center is selected.

[0061] That is, the steps of obtaining the shortest distance between the vector data that is not a cluster center and the existing cluster center and determining the vector data with the largest shortest distance to the existing cluster center as the next cluster center are repeated until the number of cluster centers reaches a preset number.

[0062] It should be noted that in some other embodiments, a preset number of vector data may be directly randomly selected as cluster centers, or other existing clustering technologies may be used to select cluster centers, such as K-means algorithm, K-means++ algorithm, etc.

[0063] Step S102, obtaining the distance between each vector data to be classified and all cluster centers, where the vector data to be classified is the vector data that is not used as the cluster center among all the vector data. Specifically, the above distance can be the distance between vectors such as Euclidean distance and Manhattan distance, that is, obtaining the distance between all vector data that are not used as cluster centers and all cluster centers. For example, if there are ten cluster centers, then for all vector data that are not used as cluster centers, obtain ten distances between the vector data and the ten cluster centers.

[0064] Step S103: according to the distance between each vector data to be classified and all the cluster centers, the vector data to be classified is recorded into at least one cluster including the cluster corresponding to the cluster center closest to the vector data to be classified.

[0065] Specifically, this step includes recording the vector data to be classified into the cluster corresponding to all cluster centers whose distance is less than or equal to the reference distance multiplied by the preset correction value, where the reference distance is the distance between the vector data to be classified and the nearest cluster center multiplied by the preset correction value, and the preset correction value is greater than 1 and less than or equal to 1.5. Exemplarily, the preset correction value can be 1.05, 1.1, 1.15, 1.2, 1.25, 1.3, 1.35, 1.4, 1.45 or 1.5, etc. Preferably, the preset correction value is greater than or equal to 1.15 and less than or equal to 1.25, for example, 1.15, 1.17, 1.2, 1.22 or 1.25, etc.

[0066] Specifically, for each vector data that is not a cluster center, after obtaining the distance between the vector data and all cluster centers, multiply the smallest distance by a preset correction value greater than 1 and less than or equal to 1.5, and the obtained value is recorded as the reference distance. If the distance between the vector data and a cluster center is less than or equal to the reference distance, the vector data is recorded in the cluster corresponding to the cluster center. Because the distance between the vector data to be classified and the nearest cluster center is definitely less than the reference distance, the vector data to be classified must at least be recorded in the cluster corresponding to the nearest cluster center.

[0067] Exemplarily, if there are ten cluster centers, the preset correction value is 1.2. Taking a vector data to be classified that is not a cluster center as an example, its distances from the ten cluster centers are obtained, namely, C1, C2, C3, C4, C5, C6, C7, C8, C9 and C10. Among them, C3 is the minimum value, because C3 must be less than or equal to 1.2C3, so the vector data to be classified is recorded in the cluster corresponding to the cluster center at distance C3. For distances other than C3, if the distance is less than or equal to 1.2C3, the vector data to be classified is also recorded in the cluster corresponding to the cluster center. For example, if C1 is less than or equal to 1.2C3, the vector data to be classified is also recorded in the cluster corresponding to the cluster center at distance C1. If the distance is greater than 1.2C3, the vector data to be classified is not recorded in the cluster corresponding to the cluster center. For example, if C2 is greater than 1.2C3, the vector data to be classified is not recorded in the cluster corresponding to the cluster center at distance C2.

[0068] In this way, a vector data to be classified may be classified into multiple groups. Moreover, the vector data classified into multiple groups must be located at the edge of the groups in multiple groups.

[0069] Combination Figure 4 As shown in the figure, it is a simple example of classification completion. Figure 4 In the figure, the range enclosed by the dotted circle represents the two groups divided by the vector database. The square block in each dotted circle represents the cluster center of the group, and the circular block in each dotted circle represents the vector data divided into the group represented by the dotted circle. The red circular block in the left dotted circle represents the vector data that is only divided into the group represented by the left dotted circle, the blue circular block in the right dotted circle represents the vector data that is only divided into the group represented by the right dotted circle, and the green circular block in the intersection of the two dotted circles represents the vector data that is simultaneously divided into the group represented by the left dotted circle and the group represented by the right dotted circle, and is at the edge of both the group represented by the left dotted circle and the group represented by the right dotted circle. Therefore, when the vector to be queried is at the edge of a group, it is highly likely that similar vector data in other groups can be queried during the query.

[0070] Continue to refer to Figure 4 As shown in the figure, the five-pointed star block represents the query vector, which is closer to the cluster center of the group represented by the dotted circle on the left, and close to the edge of the group represented by the dotted circle on the left. However, because the vector data located at the edge of the group represented by the dotted circle on the right and close to the group represented by the dotted circle on the left will also be classified into the group represented by the dotted circle on the left, only searching for several vector data most similar to the query vector from the group represented by the dotted circle on the left can also obtain vector data similar to the query vector in the group represented by the dotted circle on the right, which should only belong to the group represented by the dotted circle on the right according to the classification of the prior art, so as to make the query result better.

[0071] In the scheme of this embodiment, a preset number of vector data are selected from all vector data in the vector database as cluster centers, and the distance between each vector data to be classified and all cluster centers is obtained. The vector data to be classified is the vector data that is not used as the cluster center among all vector data, and the vector data to be classified is recorded into at least one group including the group corresponding to the cluster center closest to it according to the distance between each vector data to be classified and all cluster centers. Specifically, based on the distance from the cluster center, a vector data to be classified is at least classified into the group corresponding to the cluster center closest to it, and may also be classified into the group corresponding to other cluster centers whose distance is close to it and its distance to the nearest cluster center, and in the case where a vector data to be classified is classified into multiple groups, the vector data must be located at the edge of the group in multiple groups. In this way, when the vector to be queried is at the edge of the cluster corresponding to the nearest cluster center, even if only a few vector data most similar to the vector to be queried are searched from the cluster corresponding to the nearest cluster center, vector data that is relatively similar to the vector to be queried can be obtained, which should only belong to the edges of other clusters according to the existing technology classification, effectively reducing the occurrence of missing vector data that is more similar to the vector to be queried, and helping to make the query results better.

[0072] Furthermore, the vector data to be classified is recorded in the clusters corresponding to all cluster centers whose distances are less than or equal to the reference distance. The reference distance is obtained by multiplying the distance between the vector data to be classified and the nearest cluster center by a preset correction value. The preset correction value is greater than 1 and less than or equal to 1.5, so that the distances between the vector data classified into multiple clusters and the cluster centers of the clusters to which they belong are relatively close, avoiding the classification of the vector data to be classified into the cluster corresponding to the cluster center that is too far away, thereby affecting the classification effect. Preferably, the preset correction value is greater than or equal to 1.15 and less than or equal to 1.25, so that the preset correction value is more appropriate, which can avoid the preset correction value being too small, resulting in the vector data being basically not classified into multiple clusters, thereby affecting the final query effect, and can also avoid the preset correction value being too large, resulting in the vector data to be classified into the cluster corresponding to the cluster center that is too far away, thereby affecting the classification effect.

[0073] In addition, by randomly selecting a vector data as a cluster center from all the vector data in the vector database, obtaining the shortest distance between the vector data that is not used as the cluster center and the existing cluster center, the vector data with the largest shortest distance to the existing cluster center is determined as the next cluster center, and the steps of obtaining the shortest distance between the vector data that is not used as the cluster center and the existing cluster center and determining the vector data with the largest shortest distance to the existing cluster center as the next cluster center are repeated until the number of cluster centers reaches a preset number. In this way, the distances between the preset number of cluster centers are far enough, which helps to improve the classification effect of subsequent vector data.

[0074] like Figure 5 As shown, in one embodiment, the step of recording the vector data to be classified into the clusters corresponding to all cluster centers whose distances are less than or equal to the reference distance generally includes:

[0075] Step S301, sorting the distances between the vector data to be classified and all the cluster centers according to their sizes. Specifically, after obtaining the distances between each vector data to be classified and all the cluster centers, sorting all the distance values ​​according to their sizes.

[0076] Step S302, recording the vector data to be classified into the cluster corresponding to the cluster center closest to it.

[0077] Step S303, judging whether the distance between the cluster center to be determined and the vector data to be classified is less than or equal to the reference distance in order from small to large, if so, executing step S304, if not, executing step S305. The cluster center to be determined is the remaining cluster centers except the cluster center closest to the vector data to be classified. Specifically, starting from the distance between the cluster center closest to the vector data to be classified and the vector data to be classified, the sorted distances are compared with the reference distances in order, and the reference distances refer to the above description.

[0078] Step S304, record the vector data to be classified into the group corresponding to the cluster center to be determined. If the distance between the cluster center to be determined currently compared with the reference distance and the vector data to be classified is less than or equal to the reference distance, then record the vector data to be classified into the group corresponding to the current cluster center to be determined.

[0079] Step S305, determine whether the classification of the vector data to be classified is completed. If the distance between the cluster center to be determined and the vector data to be classified that is currently compared with the reference distance is greater than the reference distance, the vector data to be classified will not be recorded in the group corresponding to the current cluster center to be determined. Moreover, because the distance is compared with the reference distance from small to large, the subsequent distance must be greater than the reference distance, so there is no need to compare the relationship between the remaining distances and the reference distance, and the classification of the current vector data to be classified is directly terminated.

[0080] Exemplarily, if there are five cluster centers, take a vector data to be classified that is not a cluster center as an example, obtain its five distances from the five cluster centers and sort them according to size, the five distances are Cj, Ck, Cl, Cm, Cn from small to large, that is, Cj is the smallest and Cn is the largest. First, the vector data to be classified is recorded in the group corresponding to the cluster center at distance Cj. Then, starting from Ck, compare the relationship with the reference distance. If the distance is less than or equal to the reference distance, the vector data to be classified is simultaneously recorded in the group corresponding to the cluster center. For example, if Ck is less than or equal to the reference distance, the vector data to be classified is simultaneously recorded in the group corresponding to the cluster center at distance Ck. If the distance is greater than the reference distance, the vector data to be classified is not recorded in the group corresponding to the cluster center and it is determined that the classification work of the current vector data to be classified is completed. For example, if Cl is greater than the reference distance, the vector data to be classified is not recorded in the group corresponding to the cluster center at distance Cl, and there is no need to compare the relationship between Cm and Cn and the reference distance.

[0081] In the scheme of this embodiment, the distances between the vector data to be classified and all the cluster centers are sorted according to size, the vector data to be classified is recorded in the group corresponding to the cluster center closest to it, and it is judged from small to large whether the distance between the cluster center to be determined and the vector data to be classified is less than or equal to the reference distance. If so, the vector data to be classified is recorded in the group corresponding to the cluster center to be determined. If not, it is determined that the classification of the vector data to be classified is completed. That is, after sorting, starting from the distance between the cluster center closest to the vector data to be classified and the vector data to be classified, the sorted distances are compared with the reference distances in sequence. When the distance between the cluster center to be determined and the vector data to be classified that is currently compared with the reference distance is greater than the reference distance, because the distance is compared with the reference distance from small to large, the subsequent distance is definitely greater than the reference distance, so the classification of the current vector data to be classified can be directly ended. In this way, it is helpful to reduce the distances that need to be compared and help improve the classification efficiency of the vector data to be classified.

[0082] It should be noted that, in this embodiment, step S302 may also be performed before step S301.

[0083] like Figure 6As shown, in one embodiment, after the step of recording the vector data to be classified into at least one group including the group corresponding to the cluster center closest to it according to the distance between each vector data to be classified and all the cluster centers, it includes: taking the average of all the vector data in each group as the new cluster center; repeating the step of obtaining the distance between each vector data to be classified and all the cluster centers to the step of taking the average of all the vector data in each group as the new cluster center, until the cluster center of each group no longer changes.

[0084] In this embodiment, the data processing method of the vector database generally includes:

[0085] Step S401: Select a preset number of vector data from all vector data in a vector database as cluster centers.

[0086] Step S402, obtaining the distance between each vector data to be classified and all cluster centers, where the vector data to be classified is the vector data that is not used as a cluster center among all the vector data.

[0087] Step S403: according to the distance between each vector data to be classified and all the cluster centers, the vector data to be classified is recorded into at least one cluster including the cluster corresponding to the cluster center closest to the vector data to be classified.

[0088] The contents of step S401 to step S403 are the same as those described in the above embodiment.

[0089] Step S404, taking the average value of all vector data in each cluster as a new cluster center. Specifically, after all vector data are classified, taking the average value of all vector data in each cluster, and taking the average value as the new cluster center of the cluster.

[0090] Step S405, determine whether the cluster centers of all clusters have changed, if so, return to step S402, if not, execute step S406. Specifically, after taking the average value of the vector data in each cluster as the new cluster center, check whether the new cluster centers of all clusters are the same as the original cluster centers, if different, execute steps S403 to S405 based on the newly generated cluster centers.

[0091] Step S406: Determine whether cluster construction is complete. If the new cluster centers of all clusters are the same as the original cluster centers after the average value of the vector data in each cluster is used as the new cluster center, cluster construction is complete and the process does not return to step S402.

[0092] In the scheme of this embodiment, after completing the clustering of all vector data, the average value of the vector data in each cluster is taken as the new cluster center, and the vector data is clustered again, and this process is repeated until the cluster center no longer changes, so that the cluster center can better represent the data distribution of each cluster, and the vector data in the cluster is more concentrated, which helps to improve the clustering effect of the vector data.

[0093] like Figure 7 As shown, in one embodiment, after completing the cluster construction, the data processing method of the vector database generally includes:

[0094] Step S501, obtain the vector to be queried, that is, obtain a vector value, and query similar data in the vector database according to the vector value.

[0095] Step S502, obtaining the distance between the query vector and all cluster centers, and finding the cluster center with the closest distance, that is, finding the cluster center with the smallest distance to the query vector.

[0096] Step S503, obtaining the distance between the query vector and all vector data in the cluster corresponding to the nearest cluster center. After finding the nearest cluster center, similar vector data is searched from the cluster corresponding to the nearest cluster center according to the distance to the query vector.

[0097] Step S504: extract a set number of vector data in ascending order of distance. That is, extract a set number of vector data that are relatively closer to the vector to be queried. The set number can be flexibly set as needed.

[0098] Because when the query vector is located at the edge of the cluster corresponding to the nearest cluster center, even if only a few vector data most similar to the query vector are found from the cluster corresponding to the nearest cluster center, it is also possible to obtain vector data that is relatively similar to the query vector and should only belong to the edges of other clusters according to the existing technology classification, thereby effectively reducing the occurrence of missing vector data that is more similar to the query vector, and helping to make the query results better.

[0099] This embodiment also provides a computer device and a computer-readable storage medium. Figure 8 is a schematic diagram of a computer device 10 according to one embodiment of the present invention. Fig. 9 is a schematic diagram of a computer-readable storage medium 20 according to one embodiment of the present invention.

[0100] The computer device 10 may include a memory 110, a processor 120, and a computer executable program 11 stored in the memory 110 and running on the processor 120, and the processor 120 implements the data processing method of the vector database of any of the above embodiments when executing the computer executable program 11.

[0101] The computer-readable storage medium 20 stores a computer-executable program 11 thereon. When the computer-executable program 11 is executed by a processor, the data processing method of the vector database of any of the above-mentioned embodiments can be implemented.

[0102] This embodiment also provides a computer program product. Fig.10 1 is a schematic diagram of a computer program product 30 according to an embodiment of the present invention. The computer program product 30 includes a computer executable program 11, and when the computer executable program 11 is executed by a processor 120, any of the vector database data processing methods described above can be implemented.

[0103] Specifically, the computer executable program 11 for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, computer instructions, computer-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0104] For the purpose of the description of the present embodiment, the computer-readable storage medium 20 can be any device that can contain, store, communicate, propagate or transmit a program for use with an instruction execution system, device or equipment or in conjunction with these instruction execution systems, devices or equipment. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium 20 can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting or processing in other suitable ways as necessary, and then stored in a computer memory.

[0105] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0106] The computer device 10 may be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smart phone. In some examples, the computer device 10 may be a cloud acquisition node. The computer device 10 may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, a program module may include routines, programs, object programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer device 10 may be implemented in a distributed cloud acquisition environment where remote processing devices linked via a communication network perform tasks. In a distributed cloud acquisition environment, program modules may be located on a local or remote acquisition system storage medium including a storage device.

[0107] The computer device 10 may include a processor 120 adapted to execute stored instructions, and a memory 110 that provides temporary storage space for the operation of the instructions during operation. The processor 120 may be a single-core processor, a multi-core processor, an acquisition cluster, or any number of other configurations. The memory 110 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0108] The processor 120 may be connected to an I / O interface (input / output interface) suitable for connecting the computer device 10 to one or more I / O devices (input / output devices) via a system interconnect (e.g., PCI, PCI-Express, etc.). The I / O devices may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touch pad or a touch screen, etc. The I / O devices may be built-in components of the computer device 10, or may be devices externally connected to an acquisition device.

[0109] Processor 120 can also be linked to a display interface suitable for connecting computer device 10 to a display device through a system interconnection. Display device can include a display screen as a built-in component of computer device 10. Display device can also include a computer monitor, a television or a projector, etc., which are externally connected to computer device 10. In addition, a network interface controller (NIC) can be suitable for connecting computer device 10 to a network through a system interconnection. In some embodiments, NIC can use any suitable interface or protocol (such as an Internet small computer system interface, etc.) to transmit data. The network can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN) or the Internet, etc. A remote device can be connected to a computer device through a network.

[0110] At this point, those skilled in the art should recognize that, although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived based on the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and recognized as covering all these other variations or modifications.

Claims

1. A data processing method for a vector database, comprising: Selecting a preset number of vector data from all the vector data in the vector database as cluster centers; Obtaining the distance between each vector data to be classified and all cluster centers, wherein the vector data to be classified is the vector data that is not used as the cluster center among all the vector data; According to the distance between each of the vector data to be classified and all the cluster centers, the vector data to be classified is recorded into at least one cluster including the cluster corresponding to the cluster center closest to the vector data.

2. The data processing method of the vector database according to claim 1, wherein: The step of recording the vector data to be classified into at least one group including the group corresponding to the cluster center closest to the vector data to be classified according to the distance between the vector data to be classified and all the cluster centers comprises: The vector data to be classified is recorded in the cluster corresponding to all cluster centers whose distance is less than or equal to the reference distance, wherein the reference distance is obtained by multiplying the distance between the vector data to be classified and the nearest cluster center by a preset correction value, and the preset correction value is greater than 1 and less than or equal to 1.

5.

3. The data processing method of the vector database according to claim 2, wherein: The preset correction value is greater than or equal to 1.15 and less than or equal to 1.

25.

4. The data processing method of the vector database according to claim 2, wherein: The step of recording the vector data to be classified into the clusters corresponding to all cluster centers whose distances are less than or equal to the reference distance comprises: Sorting the distances between the vector data to be classified and all cluster centers according to their sizes; Recording the vector data to be classified into the cluster corresponding to the cluster center closest to it; Determine in order from small to large whether the distance between the cluster center to be determined and the vector data to be classified is less than or equal to the reference distance; if so, record the vector data to be classified into the group corresponding to the cluster center to be determined; if not, determine that the classification of the vector data to be classified is completed; The cluster centers to be determined are the remaining cluster centers except the cluster center closest to the vector data to be classified.

5. The data processing method of the vector database according to claim 1, wherein: After the step of recording the vector data to be classified into at least one group including the group corresponding to the cluster center closest to the vector data to be classified according to the distance between each vector data to be classified and all cluster centers, the following steps are included: Take the average of all vector data in each cluster as the new cluster center; Repeat the steps of obtaining the distance between each vector data to be classified and all cluster centers to taking the average value of all vector data in each group as the new cluster center until the cluster center of each group no longer changes.

6. The data processing method of the vector database according to claim 1, wherein: The step of selecting a preset number of vector data as cluster centers from all the vector data in the vector database comprises: Randomly select one vector data from all the vector data in the vector database as a cluster center; Get the shortest distance between the vector data that is not used as the cluster center and the existing cluster center; The vector data with the largest shortest distance to the existing cluster center is determined as the next cluster center; Repeat the steps of obtaining the shortest distance between the vector data that is not used as the cluster center and the existing cluster center and determining the vector data with the largest shortest distance to the existing cluster center as the next cluster center until the number of cluster centers reaches a preset number.

7. The data processing method of the vector database according to claim 1, wherein: After completing the cluster construction, the data processing method of the vector database also includes: Get the vector to be queried; Obtain the distance between the vector data to be queried and all cluster centers, and find the cluster center with the closest distance; Obtaining the distance between the query vector and all vector data in the cluster corresponding to the nearest cluster center; Take out a set amount of vector data according to the distance from small to large.

8. A computer device comprising a memory, a processor, and a computer executable program stored in the memory and running on the processor, wherein the processor implements the data processing method of the vector database according to any one of claims 1 to 7 when executing the computer executable program.

9. A computer-readable storage medium having a computer executable program stored thereon, wherein the computer executable program, when executed by a processor, implements the data processing method for a vector database according to any one of claims 1 to 7.

10. A computer program product, comprising a computer executable program, wherein when the computer executable program is executed by a processor, the method for processing vector database data according to any one of claims 1 to 7 is implemented.

Images