Fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm

Abstract

The present application relates to the technical field of smart home, and especially relates to a fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm. The steps are as follows: collecting historical voice instruction information of a user to construct a cloud database, performing voice feature extraction on data in the cloud database, and saving the voice feature extraction to a cloud vector database; clustering and layering voice feature vectors in the cloud vector database, and introducing a multi-level balance degree adjustment strategy to adjust the balance degree; using an IVF-HNSW framework to manage the clustering center; migrating data in the voice feature extraction model, extracting user voice instruction feature vectors and historical voice instruction information of the user as auxiliary information, and delivering the auxiliary information to the cloud; querying the user voice instruction feature vectors through an IVF-HNSW index structure, obtaining a matching voice feature key value, and returning the instruction to a local intelligent voice terminal. The present application can realize efficient index construction and approximate nearest neighbor search.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart home, and particularly relates to a fast intelligent voice assistant response method based on an approximate nearest neighbor search algorithm. BACKGROUND

[0002] With the improvement of living standards, more and more families use intelligent voice assistants to control smart home appliances, and the response speed of the intelligent voice assistant and the accuracy of the smart home appliance control are important links affecting user experience. Due to differences in voiceprint information such as user speaking manner, accent, and environmental noise, the same operation instruction may produce different voice data in the use process of the intelligent terminal. Therefore, in order to realize the recognition of the voice operation instruction of the specific user to the smart home appliance, a large-scale database needs to be maintained, which can be used for training a voice feature extraction model on the one hand, and can more accurately match specific voice information with high similarity in the database on the other hand. The two aspects complement each other to improve the voice assistant's recognition ability of the sound signal. In addition, with the increase of smart home appliances, the same instruction may control multiple intelligent terminals. How to effectively and quickly feed back the operation instruction and realize the quasi-cooperative operation of multiple terminals is a major challenge faced by the intelligent voice assistant.

[0003] Approximate Nearest Neighbor Search (ANNS) is a vector similarity calculation method that has emerged in recent years. This method greatly improves the retrieval efficiency while ensuring high retrieval accuracy, and has attracted widespread attention from researchers. According to the difference of index structure, the existing ANNS algorithm can be roughly divided into: IVF-Based, Graph-Based and Tree-Based. Compared with other methods, IVF-Based can realize efficient compression of large-scale data sets, and under the premise of ensuring retrieval accuracy, the index structure is loaded into limited high-speed memory at one time, thereby improving retrieval efficiency.

[0004] The construction process of IVF-Based index is first to use a clustering algorithm to divide the data set into clusters and generate the corresponding cluster centers. The cluster centers obtained are used as index heads to construct an inverted table structure. Then, a quantization algorithm is used to compress the vectors in the data set into short codes, and the short codes are added to the corresponding index heads to form an inverted index. In the retrieval process, the query vector will first be located to the nearest index head, then the vectors associated with the index head will be queried, and the final query result will be obtained. This method narrows down the query range, avoids global vector query, and thus improves query efficiency. However, the core clustering algorithm of the inverted index-based method faces the following two challenges when facing large-scale retrieval tasks:

[0005] Clustering time affects index construction efficiency: how to reduce the time of selecting initial clustering center and the time complexity of clustering process while ensuring retrieval accuracy to improve index construction efficiency is very challenging;

[0006] Clustering balance affects retrieval efficiency: how to adjust the clustering process to achieve a relatively balanced clustering result and improve retrieval efficiency is also challenging.

[0007] Therefore, the present application provides a fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm to solve the above problems. SUMMARY

[0008] The present application is aimed at the deficiencies of the prior art, and develops a fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm, the main purpose of which is to apply the approximate nearest neighbor method based on inverted index to intelligent voice assistants and improve the core clustering algorithm for large-scale data.

[0009] The technical solution of the present application solves the technical problem of a fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm, which is as follows:

[0010] S1, collect the user's historical voice command information to build a cloud database, then input the voice command in the cloud database into the voice feature extraction model for voice feature extraction, the form of voice feature is <voice feature vector, command pair>, and then save the voice feature vector to the cloud vector database ;

[0011] S2, cluster and layer the voice feature vectors in the cloud vector database, and calculate the clustering center set of each layer, then select the initial clustering center for each layer, and make the number of initial clustering centers the same as the number of clustering centers in the clustering center set, then introduce a multi-level balance adjustment strategy for balance adjustment, the multi-level balance adjustment strategy includes an intra-layer balance adjustment strategy and an inter-layer balance adjustment strategy;

[0012] S3, use the IVF-HNSW framework to manage the voice feature vectors and the calculated clustering centers;

[0013] S4, migrate the data in the voice feature extraction model, then extract the user voice command feature vector, and transfer the user voice command feature vector and the user's historical voice command information as auxiliary information to the cloud;

[0014] S5, after the cloud receives the user query vector and voice information, the user voice instruction feature vector is queried through the IVF-HNSW index structure, the similar voice vector stored in the cloud database is queried and output, the vector number in the query result is matched to the corresponding <voice feature vector, instruction pair> key value of the vector, and the instruction is returned to the local intelligent voice terminal.

[0015] S1 is specifically as follows:

[0016] The historical voice instruction information in the cloud database is input into the voice feature extraction module, the voice feature is obtained, and then the voice feature is saved to the cloud vector database, and the voice feature is represented as , , represents any one historical voice instruction information in the cloud database, represents the voice feature vector obtained according to the historical voice instruction information , represents the instruction pair obtained according to the historical voice instruction information , and respectively represent the object of the instruction action and the instruction requirement action obtained according to the historical voice instruction information , , represents the i-th voice feature vector in the cloud vector library, , , .

[0017] S2 is specifically as follows:

[0018] S2.1, the voice feature vectors are clustered and layered:

[0019] The i voice feature vectors in the cloud vector database are subjected to hierarchical clustering of L layers, the number of hierarchical clustering layers is , , , represents the number of clustering centers of the L-th layer, , , The final clustering center output after clustering the data in the cloud vector database is the clustering center of the last layer, and the number of clustering centers of the last layer is , and satisfy ;

[0020] When The number of clustering centers of each layer and the number of clustering centers of the next layer satisfy the following condition:

[0021] ,

[0022] wherein, , , , , , , represent the number of clustering centers of the 1st, 2nd, 3rd, …, , … layer, respectively;

[0023] When , a two-layer clustering is used, at this time, the number of clustering centers of the first layer is , and the number of clustering centers of the second layer is .

[0024] The clustering hierarchical process is specifically as follows:

[0025] S2.1.1, before clustering in each layer, the clustering center position of the layer is initialized, the number of clustering centers of the first layer is , the cloud vector database to be clustered has data, the cloud vector database is first divided into disjoint subsets , represents the disjoint subset, , the number of representative nodes of the first layer of is calculated by the k-means algorithm , and the calculation formula is as follows:

[0026] ,

[0027] wherein, represents an intra-layer balance adjustment parameter, and is set to ;

[0028] After clustering of all subsets of the first layer is completed, the representative nodes obtained from each subset are combined into a representative node set, and the size of the representative node set is , ;

[0029] Then, the k-means++ clustering algorithm is used to select These nodes serve as the initial set of cluster centers for the first-level clustering. , Indicates the first layer 1 initial cluster center vector, ;

[0030] Similarly, we obtain the first... Initial set of cluster centers for hierarchical clustering , Indicates the first Layer 1 initial cluster center vector, Iterate multiple times until the first result is obtained. Initial set of cluster centers for hierarchical clustering , Indicates the first Layer 1 initial cluster center vector, ;

[0031] S2.1.2 The initial set of cluster centers for the first layer of clusters is obtained using the minibatch k-means clustering algorithm. and cloud vector database Perform multiple clustering iterations, with each iteration loading a minibatch of k-means. Clustering is performed on the bar vectors. The clustering results are checked every 10 iterations. The first layer of clustering is completed when the clustering converges or the maximum number of iterations is reached.

[0032] Similarly, the minibatch k-means clustering algorithm is used to cluster the th... Initial set of cluster centers for hierarchical clustering and cloud vector data Perform multiple clustering iterations until the first clustering iteration is completed. Layered clustering, multiple iterations, to complete the first... Hierarchical clustering;

[0033] In arbitrary-level clustering, every 10 iterations constitute an iterative unit, and each layer of clustering consists of multiple iterative units. The cluster centers within the layer are adjusted based on the results of each iterative unit. For any iterative unit, the input is the cluster centers output by the previous iterative unit, and the output clustering result is the set of output cluster centers. For any iterative unit, its input set of cluster centers is... , The first The vector representation of each input cluster center; its output set of cluster centers is... , The first The vector representation of each output cluster center.

[0034] S2.2, introduce multi-level balance adjustment strategy to adjust the balance of the iteration unit:

[0035] Balance adjustment includes intra-layer balance adjustment and inter-layer balance adjustment;

[0036] S2.2.1, intra-layer balance adjustment:

[0037] For any layer clustering, the process of intra-layer balance adjustment of each iteration unit includes three steps: (1) complete iteration to obtain temporary clusters; (2) calculate the average cluster size and define large clusters and small clusters; (3) split large clusters and eliminate small clusters, the specific process is as follows:

[0038] First, in the iteration unit, after completing 10 iterations, a set of temporary clustering centers is obtained , The first temporary clustering center is represented by a vector, and the cloud vector database is assigned to the nearest temporary clustering center to obtain temporary clusters , The first temporary clustering center is represented by a vector, and the cloud vector database is assigned to the nearest temporary clustering center to obtain temporary clusters

[0039] ,

[0040] ,

[0041] Wherein, represents the average size of the temporary cluster in the first layer;

[0042] When , the temporary cluster is a large cluster, , the temporary cluster is a small cluster;

[0043] Third, for large clusters, split the large clusters into subsets, and calculate the average value of all vectors in each subset, and take the average value as the clustering center of the subset, then delete the temporary clustering center from the clustering center set , and add the newly calculated new clustering center to , the number of clustering centers added to is more than the number of clustering centers deleted from , so as to keep , , The number of cluster centers is consistent with the elimination of small clusters;

[0044] The number of additional cluster centers is calculated The calculation process is as follows:

[0045] ,

[0046] The cluster centers in the large cluster obtained after the splitting operation are sorted in descending order of cluster size, and then the cluster centers corresponding to the smallest cluster size are deleted The last clusters are deleted to form the final output .

[0047] S2.2.2, interlayer balance adjustment:

[0048] The cluster centers of any layer after clustering are , , The vector representation of the th cluster center of the th layer is respectively assigned to the th cluster center of the cloud vector database to obtain the th layer cluster , The th cluster center corresponds to a cluster, and each cluster represents a vector set.

[0049] According to and formula , the number of cluster centers of the next layer is obtained , the average size of the cluster size corresponding to each cluster center is obtained by , and the average size of the cluster size corresponding to each cluster center is obtained by The number of temporary cluster centers to be assigned in the th layer clustering is calculated , and the calculation formula is as follows:

[0050] ,

[0051] The temporary cluster centers of the th layer are calculated according to the th cluster of the th layer, and the sum is The calculation process is as follows:

[0052] , ​​​

[0053] at this time, Then for the first layer The allocation of temporary cluster centers will be carried out, and the first temporary cluster center will be allocated. Layer Cluster Arrange them in descending order, then... Clusters sorted by layer are added to the first layer. In the layer, the first In the layer Add a first one after each cluster in sequence A cluster arranged in order within the layer, after updating As the first The final cluster center assignment result for the layer.

[0054] S3 is as follows:

[0055] The cloud vector database is compressed using the PQ quantization algorithm to obtain the quantized cloud vector database. ,according to The cluster centers of the last layer obtained through iterative steps in step S2 Construct the IVF-HNSW graph index structure. Indicates the first The first in the layer Cluster centers;

[0056] Specifically, the quantized cloud vector database Clusters are assigned to the nearest cluster centers to form a cluster structure. Each cluster center represents a cluster, thus enabling the construction of the IVF inverted index. Then, among all cluster centers... Based on this, the HNSW algorithm is used to construct a graph index to manage cluster centers. The speech feature vectors and the calculated cluster centers are managed from the IVF-HNSW framework structure. The organization and management strategy of vectors and cluster centers is IVF, and the organization and management strategy of cluster centers is HNSW.

[0057] S4 is as follows:

[0058] By using mature model scaling technology, the data in the voice feature extraction model is migrated to the small terminal model. The small terminal model is then deployed on the smart voice terminal to extract the user's voice command feature vector. The extracted user voice command feature vector is used as a query vector, which is then sent to the cloud via WIFI / 5G. The user's historical voice command information is also transmitted to the cloud as auxiliary information.

[0059] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects:

[0060] The application uses an approximate nearest neighbor (ANNS) technology, combines a cloud vector database and a terminal lightweight feature extraction deep model, and improves the recognition efficiency for complex semantic instructions and the recognition accuracy of user purposes.

[0061] In the ANNS technology based on an inverted index, the application uses a hierarchical clustering framework to replace the original single-layer clustering in the inverted index, and uses a minibatch k-means method in the hierarchical clustering framework, thereby greatly reducing the time complexity of the clustering operation in the index construction process and improving the inverted index construction efficiency; on the other hand, in each layer of clustering, a divide-and-conquer initialization point selection strategy is used to reduce the initial selection point time of the clustering algorithm while ensuring the clustering accuracy, optimize the overall clustering time, and further shorten the index construction time.

[0062] The application proposes an intra / inter-layer balance adjustment strategy to realize multi-level cluster size balance adjustment; the intra-layer balance adjustment effectively controls the cluster size in the clustering iteration process to prevent the cluster size from being too large or too small; the inter-layer balance adjustment strategy proportionally allocates the cluster centers to each cluster to avoid the underfitting of large clusters and the overfitting of small clusters; the two strategies are matched with each other to ensure that the cluster sizes are roughly equal, and the inverted index retrieval efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate the application together with the embodiments thereof, and explain the application without limiting the application.

[0064] Figure 1 is a method flowchart of the application. DETAILED DESCRIPTION

[0065] In order to clearly illustrate the technical features of the present application, the application will be described in detail below with reference to the specific embodiments and the accompanying drawings.

[0066] Embodiment 1

[0067] As shown in Figure 1 , a fast intelligent voice assistant response method based on an approximate nearest neighbor search algorithm is as follows:

[0068] S1, collect the historical voice instruction information of the user to construct a cloud database, then input the voice instruction in the cloud database into a voice feature extraction model to extract voice features, the form of the voice features is <voice feature vector, instruction pair>, and then save the voice feature vector to a cloud vector database .

[0069] S2, the voice feature vectors in the cloud vector database are clustered and layered, and the cluster center set of each layer is calculated, then the initial cluster center of each layer is selected, and the number of initial cluster centers is the same as the number of cluster centers in the cluster center set, then a multi-level balance adjustment strategy is introduced for balance adjustment, the multi-level balance adjustment strategy includes an intra-layer balance adjustment strategy and an inter-layer balance adjustment strategy;

[0070] S3, the IVF-HNSW framework is used to manage the voice feature vectors and the calculated cluster centers;

[0071] S4, the data in the voice feature extraction model is migrated, then the user voice instruction feature vector is extracted, and the user's historical voice instruction information is transmitted to the cloud as auxiliary information;

[0072] S5, after the cloud receives the user query vector and voice information, the user voice instruction feature vector is queried through the IVF-HNSW index structure, similar voice vectors stored in the cloud database are queried and output, the vector number in the query result is matched to the corresponding <voice feature vector, instruction pair> key value, and the instruction is returned to the local intelligent voice terminal.

[0073] S1 is as follows:

[0074] The historical voice instruction information in the cloud database is input into the voice feature extraction module, the voice feature is obtained, and then the voice feature is saved to the cloud vector database, the voice feature is represented as , represents any one of the historical voice instruction information in the cloud database, represents the voice feature vector obtained according to the historical voice instruction information , represents the instruction pair obtained according to the historical voice instruction information , and respectively represent the object of the instruction and the instruction requirement action obtained according to the historical voice instruction information , the cloud vector database is represented as , represents the th voice feature vector in the cloud vector database, , .

[0075] S2 is as follows:

[0076] S2.1, the voice feature vectors are clustered and layered:

[0077] the cloud vector database In Each speech feature vector is used for Hierarchical clustering of layers, number of hierarchical clustering layers The number of cluster centers in each layer after clustering is , Indicates the first The number of cluster centers in the layer For cloud-based vector databases The final cluster centers output after clustering the data are the cluster centers of the last layer, and the number of cluster centers in the last layer is... , and Between satisfy ;

[0078] when At that time, the number of cluster centers in each layer and the number of cluster centers in the next layer satisfy the following condition:

[0079] ,

[0080] in, , , ... , ... , They represent the 1st, 2nd, 3rd, ... , ... The number of cluster centers in the layer;

[0081] when When using two-level clustering, At this point, the number of cluster centers in the first layer is The number of cluster centers in the second layer is .

[0082] The clustering and hierarchical process is as follows:

[0083] S2.1.1 Before each clustering layer, the cluster center positions for that layer are initialized. The number of cluster centers in the first layer is... Cloud-based vector database to be clustered Total The data will first be stored in the cloud vector database. Divided into Disjoint subsets , Indicates the first A disjoint subset, Calculated using the k-means algorithm Number of representative nodes in the first layer The calculation formula is as follows:

[0084] ,

[0085] in, Indicates the parameters for adjusting the intra-layer equalization, setting... ;

[0086] After clustering all subsets in the first layer, the representative nodes obtained from each subset are combined into a representative node set, the size of which is... , ;

[0087] Then, the k-means++ clustering algorithm is used to select representative nodes from the set. These nodes serve as the initial set of cluster centers for the first-level clustering. , Indicates the first layer 1 initial cluster center vector, ;

[0088] Similarly, we obtain the first... Initial set of cluster centers for hierarchical clustering , Indicates the first Layer 1 initial cluster center vector, Iterate multiple times until the first result is obtained. Initial set of cluster centers for hierarchical clustering , Indicates the first Layer 1 initial cluster center vector, ;

[0089] S2.1.2 The initial set of cluster centers for the first layer of clusters is obtained using the minibatch k-means clustering algorithm. and cloud vector database Perform multiple clustering iterations, with each iteration loading a minibatch of k-means. Clustering is performed on the bar vectors. The clustering results are checked every 10 iterations. The first layer of clustering is completed when the clustering converges or the maximum number of iterations is reached.

[0090] Similarly, the minibatch k-means clustering algorithm is used to cluster the th... Initial set of cluster centers for hierarchical clustering and cloud vector data Perform multiple clustering iterations until the first clustering iteration is completed. Layer clustering, multiple iterations, complete the first Layer clustering

[0091] Any layer clustering, every 10 iterations for a iteration unit, each layer clustering consists of multiple iteration units, according to the results of each iteration unit to adjust the layer clustering center, for any iteration unit, the input is the output of the last iteration unit, the output of the clustering result is the output clustering center set, for any iteration unit, its input clustering center set is , The first Input clustering center vector representation; its output clustering center set is , The first Output clustering center vector representation.

[0092] S2.2, introduce multi-level balance adjustment strategy to adjust the iteration unit balance:

[0093] Balance adjustment includes intra-layer balance adjustment and inter-layer balance adjustment;

[0094] S2.2.1, intra-layer balance adjustment:

[0095] For any layer clustering, the process of intra-layer balance adjustment of each iteration unit includes three steps: (1) complete iteration, get temporary cluster; (2) calculate the average cluster size, define large cluster and small cluster; (3) split the large cluster and eliminate the small cluster, the specific process is as follows:

[0096] First, in the iteration unit, after completing 10 iterations, the temporary clustering center set , The first Temporary clustering center vector representation, the cloud vector database Is assigned to the nearest temporary clustering center, and the temporary cluster is obtained as , The first Temporary clustering center vector corresponding to the temporary cluster, each temporary cluster represents a vector set;

[0097] Second, calculate the average size of each temporary cluster size, the calculation process is as follows:

[0098] ,

[0099] Wherein, Indicates the average size of the first Layer temporary cluster;

[0100] When At that time, the temporary cluster was a large cluster. At that time, the temporary cluster is a small cluster;

[0101] The third step is to perform a splitting operation on the large cluster, dividing it into equal parts. Each subset is processed, and the average of all vectors within that subset is calculated. This average is used as the cluster center of the subset, and then the temporary cluster centers corresponding to the larger clusters are calculated. From the cluster center set Delete it and add the newly calculated cluster centers. New additions Cluster centers compared to The number of cluster centers deleted is large, in order to maintain , , If the number of cluster centers is consistent, small clusters are eliminated.

[0102] Calculate the number of extra cluster centers The calculation process is as follows:

[0103] ,

[0104] The result after the large cluster splitting operation The cluster centers were sorted according to their corresponding cluster sizes from largest to smallest, and then deleted. The smallest cluster size The cluster centers corresponding to each cluster form the final output. .

[0105] S2.2.2, Interlayer Equilibrium Adjustment:

[0106] After clustering is completed at any layer, the cluster centers of that layer are obtained. , , The first Layer The vector representation of each cluster center will be stored in a cloud-based vector database. Assigned to the The layer cluster centers are obtained as follows: Clusters of hierarchical clustering , The first Each cluster corresponds to a cluster of cluster centers, and each cluster represents a set of vectors.

[0107] according to and formula Obtain the number of cluster centers in the next layer. ,pass The average size of the cluster corresponding to each cluster center is obtained. Calculate the first Cluster Each cluster in In the Hierarchical clustering is the number of temporary cluster centers that should be assigned. The calculation formula is as follows:

[0108] ,

[0109] According to the Cluster The cluster calculation yields the th... Temporary cluster centers of the layer and for The calculation process is as follows:

[0110] ,

[0111] at this time, Then for the first layer The allocation of temporary cluster centers will be carried out, and the first temporary cluster center will be allocated. Layer Cluster Arrange them in descending order, then... Clusters sorted by layer are added to the first layer. In the layer, the first In the layer Add a first one after each cluster in sequence A cluster arranged in order within the layer, after updating As the first The final cluster center assignment result for the layer.

[0112] S3 is as follows:

[0113] The cloud vector database is compressed using the PQ quantization algorithm to obtain the quantized cloud vector database. ,according to The cluster centers of the last layer obtained through iterative steps in step S2 Construct the IVF-HNSW graph index structure. Indicates the first The first in the layer Cluster centers;

[0114] Specifically, the quantized cloud vector database Clusters are assigned to the nearest cluster centers to form a cluster structure. Each cluster center represents a cluster, thus enabling the construction of the IVF inverted index. Then, among all cluster centers... On the basis of the above, the HNSW algorithm is used to construct a graph index to manage the cluster centers, and the IVF-HNSW framework structure is used to manage the speech feature vectors and the calculated cluster centers, wherein the vector and the cluster center are managed by the IVF strategy, and the cluster center is managed by the HNSW strategy.

[0115] S4 is specifically as follows:

[0116] The data in the speech feature extraction model is migrated to a small terminal model through a mature model scaling technology, the small terminal model is deployed to an intelligent speech terminal to extract a user speech instruction feature vector, the extracted user speech instruction feature vector is used as a query vector, the query vector is sent to the cloud through WIFI / 5G, and the historical speech instruction information of the user is transmitted to the cloud as auxiliary information.

[0117] Embodiment 2

[0118] A large number of performance comparison experiments are performed on open-source benchmark data, and the experimental results can prove that the clustering framework provided in the application can realize efficient index construction and approximate nearest neighbor search.

[0119] Specifically as follows:

[0120] The application of the method in the application to a plurality of internationally advanced inverted index algorithms and the performance of the method are described in detail in this embodiment, as shown in Table 1, the method in the application can be integrated into the following two types of inverted index algorithm architectures, the first type is a common IVF structure combined with different quantization algorithms, wherein the quantization algorithms include four types, namely PQfs (product quantization), RQ (residual quantization), LSQ (local scalar quantization) and SQ (scalar quantization), thereby forming four different IVF+quantization index structures, IVF-PQfs, IVF-RQ, IVF-LSQ and IVF-SQ; the method in the application can also be combined with another type of IVF structure, namely an IVF+graph index structure, which has two representative implementation manners, IVF-HNSW and IVF-NSG, IVF-HNSW is an Inverted File System (IVF) with Hierarchical Navigable Small Worlds (HNSW), namely an index structure combining an inverted index and an approximate nearest neighbor algorithm, and IVF-NSG is an Inverted File System (IVF) with Navigating Spreading-out Graph (NSG), namely an index structure combining an inverted index and an approximate nearest neighbor search.

[0121] The effectiveness of the method of the present application can be verified by the method of the present application and the commonly used clustering algorithms in the above two types of six different IVF structures.

[0122] In this embodiment, two commonly used comparison algorithms, K-means (K-means clustering algorithm) and Mini-Batch K-means (a variant of K-means clustering algorithm), are selected respectively, and the international general multi-modal vector dataset is selected as the comparison dataset, which includes four 1B (billion) scale datasets BIGANN-1B, DEEP-1B, MSTuring-1B, and MSSPACEV-1B, and four 10M (ten million) scale datasets BIGANN-10M, DEEP-10M, MSTuring-10M, and MSSPACEV-10M.

[0123] Among them, the BIGANN (Big Artificial Neural Network) dataset is a large-scale artificial neural network dataset used to evaluate and compare different approximate nearest neighbor search algorithms; DEEP-1B is a large-scale vector dataset published by Yandex, which contains descriptor data composed of the projection and normalized output of the last fully connected layer of the GoogLeNet model; the MSTuring dataset may refer to the vector dataset published by the Microsoft Turing team; the Microsoft SPACEV-1B is a dataset related to web search published by Microsoft Bing, which is composed of document and query vectors encoded by the Microsoft SpaceV Superior model, aiming to capture general intent representation.

[0124] Table 2 compares the clustering efficiency on different scale datasets, and the clustering efficiency is shown by the clustering time, the shorter the clustering time, the higher the efficiency. Two groups are set in the embodiment, the first group of experiments realizes clustering with 65,536, 262,144, and 1,048,576 clustering centers on four 1B scale datasets respectively. Compared with K-means and Mini-Batch K-means, the method in the present application has a great efficiency improvement on the four datasets, and the clustering time is greatly shortened. For example, on the BIGANN-1B, to complete clustering with 1,048,576 clustering centers, the K-means algorithm needs 1,252.92 seconds, the Mini-Batch K-means needs 1,665.25 seconds, and the method in the present application only needs 956.57 seconds.

[0125] The second group of experiments respectively implement clustering of 4,096, 16,384 and 65,536 clustering centers on four 10 million scale data sets, which are consistent with 1 billion scale. Compared with K-means and Mini-Batch K-means, the method in the application has greater efficiency improvement on the four data sets, and the clustering time is greatly shortened. For example, on BIGANN-1B, K-means algorithm needs 41.98 seconds to complete clustering of 65,536 clustering centers, Mini-Batch K-means needs 43.36 seconds, and the method in the application only needs 36.49 seconds.

[0126] Table 3 compares the effect of applying the method in the application to different inverted indexes. The effect is measured by evaluation indexes Recall@10 and QPS. Recall@10 is the correct retrieval rate, and the calculation method is the percentage of the coincidence of the TOP-10 nearest neighbors output by the algorithm and the real TOP-10 nearest neighbors. QPS (Queries Per Second) represents the number of query vectors processed per second.

[0127] The embodiment is set to two groups. The first group of experiments compares the effects of different clustering methods on six IVF indexes on four 10 million scale data sets. The evaluation results show that the method in the application is significantly better than the traditional K-means and Mini-batch K-means in retrieval efficiency, and the number of query vectors processed per second is significantly improved while maintaining high retrieval accuracy. For example, on BIGANN-10M, when the IVF-LSQ index is constructed, the Recall@10 obtained by using kmeans clustering is 0.829, and the QPS is 4,711.86. When Mini-batch K-means is used, the Recall@10 is 0.875, and the QPS is 5,034.87. The Recall@10 obtained by using the method in the application is 0.877, and the QPS is 5,162.29.

[0128] The second group of experiments respectively compares the effects of different clustering methods on three IVF indexes on four 1 billion scale data sets, and the evaluation results show that the method in the application is significantly better than the traditional K-means and Mini-batch K-means in retrieval efficiency, while maintaining high retrieval accuracy, the number of query vectors processed per second is significantly improved; for example, on DEEP-1B, when building an IVF-PQfs index, the Recall@10 obtained by using K-means clustering is 0.664, and the QPS is 767.56; when using Mini-batch K-means, the Recall@10 is 0.673, and the QPS is 789.32; and when using the method in the application, the Recall@10 is 0.674, and the QPS is 866.05.

[0129] Table 1: Introduction of comparison algorithms of the application

[0130]

[0131] Table 2: Comparison results of index construction efficiency of the application

[0132]

[0133] Table 3: Comparison results of retrieval accuracy and query efficiency of the application

[0134]

[0135] Therefore, the hierarchical clustering framework proposed in the application, which fuses the initial point selection strategy of divide and conquer and the intra / inter layer balance adjustment strategy, can be applied to various inverted index structures and has universality. After obtaining the clustering centers by the clustering framework, the data set can be compressed by combining various quantization methods, and for large-scale vector data sets (1 billion scale), the index can be directly loaded in limited memory, so as to improve the retrieval efficiency of the index by using high-speed memory.

[0136] Although the specific embodiments of the application are described above with reference to the accompanying drawings, the description is not a limitation on the protection scope of the application. Various modifications or variations made by those skilled in the art on the basis of the technical solutions of the application without creative labor are still within the protection scope of the application.

Claims

1. A fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm, characterized in that, Comprise the following steps: S1, collect the historical voice instruction information of the user to construct a cloud database, then input the voice instruction in the cloud database into a voice feature extraction model for voice feature extraction, the form of the voice feature is <voice feature vector, instruction pair>, and then save the voice feature vector to a cloud vector database ; S2, the voice feature vectors in the cloud vector database are clustered and layered, and the cluster center set of each layer is calculated, then the initial cluster center of each layer is selected, the number of initial cluster centers is the same as the number of cluster centers in the cluster center set, then a multi-level balance adjustment strategy is introduced for balance adjustment, the multi-level balance adjustment strategy includes an intra-layer balance adjustment strategy and an inter-layer balance adjustment strategy; S3, the IVF-HNSW framework is used to manage the voice feature vectors and the calculated cluster centers; S3 is specifically as follows: The cloud vector database is compressed using the PQ quantization algorithm to obtain the quantized cloud vector database. ,according to The cluster centers of the last layer obtained through iterative steps in step S2 Construct the IVF-HNSW graph index structure. Indicates the first The first in the layer Cluster centers; Specifically quantifying the cloud vector database The distribution to the nearest cluster center forms a cluster structure, the cluster center is used to represent the cluster, thereby realizing the construction of the IVF inverted index, and then on the basis of all cluster centers , the HNSW algorithm is used to construct a graph index to manage the cluster centers, and the speech feature vectors and the calculated cluster centers are managed from the IVF-HNSW framework structure, wherein the organization and management strategy of the vectors and the cluster centers is IVF, and the organization and management strategy of the cluster centers is HNSW; S4, the data in the voice feature extraction model is migrated, then the user voice instruction feature vector is extracted, and the historical voice instruction information of the user is transmitted to the cloud as auxiliary information; S5, after the cloud receives the user query vector and the voice information, the user voice instruction feature vector is queried through the IVF-HNSW index structure, similar voice vectors stored in the cloud database are queried and output, the vector corresponding to the <voice feature vector, instruction pair> key value is matched according to the vector number in the query result, and the instruction is returned to the local intelligent voice terminal.

2. The fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm according to claim 1, wherein, S1 is specifically as follows: The historical voice instruction information in the cloud database is input into a voice feature extraction module to obtain voice features, and then the voice features are saved to a cloud vector database. The voice features are represented as , , represents any one of the historical voice instruction information in the cloud database, represents the voice feature vector obtained according to the historical voice instruction information , represents the instruction pair obtained according to the historical voice instruction information , and respectively represent the object of the instruction action and the instruction required action obtained according to the historical voice instruction information , , represents the voice feature vector in the cloud vector database, , , .

3. The fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm according to claim 2, characterized in that, S2 is specifically as follows: S2.1, the voice feature vectors are clustered and layered: Hierarchical clustering of layers of layers of voice feature vectors in a cloud vector database ;​​​​​​​​​​​​ When the number of cluster centers of each layer and the number of cluster centers of the next layer satisfy the following condition: ， wherein, , , , , , , denote the number of cluster centers of the 1st, 2nd, 3rd, …, , … layer, respectively; When a double layer clustering is used, the number of cluster centers in the first layer is and the number of cluster centers in the second layer is .

4. The fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm according to claim 3, wherein, The clustering and layering process is specifically as follows: S2.1.1, before clustering each layer, the cluster center position is initialized, the number of cluster centers of the first layer is , the cloud vector database to be clustered has a total of pieces of data, first divide the cloud vector database into non-intersecting subsets , , the non-intersecting subset is represented as , the number of representative nodes of the first layer of is calculated by the k-means algorithm , the calculation formula is as follows: ， wherein, represents the intra-layer balance adjustment parameter, set ; After all subsets of the first layer complete clustering, representative node sets of each subset are combined into a representative node set, and the size of the representative node set is , ; Then a k-means++ clustering algorithm is used to select a number of nodes from the representative node set as the initial cluster center set of the first layer clustering , represents the initial cluster center vector of the first layer ;​​ Similarly, the initial cluster center set of the 1th layer clustering is obtained the initial cluster center set of the 1th layer clustering , the 1th initial cluster center vector of the 1th layer the 1th initial cluster center vector of the 1th layer the 1th initial cluster center vector of the 1th layer , multiple iterations until the initial cluster center set of the 1th layer clustering is obtained the initial cluster center set of the 1th layer clustering , the 1th initial cluster center vector of the 1th layer the 1th initial cluster center vector of the 1th layer the 1th initial cluster center vector of the 1th layer ; S2.1.2, the initial clustering center set of the first layer clustering is clustered by a minibatch k-means clustering algorithm and a cloud vector database perform multiple clustering iterations, and each time the minibatch k-means loads a strip vector to cluster, and after 10 iterations, the clustering result is checked once, and when the clustering converges or reaches the maximum number of iterations, the first layer clustering is completed; Similarly, the initial cluster center set of the first layer clustering is obtained by using the minibatch k-means clustering algorithm on the first layer data layer clustering and cloud vector data layer clustering, the multiple iterations are performed until the first layer clustering is completed layer clustering, the multiple iterations are performed until the first layer clustering is completed layer clustering Each iteration unit is composed of multiple layer clustering, and the clustering centers in each layer are adjusted according to the result of each iteration unit. For any iteration unit, the input is the clustering center output by the previous iteration unit, and the output is the output clustering center set. For any iteration unit, the input clustering center set is , respectively the vector representation of the first input clustering center; and the output clustering center set is , respectively the vector representation of the first output clustering center.

5. The fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm according to claim 4, wherein, S2 is specifically as follows: S2.2, a multi-level balance adjustment strategy is introduced to adjust the balance of the iteration unit: The balance adjustment includes intra-layer balance adjustment and inter-layer balance adjustment; S2.2.1, intra-layer balance adjustment: For any layer of clustering, the process of intra-layer balance adjustment of each iteration unit includes three steps: (1) complete iteration to obtain a temporary cluster; (2) calculate the average cluster size and define large clusters and small clusters; (3) split the large clusters and eliminate the small clusters, the specific process is as follows; First, in the iteration unit, after 10 iterations, a temporary cluster center set is obtained , are respectively the vector representations of the first temporary cluster centers, the cloud vector database is assigned to the nearest temporary cluster center, and a temporary cluster is obtained , are respectively the temporary clusters corresponding to the vectors of the first temporary cluster centers, and each temporary cluster represents a vector set; Secondly, the average size of each temporary cluster size is calculated as follows: ， wherein, represents the average size of the temporary clusters of the first layer layer temporary clusters; When the temporary cluster is a large cluster, the temporary cluster is a small cluster; Thirdly, the large cluster is split into subsets, and the average of all vectors in each subset is calculated as the cluster center of the subset, then the temporary cluster center of the large cluster is deleted from and the newly calculated cluster center is added to The number of cluster centers added to is more than the number of cluster centers deleted from To keep unchanged, the number of cluster centers in is kept unchanged, and the number of cluster centers in is kept unchanged. The small cluster is eliminated by keeping the number of cluster centers in unchanged. Calculating the number of additional cluster centers The calculation process is as follows: ， The result after the large cluster splitting operation The cluster centers were sorted according to their corresponding cluster sizes from largest to smallest, and then deleted. The smallest cluster size The cluster centers corresponding to each cluster form the final output. .

6. The fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm according to claim 5, characterized in that, S2 is specifically as follows: S2.2.2, inter-layer balance adjustment: The clustering center of the layer is obtained after any layer is completed , , The vector representation of the first layer clustering center is respectively The cloud vector database is assigned to the first layer clustering center to obtain the cluster of the first layer clustering , , The cluster corresponding to the first clustering center is respectively, and each cluster represents a vector set; According to and the formula the number of cluster centers of the next layer , by the average size of the cluster size corresponding to each cluster center , calculate the first layer cluster each cluster in the first layer clustering is the number of temporary cluster centers to be assigned , the calculation formula is as follows: ， According to the Cluster The cluster calculation yields the th... Temporary cluster centers of the layer and for The calculation process is as follows: ， At this time, The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer Layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer Layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer The first layer is assigned to the temporary clustering centers, and the second layer is assigned to the first layer 7. The fast intelligent voice assistant response method based on approximate nearest neighbor search algorithm according to claim 6, characterized in that, S4 is specifically as follows: The data in the voice feature extraction model is migrated to a small terminal model through mature model scaling technology, the small terminal model is deployed to the intelligent voice terminal to extract the user voice instruction feature vector, the extracted user voice instruction feature vector is used as a query vector, the query vector is sent to the cloud through WIFI / 5G, and the historical voice instruction information of the user is transmitted to the cloud as auxiliary information.

Images