Neighbor graph indexing method and system based on federal learning and electronic equipment

Abstract

The invention provides a federated learning-based neighbor graph indexing method and system and electronic equipment. The method comprises the following steps: each client participating in federated learning trains a local performance prediction model used for predicting the retrieval performance of the neighbor graph based on a local sample data set of the client to obtain a corresponding local model parameter; the federation server participating in federation learning obtains local model parameters and test data sets of all the clients, a global test data set is obtained based on the test data sets of all the clients, and contribution quantized values corresponding to all the clients are calculated through the global test data set; and the federation server side aggregates all the local model parameters based on the contribution quantized value to obtain global model parameters, so that each client side updates a local performance prediction model based on the global model parameters, and constructs a corresponding neighbor graph index for the to-be-queried data set by using the updated local performance prediction model. According to the technical scheme, neighbor graph intelligent indexing supporting distributed data can be realized.

Description

Technical Field

[0001] This application relates to the field of information retrieval technology, and in particular to a nearest neighbor graph indexing method, system, and electronic device based on federated learning. Background Technology

[0002] With the growth of data scale and the widespread application of artificial intelligence (AI), vector retrieval has become a fundamental supporting technology for many business systems. Achieving fast and accurate similarity retrieval in massive high-dimensional vector data is crucial for improving the performance of applications such as search, recommendation, and image recognition. Nearest neighbor graph-based retrieval methods have become the mainstream solution due to their good balance between accuracy and efficiency. Starting from the basic K-nearest neighbor graph, to overcome its low navigation efficiency in high-dimensional spaces, the industry has successively proposed a series of improved nearest neighbor graph index structures, such as HNSW (Hierarchical Navigable SmallWorld) and NSG (Navigating Spreading-out Graph), improving retrieval speed and scalability.

[0003] However, the performance of nearest neighbor graph indexes is highly dependent on the adaptability of their structure and parameters. In traditional solutions, the determination of index parameters mainly relies on expert experience or grid search, which is not only time-consuming and labor-intensive, but also difficult to adapt to the dynamic changes of different data distributions and business scenarios. Although intelligent indexing methods that have emerged in recent years can automatically output parameters, they are limited to supporting only local data and cannot be implemented in real-world business environments with multiple participants and distributed data storage. Summary of the Invention

[0004] In view of this, this application provides a nearest neighbor graph indexing method, system, and electronic device based on federated learning to realize an intelligent indexing of nearest neighbor graphs that supports distributed data.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of the embodiments of this specification, a nearest neighbor graph indexing method based on federated learning is provided. The method includes: each client participating in the federated learning constructs its local nearest neighbor graph index based on a preset nearest neighbor graph parameter configuration; obtains the ground truth value of nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter by executing the nearest neighbor graph index; constructs a local sample dataset for each client based on the ground truth value of nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter configuration; each client trains a local performance prediction model for predicting nearest neighbor graph retrieval performance based on its local sample dataset, and obtains corresponding local model parameters; wherein the local performance prediction models of each client have the same network architecture, and a portion of the samples in the local sample dataset of each client forms a local test dataset for the client; a federated server participating in the federated learning obtains the local model parameters and test datasets of each client, obtains a global test dataset based on the test datasets of each client, and calculates the contribution quantification value corresponding to each client using the global test dataset; the federated server aggregates all local model parameters based on the contribution quantification value to obtain global model parameters, so that each client updates its local performance prediction model based on the global model parameters, and uses the updated local performance prediction model to construct a corresponding nearest neighbor graph index for the dataset to be queried.

[0007] According to a second aspect of the embodiments of this specification, a nearest neighbor graph indexing system is provided, comprising: multiple clients and a federated server; the clients are configured to construct their local nearest neighbor graph index based on preset nearest neighbor graph parameter configurations, obtain the ground truth value of nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter by executing the nearest neighbor graph index, and construct a local sample dataset for each client based on the ground truth value of nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter configuration; and each client trains a local performance prediction model for predicting nearest neighbor graph retrieval performance based on its local sample dataset to obtain corresponding local model parameters; wherein the local performance prediction models of each client have the same network architecture, and a portion of the samples in the local sample dataset of each client forms a local test dataset for the client; the federated server is configured to obtain the local model parameters and test datasets of each client, obtain a global test dataset based on the test datasets of each client, calculate the contribution quantification value corresponding to each client using the global test dataset; and aggregate all local model parameters based on the contribution quantification value to obtain global model parameters, so that each client updates its local performance prediction model based on the global model parameters, and uses the updated local performance prediction model to construct a corresponding nearest neighbor graph index for the dataset to be queried.

[0008] According to a third aspect of the embodiments of this specification, an electronic device is provided, including a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method described in the first aspect.

[0009] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0010] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0011] In this embodiment, each client trains a local performance prediction model with the same network architecture to learn the complex mapping relationship between their respective data features and the performance of the nearest neighbor graph index. The federated server constructs a global test dataset, calculates contribution quantification values, and aggregates the local model parameters of each client accordingly to generate unified global model parameters, which are then distributed to all clients. This enables the local performance prediction model updated based on the global model parameters to integrate the distribution characteristics and knowledge from multiple participants and multimodal data, thereby accurately predicting and recommending the optimal nearest neighbor graph type and parameter configuration for new query datasets.

[0012] Compared to traditional methods that rely on expert experience or brute-force grid search, the embodiments of this application can reduce the time cost and computational overhead of building and optimizing nearest neighbor graph indexes, and effectively overcome the shortcomings of traditional indexing methods that are limited to a single data source and have insufficient generalization ability.

[0013] Furthermore, this embodiment operates within a federated learning framework, where each participant only shares local model parameters and test datasets used for evaluation, without sharing any original business data. This ensures data privacy and protects the data rights of each participant, resolving the contradiction between data privacy protection and global retrieval performance optimization in a distributed environment. It provides a reliable technical path for building efficient and secure vector retrieval services across domains and modalities. Attached Figure Description

[0014] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0015] Figure 1 This is a flowchart illustrating an exemplary embodiment of a nearest neighbor graph indexing method based on federated learning.

[0016] Figure 2 This is a schematic diagram of an intelligent retrieval process illustrated in an exemplary embodiment of this application;

[0017] Figure 3 This is a block diagram illustrating a nearest neighbor graph indexing system according to an exemplary embodiment of this application;

[0018] Figure 4 This is a block diagram illustrating an electronic device according to an exemplary embodiment of this application;

[0019] Figure 5 This is a block diagram illustrating a nearest neighbor graph indexing device based on federated learning, as shown in an exemplary embodiment of this application. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] This application provides a nearest neighbor graph indexing method based on federated learning. This nearest neighbor graph indexing method is executed by a nearest neighbor graph indexing system, which includes multiple clients and a federated server. Figure 1 This is a flowchart illustrating an exemplary embodiment of a nearest neighbor graph indexing method based on federated learning, as shown in this application. Figure 1 As shown, the nearest neighbor graph indexing method 100 includes at least the following steps S110 to S140:

[0023] Step S110: Each client participating in federated learning constructs its local nearest neighbor graph index based on the preset nearest neighbor graph parameter configuration. By executing the nearest neighbor graph index, the true value of the nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter is obtained. Based on the true value of the nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter configuration, the local sample dataset of each client is constructed.

[0024] Each client constructs the sample dataset using its local business dataset. For example, the business dataset is vectorized to obtain a vector dataset. Multiple nearest neighbor graph parameter configurations are constructed for the vector dataset, and the ground truth value of the nearest neighbor graph retrieval performance under each nearest neighbor graph parameter configuration is obtained. In this way, multiple samples can be constructed through multiple nearest neighbor graph parameter configurations. Each sample includes the vector dataset, the nearest neighbor graph parameter configuration, and the ground truth value of the nearest neighbor graph retrieval performance corresponding to the nearest neighbor graph parameter configuration.

[0025] In step S120, each client trains a local performance prediction model based on its local sample dataset to predict the performance of nearest neighbor graph retrieval, and obtains the corresponding local model parameters; wherein the local performance prediction models of each client have the same network architecture, and a portion of the samples in the local sample dataset of each client form the local test dataset of the client.

[0026] In some embodiments, the local sample dataset of each client is divided into a training dataset and a test dataset, wherein the number of samples in the test dataset is less than a set value, for example, the sample dataset is divided into a training dataset and a test dataset in a ratio of 85% and 15%.

[0027] It is worth noting that in this embodiment, the local performance prediction models of each client have the same network architecture, so as to provide an implementation basis for model aggregation in subsequent steps. This embodiment does not limit the network structure of the local performance prediction model, and those skilled in the art can set it as needed.

[0028] In step S130, the federated server participating in the federated learning obtains the local model parameters and test datasets of each client, and obtains a global test dataset based on the test datasets of each client, so as to calculate the contribution quantification value corresponding to each client using the global test dataset.

[0029] After each client completes the training of its local index model, it uploads the trained local model parameters (also known as model weights) and its test dataset to the federated server. The federated server aggregates all the test datasets uploaded by clients, merging them into a unified global test dataset that covers multimodal data features. This global test dataset provides a public and objective benchmark for subsequently evaluating the performance and contribution of each client's model. Based on this benchmark, the federated server calculates the marginal contribution of a client by comparing the performance differences of federated models containing that specific client, thus laying the foundation for calculating its contribution quantification. This process only exchanges the test dataset used for evaluation and does not involve the original business data, ensuring data privacy.

[0030] The contribution quantification value refers to a numerical value used to measure the relative contribution of each client's local model parameters to the construction of the global model parameters.

[0031] In some embodiments, the contribution quantification value of each client is calculated based on the Shapley value in cooperative game theory.

[0032] In step S140, the federated server aggregates all local model parameters based on the contribution quantification value to obtain global model parameters, so that each client updates its local performance prediction model based on the global model parameters, and uses the updated local performance prediction model to build a corresponding nearest neighbor graph index for the dataset to be queried.

[0033] Since the data modalities and distributions of the sample datasets from different clients are different, a single client model can only learn the local characteristics of its local sample dataset. The federated server aggregates the model parameters trained from different sample datasets, and the resulting global model parameters actually encode the common rules and complementary knowledge across modalities. This enables the updated model to make more general and accurate performance predictions and parameter recommendations when faced with new query datasets of any modality, in order to improve the generalization ability and global optimality of the indexing strategy.

[0034] Furthermore, the federated server only operates on model parameters and test datasets, and the aggregation process takes place in the parameter space without touching any original business data. This meets the strict protection requirements for data privacy and trade secrets in multi-participant scenarios such as e-commerce, making intelligent indexing collaboration across departments and institutions possible, and fundamentally resolving the contradiction between privacy sensitivity and global performance optimization.

[0035] In some embodiments, the federated server distributes the global model parameters to each client, and each client updates its local performance prediction model based on the global model parameters, so as to use the updated local performance prediction model to build the corresponding nearest neighbor graph index for the dataset to be queried.

[0036] like Figure 1 As shown in the nearest neighbor graph indexing method, this embodiment trains local performance prediction models with the same network architecture on each client, learning the complex mapping relationship between their respective data features and the performance of the nearest neighbor graph index. The federated server constructs a global test dataset, calculates contribution quantification values, and aggregates the local model parameters of each client accordingly, generating unified global model parameters which are then distributed to all clients. This allows the local performance prediction model, updated based on the global model parameters, to integrate the distribution characteristics and knowledge from multiple participants and multimodal data, thereby accurately predicting and recommending the optimal nearest neighbor graph type and parameter configuration for new query datasets. Compared to traditional methods that rely on expert experience or brute-force grid search, this embodiment reduces the time cost and computational overhead of nearest neighbor graph index construction and tuning, effectively overcoming the shortcomings of traditional indexing methods that are limited to a single data source and lack generalization ability. Furthermore, this embodiment operates within a federated learning framework, where each participant only shares local model parameters and test datasets used for evaluation, without sharing any original business data. This ensures data privacy and protects the data rights of each participant, resolving the contradiction between data privacy protection and global retrieval performance optimization in a distributed environment. It provides a reliable technical path for building efficient and secure vector retrieval services across domains and modalities.

[0037] In some embodiments, the samples include local vector datasets for each client, nearest neighbor graph parameter configurations, and ground truth values ​​of nearest neighbor graph retrieval performance corresponding to the nearest neighbor graph parameter configurations. Different samples correspond to different nearest neighbor graph parameter configurations.

[0038] This embodiment can construct various nearest neighbor graphs with different graph types and parameter configurations for each client participating in federated learning, based on their local vector datasets, and collect vector retrieval performance metrics corresponding to each nearest neighbor graph. Sample data is formed based on the vector dataset, the parameter configuration of the nearest neighbor graph, and the ground truth value of its corresponding nearest neighbor graph retrieval performance. Training datasets and test datasets are constructed according to a preset partitioning ratio.

[0039] In one example, the nearest neighbor graph parameters include at least one of the following: neighbor graph index type, maximum number of neighbors, number of nearest neighbors, and candidate set size during graph construction. The neighbor graph index type may include, for example, HNSW, NSG, Vamana (a basic graph construction algorithm), etc., and those skilled in the art can flexibly set it according to their needs. Vamana is a basic graph construction algorithm; its specific implementation scheme can be found in related technologies, and will not be elaborated upon in this embodiment.

[0040] In one example, the nearest neighbor graph retrieval performance includes at least one of the following metrics: recall rate, number of queries per second, and distance calculation count. The recall rate, also known as Recall, refers to the proportion of true nearest neighbors among the top K results returned by the system in a single query. The number of queries per second, also known as QPS (Query Per Second), indicates how many query requests the system can successfully process per second. A high QPS indicates that the system can support a higher number of concurrent users and a faster response time. For interactive applications requiring millisecond-level response times, such as real-time recommendations or sound-based object recognition, QPS is a key indicator for ensuring system smoothness. The distance calculation count, also known as DistComp, refers to the total number of vector similarity (or distance) calculations performed during index building and query execution.

[0041] In one example, the nearest neighbor graph parameters of the sample are generated based on grid search enumeration. The specific generation process can be found in relevant technologies, and will not be described in detail in this embodiment.

[0042] This embodiment incorporates performance metrics such as Recall, QPS, and DistComp into the same prediction framework, and balances different business objectives through adjustable weights, enabling the local performance prediction model to both learn performance patterns and support subsequent on-demand decision-making.

[0043] In some embodiments, the local performance prediction model includes a feature extraction network and a performance prediction network; the feature extraction network extracts latent features from the vector dataset through a self-attention mechanism and a graph neural network, and extracts explicit features from the vector dataset through a preset statistical method, and outputs a fused feature formed by concatenating the latent features and the explicit features; the performance prediction network takes the combination of the fused feature output by the feature extraction network and the corresponding nearest neighbor graph parameter configuration as input, and outputs the nearest neighbor graph retrieval performance prediction value corresponding to the nearest neighbor graph parameter configuration.

[0044] In one example, the feature extraction network extracts the latent features of the vector dataset through a graph Transformer network. The preset statistical method can be flexibly set, and this embodiment does not limit it.

[0045] In one example, the performance prediction network includes a feedforward neural network, which can be referenced in related technologies, and will not be described in detail in this embodiment.

[0046] This embodiment captures the implicit structural features of the vector dataset using a graph Transformer and extracts the explicit distribution features using statistical methods, enabling the local performance prediction model to fully perceive the characteristics of different modalities and provide a reliable basis for performance prediction.

[0047] In some embodiments, the training process of the local performance prediction model for each client includes: inputting samples from the training dataset of each client into its local performance prediction model to obtain the nearest neighbor graph retrieval performance prediction value corresponding to the sample. The ground truth of retrieval performance based on the nearest neighbor graph of this sample. Predicted performance values ​​of nearest neighbor graph retrieval Calculate the loss value corresponding to each nearest neighbor graph retrieval performance metric; derive a weighted loss function based on the loss values ​​corresponding to all nearest neighbor graph retrieval performance metrics. .

[0048] For example, the loss value of Recall can be calculated based on the Mean Squared Error (MSE). QPS loss value The loss value of Distcomp The weighted loss function can then be calculated using the following formula. .

[0049] (1)

[0050] in, These are adjustable hyperparameters. The dependence of each nearest neighbor graph retrieval performance metric can be initialized according to the specific task. Thus, through a weighted loss function... The original three prediction modules were optimized into a single multi-objective prediction module.

[0051] In some embodiments, step S120 above, which involves calculating the contribution quantification value corresponding to each client using the global test dataset, includes: calculating the contribution quantification value of each client based on the Shapley value in cooperative game theory; wherein, for any client i, its contribution quantification value is calculated based on the marginal contribution of client i to different client alliances, the marginal contribution being obtained by comparing a first performance loss and a second performance loss; the first performance loss is the weighted loss of the aggregation model corresponding to any client alliance on the global test dataset regarding the retrieval performance of all nearest neighbor graphs, the aggregation model being obtained by aggregating the local model parameters of all clients in the client alliance; the second performance loss is the weighted loss of the new aggregation model corresponding to the new client alliance formed after client i joins the client alliance on the global test dataset regarding the retrieval performance of all nearest neighbor graphs, the new aggregation model being obtained by aggregating the local model parameters of all clients in the new client alliance.

[0052] Any client federation of client i refers to the set of clients that, among the n clients participating in federated learning, consists of the other clients excluding client i.

[0053] In one example, for any client federation, the federation server trains a local performance prediction model of the initial state using the test dataset of all clients in the client federation to obtain the aggregate model corresponding to that client federation. The computational complexity in this case is exponential, reaching [a certain level of complexity]. .

[0054] In one example, the aggregated model is obtained through a gradient reuse strategy, which includes: for any client consortium, obtaining the gradient loss information generated by all clients in the client consortium during their local performance prediction model training phase; obtaining the aggregated model corresponding to the client consortium by linearly combining the gradient loss information of all clients in the client consortium; the gradient loss information is obtained through the gradient information generated by the clients during the backpropagation process when training their local performance prediction models. During model training, each client generates gradient information during the backpropagation process of each iteration, and the gradient loss information relative to the initial state of the model in the last iteration can be calculated using the gradient information generated in each iteration. This embodiment uses the gradient reuse method to reconstruct the aggregated model, which can reduce the computational complexity to linear complexity, for example, achieving... .

[0055] This embodiment uses a global test dataset to evaluate the generalization performance of each client model, and efficiently calculates the contribution quantification value of each client through a gradient reuse strategy, ensuring that the aggregate weights can truly reflect the data value and achieve high-quality federated aggregation.

[0056] After obtaining the aggregation model of the client consortium, the weighted loss of the aggregation model on the global test dataset can be calculated according to the above formula (1) to obtain the first performance loss. Similarly, for the new client consortium formed after client i joins the client consortium, the weighted loss of the new aggregation model on the global test dataset can also be calculated according to the above formula (1) to obtain the second performance loss.

[0057] In some embodiments, the federated server in step S130 above aggregates all local model parameters based on the contribution quantification value to obtain global model parameters, specifically including: allocating aggregation weights to the local model parameters of each client based on the contribution quantification value; and weighting the local model parameters of all clients based on the aggregation weights to obtain the global model parameters.

[0058] This embodiment uses the contribution quantification value to objectively measure the marginal contribution of each client's sample data to improving model prediction performance. The federated server assigns differentiated aggregation weights to each client's local model parameters based on this contribution quantification value. In subsequent model parameter aggregation, model parameters from clients with higher contributions will be assigned greater aggregation weights. Through this contribution-based weighted aggregation mechanism, the global model parameters can fully integrate verified and more effective feature representations and mapping relationships from different clients, thereby optimizing the overall prediction performance of the global model parameters on cross-modal query datasets.

[0059] In one example, the process of allocating aggregate weights to the local model parameters of each client based on the contribution quantification value includes: normalizing the contribution quantification value to obtain a normalized contribution value; obtaining the sample size proportion of each client based on the number of samples in the training dataset of each client and the total number of samples in the training dataset of all clients; and obtaining the aggregate weights corresponding to the local model parameters of each client based on the sample size proportion and the normalized contribution value.

[0060] To avoid weight imbalance caused by individual client contribution values ​​being too large or too small, this example uses a weight normalization mechanism to normalize the contribution values. The normalization algorithm can be flexibly set, for example, to the Min-Max normalization algorithm.

[0061] In this example, the allocation of aggregation weights considers both data quality and data size, with the contribution quantification primarily reflecting data quality. However, if a client has a large data volume, its local model parameters can typically learn more stable and universal data distribution characteristics from a richer dataset. Ignoring the data size factor may dilute this statistically significant knowledge in the global model parameters. Therefore, this example introduces a sample size ratio, which helps preserve these more generalized patterns learned from large amounts of data when allocating weights.

[0062] In some embodiments, step S130 above uses an updated local performance prediction model to construct a corresponding nearest neighbor graph index for the dataset to be queried, specifically including: inputting the dataset to be queried and a variety of preset nearest neighbor graph parameter configurations into the updated local performance prediction model to obtain the nearest neighbor graph retrieval performance prediction value corresponding to each set of nearest neighbor graph parameter configurations; in response to the target nearest neighbor graph parameter configuration determined according to the nearest neighbor graph retrieval performance prediction value, constructing a nearest neighbor graph index that matches the target nearest neighbor graph parameter configuration based on the dataset to be queried.

[0063] In practical applications, if a client has a vector retrieval requirement, the client vectorizes the dataset to be queried. It then inputs the vectorized dataset and several preset nearest neighbor graph parameter configurations into an updated local performance prediction model, obtaining predicted values ​​for Recall, QPS, and Distcom metrics for each configuration. The client filters out suitable configurations based on the performance metrics required for vector retrieval and selects the relatively optimal configuration as the target nearest neighbor graph parameter configuration. Alternatively, the client can display the predicted values ​​for Recall, QPS, and Distcom metrics for each configuration to the user, allowing the user to choose the target nearest neighbor graph parameter configuration. A nearest neighbor graph index is then generated based on the target nearest neighbor graph parameter configuration, enabling vector retrieval of the vectorized dataset to be queried using this index.

[0064] The nearest neighbor graph indexing method of this embodiment will be explained in detail below, taking the application of the nearest neighbor graph indexing method to a cross-modal similarity retrieval system for e-commerce platforms as an example. The platform comprises multiple business departments, each holding product-related data in different modalities. To achieve unified and efficient retrieval services such as "image-based product search," "semantic product search," and "sound-based object identification," the platform needs to construct a nearest neighbor graph index capable of supporting cross-modal vector similarity search.

[0065] Assume the product cross-modal similarity retrieval system comprises a federated server and five clients, each client corresponding to a specific business department or data source, holding vectorized datasets of different modalities; where:

[0066] Client 1 corresponds to the user behavior analysis data source. Its local data consists of a high-dimensional sparse vector generated from user-product interaction behavior, with a dimension of 1024.

[0067] Client 2 corresponds to the product text information data source. Its local data includes product titles and descriptions, which are encoded by large language models such as BERT-large to generate 768-dimensional dense vectors and contain some synthetic augmented samples.

[0068] Client 3 corresponds to the product visual information data source, and its local data is product images. A 2048-dimensional image feature vector is extracted through a ResNet-152 neural network.

[0069] Client 4 corresponds to the product audio information data source. Its local data consists of audio clips from short videos introducing products, which are processed and projected by the Whisper speech encoder and the CLAP model to obtain a 512-dimensional audio semantic vector.

[0070] Client 5 corresponds to the comprehensive attribute data source. Its local data consists of structured and semi-structured data such as SKU attributes and comment sentiment analysis results. After multilayer perceptron fusion processing, a 384-dimensional comprehensive feature vector is obtained.

[0071] All clients do not share the original data; they only collaborate to optimize the nearest neighbor graph index parameters through a federated server.

[0072] It is worth noting that in this embodiment, all clients only participate in the federated collaborative training of the local performance prediction model and do not share any original business data.

[0073] The following will be combined with the appendix Figure 2 This paper details the implementation process of the nearest neighbor graph indexing method in this embodiment of the product cross-modal similarity retrieval system.

[0074] First, execute step 1. Each client restarts its local vector dataset and performs a full combination enumeration of the following preset nearest neighbor graph parameter configurations:

[0075] Neighbor graph index type

[0076] Maximum number of neighbors

[0077] Nearest neighbor count

[0078] For each configuration (GraphType, NN, K), each client uses this configuration to build a local nearest neighbor graph index, performs Top-K nearest neighbor retrieval, and calculates three performance metrics: the proportion of the true nearest neighbor appearing in the top K positions in the returned Top-K results to obtain the Recall ground truth. Where K takes one of the values ​​1, 10, or 30 from the configuration. The true QPS (queries per second) is obtained by measuring the number of queries that can be processed under single-threaded, no-caching, and steady-state load conditions. Simultaneously, the total number of distance calculations performed during the indexing process is obtained to determine the truth value of DistComp. Therefore, each client constructs its local sample dataset, with each sample containing a complete vector dataset, nearest neighbor graph parameter configuration, and ground truth values ​​for nearest neighbor graph retrieval performance. Client i's sample dataset is then divided into training datasets at a ratio of 85% and 15%. With the test dataset .

[0079] It is worth noting that this embodiment does not limit the method of obtaining the true nearest neighbor. For example, a K-nearest neighbor graph can be constructed for the local vector dataset of each client, and the true nearest neighbor can be obtained based on the K-nearest neighbor graph.

[0080] Then, step 2 is performed, where each client constructs a local performance prediction model based on its local vector dataset. This local performance prediction model includes a feature extraction network and a performance prediction network. The feature extraction network is used to extract a comprehensive representation that combines structural and statistical characteristics from each client's local vector dataset.

[0081] For example, the vector dataset of client i A fully connected graph is constructed, where each vector serves as a node, and the edge weights between nodes are calculated using cosine similarity. A graph Transformer network is then used to encode this fully connected graph. For example, by employing multi-layer self-attention mechanisms and message passing mechanisms in graph neural networks, complex nonlinear dependencies in the high-dimensional space are captured, ultimately outputting... The latent features are denoted as Meanwhile, statistical methods were used to analyze the vector dataset. Extract statistical attributes, such as vector dimension, mean L2 norm, standard deviation, mean nearest neighbor distance, local density estimation, and clustering coefficient, to obtain, for example, The dominant trait is denoted as Finally, the latent and explicit features are concatenated along the channel dimension to obtain the fused features output by the feature extraction network. .

[0082] The performance prediction network input includes the fused features from the feature extraction network output. and the current nearest neighbor graph parameter configuration This embodiment configures the current nearest neighbor graph parameters. Encoding is performed; for example, GraphType is one-hot encoded into a 3D vector, and NN and K are normalized to the [0,1] interval to form a 6D vector, denoted as . The above-mentioned fusion features With 6-dimensional vectors The concatenated data is then input into the performance prediction network.

[0083] In one example, the performance prediction network is a feedforward neural network consisting of three fully connected layers with hidden layer dimensions of 256, 128, and 64, respectively, and the ReLU activation function is used. This performance prediction network outputs a three-dimensional vector, representing the predicted values ​​for Recall, QPS, and DistComp. .

[0084] Thus, step 2 completes the construction of the local performance prediction model.

[0085] Next, step 3 is performed, where each client trains the constructed local performance prediction model using its local training dataset. The training process is as follows:

[0086] For any sample in the local training dataset of client i, the nearest neighbor graph parameters of that sample are configured in the pre-vector dataset and input into the local performance prediction model to obtain the predicted values ​​of the above three performance metrics. , and For the predicted and true values ​​of these three performance indicators, calculate the mean squared error loss of each performance indicator. Because e-commerce platforms rely on various performance metrics to varying degrees in different business scenarios—for example, product search focuses more on recall rate, while real-time recommendation prioritizes QPS—this embodiment introduces adjustable hyperparameters. The weighted loss function is calculated based on Formula 1 above. .

[0087] In this embodiment, for cross-modal product retrieval tasks, the following settings can be configured: By minimizing The model parameters of the local performance prediction model are optimized using backpropagation and the Adam optimizer. After the local performance prediction model is trained, the local test dataset is compared with the local model parameters of the trained local performance prediction model. This includes encrypting graph Transformer weights, statistical feature processing layers, and feedforward neural network weights before uploading them to the federated server, instead of uploading the original business data, index structure, or original performance metric values, thus ensuring data privacy and security.

[0088] Then, step 4 is executed, where the federated server receives the local model parameters uploaded by the five clients. Afterwards, the federated server starts the contribution evaluator to objectively measure the value of each client's local model parameters to the global model parameters.

[0089] For example, this contribution evaluator consists of two parts: a model testing network and a contribution quantification network. The model testing network merges the test datasets from each client into a global test dataset. For each client, create its corresponding client alliance S and the aggregation model corresponding to each client alliance S, and then test the model on the global test dataset. The above inference is performed on the aggregation model to calculate the loss between the predicted and actual values ​​of its performance metrics. The contribution quantification network uses the Shapley value from cooperative game theory to fairly distribute contributions. Let N = {1, 2, 3, 4, 5}, then n = |N|. If S is a subset of any other client that does not include client i, then the contribution quantification value of client i can be calculated using the following formula (2):

[0090] (2)

[0091] Among them, in the above formula (2) For the first performance loss, For the second performance loss, and The calculation process can be found in the previous text. The calculation process will not be described in detail in this embodiment.

[0092] To avoid the high communication and computational costs associated with repetitive training and frequent model aggregation, this embodiment employs a gradient reuse strategy. Each client caches its gradient vectors during model training to construct gradient loss information before and after model training based on the gradient vectors from each iteration. Each client synchronously uploads this gradient loss information to the federated server, eliminating the need for additional computational overhead. Thus, for any client consortium S, its aggregated model can be approximated by a linear set of gradient loss information from all clients in consortium S, thereby reducing computational complexity from exponential to linear. Reduced to linear level This allows for the calculation of contribution quantification values ​​to be completed within seconds on a scale of five clients.

[0093] Next, step 5 is executed, where the federated server quantifies the contribution value of each client. Assign aggregation weights to the local model parameters of each client, and perform parameter aggregation based on the aggregation weights to obtain the global model parameters. .

[0094] In this embodiment, the aggregation weight allocation takes into account both data quality and data size. Data quality is represented by a contribution quantification value, while data size is represented by the proportion of the sample size.

[0095] In one example, the contribution quantification value is normalized using the following formula (3) to obtain the normalized contribution value of client i. :

[0096] (3)

[0097] In one example, the aggregate weight of client i is calculated using the following formula (4):

[0098] (4)

[0099] In this way, the aggregation weights comprehensively consider both data quality and data scale to achieve more accurate and efficient aggregation of local model parameters. The federated server can then apply these aggregation weights to the local model parameters. Weighted fusion is performed to obtain the final global model parameters. global model parameters The global model parameters are then distributed to each client. It has integrated the knowledge and data characteristics of all clients, based on global model parameters. The constructed local performance prediction model has cross-modal generalization ability and can accurately predict the predicted values ​​of performance metrics under different nearest neighbor graph parameter configurations for any new query dataset, regardless of whether it comes from text, images, audio or a combination thereof.

[0100] Finally, in step 6, when there is a new query dataset, such as a set of product image features that needs to be used to build an efficient vector index, the federated server inputs the statistical features of the query dataset and all the preset nearest neighbor graph parameter configurations into the updated local performance prediction model to obtain the predicted values ​​of the three performance indicators corresponding to each configuration.

[0101] The optimal configuration can be selected based on actual business needs. If the search accuracy is emphasized, such as "searching for goods by image", the nearest neighbor graph parameter configuration corresponding to the highest Recall prediction value can be selected first. If the response speed is emphasized, such as real-time recommendation, the nearest neighbor graph parameter configuration corresponding to the highest QPS prediction value and the lower DistComp prediction value can be selected.

[0102] After selecting the nearest neighbor graph parameter configuration, the client can use this configuration to generate or load the corresponding nearest neighbor graph index on its local data for subsequent vector retrieval.

[0103] This application also provides a nearest neighbor graph indexing system. Figure 3 This is a block diagram illustrating a nearest neighbor graph indexing system in an exemplary embodiment of this application, such as... Figure 3 As shown, the system includes a federated server and multiple clients:

[0104] The client is configured to construct its local nearest-neighbor graph index based on preset nearest-neighbor graph parameter configurations, obtain the ground truth value of nearest-neighbor graph retrieval performance corresponding to each nearest-neighbor graph parameter by executing the nearest-neighbor graph index, construct the local sample dataset of each client based on the ground truth value of nearest-neighbor graph retrieval performance corresponding to each nearest-neighbor graph parameter configuration, and train a local performance prediction model for predicting the performance of nearest-neighbor graph retrieval based on its local sample dataset to obtain the corresponding local model parameters; wherein the local performance prediction models of each client have the same network architecture, and a portion of the samples in the local sample dataset of each client forms the local test dataset of the client.

[0105] The federated server is used to obtain the local model parameters and test datasets of each client, obtain a global test dataset based on the test datasets of each client, and use the global test dataset to calculate the contribution quantification value corresponding to each client; and aggregate all local model parameters based on the contribution quantification value to obtain global model parameters, so that each client updates its local performance prediction model based on the global model parameters, and uses the updated local performance prediction model to build a corresponding nearest neighbor graph index for the dataset to be queried.

[0106] In some embodiments, the federated server is used to calculate the contribution quantification value of each client based on the Shapley value in cooperative game theory; wherein, for any client i, its contribution quantification value is calculated based on the marginal contribution of client i to different client federations, the marginal contribution being obtained by comparing a first performance loss and a second performance loss; the first performance loss is the weighted loss of the aggregation model corresponding to any client federation on the global test dataset with respect to the retrieval performance of all nearest neighbor graphs, the aggregation model being obtained by aggregating the local model parameters of all clients in the client federation; the second performance loss is the weighted loss of the new aggregation model corresponding to the new client federation formed after client i joins the client federation on the global test dataset with respect to the retrieval performance of all nearest neighbor graphs, the new aggregation model being obtained by aggregating the local model parameters of all clients in the new client federation.

[0107] In some embodiments, the federated server is configured to allocate aggregate weights to the local model parameters of each client based on the contribution quantification value; and to weight the local model parameters of all clients based on the aggregate weights to obtain the global model parameters.

[0108] In some embodiments, the federated server is configured to normalize the contribution quantification value to obtain a normalized contribution value; obtain the sample size proportion of each client based on the number of samples in the training dataset of each client and the total number of samples in the training dataset of all clients; and obtain the aggregate weights corresponding to the local model parameters of each client based on the sample size proportion and the normalized contribution value.

[0109] In some embodiments, the client is configured to input the dataset to be queried and a variety of preset nearest neighbor graph parameter configurations into the updated local performance prediction model to obtain the nearest neighbor graph retrieval performance prediction value corresponding to each set of nearest neighbor graph parameter configurations; in response to the target nearest neighbor graph parameter configuration determined based on the nearest neighbor graph retrieval performance prediction value, a nearest neighbor graph index matching the target nearest neighbor graph parameter configuration is constructed based on the dataset to be queried.

[0110] Figure 4This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, a hardware acceleration device 410, and non-volatile memory 412, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 412 into memory 408 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0111] Figure 5 This is a block diagram illustrating an exemplary embodiment of a nearest neighbor graph indexing device based on federated learning, which can be applied to, for example... Figure 4 The electronic device shown implements the technical solution of this application. The nearest neighbor graph indexing device may include a model training unit 520, a contribution calculation unit 530, and a federated aggregation unit 540, wherein:

[0112] The dataset creation unit 510 is used to construct a local nearest graph index for each client participating in federated learning based on a preset nearest graph parameter configuration. By executing the nearest graph index, the true value of the nearest graph retrieval performance corresponding to each nearest graph parameter is obtained, and a local sample dataset for each client is constructed based on the true value of the nearest graph retrieval performance corresponding to each nearest graph parameter configuration.

[0113] Model training unit 520 is used to train a local performance prediction model for predicting the performance of nearest neighbor graph retrieval based on the local sample dataset of each client, and obtain the corresponding local model parameters; wherein the local performance prediction models of each client have the same network architecture, and a portion of the samples in the local sample dataset of each client form the local test dataset of the client.

[0114] The contribution calculation unit 530 is used to obtain the local model parameters and test datasets of each client by the federated server participating in the federated learning, and to obtain a global test dataset based on the test datasets of each client, so as to calculate the contribution quantification value corresponding to each client using the global test dataset.

[0115] The federated aggregation unit 540 is used by the federated server to aggregate all local model parameters based on the contribution quantification value to obtain global model parameters, so that each client updates its local performance prediction model based on the global model parameters, and uses the updated local performance prediction model to build a corresponding nearest neighbor graph index for the dataset to be queried.

[0116] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0117] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0118] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0119] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0120] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0121] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0122] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0123] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0124] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0125] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A nearest neighbor graph indexing method based on federated learning, characterized in that, The method includes: Each client participating in federated learning constructs its local nearest neighbor graph index based on the preset nearest neighbor graph parameter configuration. By executing the nearest neighbor graph index, the true value of the nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter is obtained. Based on the true value of the nearest neighbor graph retrieval performance corresponding to each nearest neighbor graph parameter configuration, the local sample dataset of each client is constructed. Each client trains a local performance prediction model based on its local sample dataset to predict the performance of nearest neighbor graph retrieval, and obtains the corresponding local model parameters. The local performance prediction models of each client have the same network architecture, and a portion of the samples in each client's local sample dataset forms the client's local test dataset. The federated server participating in the federated learning obtains the local model parameters and test datasets of each client, and obtains a global test dataset based on the test datasets of each client, so as to calculate the contribution quantification value corresponding to each client using the global test dataset; The federated server aggregates all local model parameters based on the contribution quantification value to obtain global model parameters, so that each client updates its local performance prediction model based on the global model parameters, and uses the updated local performance prediction model to build a corresponding nearest neighbor graph index for the dataset to be queried.

2. The method according to claim 1, characterized in that, The calculation of the contribution quantification value for each client using the global test dataset includes: The contribution quantification value of each client is calculated based on the Shapley value in cooperative game theory; Wherein, for any client i, its contribution quantification value is calculated based on the marginal contribution of client i to different client alliances, and the marginal contribution is obtained by comparing the first performance loss and the second performance loss; The first performance loss is the weighted loss of the aggregation model corresponding to any client consortium on the global test dataset with respect to the retrieval performance of all nearest neighbor graphs. The aggregation model is obtained by aggregating the local model parameters of all clients in the client consortium. The second performance loss is the weighted loss of the new aggregation model corresponding to the new client alliance formed after client i joins the client alliance, on the global test dataset regarding the retrieval performance of all nearest neighbor graphs. The new aggregation model is obtained by aggregating the local model parameters of all clients in the new client alliance.

3. The method according to claim 1, characterized in that, The federated server aggregates all local model parameters based on the contribution quantification value to obtain global model parameters, including: Based on the contribution quantification value, aggregate weights are assigned to the local model parameters of each client; The local model parameters of all clients are weighted based on the aggregate weights to obtain the global model parameters.

4. The method according to claim 3, characterized in that, The allocation of aggregate weights based on the contribution quantification value to the local model parameters of each client includes: The quantified contribution value is normalized to obtain a normalized contribution value; The sample size proportion of each client is obtained by comparing the number of samples in the training dataset of each client with the total number of samples in the training dataset of all clients. Based on the sample size ratio and the normalized contribution value, the aggregate weights corresponding to the local model parameters of each client are obtained.

5. The method according to claim 1, characterized in that, The step of constructing a corresponding nearest neighbor graph index for the dataset to be queried using the updated local performance prediction model includes: The dataset to be queried and the preset multiple nearest neighbor graph parameter configurations are input into the updated local performance prediction model to obtain the nearest neighbor graph retrieval performance prediction value corresponding to each set of nearest neighbor graph parameter configurations; In response to the target nearest neighbor graph parameter configuration determined based on the predicted performance value of the nearest neighbor graph retrieval, a nearest neighbor graph index matching the target nearest neighbor graph parameter configuration is constructed based on the dataset to be queried.

6. The method according to claim 1, characterized in that, The samples in the sample dataset include the client's local vector dataset, the nearest neighbor graph parameter configuration, and the ground truth value of the nearest neighbor graph retrieval performance corresponding to the nearest neighbor graph parameter configuration; The nearest neighbor graph parameters include at least one of the following: neighbor graph index type, maximum number of neighbors, number of nearest neighbors, and candidate set size during graph construction; The nearest neighbor graph retrieval performance includes at least one of the following metrics: recall rate, number of queries processed per second, and number of distance calculations.

7. The method according to claim 6, characterized in that, The local performance prediction model includes a feature extraction network and a performance prediction network; The feature extraction network extracts latent features from the vector dataset through a self-attention mechanism and a graph neural network, and extracts explicit features from the vector dataset through a preset statistical method, and outputs a fused feature formed by splicing the latent features and the explicit features. The performance prediction network takes the combination of the fused features output by the feature extraction network and the corresponding nearest neighbor graph parameter configuration as input, and outputs the nearest neighbor graph retrieval performance prediction value corresponding to the nearest neighbor graph parameter configuration.

8. A nearest neighbor graph indexing system, characterized in that, The system includes multiple clients and a federated server; The client is used to construct its local nearest graph index based on the preset nearest graph parameter configuration, obtain the true value of the nearest graph retrieval performance corresponding to each nearest graph parameter by executing the nearest graph index, and construct the local sample dataset of each client based on the true value of the nearest graph retrieval performance corresponding to each nearest graph parameter configuration. Each client trains a local performance prediction model based on its local sample dataset to predict the performance of nearest neighbor graph retrieval, and obtains the corresponding local model parameters. Each client's local performance prediction model has the same network architecture, and a portion of the samples in each client's local sample dataset forms the client's local test dataset. The federated server is used to obtain the local model parameters and test datasets of each client, obtain a global test dataset based on the test datasets of each client, and use the global test dataset to calculate the contribution quantification value corresponding to each client; and aggregate all local model parameters based on the contribution quantification value to obtain global model parameters, so that each client updates its local performance prediction model based on the global model parameters, and uses the updated local performance prediction model to build a corresponding nearest neighbor graph index for the dataset to be queried.

9. An electronic device, characterized in that, include: processor; as well as A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images