A data space-oriented private domain data federated retrieval method and system

Abstract

The embodiment of the application discloses a private domain data federated search method and system for data space, relates to the technical field of computers, and can realize effective and accurate query of scattered private domain data in a data space while meeting the requirement of a data owner in local control and management of private domain data. The method calls the search capability of a local private domain data source of a federation, thereby realizing effective search of scattered private domain data in the data space without the need of establishing centralized indexes. Based on a probability local sensitive hash table, the method can fuzzily select a target data source in which stored private domain data is consistent with a user query intention without knowing original data in the private domain data source, and sends a search request to the target data source, thereby reducing the risk of leakage of the original data in the private domain data source, and realizing accurate query of scattered private domain data in the data space.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and system for private domain data federation retrieval oriented towards data space. Background Technology

[0002] The core concept of Data Space is to break down the silos formed by heterogeneous and scattered data by adopting the "on-demand integration" approach, so as to achieve logical consistency in the discovery and utilization of cross-domain data, while meeting the requirements of data owners to control and manage their private domain data locally, thereby protecting data rights.

[0003] However, the local control and management requirements of data owners over private domain data limit the possibility of building traditional centralized indexes, making it difficult to achieve effective and accurate queries of private domain data scattered throughout the data space. Summary of the Invention

[0004] The purpose of this application is to provide a private domain data federation retrieval method and system for data space, which can achieve effective and accurate querying of scattered private domain data in data space while meeting the requirements of data owners to control and manage their private domain data locally.

[0005] In a first aspect, embodiments of this application provide a private domain data federation retrieval method oriented towards data space, the method comprising:

[0006] The global retrieval agent responds to the query string sent by the user by mapping the query string to the Hamming space to obtain the first Hamming code corresponding to the query string;

[0007] The global retrieval agent obtains each second Hamming code from multiple preset Hamming codes, wherein the Hamming distance between the second Hamming code and the first Hamming code is not greater than the preset Hamming distance. It also selects each candidate data source associated with each second Hamming code from the private domain data sources of each federation in the data space. Different preset Hamming codes are associated with different subspaces in the semantic space, and different subspaces are associated with different private domain data in the private domain data sources of each federation.

[0008] The global retrieval agent obtains the target data distribution features corresponding to each candidate data source from a preset probabilistic locality sensitive hash table. The target data distribution features are used to characterize the data distribution of the candidate data source in each subspace associated with each second Hamming code. The probabilistic locality sensitive hash table contains the data distribution features corresponding to each private domain data source of each federation. The data distribution features are used to characterize the data distribution of the private domain data source of the federation in different subspaces of the semantic space.

[0009] The global retrieval agent determines the semantic relevance score between each candidate data source and the query string based on the obtained target data distribution characteristics;

[0010] The global retrieval agent selects one or more target data sources from the candidate data sources based on the semantic relevance scores between each candidate data source and the query string.

[0011] The global retrieval agent distributes retrieval requests carrying the query string to the one or more target data sources, integrates the various retrieval result lists returned by the one or more target data sources, and returns them as a query result list to the user.

[0012] A second aspect of this application provides a private domain data federation retrieval system for a data space, the system comprising: a global retrieval agent and private domain data sources of various federations in the data space that are communicatively connected to the global retrieval agent, wherein:

[0013] The global retrieval agent is used to respond to a query string sent by a user, map the query string to a Hamming space, and obtain the first Hamming code corresponding to the query string. Different subspaces in the semantic space are associated with different preset Hamming codes.

[0014] The global retrieval agent is also used to obtain each second Hamming code from multiple preset Hamming codes, wherein the Hamming distance between the second Hamming code and the first Hamming code is not greater than the preset Hamming distance, and to select each candidate data source associated with each second Hamming code from the private domain data sources of each federation in the data space. Different preset Hamming codes are associated with different subspaces in the semantic space, and different subspaces are associated with different private domain data in the private domain data sources of each federation.

[0015] The global retrieval agent is also used to obtain the target data distribution features corresponding to each of the candidate data sources from a preset probabilistic locality sensitive hash table. The target data distribution features are used to characterize the data distribution of the candidate data sources in each subspace associated with each second Hamming code. The probabilistic locality sensitive hash table contains the data distribution features corresponding to each private domain data source of each federation. The data distribution features are used to characterize the data distribution of the private domain data sources of the federation in different subspaces of the semantic space.

[0016] The global retrieval agent is also used to determine the semantic relevance score between each candidate data source and the query string based on the obtained target data distribution characteristics;

[0017] The global retrieval agent is also used to select one or more target data sources from the candidate data sources based on the semantic relevance scores between each candidate data source and the query string, and to distribute retrieval requests carrying the query string to the one or more target data sources;

[0018] Each federation's private domain data source is used to respond to a received retrieval request by retrieving its own stored private domain data based on the query string in the retrieval request, generating a list of retrieval results and returning it to the global retrieval agent;

[0019] The global retrieval agent is also used to integrate the retrieval result lists returned by the one or more target data sources and return them as a query result list to the user.

[0020] A third aspect of this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the private domain data federation retrieval method for data space as described in the first aspect.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the private domain data federation retrieval method for data space as described in the first aspect.

[0022] A fifth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the private domain data federation retrieval method for data space as described in the first aspect.

[0023] As can be seen from the above technical solution, this application introduces federated retrieval technology. The global retrieval agent calls the local retrieval capabilities of the federated private domain data sources, thereby achieving effective retrieval of scattered private domain data in the data space without the need to build a centralized index. Moreover, based on a probabilistic locality-sensitive hash table, the global retrieval agent can fuzzily select target data sources whose stored private domain data matches the user's query intent (i.e., is semantically related to the query string) without knowing the original data in the private domain data source, and send retrieval requests to them. This can reduce the risk of leakage of the original data in the private domain data source, and avoid the waste of network bandwidth and computing resources, as well as the problems of low query efficiency and decreased accuracy caused by private domain data sources unrelated to the user's query intent responding to retrieval requests. Thus, while meeting the requirements of data owners to control and manage their private domain data locally, it realizes effective and accurate querying of scattered private domain data in the data space. Attached Figure Description

[0024] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating the implementation of a private domain data federation retrieval method oriented towards data space, provided in an embodiment of this application;

[0026] Figure 2 A schematic diagram illustrating a data source selection process provided in an embodiment of this application;

[0027] Figure 3 A schematic diagram illustrating the construction process of a fuzzy locality-sensitive hash table provided in an embodiment of this application;

[0028] Figure 4 A schematic diagram of the structure of a private domain data federation retrieval system oriented towards data space provided in this application embodiment;

[0029] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0031] In recent years, numerous enterprises, research institutes, and other organizations have accumulated vast amounts of private domain data during their operations. This data is stored by its respective data owners on scattered hardware and software. To unlock the value of this heterogeneous and multi-source private domain data, much research in industry and academia has focused on employing data space technologies to facilitate the discovery, flow, and utilization of large-scale data.

[0032] In the data space, the primary concern is how to efficiently discover scattered data (i.e., how to effectively and accurately query scattered data). Traditional discovery techniques for scattered data utilize centralized search engines, which periodically crawl data across the entire network and build a centralized index to achieve efficient data discovery. However, data owners in the data space need to maintain control over their local data, thus preventing their data from leaving their local area to build a centralized index, making it difficult to achieve effective and accurate queries for scattered private data in the data space.

[0033] Based on the above analysis, and addressing the difficulty in effectively and accurately querying dispersed private domain data in the data space in related technologies, this application provides a private domain data federated retrieval method and system for the data space. It introduces federated retrieval technology to achieve effective retrieval of dispersed private domain data in the data space without the need to build a centralized index. Furthermore, it selects target data sources that match the user's query intent based on a probability locality sensitive hash table and sends retrieval requests to them, which can effectively improve the accuracy of the final query result list.

[0034] First, to facilitate understanding of the technical solutions provided in this application, the main technical concepts involved in the embodiments of this application will be briefly explained below.

[0035] Data Space: It is a critical infrastructure supporting the secure and reliable flow of large-scale, distributed, and heterogeneous data. By building a multi-party consensus mechanism and a security technology system, it achieves secure interoperability and sovereignty control of heterogeneous data. Its core value lies in achieving logical-layer data fusion across systems and organizations while ensuring the data owner's control through standardized interface protocols and dynamic integration demand response mechanisms. Based on metadata mutual recognition, semantic alignment, and virtualization adapters, it enables interoperability of multi-source heterogeneous data such as databases, the Internet of Things (IoT), and documents without physical migration.

[0036] Federated search, also known as federated search or consortium search, uses a global search agent built on top of one or more search engines to retrieve information from various sources. A user simply submits a query, which is then distributed by the global search agent to the participating search engines (or databases, etc.). The global search agent then integrates the search results received from the search engines and presents them to the user. Federated search is primarily used to integrate different information resources within a single large organization (such as a company) or an entire network. Unlike distributed search, federated search does not require a centralized index, but it does require centralized coordination of searchable resources. This includes coordinating queries distributed to various search engines and merging the results returned by each search engine.

[0037] See Figure 1 The diagram shown is an implementation flowchart of a private domain data federation retrieval method for data space provided in this application embodiment. The method may include the following steps:

[0038] Step S101: In response to receiving the query string sent by the user, the global retrieval agent maps the query string to the Hamming space to obtain the first Hamming code corresponding to the query string.

[0039] In specific implementation, the global retrieval agent receives a query request sent by the user carrying a query string (which represents the user's query intent), and then maps the query string to the Hamming space (i.e. converts it into binary form) according to a preset Hamming code encoding scheme (such as a preset hash function or character encoding standard) to obtain the first Hamming code corresponding to the query string.

[0040] Step S102: The global retrieval agent obtains each second Hamming code from multiple preset Hamming codes whose Hamming distance from the first Hamming code is no greater than the preset Hamming distance, and selects each candidate data source associated with each second Hamming code from the private domain data sources of each federation in the data space. Different preset Hamming codes are associated with different subspaces in the semantic space, and different subspaces are associated with different private domain data in the private domain data sources of each federation.

[0041] In practice, for each preset Hamming code, the global retrieval agent detects whether the preset Hamming code and the first Hamming code have the same value at each position, and counts the number of different values ​​at the same position to obtain the Hamming distance between the preset Hamming code and the first Hamming code.

[0042] The global retrieval agent obtains each preset Hamming code from multiple preset Hamming codes whose Hamming distance to the first Hamming code is no greater than a preset Hamming distance (e.g., 1) as each second Hamming code, and selects each private domain data source associated with each of the second Hamming codes from the private domain data sources of each federation in the data space as candidate data sources, that is, selects private domain data sources with private domain data that are semantically related to the query string as candidate data sources.

[0043] Step S103: The global retrieval agent obtains the target data distribution features corresponding to each candidate data source from a preset probabilistic locality sensitive hash table. The target data distribution features are used to characterize the data distribution of the candidate data source in each subspace associated with each second Hamming code. The probabilistic locality sensitive hash table contains the data distribution features corresponding to each private domain data source of each federation. The data distribution features are used to characterize the data distribution of the private domain data source of the federation in different subspaces of the semantic space.

[0044] In practical implementation, the probability-locality-sensitive hash table can contain data distribution characteristics such as the amount of data or the probability of data distribution in different subspaces of the semantic space for each private domain data source in the federation. By using the data distribution characteristics of the private domain data sources in the semantic space recorded by the probability-locality-sensitive hash table to semantically represent the private domain data sources, it is possible to efficiently select target data sources whose stored private domain data is more consistent with the user's query intent (i.e., highly semantically related to the query string).

[0045] Therefore, this application realizes a semantic representation of data sources based on locality-sensitive hashing. By comparing the data distribution characteristics of each candidate data source in the probabilistic locality-sensitive hash table, the differences in data semantic distribution between data sources with strong heterogeneity in the data space can be effectively measured. Furthermore, through fuzzing, the probabilistic locality-sensitive hash table does not contain the original data of private domain data sources (i.e., private domain data), thereby avoiding the leakage of original data to the global retrieval agent and thus protecting data rights.

[0046] In this embodiment, the present application utilizes the concept of Locality Sensitive Hashing (LSH) to generate data distribution characteristics of private data sources in different subspaces of the semantic space as semantic features of the private data sources. This allows for the fuzzification of the data source semantic representation (i.e., the probabilistic LSH table) based on the data distribution of the data source in different subspaces, thus preventing the leakage of private data to the global retrieval agent. The global retrieval agent then searches the fuzzified semantic representation of the data source for the data distribution characteristics associated with the subspaces associated with each second Hamming code, quickly obtaining the target data distribution characteristics corresponding to each candidate data source.

[0047] Step S104: The global retrieval agent determines the semantic relevance score between each candidate data source and the query string based on the obtained target data distribution characteristics.

[0048] In specific implementation, the global retrieval agent integrates the distribution features of each target data corresponding to each candidate data source to obtain the semantic relevance score of each candidate data source; for example, it calculates the statistics (such as sum or mean) of the amount of data (or data distribution probability, etc.) of each candidate data source in each subspace associated with each second Hamming code, and then determines the statistics as the semantic relevance score between each candidate data source and the query string, so as to quantify the semantic relevance between the query string and the private domain data stored in each candidate data source.

[0049] Step S105: The global retrieval agent selects one or more target data sources from the candidate data sources based on the semantic relevance scores between each candidate data source and the query string.

[0050] In practice, the global retrieval agent selects one or more candidate data sources with high semantic relevance scores (and that meet the set rules, such as the set number of data sources to be selected) from the candidate data sources as target data sources, thereby obtaining target data sources with high semantic relevance to the query string.

[0051] Step S106: The global retrieval agent distributes retrieval requests carrying the query string to the one or more target data sources, integrates the various retrieval result lists returned by the one or more target data sources, and returns them as a query result list to the user.

[0052] In practice, the one or more target data sources retrieve their own stored private domain data based on the query string in the retrieval request. They then return the retrieved private domain data semantically related to the query string as retrieval results in a list format to the global retrieval agent. The global retrieval agent then summarizes these retrieval result lists and returns the resulting query result list to the user. Thus, the global retrieval agent distributes retrieval requests carrying the query string to the one or more target data sources, thereby utilizing the local retrieval capabilities of the private domain data sources to retrieve dispersed private domain data. Based on federated retrieval technology, this enables effective retrieval of dispersed private domain data in the data space without the need for a centralized index.

[0053] Optionally, the one or more target data sources return the data identifiers of each private domain data that they retrieve and that are semantically related to the query string as search results to the global search agent.

[0054] It should be noted that, considering that private domain data generated by enterprises and other organizations often contains a large amount of privacy information, data owners often require their private domain data to meet the requirement of "usable but not visible". Therefore, in the data space scenario, for the global retrieval agent, which is usually considered an "honest but curious" third-party component, by introducing a probabilistic locality sensitive hash table and further introducing data identifiers (such as digital object identifiers), the global retrieval agent can avoid obtaining or accessing private domain data throughout the entire federated retrieval process, thereby better protecting the data rights of data owners in the data discovery process.

[0055] As can be seen from the above technical solution, this application introduces federated retrieval technology. The global retrieval agent calls the local retrieval capabilities of the federated private domain data sources, thereby achieving effective retrieval of scattered private domain data in the data space without the need to build a centralized index. Moreover, based on a probabilistic locality-sensitive hash table, the global retrieval agent can fuzzily select target data sources whose stored private domain data matches the user's query intent (i.e., is semantically related to the query string) without knowing the original data in the private domain data source, and send retrieval requests to them. This can reduce the risk of leakage of the original data in the private domain data source, and avoid the waste of network bandwidth and computing resources, as well as the problems of low query efficiency and decreased accuracy caused by private domain data sources unrelated to the user's query intent responding to retrieval requests. Thus, while meeting the requirements of data owners to control and manage their private domain data locally, it realizes effective and accurate querying of scattered private domain data in the data space.

[0056] In an optional embodiment, before the global retrieval agent obtains the distribution features of each target data corresponding to each candidate data source from a preset probability-locality sensitive hash table, the method further includes:

[0057] The global retrieval agent receives fuzzy local sensitive hash tables sent by the private domain data sources of each federation in the data space. The fuzzy local sensitive hash tables are used to record the amount of data of the private domain data sources in different subspaces of the semantic space.

[0058] The global retrieval agent determines the total amount of data for each data source in each federation based on the received fuzzy local sensitive hash tables.

[0059] For each federation's private domain data source, the global retrieval agent performs normalization calculations based on the total amount of data from the private domain data source and the amount of data from the private domain data source in different subspaces of the semantic space, to obtain the data distribution probability of the private domain data source in different subspaces of the semantic space.

[0060] The global retrieval agent uses the data distribution probability of each private domain data source of each federation in different subspaces of the semantic space as data distribution features, and integrates each data distribution feature with the preset Hamming code associated with different subspaces of the semantic space to obtain the probabilistic locality sensitive hash table.

[0061] In practice, the global retrieval agent performs normalization calculations based on the fuzzy local sensitive hash tables sent by each private domain data source. That is, it calculates the probability that the data of each private domain data source is distributed in each subspace. Thus, the amount of data distributed in each subspace reflects the overall distribution of the corresponding data source in the semantic space. This allows the global retrieval agent to construct the semantic representation of the data source without extracting private domain data, which is beneficial to protecting data rights.

[0062] In this embodiment, unlike traditional locality-sensitive hashing, the hash bucket mapped to each preset Hamming code in this application is not a set of vectors, but a mapping from the data source (or data source identifier) ​​to the probability of data distribution in the hash bucket (i.e., a subspace of the semantic space). Therefore, by subsequently determining the semantic relevance score between the private domain data source and the query string based on the probability of data distribution in the hash bucket, it is possible to avoid leaking private domain data to the global retrieval agent, while effectively reflecting the semantic feature differences of each data source.

[0063] Optionally, the global retrieval agent comprises two components: a global retrieval unit and a data source semantic representation module. The global retrieval unit performs federated retrieval, enabling users to discover private domain data scattered throughout the data space simply by sending a query string to the global retrieval agent. The data source semantic representation module constructs a semantic representation model for each data source (i.e., constructs a probabilistic locality-sensitive hash table) based on the fuzzy locality-sensitive hash tables sent by each private domain data source. This allows for accurate semantic representation of each data source without disclosing its original data, thus providing a basis for subsequent data source selection.

[0064] In this embodiment, this application designs a semantic representation model of the data source with fuzziness and a semantic representation model construction process without the participation of the original data, so as to achieve fuzziness processing in all stages of federated retrieval, thereby avoiding the leakage of the original data and protecting data rights.

[0065] Optionally, the global retrieval agent determines the semantic relevance score between each candidate data source and the query string based on the obtained target data distribution characteristics, including:

[0066] For each candidate data source, the global retrieval agent determines the sum of the data distribution probabilities of the candidate data source in each subspace associated with each second Hamming code as the semantic relevance score between the candidate data source and the query string.

[0067] In practice, the global retrieval agent uses the sum of probabilities that data is distributed in each subspace (i.e., each hash bucket) associated with each second Hamming code as the semantic relevance score between the candidate data source and the query string. This ensures that the semantic relevance score can accurately reflect the data semantic distribution of each candidate data source, thereby guaranteeing the reliability of subsequent data source selection.

[0068] Optionally, the private domain data source encodes its stored private domain data into dense vectors and stores the private domain data and its corresponding dense vectors. When a user sends a query string to the global retrieval agent, the global retrieval agent selects several private domain data sources that are most similar to the dense vectors of the query string (i.e., the overall dense vectors stored by the private domain data source are most similar to the dense vectors of the query string) based on the preset Hamming distance and the semantic relevance score, and uses them as target data sources.

[0069] Taking a preset Hamming distance of 1 and requiring the selection of one private domain data source as the target data source as an example, see [link to example]. Figure 2 The diagram illustrates the data source selection process. The user query (i.e., the query string) is encoded into a dense vector and then mapped to a semantic space. The preset Hamming code associated with the subspace to which the mapped position of this dense vector belongs is determined as the first Hamming code. Then, preset Hamming codes with a Hamming distance of less than 1 from the first Hamming code and their associated private domain data sources are detected, resulting in three second Hamming codes: 010101, 011101, and 011111, and two candidate data sources: data source 1 and data source 2. For each candidate data source, the sum of the data distribution probabilities in each subspace associated with each second Hamming code is calculated. For data source 1, the sum of its associated data distribution probabilities (i.e., semantic relevance score) is calculated to be 0.1 + 0.2 + 0.4 = 0.7. For data source 2, the sum of its associated data distribution probabilities (i.e., semantic relevance score) is calculated to be 0.2 + 0.1 + 0.1 = 0.4. Since 0.7 > 0.4, data source 1 is selected as the target data source.

[0070] It is understandable that, since private domain data is mapped from semantic space to Hamming space, Hamming distance can be used to measure semantic relevance and thus serve as a basis for data source selection. Therefore, this application does not directly use the query string, but instead calculates all preset Hamming codes within a certain Hamming distance of the first Hamming code. That is, it uses a depth-first search algorithm to traverse the subspace associated with each preset Hamming code within the preset Hamming distance, which can improve the robustness of data source selection. Furthermore, the semantic relevance score is obtained by summing the data source relevance scores (i.e., data distribution probabilities) corresponding to these subspaces. This allows for the selection of several private domain data sources that are semantically most relevant to the user query. As a result, the private domain data sources that best match the user's query intent can be selected efficiently and accurately to distribute retrieval requests.

[0071] Optionally, the target data source is determined by converting the query string into a dense vector and matching the resulting dense vector with the dense vectors corresponding to various private domain data stored in the data source. Thus, by introducing a dense index (i.e., creating a set of dense vectors and indexing accordingly) in the data owner's local storage, high semantic relevance between the various search results obtained during local retrieval and the query string can be ensured.

[0072] Optionally, before the global retrieval agent receives the fuzzy locality-sensitive hash tables sent by the private domain data sources of each federation in the data space, the method further includes:

[0073] The global retrieval agent initializes and saves a global random hyperplane group, which includes the number of embedding dimensions, the number of Hamming code dimensions, and the number of locality-sensitive hash tables.

[0074] The global retrieval agent sends the global random hyperplane group to the private domain data sources of each federation in the data space, so that each private domain data source executes the local sensitive hash algorithm according to the global random hyperplane group to obtain a fuzzy local sensitive hash table, and sends the fuzzy local sensitive hash table to the global retrieval agent;

[0075] The locality-sensitive hash algorithm includes the following steps:

[0076] Define a hash function, and divide the semantic space into multiple subspaces associated with preset Hamming codes based on the hash function;

[0077] Each piece of private data in the private data source is encoded into a dense vector and stored.

[0078] After mapping each of the dense vectors to the semantic space, the number of dense vectors associated with different subspaces in the semantic space is counted.

[0079] The number of dense vectors associated with different subspaces in the semantic space and the preset Hamming codes associated with different subspaces in the semantic space are integrated to obtain the fuzzy local sensitive hash table.

[0080] In practice, the global retrieval agent first initializes and saves a set of global random hyperplanes (i.e., a set of global random hyperplanes containing three parameters: the number of embedding dimensions of the predefined dense retrieval code, the number of Hamming code dimensions, and the number of locality-sensitive hash tables).

[0081] Then, the global retrieval agent sends the global random hyperplane group to the private domain data sources of each federation in the data space. After receiving the global random hyperplane group, the private domain data source immediately executes the locality-sensitive hashing algorithm based on the global random hyperplane group.

[0082] Specifically, the private domain data source first initializes the local sensitive hash table, for example, by initializing the amount of data in each subspace of the private domain data source to 0; then it enters the construction phase, defines a hash function, encodes each piece of private domain data in the private domain data source into a dense vector to obtain a set of dense vectors and stores them (for example, in a vector database), so that subsequent local retrieval based on this set of dense vectors can be performed to obtain a list of retrieval results corresponding to the retrieval request. At the same time, for each dense vector in the set of dense vectors in the private domain data source, it is mapped to the hash space (i.e., mapped to the corresponding hash bucket, with each hash bucket corresponding one-to-one with a subspace in the semantic space) according to the global random hyperplane set and hash function, and the amount of data distributed in the corresponding hash bucket (i.e., hash bucket value) is updated. Based on this, a fuzzy local sensitive hash table (i.e., containing different preset Hamming codes and their associated hash bucket values) is constructed, and the constructed fuzzy local sensitive hash table is returned to the global retrieval agent.

[0083] For example, see Figure 3 The diagram illustrates the construction process of a fuzzy locality-sensitive hash table. The global retrieval agent defines a mapping from the semantic space to the Hamming space and partitions the Hamming space using a predefined hash function (i.e., constructing different preset Hamming codes and their mapped hash buckets). This Hamming space is then used to replace the original semantic space to describe the semantic distribution of the data, thereby reducing spatial dimensionality while achieving initial semantic fuzziness. Within the Hamming space, the global retrieval agent uses a probabilistic locality-sensitive hash table to represent the semantic features of each data source, such as... Figure 3As shown, in the probabilistic locality sensitive hash table, the hash bucket values ​​corresponding to the preset Hamming codes 010101, 010111, and 010110 are 3, 1, and 5, respectively, indicating that the vector database of the relevant private domain data source has 3, 1, and 5 dense vectors in the subspaces corresponding to 010101, 010111, and 010110 in the semantic space, respectively.

[0084] Optionally, the global retrieval agent sends the global random hyperplane group to the private domain data sources of each federation in the data space, including:

[0085] In response to receiving a request to join the federation from a private domain data source in the data space, the global retrieval agent records the metadata of the private domain data source to identify it as a private domain data source in the federation. The agent then sends the global random hyperplane group and the first response packet corresponding to the request to join the federation to the private domain data source. This allows the private domain data source to generate a fuzzy local sensitive hash table based on the global random hyperplane group, and then send the fuzzy local sensitive hash table and the second response packet corresponding to the first response packet to the global retrieval agent.

[0086] In this embodiment, considering the dynamic nature of private domain data sources joining the federation, this application designs a method for private domain data sources to construct a semantic representation of the data source when sending a request to join the federation to the global retrieval agent.

[0087] Specifically, the global retrieval agent initializes and saves the global random plane group before it begins receiving requests from private domain data sources to join the federation.

[0088] When a private data source in the data space sends a request to join the federation to the global retrieval agent, the global retrieval agent records the metadata of the private data source (including data source identifier, address, and possible descriptive information, etc.) and sends the saved global random hyperplane group along with the response packet (i.e., the first response packet) to the private data source. Subsequently, the private data source executes the Locality Sensitive Hash algorithm based on the global random hyperplane group to construct a fuzzy Locality Sensitive Hash table, and after the fuzzy Locality Sensitive Hash table is constructed, it sends it along with the response packet (i.e., the second response packet) to the global retrieval agent.

[0089] Optionally, the method further includes:

[0090] When a data addition operation is detected, each private domain data source of the federation encodes the newly added data as a dense vector. After determining the subspace associated with the dense vector based on the hash function, the number of dense vectors associated with the subspace is incremented by 1 in the fuzzy local sensitive hash table.

[0091] When a data deletion operation is detected, each private domain data source of the federation obtains the dense vector corresponding to the data to be deleted, determines the subspace associated with the dense vector based on the hash function, and then decrements the number of dense vectors associated with the subspace by 1 in the fuzzy local sensitive hash table.

[0092] The global retrieval agent periodically requests fuzzy locality-sensitive hash tables from the private domain data sources of each federation, and updates probabilistic locality-sensitive hash tables based on the requested fuzzy locality-sensitive hash tables.

[0093] In this embodiment, the present application designs an incremental update step to adapt to scenarios where private domain data changes dynamically. Each private domain data source only needs to fully construct the fuzzy locality-sensitive hash table once and save the random hyperplane group from the global retrieval agent; subsequently, when the private domain data in the private domain data source changes dynamically (such as being added, deleted, or modified), it is only necessary to incrementally update the fuzzy locality-sensitive hash table according to the corresponding operation, that is, to dynamically update the fuzzy locality-sensitive hash table according to the dynamic changes of the private domain data.

[0094] For example, when adding a new piece of data, the private domain data source encodes the new data as a dense vector, runs a hash function on a random hyperplane group to obtain the preset Hamming code corresponding to the new data, and then increments the hash bucket value corresponding to the preset Hamming code by 1 in the fuzzy locality sensitive hash table.

[0095] For example, before deleting a piece of old data, the private domain data source accesses the dense vector of the data, runs a hash function on a random hyperplane group to obtain the preset Hamming code corresponding to the old data, and then decrements the hash bucket value corresponding to the preset Hamming code by 1 in the fuzzy locality sensitive hash table.

[0096] For example, when modifying a piece of data, the private domain data source only needs to delete the corresponding old data and add the new data. Therefore, the incremental update operations corresponding to deleting an old piece of data and adding a new piece of data can be run in sequence to complete the incremental update of the fuzzy local sensitive hash table.

[0097] The global retrieval agent updates the semantic features of the private data source by periodically requesting a new fuzzy local sensitive hash table from the private data source, thus achieving this with relatively low construction cost.

[0098] Specifically, after the global retrieval proxy requests the fuzzy locality-sensitive hash table of the private domain data source, it updates the probabilistic locality-sensitive hash table to adjust the semantic representation of the private domain data source. That is: first, the total amount of data in the private domain data source is initialized to 0; during the update phase, each hash bucket of the fuzzy locality-sensitive hash table is traversed, the total amount of private domain data is counted, the probability of the private domain data being distributed in each hash bucket is calculated by normalization, and the corresponding fields of the probabilistic locality-sensitive hash table are adjusted accordingly to obtain the updated probabilistic locality-sensitive hash table.

[0099] In an optional embodiment, the global retrieval agent selects one or more target data sources from the candidate data sources based on the semantic relevance scores between each candidate data source and the query string, including:

[0100] The global retrieval agent selects target data sources from the candidate data sources in descending order of semantic relevance scores, with the number of target data sources not exceeding the preset maximum number of target data sources.

[0101] In practice, the global retrieval agent receives the predefined maximum number of selected data sources and the maximum hash distance (i.e., the preset Hamming distance) as data source selection parameters and initializes them to create a mapping between the private domain data source identifier and its semantic relevance score.

[0102] Then, the data source selection process is executed. If the length of the candidate data source identifier list (i.e., the number of candidate data sources) is less than or equal to the maximum number of data sources to be selected, all candidate data sources are directly selected as the target data source. Otherwise, the query string is mapped to the Hamming space, and a second Hamming code with a Hamming distance less than or equal to the maximum hash distance is obtained. The semantic relevance score of the candidate data sources associated with the second Hamming code is calculated according to the probabilistic locality sensitive hash table. Finally, the candidate data sources are sorted according to the semantic relevance score, and the top n candidate data sources with the highest scores (n is a positive integer, determined according to actual needs) are selected as the target data sources. A list of identifiers for the target data sources is generated for subsequent distribution of retrieval requests.

[0103] In an optional embodiment, the search results list includes: digital object identifiers of various digital objects that are semantically related to the query string, the digital objects being encapsulated from private domain data.

[0104] In practice, each private domain data source encapsulates the private domain data of the data owner into digital objects and stores them. The digital objects may contain digital object identifiers (used to uniquely identify digital objects), metadata (such as descriptive information such as creators), and data entities (i.e., private domain data).

[0105] Upon receiving a retrieval request from the global retrieval agent, the private domain data source returns the digital object identifiers of each digital object semantically related to the query string in the retrieval request as retrieval results, in list form, to the global retrieval agent. The global retrieval agent then integrates the digital object identifiers returned by each private domain data source, generates a list of query results, and returns it to the user. This allows the user to access the corresponding private domain data from the relevant private domain data source. This implements a federated retrieval method based on digital object identifiers, ensuring that the global retrieval agent does not need to acquire private domain data throughout the entire federated retrieval process, thus effectively protecting data rights.

[0106] Optionally, a digital object architecture is used to model the private domain data source of the data owner in the data space as two main components: a digital object registry and a digital object repository, to provide local data indexing and storage services, respectively. The digital object registry provides local retrieval services by building an index from dense vectors obtained through dense encoder embedding to digital object identifiers; the digital object repository stores data entities (i.e., private domain data) and is also responsible for constructing the semantic representation of the private domain data source (i.e., constructing a fuzzy locality-sensitive hash table) to support the data source selection of the global retrieval agent.

[0107] In this embodiment, the present application designs a federated retrieval method oriented towards digital object identifiers. The global retrieval agent returns the digital object identifiers as retrieval results to the user, thereby eliminating the need to obtain private domain data and thus protecting data rights.

[0108] Optionally, the step of integrating the lists of search results returned by the one or more target data sources and returning them as a list of query results to the user includes:

[0109] The global retrieval agent receives the semantic relevance scores between the various numeric objects associated with the retrieval result lists sent by the one or more target data sources and the query string;

[0110] The global retrieval agent reorders the identifiers of each numeric object in each retrieval result list based on the semantic relevance score between each numeric object and the query string.

[0111] The global retrieval agent sends the reordered identifiers of each digital object as a list of query results to the user, so that the user can obtain private domain data related to the semantics of the query string from the data space based on the list of query results.

[0112] In practice, the global retrieval agent reorders the digital object identifiers in the retrieval result lists sent by the one or more target data sources in descending order of the semantic relevance scores between the digital objects associated with the query result list and the query string, so as to integrate the retrieval result lists returned by the one or more target data sources, and returns the reordered list of digital object identifiers as the query result list to the user.

[0113] Taking a global retrieval agent that includes a global retrieval agent and candidate data sources that include a digital object registry and a digital object repository as an example, the re-ranking method based on distributed similarity scores provided in this application will be explained.

[0114] After the global search agent sends a query string to the digital object registry of each selected private domain data source (i.e., each target data source), each digital object registry performs an independent search based on the query string and organizes the retrieved digital object identifiers into a list, returning it to the global search agent. Once the global search agent receives the lists of digital object identifiers returned by each digital object registry, it needs to reorder them to merge them into a unified list and return it to the client (i.e., the user).

[0115] This application designs that each digital object registry, when returning a list of digital object identifiers, also needs to provide a corresponding semantic relevance score list (containing the semantic relevance scores between each digital object associated with the list of digital object identifiers and the query string). The semantic relevance scores in the semantic relevance score list are calculated by the digital object registry when retrieving data. The digital object registry can send the list of digital object identifiers and the semantic relevance score list separately to the global search engine; alternatively, the digital object registry can send both the list of digital object identifiers and the semantic relevance score list to the global search engine together to further improve the efficiency of global search result reordering.

[0116] The global search engine uses a priority queue to reorder all digital object identifiers in the received list of digital object identifiers based on the semantic relevance score list sent by each private domain data source. Thus, the global search engine can efficiently merge the search result list by relying solely on the semantic relevance score, while also meeting the requirement of reordering search results based on semantic relevance.

[0117] In this embodiment, the present application designs a reordering method based on distributed relevance scores. The global retrieval agent uses the semantic relevance scores calculated separately by each target data source to reorder the retrieval results (i.e., digital object identifiers) obtained from each target data source, thereby merging the digital object identifiers retrieved from each target data source into a list to obtain the query result list. This enables the global retrieval agent to efficiently complete the integration of retrieval results even when it can only obtain digital object identifiers.

[0118] Optionally, the list of reordered numeric object identifiers can be truncated according to a set rule (e.g., the number of query results set), and the list of truncated numeric object identifiers can be used as the query result list.

[0119] In an optional embodiment, the method further includes:

[0120] The global retrieval agent invokes a preset large model, generates a response string corresponding to the query string based on the private domain data associated with the query result list, and sends the response string to the user to provide the user with enhanced retrieval generation services based on the large model.

[0121] In this embodiment, considering the significant capabilities and application value of large language model technology in natural language understanding, generation, and information integration in recent years, this application introduces a retrieval enhancement generation technology. By leveraging the generation capabilities of the large language model (i.e., the large model), heterogeneous private domain data associated with the retrieval result list is integrated to generate the response string corresponding to the query string (i.e., the response statement to the question statement corresponding to the query string), thereby providing users with a more user-friendly service.

[0122] Understandably, this application overcomes the shortcomings of federated search technology in effectively ensuring the semantic relevance of search results in a multi-source heterogeneous data space, thus providing a reliable technical foundation for the introduction of search enhancement generation technology (i.e., it can provide large models with private domain data that is highly semantically related to the query string), thereby ensuring the accuracy of the response statements generated by the large model during the search enhancement generation process.

[0123] It is important to note that in the digital economy era, the data space, as the core infrastructure for realizing the market-based allocation of data elements, profoundly influences the development path of information retrieval technology. In data spaces represented by IDS (Independent Data Storage), private data sources under data sovereignty control exhibit a highly decentralized nature. These heterogeneous data nodes (such as enterprise databases, IoT terminals, and cloud service interfaces) controlled by different data owners often employ differentiated storage strategies, leading to several challenges in the accurate discovery of cross-domain data: First, when the number of data sources is large, if the global retrieval agent indiscriminately sends search requests to search engines of all data sources, it may result in data sources unrelated to the user's query responding to the query, thus wasting network bandwidth and computing resources, and potentially causing problems such as low search efficiency and decreased accuracy; Second, the data owners' requirements for local control and management of data (such as the "usable but not visible" principle) limit the possibility of building traditional centralized indexes. At the same time, the deep integration of emerging paradigms such as retrieval enhancement generation with large language models places higher demands on the semantic understanding capabilities of retrieval systems, requiring retrieval systems to achieve accurate semantic alignment between query intent and distributed data in a multi-source heterogeneous environment.

[0124] To address the aforementioned issues, this application proposes a two-stage optimization mechanism for federated search technology: data source selection and cross-source result reordering.

[0125] In the data source selection phase, the global retrieval agent needs to represent the data source. Unlike related technologies that rely on bag-of-words models and lightweight sampling techniques (such as query word probes) to process the raw data returned by the data source for semantic representation construction, the global retrieval agent needs to establish a shallow semantic representation at the data source level through term frequency statistics (such as extracting statistical features like Term Frequency-Inverse Document Frequency (TF-IDF)) and scale estimation (such as constructing a word distribution profile at the data source level) to prioritize data sources with larger document sizes and higher keyword matching. This application designs a data source semantic representation model based on locality-sensitive hashing, including both local and global layers:

[0126] 1) At the local level, each private domain data source maps its original data from a continuous semantic space to a discrete Hamming space to construct a fuzzy locality-sensitive hash table, such as... Figure 3 As shown, this describes the semantic characteristics of private domain data sources.

[0127] 2) At the global level, the global retrieval agent aggregates the fuzzy local sensitive hash tables sent by each private domain data source and constructs a probabilistic local sensitive hash table as the basis for comparing the semantic features of each data source. In this way, the overall semantic features of each private domain data source are characterized by describing the distribution of vector data.

[0128] Therefore, this application overcomes the shortcomings of related technologies, such as the reliance on data sources to return original data content during the semantic representation construction process, which violates the control principle that data in the data space should not leave the local area, leading to privacy leaks and compliance risks, and the fact that data source selection methods based on word frequency statistics (such as TF-IDF) can only capture word-level co-occurrence features and cannot construct deep semantic representations, resulting in semantic deviation between the selected data source and the query intent.

[0129] In the cross-source result re-ranking stage, unlike related technologies that rely on a centralized and unified re-ranking method using a global retrieval agent (i.e., relying on the prerequisite that the data source can return the original data), such as the global retrieval agent needing to use pre-trained Transformer series models to perform relevance calculation and deep semantic re-ranking on documents returned from multiple sources in order to obtain a list of query results based on content-based re-ranking, this application designs a re-ranking method based on distributed similarity scores. The global retrieval agent re-ranks each retrieval result according to the semantic similarity scores calculated separately by each private domain data source. This overcomes the shortcomings of related technologies, which rely on the data source to return the original data content, violate the control principle that data in the data space should not leave the local area, and cause privacy leaks and compliance risks.

[0130] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0131] This application also provides a private domain data federated retrieval system for data space, which is used to implement the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application. Figure 4 As shown, the system includes: a global retrieval agent and private domain data sources of various federations in the data space that are communicatively connected to the global retrieval agent, wherein:

[0132] The global retrieval agent is used to respond to a query string sent by a user, map the query string to the Hamming space, and obtain the first Hamming code corresponding to the query string;

[0133] The global retrieval agent is also used to obtain each second Hamming code from multiple preset Hamming codes, wherein the Hamming distance between the second Hamming code and the first Hamming code is not greater than the preset Hamming distance, and to select each candidate data source associated with each second Hamming code from the private domain data sources of each federation in the data space. Different preset Hamming codes are associated with different subspaces in the semantic space, and different subspaces are associated with different private domain data in the private domain data sources of each federation.

[0134] The global retrieval agent is also used to obtain the target data distribution features corresponding to each of the candidate data sources from a preset probabilistic locality sensitive hash table. The target data distribution features are used to characterize the data distribution of the candidate data sources in each subspace associated with each second Hamming code. The probabilistic locality sensitive hash table contains the data distribution features corresponding to each private domain data source of each federation. The data distribution features are used to characterize the data distribution of the private domain data sources of the federation in different subspaces of the semantic space.

[0135] The global retrieval agent is also used to determine the semantic relevance score between each candidate data source and the query string based on the obtained target data distribution characteristics;

[0136] The global retrieval agent is also used to select one or more target data sources from the candidate data sources based on the semantic relevance scores between each candidate data source and the query string, and to distribute retrieval requests carrying the query string to the one or more target data sources;

[0137] Each federation's private domain data source is used to respond to a received retrieval request by retrieving its own stored private domain data based on the query string in the retrieval request, generating a list of retrieval results and returning it to the global retrieval agent;

[0138] The global search agent is also used to integrate the various search result lists returned by the one or more target data sources and return them as a query result list to the user.

[0139] Optionally, the system is built on a client-server (C / S) architecture, and the backend program is written in Python.

[0140] Optionally, the private domain data source includes a digital object registry and a digital object repository, which are used to provide local data indexing and storage services, respectively.

[0141] Optionally, the global retrieval agent includes a global retrieval unit and a data source semantic representation module, wherein the global retrieval unit is used to perform federated retrieval to obtain query results, and the data source semantic representation module is used to construct a probability locality sensitive hash table.

[0142] This application also provides an electronic device, see embodiments thereof. Figure 5 , Figure 5 This is a schematic diagram of the electronic device proposed in an embodiment of this application. Figure 5 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the private domain data federation retrieval method for data space disclosed in the embodiments of this application.

[0143] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the private domain data federation retrieval method for data space as disclosed in this application.

[0144] This application also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the private domain data federation retrieval method for data space as disclosed in this application.

[0145] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, systems, devices, storage media, and program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0151] The above provides a detailed description of a private domain data federation retrieval method and system for data space provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data space-oriented private domain data federated retrieval method, characterized in that, The method comprises: The global search agent maps the query string sent by the user to a Hamming space to obtain a first Hamming code corresponding to the query string in response to receiving the query string; The global search agent obtains each second Hamming code with a Hamming distance from the first Hamming code not greater than a preset Hamming distance from a plurality of preset Hamming codes, and selects each candidate data source associated with the second Hamming code from the private domain data sources of each federation in the data space, different preset Hamming codes are associated with different subspaces in the semantic space, and different subspaces are associated with different private domains in the private domain data sources of the federations; The global search agent obtains each target data distribution feature corresponding to each candidate data source from a preset probabilistic local sensitive hash table, wherein the target data distribution feature is used to represent the data distribution of the candidate data source in each subspace associated with the second Hamming code, the probabilistic local sensitive hash table contains each data distribution feature corresponding to each private domain data source of the federation, and the data distribution feature is used to represent the data distribution of the private domain data source of the federation in different subspaces of the semantic space; The global search agent determines the semantic relevance score between each candidate data source and the query string according to the obtained target data distribution feature; wherein the global search agent determines the sum of the data distribution probability of each candidate data source in each subspace associated with the second Hamming code as the semantic relevance score between the candidate data source and the query string; The global search agent selects one or more target data sources from the candidate data sources according to the semantic relevance score between each candidate data source and the query string; The global search agent distributes a search request carrying the query string to the one or more target data sources, integrates the search result list returned by the one or more target data sources, and returns it to the user as a query result list.

2. The method of claim 1, wherein, Before the global search agent obtains each target data distribution feature corresponding to each candidate data source from a preset probabilistic local sensitive hash table, the method further comprises: The global search agent receives a fuzzy local sensitive hash table sent by each private domain data source of each federation in the data space, and the fuzzy local sensitive hash table is used to record the data amount of the private domain data source in different subspaces of the semantic space; The global search agent determines the total amount of data of each data source of each federation according to the received fuzzy local sensitive hash table; The global search agent performs normalization calculation on the total amount of data of each private domain data source of each federation and the data amount of the private domain data source in different subspaces of the semantic space to obtain the data distribution probability of the private domain data source in different subspaces of the semantic space. The global search agent takes the data distribution probability of each private domain data source of each federation in different subspaces of the semantic space as a data distribution feature respectively, integrates each data distribution feature with the preset Hamming code associated with different subspaces in the semantic space, and obtains the probability local sensitive hash table.

3. The method of claim 2, wherein, The target data source is obtained by converting the query string into a dense vector, and matching the converted dense vector with the dense vector corresponding to each private domain data stored by the target data source, to determine the search result list corresponding to the query string. Before the global search agent receives the fuzzy local sensitive hash table sent by each private domain data source of each federation in the data space, the method further comprises: The global search agent initializes a global random hyperplane group and saves it, wherein the global random hyperplane group includes embedding dimension number, Hamming code dimension number and local sensitive hash table number; The global search agent sends the global random hyperplane group to each private domain data source of each federation in the data space, so that each private domain data source performs a local sensitive hash algorithm based on the global random hyperplane group to obtain a fuzzy local sensitive hash table, and sends the fuzzy local sensitive hash table to the global search agent; The local sensitive hash algorithm comprises the following steps: Defining a hash function, and dividing the semantic space into a plurality of subspaces associated with preset Hamming codes based on the hash function; Each private domain data in the private domain data source is encoded into a dense vector and stored; After mapping each dense vector to the semantic space, the number of dense vectors associated with different subspaces in the semantic space is counted; Integrating the number of dense vectors associated with different subspaces in the semantic space and the preset Hamming code associated with different subspaces in the semantic space, to obtain the fuzzy local sensitive hash table.

4. The method of claim 3, wherein, The global search agent sends the global random hyperplane group to each private domain data source of each federation in the data space, comprising: In response to receiving a request to join the federation from a private domain data source in the data space, the global search agent records the metadata of the private domain data source to determine the private domain data source as a private domain data source of the federation, and sends the global random hyperplane group and a first response packet corresponding to the request to join the federation to the private domain data source, so that the private domain data source sends the fuzzy local sensitive hash table generated based on the global random hyperplane group and a second response packet corresponding to the first response packet to the global search agent.

5. The method of claim 3, wherein, The method further comprises: When each private domain data source of each federation detects a data increase operation, the increased new data is encoded into a dense vector, and after determining the subspace associated with the dense vector based on the hash function, the number of dense vectors associated with the subspace in the fuzzy local sensitive hash table is increased by 1. In response to detecting a data deletion operation, each federated private domain data source obtains a dense vector corresponding to the data to be deleted, determines a subspace associated with the dense vector based on the hash function, and then decrements the number of dense vectors associated with the subspace in the fuzzy local sensitive hash table by 1; The global search agent periodically requests the fuzzy local sensitive hash table from each federated private domain data source, and updates the probabilistic local sensitive hash table based on the requested fuzzy local sensitive hash table.

6. The method according to any one of claims 1 to 5, characterized in that, The global search agent selects one or more target data sources from the candidate data sources based on the semantic relevance scores between the respective candidate data sources and the query string, including: The global search agent selects target data sources from the candidate data sources in descending order of the semantic relevance scores, with the number of selected target data sources not exceeding a preset maximum number of data sources.

7. The method according to any one of claims 1 to 5, characterized in that, The search result list includes digital object identifiers of digital objects semantically related to the query string, which are encapsulated by private domain data; The global search agent receives the semantic relevance scores between the digital objects associated with the search result lists sent by the one or more target data sources and the query string, and reorders the digital object identifiers in the search result lists based on the semantic relevance scores between the digital objects and the query string. The global search agent sends the reordered digital object identifiers to the user as a query result list, so that the user can obtain private domain data semantically related to the query string from the data space based on the query result list. The method further includes: The global search agent invokes a preset large model to generate a reply string corresponding to the query string based on the private domain data associated with the query result list, and sends the reply string to the user.

8. The method according to any one of claims 1 to 5, characterized in that, The system includes a global search agent and each federated private domain data source in the data space that is communicatively connected to the global search agent, wherein: In response to receiving a query string sent by a user, the global search agent maps the query string to a Hamming space to obtain a first Hamming code corresponding to the query string; 9. A data space oriented private domain federated search system, characterized in that, The global search agent further obtains, from a plurality of preset Hamming codes, each second Hamming code having a Hamming distance from the first Hamming code not greater than a preset Hamming distance, and selects, from each federated private domain data source in the data space, each candidate data source associated with the second Hamming code, different preset Hamming codes associated with different subspaces in the semantic space, and different subspaces associated with different private domain data in the federated private domain data sources. ​ ​ The global search agent is further configured to obtain respective target data distribution features corresponding to the candidate data sources from a preset probabilistic local sensitive hash table, the target data distribution features being used to represent data distribution of the candidate data sources in respective subspaces associated with the second Hamming codes, and the probabilistic local sensitive hash table containing respective data distribution features corresponding to private domain data sources of each federation, the data distribution features being used to represent data distribution of the private domain data sources of the federation in different subspaces of the semantic space. The global search agent is further configured to determine semantic relevance scores between the candidate data sources and the query string according to the obtained target data distribution features, wherein the global search agent determines a sum of data distribution probabilities of each candidate data source in the subspaces associated with the second Hamming codes as the semantic relevance score between the candidate data source and the query string. The global search agent is further configured to select one or more target data sources from the candidate data sources according to the semantic relevance scores between the candidate data sources and the query string, and distribute a search request carrying the query string to the one or more target data sources. The private domain data source of each federation is configured to search the private domain data stored therein according to the query string in the search request in response to receiving the search request, generate a search result list and return the search result list to the global search agent. The global search agent is further configured to integrate the search result lists returned by the one or more target data sources and return the integrated search result list as a query result list to the user.

Images