System and method for spatially calibrated unstructured data connection

Abstract

The application discloses a kind of unstructured data connection system and method based on space calibration, belong to vector similarity retrieval and machine learning model inference field.The application includes: the system carries out normalization to the embedding vector of first data set and second data set and is calibrated by residual adaptation network, generates candidate connection pair based on inner product Top-k retrieval and radius threshold filtering;To candidate connection pair constructs splicing feature and is output by fully connected network Matching probability, and is divided by double threshold system;Wherein review area calls decision function and verifies connection key consistency.The application significantly reduces the calculation cost while ensuring recall rate through two-stage screening and dynamic threshold decision.

Description

Technical Field

[0001] This invention relates to the fields of database connection processing, vector similarity retrieval, and machine learning model inference, and particularly to an unstructured data connection system and method based on spatial calibration. Background Technology

[0002] In database systems and big data processing, joins are one of the most fundamental and time-consuming operators in query processing. Common implementations include nested loop joins, hash joins, and sorted merge joins. For structured equi-key joins, the optimizer can use statistics, indexes, and predicate pushdown to reduce the size of intermediate results and select an appropriate execution plan. However, when dealing with unstructured data such as images and text, join predicates often exhibit semantic similarity or consistent rules. In engineering practice, objects are usually first encoded as vector embeddings, and then the join relationship is determined based on vector similarity and rules.

[0003] In the scenarios described above, connection determination often requires calling user-defined functions (UDFs) or external services to generate connection keys, perform semantic matching, or complete complex validations. UDFs may involve model inference or cross-system access, resulting in high and fluctuating overhead per call, which is difficult to accurately estimate using traditional cost models. When performing full determination on large-scale first and second datasets, the number of candidate connection pairs increases quadratically with the scale, leading to a rapid amplification of latency and resource consumption.

[0004] To reduce the number of UDF calls, existing systems typically use vector similarity retrieval to generate candidate sets, such as Top-k nearest neighbor retrieval or radius / threshold filtering, combined with approximate nearest neighbor indexes to improve retrieval efficiency. In this process, the Top-k, threshold, or radius parameters are sensitive to recall and candidate size: values ​​that are too small easily lead to missed recalls, while values ​​that are too large introduce a large number of invalid candidates and increase subsequent inference and verification costs. Furthermore, model updates or changes in data distribution can further amplify the risk of parameter mismatch. Since there may be discrepancies between the vector space and the actual connection predicates, engineering systems often perform lightweight model scoring and re-ranking of candidate pairs, and call the UDF for final determination of uncertain candidates entering the verification area, while also using caching to reduce duplicate calls. A stable and universal parameter optimization mechanism is still lacking to uniformly select Top-k, threshold, and verification strategies while minimizing the overall cost under the constraints of minimum recall and minimum precision.

[0005] In actual deployment, the parameter combinations obtained from offline evaluation may be affected by changes in hardware resources, concurrency, and external service status, resulting in deviations during the online phase. Furthermore, incremental data updates can lead to index updates and parameter drift issues, requiring periodic re-evaluation and adjustment of strategies. Therefore, engineering systems typically require corresponding cost measurements and operational statistics to support continuous optimization of parameters and strategies.

[0006] In summary, the following problems urgently need to be addressed in existing technologies: In unstructured data connection scenarios, when connection determination requires calling user-defined decision functions (UDFs) or external services with high costs and fluctuating execution times, how can the candidate pair size be effectively controlled and the number of UDF calls reduced while meeting accuracy constraints such as minimum recall and minimum precision? How can the vector similarity retrieval candidate generation, model probability scoring, and UDF verification and determination be organized into a structured and executable connection processing chain, so that the connection process can output connection results and generate statistical information such as the number of candidates, the number of model inferences, and the number of UDF calls? And how can cost measurement and accuracy evaluation be combined in the offline stage to select and adjust the Top-k parameters, radius parameters, and upper and lower thresholds to reduce the overall connection cost and reduce the instability caused by relying solely on empirical parameter settings? Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide an unstructured data linking system and method based on spatial calibration.

[0008] The objective of this invention is achieved through the following technical solution: an unstructured data link system based on spatial calibration, comprising:

[0009] The spatial calibration module is used to perform L2 normalization on the embedding vectors of the first and second input data sets, and to perform spatial calibration through the residual adaptation network to obtain the calibration vector.

[0010] The candidate generation module is configured to perform a Top-k search of inner product similarity (where k is a predetermined positive integer) on each calibration vector in the first dataset as a query, and combine this with a similarity radius parameter. Generate a similarity threshold to filter and generate candidate connection pairs;

[0011] The probability scoring module is used to construct a feature vector for each candidate connection pair and input the feature vector into a fully connected network to obtain the connection matching probability.

[0012] The threshold decision and verification module is used to obtain the next threshold. With upper threshold The candidate connection pairs are divided into a rejection zone, an acceptance zone, and a verification zone based on the connection matching probability. The candidate connection pairs in the acceptance zone are output as connection results. The candidate connection pairs in the rejection zone are considered to not meet the connection conditions, are judged to have failed, and are directly discarded. The candidate connection pairs in the verification zone are configured to call a user-defined judgment function to obtain the connection key between the first data set and the second data set, and output as connection results when the connection keys match.

[0013] The parameter optimization module is configured to optimize the Top-k parameters based on a recall constraint and cost model. Similarity radius parameter and the lower threshold With upper threshold Perform a search to obtain the target parameters;

[0014] The connection execution module is configured to perform the connection in the order of "calibration-candidate generation-probability scoring-threshold decision and review" and output the connection results.

[0015] Furthermore, the spatial calibration module includes a normalization unit and a residual adaptation network. The residual adaptation network includes a first fully connected layer, a batch normalization layer, a ReLU activation layer, and a second fully connected layer connected in sequence. The residual adaptation network is configured to be trained by minimizing the triplet interval loss function.

[0016] Furthermore, the residual adaptation network is trained using triplet samples, which include anchor points, positive samples, and negative samples, wherein the positive samples have the same connection bond as the anchor points, and the negative samples have a different connection bond than the anchor points.

[0017] Furthermore, the candidate generation module is configured to set the similarity threshold to 1 minus the radius parameter. And when generating candidates, only the Top-k results with a similarity not less than the similarity threshold are retained.

[0018] Furthermore, the fully connected network of the probability scoring module includes a first fully connected layer, a ReLU activation layer, a second fully connected layer, and a Sigmoid activation layer, wherein the input dimension of the first fully connected layer is... , The dimension of the vector embedding; the feature vector is obtained by concatenating the first vector, the second vector, the dimension-wise absolute difference between the two vectors, and the dimension-wise product of the two vectors.

[0019] Furthermore, the threshold decision and review module is configured to: when calling the user-defined decision function for the candidate connection pairs in the review area, adopt a caching mechanism to avoid repeated calls to the same data object and the same row index.

[0020] Furthermore, the parameter optimization module searches for the lower threshold. With upper threshold The optimization objective is to minimize the total cost, which includes the cost of the probability scoring module inferring candidate connection pairs, the cost of the verification region calling the user-defined decision function, and the penalty for false matches generated in the acceptance region. The parameter optimization module is configured to select the threshold pair that minimizes the total cost as the target parameter when the minimum recall and minimum precision constraints are met.

[0021] Furthermore, the parameter optimization module is configured to adjust the Top-k parameters on the validation data based on the recall target during the offline phase. With the similarity radius parameter Estimate and set the lower threshold. With upper threshold Perform a grid search.

[0022] Furthermore, the parameter optimization module constructs the total cost by measuring the average call time of the user-defined decision function and the unit candidate pair inference time of the probability scoring module.

[0023] This invention also provides a method for unstructured data linking based on spatial calibration, comprising the following steps:

[0024] The vector embeddings of the first and second input data sets are L2 normalized and spatially calibrated using a residual adaptation network to obtain the calibration vector.

[0025] Using each calibration vector in the first dataset as a query, a Top-k search of inner product similarity is performed in the calibration vector set of the second dataset (where k is a predetermined positive integer), combined with the radius parameter. Generate a similarity threshold to generate candidate connection pairs;

[0026] For each candidate connection pair, a feature vector is constructed, and the feature vector is input into a fully connected network to obtain the connection matching probability. The feature vector is obtained by concatenating a first vector, a second vector, the dimension-wise absolute difference between the two vectors, and the dimension-wise product of the two vectors.

[0027] Get the lower threshold With upper threshold The candidate connection pairs are divided into a rejection region, an acceptance region, and a verification region based on the connection matching probability.

[0028] The candidate connection pairs in the receiving region are directly output as connection results; for the candidate connection pairs in the verification region, a user-defined judgment function is called to obtain the connection key between the first data set and the second data set, and the connection result is output when the connection key is consistent.

[0029] Furthermore, the method also includes a parameter optimization step: based on the recall constraint and cost model, optimizing the Top-k parameters. Similarity radius parameter and the lower threshold With upper threshold Perform a search to obtain the target parameters.

[0030] Furthermore, the parameter optimization step includes: searching for the lower threshold. With upper threshold The optimization objective is to minimize the total cost, which includes the cost of the probability scoring module inferring candidate connection pairs, the cost of calling the user-defined decision function, and the penalty for false matches generated in the acceptance region. The threshold pair that minimizes the total cost is selected when the minimum recall and minimum precision constraints are met.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] 1. Two-stage threshold partitioning and review mechanism: Through the processing chain of "candidate generation - probability scoring - upper and lower threshold partitioning - review", the call of user-defined decision function (UDF) is strictly limited to candidate pairs in the review area, and the time overhead and resource consumption caused by repeated decision are significantly reduced by caching and reusing decision results.

[0033] 2. Accelerated connection based on vector operations: Candidate generation adopts Top-k retrieval combined with radius threshold screening, and probability scoring adopts vector feature concatenation and batch inference, so that the main computation is concentrated in the vector retrieval and tensor operation stages, which makes it easier to use vector indexing and parallel computing resources to reduce the cost of full comparison.

[0034] 3. Cost-aware adaptive parameter selection: In the offline stage, a total cost model is constructed by measuring the average UDF time and model inference time. Parameters are searched and target parameter combinations are selected under accuracy constraints to keep the execution cost under different data scales and hardware conditions predictable and output statistical information.

[0035] 4. Wide applicability of data types: This invention abstracts the processing objects into a first data set and a second data set, which is not only applicable to traditional database table joins, but also to unstructured data join scenarios based on vector embedding, such as document matching and entity alignment. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the overall architecture of an unstructured data connectivity optimization system based on spatial calibration and dual-threshold partitioning;

[0037] Figure 2 This is a schematic diagram of the structure and processing flow of the space calibration module;

[0038] Figure 3 This is a schematic diagram of the candidate generation module generating candidate connection pairs based on Top-k retrieval and radius threshold filtering;

[0039] Figure 4 This is a schematic diagram showing the connection between the feature construction of the probability scoring module and the connection between the neural network inference output and the probability matching.

[0040] Figure 5 This is a schematic diagram of the partitioning decision and review mechanism based on upper and lower thresholds for candidate connections;

[0041] Figure 6 This is a schematic diagram of the offline training and parameter optimization process. Detailed Implementation

[0042] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are used to explain the technical solutions and operation processes of the present invention, but do not constitute a limitation on the scope of protection of the present invention. Where there is no conflict, the technical features in the following embodiments and implementations can be combined with each other.

[0043] like Figure 1 As shown, this embodiment of the invention provides an unstructured data connection system based on spatial calibration, whose overall architecture includes an input layer, an online connection execution subsystem (container), and an offline parameter optimization subsystem.

[0044] The system input layer includes an input interface for a first data set (Table A) and an input interface for a second data set (Table B). These interfaces provide vector representations (embedded representations) of the data objects in the first and second data sets, respectively. These vector representations can be obtained from existing representation learning models; for example, sentence vector model encoding can be used for text data, image encoder encoding can be used for image data, and cross-modal encoder encoding can be used for multimodal data to obtain a unified vector representation. To facilitate subsequent similarity calculation and threshold filtering, the vector representations can be normalized before entering the online connection execution subsystem.

[0045] The online connection execution subsystem in Figure 1 The system is presented in a top-down pipeline format, comprising a space calibration module, a candidate generation module, a probability scoring module, and a threshold decision and verification module. These modules are connected via a data bus or logical calls. The specific configurations of each module are as follows:

[0046] The spatial calibration module is used to perform L2 normalization on the embedding vectors of the first and second input data sets, and to perform spatial calibration through the residual adaptation network to obtain the calibration vector.

[0047] The candidate generation module is configured to perform a Top-k search of inner product similarity in the calibration vector set of the second data set, using each calibration vector in the first data set as the query (where...). (as a predetermined positive integer), and combined with the similarity radius parameter A similarity threshold is generated to filter the generated candidate connection pairs.

[0048] The probability scoring module is used to construct a feature vector for each candidate connection pair and input the feature vector into the fully connected network to obtain the connection matching probability. The feature vector is obtained by concatenating the first vector, the second vector, the dimension-wise absolute difference between the two vectors and the dimension-wise product of the two vectors.

[0049] The threshold decision and review module is configured to obtain the next threshold. With upper threshold The candidate connection pairs are divided into a rejection region, an acceptance region, and a verification region based on the connection matching probability. The candidate connection pairs in the acceptance region are output as connection results, and the candidate connection pairs in the verification region are configured to call a user-defined judgment function to obtain the connection key between the first data set and the second data set, and output as connection results when the connection keys match.

[0050] like Figure 1 As shown, the review channel is further connected to an external user-defined decision function interface and a cache unit. A loop is formed between the cache unit and the user-defined decision function interface to represent the caching and reuse of the decision results. When the review passes, the output of the review channel and the output of the receiving channel are combined and fed into the system output module. The system output module is used to output a set of connection results and statistical information, where the statistical information may include throughput and count metrics for each stage.

[0051] like Figure 1 As shown, the system also includes an offline parameter optimization subsystem (shown in dashed box), which comprises a cost measurement unit and a parameter search unit. The offline parameter optimization subsystem is connected to the candidate generation module and the threshold decision and verification module via configuration interfaces, and is used to process the Top-k parameters obtained offline. radius parameters Provided to the candidate generation module, the lower threshold is... With upper threshold It provides threshold decision and verification modules, thereby enabling offline training to configure parameters for online execution.

[0052] The online connection execution subsystem is used to output the connection result given a first data set and a second data set. This process corresponds to a spatial calibration-based unstructured data connection method of the present invention, and the specific steps are as follows:

[0053] S1, Receive Vector Input: Receive the vector set of the first data set and the vector set of the second data set, and maintain the mapping relationship between the vectors and the original data object identifiers (IDs).

[0054] S2, Spatial Calibration: The spatial calibration module normalizes and adapts the input vectors to residuals, making the vector space more consistent with the semantic similarity of the connecting predicates in the distribution of training data.

[0055] S3, Candidate Generation: The candidate generation module uses the vectors in the first data set as query vectors and retrieves the Top-k nearest neighbors in the vector set of the second data set based on the inner product similarity; then, it combines the radius parameter... Threshold filtering is performed to retain only those with a similarity not less than a preset threshold (e.g., 1-). Candidate pairs of ) are selected to form a set of candidate connection pairs.

[0056] S4, Probability Scoring: The probability scoring module constructs a feature vector for each candidate connection pair and infers the connection matching probability through a fully connected network. The probability This characterizes the confidence level of the candidate pair in satisfying the connective predicate.

[0057] S5, Threshold Partitioning and Review: The threshold decision and review module is based on probability. With threshold ( , The candidate pairs are split into streams. Among them, the candidate pairs in the review area trigger a user-defined decision function to perform join key extraction and consistency verification, and prioritize querying cache units to reduce redundant calculations.

[0058] S6, Output: Output the final confirmed set of connection pairs and statistical information (such as the number of candidates, the number of inferences, the number of UDF calls, etc.).

[0059] As a preferred embodiment, the present invention also provides the specific structure and processing flow of the space calibration module.

[0060] The spatial calibration module is configured to improve the consistency between the embedded space and the connection predicate, and includes a normalization unit and a residual adaptation unit. The normalization unit includes a first L2 normalization unit and a second L2 normalization unit.

[0061] like Figure 2 As shown, the spatial calibration module performs calibration on the input embedding vector. The input vector first enters the first L2 normalization unit. Subsequently, the normalized vector enters the residual adaptation network, which consists of a first fully connected layer, a batch normalization layer, a ReLU activation layer, and a second fully connected layer connected in series, used to learn the spatial calibration increment of the vector.

[0062] like Figure 2 As shown, the spatial calibration module adopts a residual connection structure, which adds the output of the first L2 normalization unit and the output of the residual adaptation network at the summing node dimension by dimension. The result after addition is then passed through the second L2 normalization unit, and finally outputs the calibration embedding vector.

[0063] During the training phase ( Figure 2 As shown in the dashed panel, this module utilizes triple samples (anchor, positive sample, negative sample) combined with triplet margin loss for optimization. Positive samples and anchors correspond to data pairs that satisfy the connection predicate, while negative samples correspond to cases that do not, thereby narrowing the distance between similar vectors and widening the distance between dissimilar vectors.

[0064] The spatial calibration module outputs calibration vector sets for the first and second data sets respectively, and provides them to the candidate generation module and the probability scoring module.

[0065] As a preferred embodiment, the present invention also provides the specific structure and processing flow of the candidate generation module.

[0066] like Figure 3 As shown, the candidate generation module uses each calibration vector in the first dataset as the query vector and the set of calibration vectors in the second dataset as the search object. The inner product similarity calculation unit calculates the inner product (Dot Product) between the query vector and the index vector, and performs a Top-k search in conjunction with a vector index library (such as HNSW or IVF index), where k is a preset positive integer.

[0067] The retrieved Top-k candidate list enters the radius / similarity threshold filtering unit. This unit filters based on the radius parameter. Determine the truncation threshold, filter out long-tail candidates with similarity below the threshold, and output the simplified candidate connection pairs.

[0068] The candidate generation module outputs the candidate pair identifier and the number of candidate pairs, and passes the candidate connection pair set to the probability scoring module.

[0069] In an alternative implementation, when the vector index is unavailable, the module can switch to "Fallback Mode," which uses matrix multiplication to directly calculate the full similarity and perform Top-k sorting, ensuring high availability of the system.

[0070] As a preferred embodiment, the present invention also provides the specific structure and training method of the probability scoring module.

[0071] like Figure 4As shown, the probability scoring module receives vectors A and B from the candidate connection pair. The feature construction unit generates a combined feature through a concatenation operation. This combined feature, for example, contains four components: vector A, vector B, and the dimension-wise absolute difference between the two vectors. and the dimension-wise product of two vectors The concatenated feature dimensions are: ( (For the embedding dimension).

[0072] The feature vector is input into the fully connected inference network, and mapped to the connection matching probability in the interval [0, 1] via a multilayer perceptron (MLP) and a sigmoid activation function. During the training phase, a binary cross-entropy loss function is used to supervise the learning of network weights based on labeled positive and negative samples. Training sample pairs can consist of candidate nearest neighbors and supplementary positive samples, so that the output probability can represent the confidence level of the candidate pair in satisfying the connection predicate. The probability scoring module associates the candidate pair identifiers with their corresponding probabilities. Output to the threshold decision and review module.

[0073] As a preferred embodiment, the present invention also provides the threshold partitioning and verification processing mechanism.

[0074] like Figure 5 As shown, the threshold decision and review module is based on probability. Divide the data into three intervals:

[0075] 1. Receiving area ( ): The connection is considered successful, and the result is output directly.

[0076] 2. Rejection Zone ( ): The connection is deemed to have failed and is discarded.

[0077] 3. Review area ( ): Enter the review process.

[0078] During the review process, the system calls user-defined decision functions (UDFs) to extract the connection key (such as ID number, specific SKU code, etc.) from the candidate pairs in the first and second data sets. If the connection keys match, the connection is considered successful; otherwise, it is rejected.

[0079] To reduce the cost of review, the threshold decision and review module includes a caching unit. The caching unit uses "data set ID + data object ID" as the key to cache the extraction results of UDF, avoiding repeated calls to expensive external services.

[0080] In a preferred embodiment, the present invention also includes an offline training and parameter optimization process, implemented through a parameter optimization module.

[0081] The parameter optimization module is configured to determine the Top-k parameters during the offline phase. radius parameters and threshold parameters and ,like Figure 6 As shown, the parameter optimization module uses the validation dataset and labeled ground truth values ​​to perform parameter search. This process includes:

[0082] 1. Accuracy assessment: Perform retrieval and scoring on the validation set, and calculate the recall and precision under different parameters.

[0083] 2. Cost Measurement: Measure the average UDF call time, model inference time, and index retrieval time during runtime.

[0084] 3. Total Cost Modeling: Construct a total cost function that includes inference cost, verification cost, and misjudgment penalty.

[0085] 4. Parameter Search: Under the constraints of minimum recall and precision, find the combination of target parameters that minimizes the total cost through grid search or heuristic algorithms. , , , ).

[0086] The finalized target parameters will be fed back into the online system to achieve adaptive optimization of connection performance and cost.

[0087] In a preferred embodiment, the present invention also includes a connection execution module for organizing the output of connection results and statistical information.

[0088] The system output consists of two parts: the first part is the set of connection results, which includes the first and second data object pairs that are determined to be matched; the second part is statistical information, including but not limited to the number of generated candidates, the number after filtering, the proportion of the verification area, the actual number of UDF calls, and the time consumed at each stage. This statistical information can be fed back to the offline module for subsequent parameter calibration and model updates.

[0089] The connection execution module is configured to perform connection processing and output connection results during the online phase. The connection execution module sequentially calls the spatial calibration module to obtain calibration vector sets for the first and second data sets; it then calls the candidate generation module based on... and Generate a set of candidate connection pairs; call the probability scoring module to infer the probability of each candidate pair. ; Call the threshold decision and review module according to and Perform partitioning decisions and only call UDFs to verify join key consistency for candidate verification zones; finally output the set of join results and statistical information.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention. Those skilled in the art can make equivalent substitutions or modifications without departing from the spirit of the present invention, and all such substitutions or modifications should fall within the protection scope of the present invention.

Claims

1. An unstructured data link system based on spatial calibration, characterized in that, include: The spatial calibration module is used to perform L2 normalization on the embedding vectors of the first and second input data sets, and to perform spatial calibration through the residual adaptation network to obtain the calibration vector. The candidate generation module is configured to use each calibration vector in the first dataset as a query, perform a Top-k search for inner product similarity in the calibration vector set of the second dataset, and generate a similarity threshold by combining the similarity radius parameter to filter and generate candidate connection pairs; The probability scoring module is used to construct a feature vector for each candidate connection pair and input the feature vector into the fully connected network to obtain the connection matching probability. The threshold decision and review module is used to obtain the lower threshold and the upper threshold, and divide the candidate connection pairs into a rejection zone, an acceptance zone and a review zone according to the connection matching probability; wherein, the candidate connection pairs in the acceptance zone are output as connection results; the candidate connection pairs in the rejection zone are considered to not meet the connection conditions and the connection is determined to fail; the candidate connection pairs in the review zone are configured to call a user-defined decision function to obtain the connection key of the first data set and the second data set, and output the connection result when the connection key matches. The parameter optimization module is configured to search for the target parameters based on the recall constraint and cost model, including the Top-k parameter k, the similarity radius parameter, and the lower and upper thresholds. The connection execution module is configured to perform the connection in the order of "calibration-candidate generation-probability scoring-threshold decision and review" and output the connection results.

2. The system according to claim 1, characterized in that, The spatial calibration module includes a normalization unit and a residual adaptation network. The residual adaptation network includes a first fully connected layer, a batch normalization layer, a ReLU activation layer, and a second fully connected layer connected in sequence. The residual adaptation network is configured to be trained by minimizing the triplet interval loss function.

3. The system according to claim 1, characterized in that, The candidate generation module is configured to: set the similarity threshold to 1 minus the radius parameter, and retain only the Top-k results with a similarity not less than the similarity threshold during candidate generation.

4. The system according to claim 1, characterized in that, The fully connected network of the probability scoring module includes a first fully connected layer, a ReLU activation layer, a second fully connected layer, and a Sigmoid activation layer. The input dimension of the first fully connected layer is 4×d, where d is the dimension of the vector embedding. The feature vector is obtained by concatenating the first vector, the second vector, the dimension-wise absolute difference between the two vectors, and the dimension-wise product of the two vectors.

5. The system according to claim 1, characterized in that, The threshold decision and review module is configured to use a caching mechanism to avoid repeated calls to the same data object and the same row index when calling the user-defined decision function for the candidate connection pairs in the review area.

6. The system according to claim 1, characterized in that, The parameter optimization module aims to minimize the total cost when searching for the lower and upper thresholds. The total cost includes the cost of the probability scoring module inferring candidate connection pairs, the cost of calling the user-defined decision function in the verification region, and the penalty for false matches in the acceptance region. The parameter optimization module is configured to select the threshold pair that minimizes the total cost as the target parameter when the minimum recall and minimum precision constraints are met.

7. The system according to claim 1, characterized in that, The parameter optimization module is configured to estimate the Top-k parameter k and the similarity radius parameter on the verification data based on the recall target during the offline phase, and to perform a grid search on the lower threshold and the upper threshold.

8. A method for unstructured data linking based on spatial calibration, characterized in that, Includes the following steps: The vector embeddings of the first and second input data sets are L2 normalized and spatially calibrated using a residual adaptation network to obtain the calibration vector. Using each calibration vector in the first dataset as a query, perform a Top-k search for inner product similarity in the calibration vector set of the second dataset, and combine the radius parameter to generate a similarity threshold to generate candidate connection pairs; For each candidate connection pair, a feature vector is constructed, and the feature vector is input into a fully connected network to obtain the connection matching probability. The feature vector is obtained by concatenating a first vector, a second vector, the dimension-wise absolute difference between the two vectors, and the dimension-wise product of the two vectors. Obtain the lower threshold and the upper threshold, and divide the candidate connection pairs into a rejection region, an acceptance region and a verification region based on the connection matching probability; The candidate connection pairs in the acceptance region are directly output as connection results; for the candidate connection pairs in the verification region, a user-defined judgment function is called to obtain the connection key between the first data set and the second data set, and the connection result is output when the connection key is consistent.

9. The method according to claim 8, characterized in that, The method also includes a parameter optimization step: based on the recall constraint and cost model, the Top-k parameter k, the similarity radius parameter, and the lower and upper thresholds are searched to obtain the target parameters.

10. The method according to claim 9, characterized in that, The parameter optimization steps include: when searching for the lower threshold and the upper threshold, the total cost is used as the optimization objective. The total cost includes the cost of the probability scoring module inferring candidate connection pairs, the cost of calling the user-defined decision function, and the penalty for false matching in the acceptance region. The threshold pair that minimizes the total cost is selected when the minimum recall and minimum precision constraints are met.

Images