Data matching method and device, wearable equipment and storage medium

Abstract

The invention discloses a data matching method, apparatus and device, and a storage medium. The method comprises the steps of constructing a to-be-discriminated data set and a training data set according to a plurality of sample pairs corresponding to a plurality of to-be-matched data sets; determining a plurality of pseudo label sample pairs and a plurality of to-be-discriminated sample pairs according to the initial discriminator and the to-be-discriminated data set; performing real label labeling on the plurality of to-be-discriminated sample pairs to obtain a plurality of labeled sample pairs, and updating the training data set in combination with a plurality of pseudo label sample pairs; and performing iterative training on the initial discriminator through the updated training data set, obtaining a sample label of each label-free sample pair according to the initial discriminator after the iterative training, and further performing data matching and storage on data samples existing in the data set. By means of the mode, the labeling cost is remarkably reduced, semi-supervised cooperative training is combined, the problem of data sample imbalance is effectively solved, the value of labeling data on model training is improved, and precise matching of complex data attributes is achieved.

Description

Technical Field

[0001] This application relates to the field of data matching technology, and in particular to data matching methods, apparatus, wearable devices and storage media. Background Technology

[0002] Currently, mainstream entity matching methods include rule-based and learning-based methods. Rule-based methods rely on prior knowledge and are sensitive to attribute measurement and threshold selection. Learning-based methods, when faced with tasks involving a large number of entities and complex user attributes in real-world scenarios, suffer from problems such as difficulty in obtaining a large number of labeled samples, imbalanced data samples, and a large number of low-value labeled samples that waste manpower. Traditional methods cannot effectively solve these problems. Summary of the Invention

[0003] The main objective of this application is to provide a data matching method, apparatus, wearable device, and storage medium, aiming to solve the technical problems of low labeled sample value, imbalanced data samples, and low data matching efficiency in entity matching tasks.

[0004] To achieve the above objectives, this application proposes a data matching method, which includes:

[0005] The dataset to be judged and the training dataset are constructed based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true label sample pairs with real labels.

[0006] Based on the initial discriminator and the dataset to be discriminated, multiple pseudo-label sample pairs and multiple sample pairs to be discriminated are determined;

[0007] Multiple pairs of samples to be judged are labeled with real labels to obtain multiple labeled sample pairs. The training dataset is then updated using multiple labeled sample pairs and multiple pseudo-label sample pairs to obtain the updated training dataset.

[0008] The initial discriminator is iteratively trained using the updated training dataset. If the initial discriminator meets the iteration stopping condition during the iterative training process, the unlabeled sample pairs in the dataset to be discriminated are classified according to the iteratively trained initial discriminator to obtain the sample labels of each unlabeled sample pair.

[0009] Based on the sample labels of each unlabeled sample pair, the data samples in each dataset to be matched are matched, and the matched data samples are stored in the target database.

[0010] In one embodiment, the step of constructing a dataset to be judged and a training dataset based on multiple sample pairs corresponding to multiple datasets to be matched includes:

[0011] Based on the data attributes of data samples existing in multiple datasets to be matched, sample pairs are constructed to determine multiple sample pairs;

[0012] The similarity is calculated based on the data attributes of the data samples in each sample pair and the similarity evaluation method corresponding to each data attribute, and the sample pair similarity is determined.

[0013] Based on the sample pair similarity and similarity threshold, each sample pair is screened to determine multiple initial sample pairs and multiple unlabeled sample pairs;

[0014] Multiple initial sample pairs are labeled with real labels to obtain multiple real label sample pairs;

[0015] A training dataset is constructed based on multiple pairs of true labeled samples, and a dataset to be judged is constructed based on multiple pairs of unlabeled samples.

[0016] In one embodiment, the step of determining multiple pseudo-label sample pairs and multiple undiscriminated sample pairs based on the initial discriminator and the dataset to be discriminated includes:

[0017] The feature vectors of multiple unlabeled sample pairs in the dataset to be judged are input into the initial discriminator to obtain the information entropy data of each unlabeled sample pair.

[0018] The uncertainty operator for each unlabeled sample pair is determined based on the information entropy data of each unlabeled sample pair. The uncertainty operator includes at least one of the average entropy value and the entropy variance.

[0019] Multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs are determined based on the uncertainty operator of each unlabeled sample pair.

[0020] In one embodiment, the initial discriminator comprises multiple classifiers;

[0021] The step of inputting the feature vectors of multiple unlabeled sample pairs in the dataset to be judged into the initial discriminator to obtain the information entropy data of each unlabeled sample pair includes:

[0022] The feature vectors of multiple unlabeled sample pairs in the dataset to be judged are respectively input into multiple classifiers to obtain the discrimination probability of each unlabeled sample pair output by each classifier.

[0023] Information entropy is calculated based on multiple discrimination probabilities of each unlabeled sample pair to obtain multiple discrimination information entropies for each unlabeled sample pair;

[0024] The information entropy data of unlabeled sample pairs are determined based on the multiple discriminative information entropies of each labeled sample pair.

[0025] In one embodiment, the step of determining multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs based on the uncertainty operator of each unlabeled sample pair includes:

[0026] The average entropy value and the entropy variance of each unlabeled sample pair are determined based on the uncertainty operator of each unlabeled sample pair, and the probability variance of each unlabeled sample pair is obtained.

[0027] The uncertainty score of each unlabeled sample pair is determined by weighting the probability variance, the average entropy value, the entropy variance, the weighting coefficient corresponding to the probability variance, the weighting coefficient corresponding to the average entropy value, and the weighting coefficient corresponding to the entropy variance.

[0028] The uncertain scores of each unlabeled sample pair are sorted, and multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs are determined based on the sorting results.

[0029] In one embodiment, the iteration stopping condition includes at least one of the following:

[0030] The total number of iterations has reached the preset number of iterations;

[0031] During iterative training, the performance index of the initial discriminator in the current iteration round is less than or equal to the performance index in the previous iteration round, and the state duration reaches a preset round and the performance index in the current iteration round is greater than or equal to the index threshold.

[0032] In one embodiment, the initial discriminator after iterative training comprises multiple classifiers after iterative training;

[0033] The steps for classifying unlabeled sample pairs in the dataset to be classified based on the initial discriminator after iterative training, and obtaining the sample labels for each unlabeled sample pair, include:

[0034] The target classifier is determined based on the performance metrics of the classifiers trained in each iteration at the target iteration round.

[0035] Based on multiple labeled sample pairs, filter the unlabeled sample pairs in the dataset to be judged to obtain multiple filtered unlabeled sample pairs;

[0036] The filtered unlabeled sample pairs are classified by a target classifier to obtain the sample labels of the filtered unlabeled sample pairs.

[0037] In one embodiment, the step of matching data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and storing the matched data samples in the target database includes:

[0038] Based on the sample labels of each unlabeled sample pair, the real labels of each true labeled sample pair in the training dataset, and the real labels of each labeled sample pair, determine the source status of multiple sample pairs corresponding to each dataset to be matched.

[0039] When the data sources of each sample pair are of the same origin, the data samples in each sample pair are merged, and the merged data samples are stored in the target database.

[0040] When the data sources of each sample pair are different, the data samples in each sample pair are stored in the target database.

[0041] Furthermore, to achieve the above objectives, this application also proposes a data matching device, which includes:

[0042] The building module is used to construct the dataset to be judged and the training dataset based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true label sample pairs with real labels.

[0043] The processing module is used to determine multiple pseudo-label sample pairs and multiple sample pairs to be discriminated based on the initial discriminator and the dataset to be discriminated;

[0044] The annotation module is used to annotate multiple pairs of samples to be judged with real labels to obtain multiple labeled sample pairs. The training dataset is then updated using the multiple labeled sample pairs and multiple pseudo-label sample pairs to obtain the updated training dataset.

[0045] The prediction module is used to iteratively train the initial discriminator using the updated training dataset. If the initial discriminator meets the iteration stopping condition during the iterative training process, it classifies the unlabeled sample pairs in the dataset to be discriminated based on the iteratively trained initial discriminator to obtain the sample labels of each unlabeled sample pair.

[0046] The matching module is used to match the data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and store the matched data samples in the target database.

[0047] In addition, to achieve the above objectives, this application also proposes a wearable device, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the data matching method described above.

[0048] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data matching method described above.

[0049] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data matching method described above.

[0050] This application constructs a discriminant dataset and a training dataset based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true-label sample pairs with real labels. Multiple pseudo-label sample pairs and multiple undiscriminated sample pairs are determined based on an initial discriminator and the datasets to be matched. Real labels are applied to the multiple undiscriminated sample pairs to obtain multiple labeled sample pairs. The training dataset is then updated using these labeled and pseudo-label sample pairs to obtain an updated training dataset. The initial discriminator is iteratively trained using the updated training dataset. If the initial discriminator meets the iteration stopping condition during iterative training, the unlabeled sample pairs in the datasets to be matched are classified based on the iteratively trained initial discriminator to obtain sample labels for each unlabeled sample pair. Data matching is performed on the data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and the matched data samples are stored in the target database. By employing the above method, the initial discriminator is used to preliminarily screen pseudo-labeled sample pairs and unlabeled sample pairs, accurately locating key samples that require real labeling, significantly reducing labeling costs. The screened unlabeled sample pairs are labeled and the training dataset is updated. Combined with semi-supervised collaborative training, the imbalance of data samples is effectively solved, enhancing the value of labeled data for model training. After iterative training, the discriminator performs probability prediction on unlabeled sample pairs, achieving accurate matching of complex data attributes, improving the discriminator's ability to identify matching samples, and finally efficiently completing data matching and storage, greatly improving model accuracy and data matching efficiency. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating an embodiment of the data matching method of this application.

[0054] Figure 2A schematic diagram showing the uncertainty score ranking of the data matching method provided in Embodiment 1 of this application;

[0055] Figure 3 A schematic diagram of semi-supervised active learning training for the data matching method provided in Embodiment 1 of this application;

[0056] Figure 4 A schematic diagram of data merging for the data matching method provided in Embodiment 1 of this application;

[0057] Figure 5 This is a flowchart illustrating Embodiment 2 of the data matching method of this application;

[0058] Figure 6 A schematic diagram illustrating the construction of feature vectors for sample pairs in the data matching method provided in Embodiment 2 of this application;

[0059] Figure 7 A schematic diagram of dataset partitioning for the data matching method provided in Embodiment 1 of this application;

[0060] Figure 8 This is a flowchart illustrating Embodiment 3 of the data matching method of this application;

[0061] Figure 9 This is a schematic diagram of the module structure of the data matching device according to an embodiment of this application;

[0062] Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the data matching method in the embodiments of this application.

[0063] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0064] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0065] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The main solution of this application embodiment is as follows: A dataset to be discriminated and a training dataset are constructed based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true-label sample pairs with real labels. Multiple pseudo-label sample pairs and multiple sample pairs to be discriminated are determined based on the initial discriminator and the datasets to be discriminated. Real labels are applied to the multiple sample pairs to be discriminated to obtain multiple labeled sample pairs. The training dataset is then updated using the multiple labeled sample pairs and multiple pseudo-label sample pairs to obtain the updated training dataset. The initial discriminator is iteratively trained using the updated training dataset. If the initial discriminator meets the iteration stopping condition during iterative training, the unlabeled sample pairs in the datasets to be discriminated are classified according to the iteratively trained initial discriminator to obtain the sample labels for each unlabeled sample pair. Data matching is performed on the data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and the matched data samples are stored in the target database.

[0067] Currently, most mainstream and effective entity matching methods are rule-based or learning-based. Rule-based entity matching methods typically rely on prior knowledge of the dataset by users or domain experts to achieve good matching results. This method is highly sensitive to the choice of attribute measurement methods and thresholds, usually requiring specialized design by domain experts to obtain satisfactory results. Learning-based entity matching methods include machine learning-based entity matching and deep learning-based entity matching. Machine learning-based entity matching methods can automatically learn the features of matching entities using existing powerful machine learning models. With the continuous development of deep learning technology, deep learning-based entity matching methods, utilizing pre-trained language models and augmented training with relevant domain text, are gradually demonstrating their superior performance.

[0068] However, in many real-world tasks, with a large number of entities (each dataset containing a wealth of user information) and complex user attributes (including various complex time series data such as EMG (Electromyogram) signals and IMU (Inertial Measurement Unit) signals, in addition to basic information), it is difficult to obtain a large number of labeled samples to train machine learning or deep learning models. Furthermore, the number of mismatched samples in the two datasets is usually much greater than the number of matched samples. This imbalanced label distribution in binary classification can lead to insufficient training on matched samples.

[0069] In summary, current entity matching methods suffer from the following drawbacks, including but not limited to: 1. When performing entity matching on two datasets, it is difficult to obtain a large number of labeled data samples. Manual labeling requires significant manpower and time, and it is usually difficult to quickly obtain a large number of effective labels in a short period of time. 2. When both datasets are very large, there may be a severe imbalance in the data samples, meaning that the number of mismatched samples is usually much greater than the number of matched samples. 3. The labeled value of many entity pairs is low.

[0070] This application provides a solution that utilizes an initial discriminator to preliminarily screen pseudo-labeled sample pairs and unlabeled sample pairs, accurately locating key samples that require real labeling, significantly reducing labeling costs. The selected unlabeled sample pairs are then labeled, and the training dataset is updated. Combined with semi-supervised collaborative training, this effectively solves the problem of imbalanced data samples, enhancing the value of labeled data for model training. The iteratively trained discriminator performs probability prediction on unlabeled sample pairs, achieving accurate matching of complex data attributes, improving the discriminator's ability to identify matching samples, and ultimately efficiently completing data matching and storage, significantly improving model accuracy and data matching efficiency.

[0071] It should be noted that the executing entity in this embodiment can be a computing service device with data matching, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a wearable device capable of performing the above functions. The following description uses a wearable device as the executing entity to illustrate this embodiment and the subsequent embodiments.

[0072] Based on this, embodiments of this application provide a data matching method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data matching method of this application.

[0073] In this embodiment, the data matching method includes steps S10 to S50:

[0074] Step S10: Construct a dataset to be judged and a training dataset based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true label sample pairs with real labels.

[0075] It should be noted that the wearable devices in this embodiment include, but are not limited to, smartwatches, smart bracelets, smart wristbands, and VR (Virtual Reality) head-mounted display devices. Entity matching refers to the process of determining whether corresponding data items in two datasets represent the same real entity. Its goal is to determine whether a pair of input data represents the same entity (e.g., a user), where the two data items in the pair come from different data sources. For example, with two datasets of resident information, the goal is to compare the attributes of entities from the two different datasets, such as name, age, and home address, to determine whether the two records represent the same entity.

[0076] It is understandable that multiple independent datasets to be matched may have data from different sources. For example, dataset 1 may have data from wearable device A, while dataset 2 may have data from wearable device B. Furthermore, the storage format or structure of each dataset may differ when storing its data samples.

[0077] In practical implementation, when performing entity matching on multiple datasets to be matched, the goal is to determine whether the data samples belong to the same entity, such as the same user, the same product, or the same event. One data sample corresponds to one data record for one entity, and the datasets to be matched may contain multiple data records for multiple users. This embodiment uses the matching of two existing datasets as an example for illustration; however, the number of datasets can be adjusted according to requirements, and this embodiment does not impose any limitations on this.

[0078] It should be noted that before data matching, for the two datasets to be matched, all data samples in dataset 1 and dataset 2 need to be paired. During the pairing process, it is necessary to ensure that the paired data samples contain at least one data item with the same attribute, forming all sample pairs. For example, the vector of a sample pair is (a, b), where a is a data sample in dataset 1 and b is a data sample in dataset 2. The specific content of a is: username, address, height, weight; the specific content of b is: username, email, fitness frequency, mobile phone number, etc. Data samples a and b contain data items with the same data attribute.

[0079] It should be understood that within each dataset to be matched, same-origin matching must first be performed to ensure that there is only one corresponding data sample for the same entity in each dataset. A data sample can contain one or more data items with one or more data attributes, and each data attribute can also contain one or more data items. The specific process is as follows: within each dataset to be matched, each data item will be accompanied by a unique identifier. The unique identifier is used to reflect the entity source of the data item. By using the unique identifier of each data item, data items belonging to the same entity are merged to form the data sample of that entity.

[0080] It is understood that, in this embodiment, the data attribute classification may include, but is not limited to, the following four items: 1. Textual attribute information, such as username, address, and email address when the entity is a user. 2. Numerical information representing the entity's characteristics or physical traits, such as height, weight, BMI (Body Mass Index), and exercise frequency when the entity is a user. 3. Basic numerical information representing the entity's identity or identifier, such as year of birth, mobile phone number, and dominant hand when the entity is a user. 4. Time-series attribute information (e.g., physiological signals or sensor signals). For example, signals including, but not limited to, EMG, IMU, and PPG (Photoplethysmography) signals when the entity is a user.

[0081] In the specific implementation, all sample pairs s are calculated. i,j The similarity is denoted by , where i represents a data sample from dataset 1 to be matched, and j represents a data sample from dataset 2 to be matched. If sample pairs s i,j If the similarity is less than the threshold α, it indicates that the sample pairs s i,j If there is a mismatch, the two samples do not belong to the same entity. In this case, the sample pair should be directly assigned to the discrete dataset.

[0082] It should be noted that if the sample pair s i,j If the similarity between samples is greater than the threshold β, it indicates that the samples are paired. i,j If the two samples match and belong to the same entity, then the sample pair s will be matched. i,j The existing data samples are merged, and the merged data samples are stored in the data record corresponding to the entity. At the same time, the data samples existing in the sample pair are deleted from the dataset to be matched. In this embodiment, the threshold α is less than the threshold β, and the value of the threshold α is determined based on the computing resources and the sample pairs s corresponding to the multiple datasets to be matched. i,jGiven a fixed amount of data, if the data volume is small but resources are abundant, the threshold α is reduced to decrease the number of mismatched data samples that are directly deleted; conversely, the threshold α is increased to delete more high-probability mismatched data samples. The threshold β is set according to the scenario, or it can be set directly to 1 as needed. In this embodiment, the similarity value ranges from [0, 1]. The higher the similarity value, the higher the similarity between the two data samples in the sample pair. In this embodiment, the target database is a newly created database used to simultaneously store multiple data samples from the dataset to be matched.

[0083] It is understandable that for sample pairs s with similarity less than or equal to threshold β and greater than or equal to threshold α, i,j Then, a small subset of sample pairs is selected and labeled with real labels. During the real labeling process, experts label the sample pairs to determine if the two data samples in each pair match, thus obtaining multiple truly labeled sample pairs. The training dataset D is then constructed using these multiple truly labeled sample pairs. L and validation dataset D V When constructing the training and validation sets, multiple pairs of true labeled samples with real labels can be randomly allocated in a 7:3 ratio. Alternatively, other methods can be used to construct the training and validation sets. This embodiment does not impose any restrictions on these methods.

[0084] In the specific implementation, for the remaining sample pairs s that have not been labeled with real tags and whose similarity is less than or equal to the threshold β and greater than or equal to the threshold α, i,j All samples were used as unlabeled sample pairs to construct the dataset D to be judged. N .

[0085] Step S20: Determine multiple pseudo-label sample pairs and multiple sample pairs to be discriminated based on the initial discriminator and the dataset to be discriminated.

[0086] It should be noted that the initial discriminator P is constructed using a training dataset D consisting of multiple pairs of truly labeled samples with real labels. L This is obtained after training the discriminator P0. In this embodiment, the discriminator P0 integrates multiple classifiers C. i The number and specific type of classifiers are not specified. Classifier types include, but are not limited to, MLP (Multi-Layer Perceptron), Randomforest, and SVM (Support Vector Machine).

[0087] Understandably, each classifier's input is the feature vector of each sample pair, and its output is the information entropy of each sample pair or the probability of a match between sample pairs. Information entropy is a measure of the amount of information contained in a sample. When classifier C... i When the probability of a predicted sample pair matching or not matching is 0.5, H i (e) = 1 (i.e., the discriminator cannot identify it, and the uncertainty is high); when the probability of predicting a match or a mismatch is 1, H i (e) = 0 (the discriminator can clearly identify it, and the uncertainty is low). Therefore, it can be seen that the value range of information entropy H(e) is 0-1. The closer it is to 1, the more uncertain it is, and the closer it is to 0, the more certain it is.

[0088] In the specific implementation, all unlabeled sample pairs in the dataset to be judged are input into the initial discriminator P. Each classifier in the initial discriminator outputs the probability of a match and the probability of a mismatch for each unlabeled sample pair. Therefore, each unlabeled sample pair has multiple match probabilities and corresponding mismatch probabilities. For each unlabeled sample pair, multiple information entropies are calculated using these multiple match probabilities and their corresponding mismatch probabilities. The uncertainty score of the unlabeled sample pair can then be calculated using these multiple information entropies. A low uncertainty score indicates that the initial discriminator can accurately identify the sample pair; a high uncertainty score indicates that the initial discriminator cannot accurately identify the sample pair.

[0089] It should be noted that the unlabeled sample pairs are sorted according to their uncertainty scores. The sorting method can be either descending or ascending. This embodiment does not restrict this. The K most certain unlabeled sample pairs are selected as pseudo-labeled sample pairs based on the sorting results, and the K most uncertain unlabeled pairs are selected as sample pairs to be judged.

[0090] In this embodiment, for pseudo-label sample pairs, the uncertainty score is low, and multiple classifiers can accurately identify the pseudo-label sample pairs. At this time, the matching probability variance and non-matching probability variance of the pseudo-label sample pairs can be calculated based on the matching probability and non-matching probability output by multiple classifiers, and pseudo-labels can be added to the pseudo-label sample pairs according to the probability variance.

[0091] Step S30: Label multiple pairs of samples to be judged with real labels to obtain multiple labeled sample pairs. Update the training dataset with multiple labeled sample pairs and multiple pseudo-label sample pairs to obtain the updated training dataset.

[0092] It should be noted that, for the selected K pairs of samples to be judged, they are labeled with real labels. During the labeling process, experts are consulted to determine whether the two data samples in the pair match. In this embodiment, the resulting pairs of samples with real labels are the labeled sample pairs.

[0093] Understandably, K labeled sample pairs with real labels and K pseudo-labeled sample pairs with pseudo-labels are taken from the dataset D to be judged. N Move into training dataset D L In this process, the updated training dataset D' is obtained. L .like Figure 2 As shown, according to the dataset D to be discriminated N The uncertainty scores of each unlabeled sample pair are sorted from low to high. Through an active learning process, K of the most uncertain sample pairs are selected for true labeling, and through a self-learning process, K of the most certain sample pairs are selected for pseudo-labeling. This process transforms the 2K sample pairs from the dataset D to be judged. N Move into training dataset D L middle.

[0094] Step S40: The initial discriminator is iteratively trained using the updated training dataset. If the initial discriminator meets the iteration stopping condition during the iterative training process, the unlabeled sample pairs in the dataset to be discriminated are classified according to the iteratively trained initial discriminator to obtain the sample labels of each unlabeled sample pair.

[0095] It should be noted that the updated training dataset D' L The initial discriminator P is trained iteratively, and the validation dataset D is used. V The initial discriminator P after iterative training is validated to obtain its performance index F1 at each training epoch. If the performance index of the initial discriminator decreases instead of increasing and fails to improve for multiple consecutive epochs, the iteration stopping condition is met. If the total number of training iterations reaches the preset number of iterations, the iteration stopping condition is also met. In this embodiment, the total number of iterations refers to the total number of training iterations.

[0096] Understandably, the performance metric refers to the F1 score calculated by the initial discriminator on the validation dataset. The F1 score reflects the classification accuracy and completeness of the initial discriminator. The initial discriminator can calculate the corresponding precision (P) and recall (R) on the validation dataset. Precision (P) is the proportion of sample pairs predicted as matches by the initial discriminator that are actually matches; recall (R) is the proportion of true matches correctly predicted as matches by the initial discriminator. The F1 score is the harmonic mean of precision and recall, calculated using the formula: F1 = 2 × P × R / (P + R). When either precision or recall is too low, the F1 score will decrease significantly. A higher F1 score indicates better performance of the initial classifier.

[0097] In the specific implementation, since the initial discriminator is an ensemble of multiple classifiers, in each round of training iteration, each classifier has a performance metric (i.e., F1 score) calculated on the validation dataset. The highest performance metric is selected as the initial discriminator's performance metric F1(D) for that iteration round. V C i In iterative training, if the performance metric of the initial discriminator meets the iteration stopping condition, the round in which the performance metric of the initial discriminator is last greater than that of the previous round is identified as having the best performance metric so far. Among the rounds in which the performance metric of the initial discriminator is last greater than that of the previous round, the classifier C with the highest F1 score calculated on the validation dataset is identified. best As the final classification model in the initial discriminator obtained after iterative training stops, it is obtained through only one classifier C. best Classify the unlabeled sample pairs in the discrimination dataset. Besides the above method, the classifier C with the highest F1 score calculated on the validation dataset at the iteration stop can also be directly used. best This embodiment does not impose any limitations on the final classification model in the initial discriminator obtained after iterative training has stopped.

[0098] It should be noted that, if the total number of iterations meets the iteration stopping condition, then the classifier C with the highest F1 score calculated on the validation dataset at the iteration in which the iteration stops will be selected. best As the final classification model in the initial discriminator obtained after iterative training stops, it is obtained through only one classifier C. best Classify the unlabeled sample pairs in the dataset to be judged.

[0099] Understandably, the initial discriminator, obtained after iterative training and when training stops, is used to classify the unlabeled sample pairs in the dataset to be discriminated. At this point, the sample pair matching classification relies solely on the classifier C in the initial discriminator model after iterative training. best The initial discriminator, after iterative training, adds a match label to each unlabeled sample pair based on the matching and non-match probabilities of each output pair.

[0100] In the specific implementation, if the performance index F1 of the initial discriminator and the total number of iterations in multiple consecutive training rounds do not meet the iteration stopping condition, then steps S20 to S40 need to be repeated until any one or more of the performance index F1 of the initial discriminator and the total number of iterations in multiple consecutive training rounds meet the iteration stopping condition.

[0101] It should be noted that the semi-supervised active learning training process is as follows: Figure 3 As shown, data source 1 and data source 2 are two datasets to be matched. First, all data samples in the two data sources are matched pairwise to obtain multiple sample pairs. The unlabeled dataset contains all sample pairs. Then, the unlabeled dataset is pruned to construct the training dataset, validation dataset, and dataset to be discriminated. The discriminator is trained using the training dataset. The discriminator adopts the method of integrating multiple classification models (i.e., classifiers). The integration can adopt classification models such as MLP and Randomforest. The input of these classification models is the feature vector F, and the output is the matching probability or information entropy. Then, the iterative training loop (3) to (8) is entered: all classifiers in the initial discriminator P are used to calculate the record pairs (i.e. sample pairs) in the dataset to be discriminated, and the uncertainty score is calculated using the obtained information entropy H(e). All sample pairs are sorted according to the uncertainty score. The K most uncertain sample pairs are handed over to experts for labeling, and the K most certain sample pairs are automatically set with pseudo-labels (semi-supervised strategy) according to the classification results. These 2K samples (with labels or pseudo-labels) are moved from the dataset to be discriminated to the training dataset. Update the initial discriminator P with the new training dataset, and then validate the new initial discriminator P on the validation dataset. If the iteration stopping condition (i.e. the stopping criterion) is met, the optimal classification model C is output and used to construct a new database for subsequent judgment of all sample pairs. If the stopping criterion is not met, repeat the above steps (3) to (8) and continue to select samples from the dataset to be discriminated for expert and semi-supervised annotation process.

[0102] In one feasible implementation, the iteration stopping condition includes at least one of the following: the total number of iterations reaches a preset number of iterations; the performance index of the initial discriminator in the current iteration round is less than or equal to the performance index in the previous iteration round; the state duration rounds reach a preset number of rounds and the performance index in the current iteration round is greater than or equal to the index threshold.

[0103] It should be noted that during the training process, if any one or more of the performance metrics and the total number of iterations meet the conditions, the current iteration stopping condition is determined to be met, and the iterative update training of the initial classifier can be stopped at this time.

[0104] Understandably, the specific discrimination process for the initial discriminator in the iterative training process, where the performance metric of the initial discriminator in the current iteration is less than or equal to the performance metric of the previous iteration, and the state duration reaches a preset number of iterations while the performance metric in the current iteration is greater than or equal to the metric threshold, is as follows: The intermediate discriminator in the training process is used to predict the samples on the validation dataset. Since the initial discriminator consists of multiple classifiers C... i Ensemble training involves each classifier having a performance metric (F1 score) computed on the validation dataset during each iteration. The highest performance metric is selected as the initial discriminator's performance metric (F1) for that iteration after training. V C i ).

[0105] It is understandable that the state duration round refers to the number of times the performance metric of the initial discriminator after iterative training in the current iteration round is not greater than the performance metric in the previous iteration round. The starting point for counting state duration rounds is when the performance metric of the initial discriminator in the current iteration round is less than or equal to the performance metric in the previous iteration round. If the performance metric of the trained initial discriminator in the current iteration round is less than or equal to the performance metric in the previous iteration round, and the state duration round reaches a preset number of rounds, then the performance metric of the trained initial discriminator in the current iteration round is compared with a metric threshold. If the performance metric in the current iteration round is greater than or equal to the metric threshold, it indicates that the iteration stopping condition is met; if the performance metric in the current iteration round is less than the metric threshold, it indicates that the iteration stopping condition is not met, and iteration will continue in either case. In this embodiment, the metric threshold T... f It is a set critical value to prevent premature exit of the iteration.

[0106] For example, in N iterations, the classifier with the highest performance index is C1, and the performance index is F1(D). V C1); In iteration N+1, the classifier with the highest performance index is C2, and the performance index is F1(D). V C2), and F1(DV C1)≥F1(D V C2); In iteration N+1, the classifier with the highest performance index is C3, with a performance index of F1(D). V C3), and F1(D V C2)≥F1(D V If C3), then the state lasts for 3 rounds, F1(D V C1) > T f If the performance metrics of the initial discriminator during the iteration process satisfy the iteration stopping condition, then classifier C1 is taken as the final classification model in the initial discriminator obtained after iterative training has stopped. best Only through a single classifier C best Classify the unlabeled sample pairs in the dataset to be judged.

[0107] In the specific implementation, the iteration process terminates when the best-performing classifier in the initial discriminator P in the i-th iteration no longer increases its F1 score on the validation dataset, or even decreases. Since each intermediate discriminator has a certain bias, this embodiment employs a continued active learning process to obtain a more stable discriminator; that is, if the highest F1 score in each round does not improve after several consecutive iterations, the iteration stops. Furthermore, a set metric threshold T is used... f To avoid premature exit during the iteration process. If the F1 score of the classifier calculated on the validation dataset is lower than T... f This means that iteration will continue regardless of the stopping condition. This approach ensures that the results of active learning are not too poor and prevents the discriminator from stopping before learning enough features.

[0108] It should be noted that the specific process for determining when the total number of iterations reaches the preset number of iterations is as follows: as the labeled data continues to increase, the F1 score of the discriminator may continue to rise during the iterative training process. Therefore, a preset number of iterations T is set. n To avoid terminating the iteration too late, the iteration will terminate when the total number of iterations N reaches the preset number of iterations, regardless of the stopping condition.

[0109] In one feasible implementation, the initial discriminator after iterative training includes multiple classifiers after iterative training; in step S40: the step of classifying the unlabeled sample pairs in the dataset to be discriminated according to the initial discriminator after iterative training to obtain the sample labels of each unlabeled sample pair may include: steps A11 to A13:

[0110] Step A11: Determine the target classifier based on the performance metrics of the classifiers trained in each iteration at the target iteration round.

[0111] It should be noted that after stopping the iterative training of the initial discriminator, if the iteration stops because the performance index of the initial discriminator meets the iteration stopping condition during the iterative training process, then the last iteration in which the performance index of the initial discriminator is greater than the performance index of the previous iteration is taken as the target iteration iteration.

[0112] It is understandable that if the iteration stops because the total number of iterations meets the iteration stopping condition, then the last iteration training round will be taken as the target iteration round.

[0113] In the specific implementation, the performance metrics of the classifiers trained in each iteration are obtained on the validation dataset at the target iteration round. The performance metrics of the classifiers trained in each iteration are then ranked, and the classifier with the highest performance metric (i.e., F1 score) is selected as the target classifier C. best .

[0114] Step A12: Filter the unlabeled sample pairs in the dataset to be judged based on the multiple labeled sample pairs to obtain multiple filtered unlabeled sample pairs.

[0115] It should be noted that, due to the dataset D to be judged... N In the initial state, it consists of multiple unlabeled sample pairs. In the subsequent training process, multiple unlabeled sample pairs are selected to be labeled with real labels, thus obtaining the corresponding labeled sample pairs, which are then moved into the training dataset.

[0116] Understandably, to reduce subsequent workload and improve the model's classification matching efficiency, labeled sample pairs in the dataset to be judged are filtered out, leaving unlabeled sample pairs. During subsequent training, although pseudo-labels are added to some unlabeled sample pairs, this process is crucial in selecting the target classifier C. best Afterwards, the unlabeled sample pairs with added pseudo-labels still need to be reclassified to obtain the sample labels corresponding to the unlabeled sample pairs with added pseudo-labels. Therefore, the unlabeled sample pairs with added pseudo-labels still belong to the multiple unlabeled sample pairs after filtering.

[0117] Step A13: Classify the filtered unlabeled sample pairs using the target classifier to obtain the sample labels for the filtered unlabeled sample pairs.

[0118] It should be noted that the target classifier will add a matching or non-matching sample label to each unlabeled sample pair based on the matching probability and non-matching probability of each unlabeled sample pair in the output.

[0119] Step S50: Perform data matching on the data samples in each dataset to be matched according to the sample labels of each unlabeled sample pair, and store the matched data samples in the target database.

[0120] It should be noted that when constructing the dataset using multiple sample pairs corresponding to each dataset to be matched, several truly labeled sample pairs with real labels in the training and validation datasets, several unlabeled sample pairs in the dataset to be judged, sample pairs with similarity less than the threshold α, and sample pairs with similarity greater than the threshold β have been selected. Sample pairs with similarity less than the threshold α have been classified as discrete datasets; for sample pairs with similarity greater than the threshold β, during the dataset construction process, the data samples in the sample pair have been merged, and the merged data samples have been stored in the target database, with the data record corresponding to the entity recorded. At the same time, the data samples in the sample pair have been deleted from the dataset to be matched.

[0121] Understandably, after using the trained model to classify all unlabeled sample pairs that lack true labels and obtain their corresponding sample labels, the data samples in all unlabeled sample pairs with matching sample labels are first merged. The merged data samples are then stored in the target database, with the data record corresponding to the entity recorded, while the data samples in the sample pair are deleted from the dataset to be matched.

[0122] In the specific implementation, if the sample labels of an unlabeled sample pair do not match, the system checks whether the data sample in the sample pair has been deleted in the dataset to be matched. If it has, it means that the existing data sample has participated in the data merging of other sample pairs, and the data sample in the unlabeled sample pair is not processed. If it has not, it means that the existing data sample has not participated in the data merging of other sample pairs, and the data sample in the unlabeled sample pair is directly inserted into the target database.

[0123] It should be noted that multiple pairs of real-labeled samples and labeled samples that exist in the training and validation datasets are also processed in the same way to complete the matching of data samples in the two datasets to be matched.

[0124] In one feasible implementation, step S50 may include steps B11 to B13:

[0125] Step B11: Based on the sample labels of each unlabeled sample pair, the real labels of each true labeled sample pair in the training dataset, and the real labels of each labeled sample pair, determine the source status of multiple sample pairs corresponding to each dataset to be matched.

[0126] It should be noted that, based on the sample labels of each unlabeled sample pair, the ground truth labels of each properly labeled sample pair in the training dataset, the ground truth labels of each labeled sample pair, and the ground truth labels of each properly labeled sample pair in the validation dataset, the source state of each sample pair in the multiple sample pairs corresponding to each dataset to be matched is determined. If the labels match, the source state is "data from the same source," meaning that the two data samples in the sample pair come from the same entity; if the labels do not match, the source state is "data from different sources," meaning that the two data samples in the sample pair come from different entities.

[0127] Step B12: If the data sources of each sample pair are of the same origin, merge the data samples in each sample pair and store the merged data samples in the target database.

[0128] It should be noted that when the data sources of each sample pair are the same, the data samples in the data pair are merged and the merged data samples are stored in the target database. The data record corresponding to the entity is recorded, and the data samples in the sample pair are deleted from the dataset to be matched.

[0129] Step B13: If the source status of each sample pair is different from the data source status, store the data samples in each sample pair into the target database.

[0130] It should be noted that when the data sources of each sample pair are different, the system checks whether the data samples in the sample pair have been deleted from the dataset to be matched. If so, it means that the existing data samples have participated in the data merging of other sample pairs, and the data samples in that sample pair will not be processed. If not, it means that the existing data samples have not participated in the data merging of other sample pairs, and the data samples in that sample pair will be directly inserted into the target database. In this case, each sample pair will be directly inserted into the target database, and the data samples in that sample pair will not be assigned to any entity.

[0131] It should be understood that after matching all unlabeled sample pairs and all true labeled sample pairs, the system searches the discrete dataset for data samples in the dataset to be matched that have not been deleted from the dataset to be matched. If a data sample pair exists in the dataset to be matched that has not been deleted, then that data pair is directly inserted into the target database. After all data samples in the dataset to be matched have been matched, the amount of data in the target database is equal to the sum of the amounts of data in the two datasets to be matched.

[0132] Understandably, taking users as an example, the process of merging two datasets to be matched is as follows: Figure 4As shown, firstly, all data samples from the two data sources are paired, resulting in multiple sample pairs. The unlabeled dataset contains all sample pairs. Then, the unlabeled dataset is pruned using a linear rule-based approach. The linear similarity score of each similarity vector is calculated. If the score exceeds a threshold β, the samples represented by this vector from the two data sources are directly considered to come from the same user, and these two sample datasets are merged and inserted into a new database. If the score is below a threshold α, the samples are directly inserted into the new database. The remaining sample pairs are used to construct the training dataset, validation dataset, and the dataset to be judged. Next, a semi-supervised active learning process is initiated. After this process, a classifier C is obtained, which can label all sample pairs in the dataset to be judged that lack true labels. Finally, all sample pairs determined to be matched are merged into a single record and stored in the new database. The remaining unmatched samples from the two data sources are directly inserted into the new database. This completes the matching of all user information from the two data sources and merges them into a new database.

[0133] This embodiment constructs a discriminant dataset and a training dataset based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true-label sample pairs with real labels. Multiple pseudo-label sample pairs and multiple undiscriminated sample pairs are determined based on the initial discriminator and the datasets to be matched. The undiscriminated sample pairs are labeled with real labels to obtain multiple labeled sample pairs. The training dataset is then updated using these labeled and pseudo-label sample pairs to obtain an updated training dataset. The initial discriminator is iteratively trained using the updated training dataset. If the initial discriminator meets the iteration stopping condition during iterative training, the unlabeled sample pairs in the datasets to be matched are classified according to the iteratively trained initial discriminator to obtain sample labels for each unlabeled sample pair. Data matching is performed on the data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and the matched data samples are stored in the target database. By employing the above method, the initial discriminator is used to preliminarily screen pseudo-labeled sample pairs and unlabeled sample pairs, accurately locating key samples that require real labeling, significantly reducing labeling costs. The screened unlabeled sample pairs are labeled and the training dataset is updated. Combined with semi-supervised collaborative training, the imbalance of data samples is effectively solved, enhancing the value of labeled data for model training. After iterative training, the discriminator performs probability prediction on unlabeled sample pairs, achieving accurate matching of complex data attributes, improving the discriminator's ability to identify matching samples, and finally efficiently completing data matching and storage, greatly improving model accuracy and data matching efficiency.

[0134] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 The data matching method further includes steps S11 to S15 in step S10:

[0135] Step S11: Based on the data attributes of the data samples in the multiple datasets to be matched, construct sample pairs for the data samples in the multiple datasets to be matched, and determine multiple sample pairs.

[0136] It should be noted that, for two datasets to be matched, all data samples in dataset 1 and dataset 2 need to be paired up using the data attributes of all data samples. During the pairing process, it is necessary to ensure that there is at least one data item with the same attribute in the two paired data samples, so as to form all sample pairs. All sample pairs are used to construct an unlabeled dataset.

[0137] Step S12: Calculate the similarity of each sample pair based on the data attributes of the data samples in each sample pair and the similarity evaluation method corresponding to each data attribute.

[0138] It should be noted that different data attributes correspond to different similarity assessment methods. When the data attribute is text-based, similarity assessment methods can include Levenshtein Sim, Jaro-Winkler, Jaccard, and others. The specific calculation formula is as follows: Where x and y represent text sequences from two datasets to be matched in a sample pair, and d(x,y) represents the minimum number of operations (such as addition, deletion, and rewriting operations) required to transform string x into y.

[0139] Understandably, when the data attribute is numerical information representing an entity's model or physical characteristics, the similarity assessment method is to calculate the similarity using relative differences. The specific calculation formula can be as follows: Where x and y represent the values ​​in a sample pair that come from two datasets to be matched.

[0140] In the specific implementation, when the data attribute is basic numerical information representing the identity or identifier of an entity, the similarity evaluation method adopts the exact matching method to calculate the similarity. That is, if the values ​​are exactly the same, the similarity is 1; otherwise, the similarity is 0.

[0141] It should be noted that when the data attributes are time-series attributes (such as physiological signals or sensor signals), the similarity assessment method uses MMD (maximum mean discrepancy) to measure the distributional similarity of the sequences. The specific calculation formula is as follows:

[0142]

[0143] Where x and y represent time series from two datasets to be matched, respectively, in a sample pair, and n x n y y represents the time series lengths of x and y, respectively, and i and j represent the sampling point positions. The values ​​of time series x and y at sampling point i / j are represented by k, which represents the kernel function. In this embodiment, a Gaussian kernel function is used.

[0144] It is understandable that for each sample pair: since there are two data samples in the sample pair, and each sample has one or more data items with different data attributes, the similarity between two data items with the same data attribute is calculated according to the similarity evaluation method corresponding to each data attribute. The similarity values ​​are all scaled down to between 0 and 1 by the formula, with closer to 1 indicating similarity and closer to 0 indicating dissimilarity.

[0145] In the specific implementation, for each sample pair, the following formula applies: Calculate the sample pair similarity for this sample pair, where s i s represents each component in the similarity vector of sample pairs. i Specifically, it represents the similarity between two data items with the same data attributes; w i The weight coefficients are the weights corresponding to different components. The sum of the weight coefficients corresponding to all components is 1. The weight coefficients corresponding to different components can be set according to the requirements. This embodiment does not impose any restrictions on this.

[0146] It should be noted that during the dataset construction phase, the data samples from the two datasets to be matched are paired and similarity vectors are calculated to construct an unlabeled dataset containing all similarity vectors. Then, all samples are sorted using the linear similarity formula, removing obviously mismatched pairs. The obviously matched pairs are directly stored in the final new database. Finally, a portion of the unlabeled samples are selected and manually labeled by experts to form the training and validation datasets. Each data sample pair in the unlabeled dataset represents the feature similarity vector between two data samples in the two datasets to be matched. The process of constructing sample pairs is as follows: Figure 6 As shown, A comes from data source 1 and B comes from data source 2. Construct a feature similarity vector for A and B, which involves calculating the similarity of all attributes of A and B according to their attribute types and saving it to the vector.ab middle.

[0147] Step S13: Filter each sample pair according to the sample pair similarity and similarity threshold to determine multiple initial sample pairs and multiple unlabeled sample pairs.

[0148] It should be noted that in this embodiment, the similarity threshold includes threshold α and threshold β. All data samples in the two datasets to be matched are paired. If dataset 1 has M data samples and dataset 2 has N data samples, there may be M×N sample pairs. When M and N are very large, the number of sample pairs will be excessive. If there are no duplicate users in M ​​and N, the actual number of matched samples will not exceed max(M, N), which is much smaller than M×N, resulting in a large amount of mismatched redundant data in the sample pairs. Therefore, a linear similarity calculation method is used for further initial screening to delete obviously mismatched sample pairs.

[0149] Understandably, sample pairs with a similarity less than the threshold α are directly identified as mismatched redundant data and a discrete dataset is constructed. For sample pairs with a similarity greater than the threshold β, they are directly identified as matched similar data. The data samples in the sample pair are then merged and stored in the target database. The data record corresponding to the entity is recorded, and the data samples in the sample pair are deleted from the dataset to be matched.

[0150] It is understandable that for the remaining sample pairs in the unlabeled dataset: sample pairs s with similarity less than or equal to the threshold β and greater than or equal to the threshold α... i,j Then, all sample pairs that meet the above conditions are sorted according to their similarity. After sorting, a preset number of sample pairs are selected from both ends and the middle as initial sample pairs. For the remaining unselected sample pairs that meet the conditions of having a similarity less than or equal to threshold β and greater than or equal to threshold α, they are treated as unlabeled sample pairs.

[0151] Step S14: Label multiple initial sample pairs with real labels to obtain multiple real label sample pairs.

[0152] It should be noted that multiple initial sample pairs are labeled with real labels. During the real labeling process, experts label the sample pairs with real labels to determine whether the two data samples in the sample pair match, thus obtaining multiple real label sample pairs with real labels.

[0153] Step S15: Construct a training dataset based on multiple pairs of true labeled samples, and construct a dataset to be judged based on multiple pairs of unlabeled samples.

[0154] It should be noted that the training dataset D is constructed using multiple pairs of truly labeled samples with real labels and the feature vectors of these samples. L and validation dataset D V When constructing the training and validation sets, multiple pairs of truly labeled samples can be randomly allocated in a 7:3 ratio, or other methods can be used to construct the training and validation sets. This embodiment does not impose any restrictions on this. The dataset to be judged is then constructed using all unlabeled sample pairs and their feature vectors.

[0155] It is understandable that the specific process of partitioning the unlabeled dataset and constructing the training dataset, validation dataset, and discriminant dataset is as follows: Figure 7 As shown, if the feature similarity vector of the sample pair Feat ab Sample pairs with a similarity less than a threshold α are directly judged as mismatched and deleted from the unlabeled dataset (the data samples corresponding to these sample pairs clearly do not come from the same user in the two data sources), forming a discrete dataset for data merging and matching in the final matching stage; if the feature similarity vector of the sample pair Feat ab If the similarity between sample pairs is greater than the threshold β, they are directly considered a match. The two data samples a and b of the sample pair are merged and saved to a new database. At the same time, all samples associated with the two samples are deleted from the unlabeled dataset. For the remaining samples with similarity values ​​between α and β, some initial sample pairs need to be selected and given to experts for labeling to obtain sample pairs with real labels. This is used to construct the training dataset and the validation dataset. The remaining unlabeled sample pairs in the unlabeled dataset form the dataset to be judged.

[0156] This embodiment constructs sample pairs from data samples in multiple datasets to be matched based on their data attributes, thus determining multiple sample pairs. Similarity is calculated for each sample pair based on its data attributes and the corresponding similarity evaluation method. Each sample pair is then filtered based on its similarity and a similarity threshold to determine multiple initial sample pairs and multiple unlabeled sample pairs. The initial sample pairs are then labeled with real labels to obtain multiple truly labeled sample pairs. A training dataset is constructed based on the truly labeled sample pairs, and a dataset to be judged is constructed based on the multiple unlabeled sample pairs. This approach reduces labeling costs, increases labeling value, and improves matching efficiency.

[0157] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 8The data matching method, step S20, also includes steps S21 to S23:

[0158] Step S21: Input the feature vectors of multiple unlabeled sample pairs in the dataset to be judged into the initial discriminator to obtain the information entropy data of each unlabeled sample pair.

[0159] It should be noted that when the initial discriminator contains only one classifier, the feature vectors of multiple unlabeled sample pairs in the dataset to be discriminated against are input into the initial discriminator. Based on the probability distribution of whether each unlabeled sample pair matches, as output by the initial discriminator, the corresponding discriminative information entropy can be calculated. In this case, there is only one discriminative information entropy for each unlabeled sample pair. Discriminative information entropy is an uncertainty metric used to measure the model's confidence in classifying the sample pair.

[0160] It is understandable that when there are multiple classifiers in the initial discriminator, the feature vectors of multiple unlabeled sample pairs in the dataset to be discriminated are input into each classifier respectively. Based on the distribution probability of whether each classifier outputs whether each unlabeled sample pair matches, the corresponding discriminative information entropy can be calculated. At this time, the information entropy data of each unlabeled sample pair contains multiple discriminative information entropies.

[0161] Step S22: Determine the uncertainty operator for each unlabeled sample pair based on the information entropy data of each unlabeled sample pair. The uncertainty operator includes at least one of the average entropy value and the entropy variance.

[0162] It should be noted that the uncertainty operator is a statistical indicator used to quantify the uncertainty of unlabeled sample pairs. In this embodiment, it includes at least one of the average entropy value and the entropy variance. When there is only one discriminant entropy in the information entropy data of each unlabeled sample pair, the average entropy value is directly equivalent to the discriminant entropy, and the entropy variance is 0.

[0163] It is understandable that when there are multiple discriminative entropies in the information entropy data of each unlabeled sample pair, for each unlabeled sample pair: the average entropy value is the arithmetic mean of the discriminative entropies of the unlabeled sample pair output by all classifiers; the entropy variance refers to the variance of the discriminative entropies of the unlabeled sample pair output by all classifiers, which measures the degree of deviation between the discriminative entropies of the unlabeled sample pair output by each classifier and the average entropy value.

[0164] In practical implementation, when the initial discriminator is an ensemble of multiple classifiers, for each unlabeled sample pair: to comprehensively consider all discriminative information entropies, the mean of the discriminative information entropies corresponding to all classifiers needs to be calculated. The average of the discriminative information entropies obtained is the average entropy value of the unlabeled sample pair. The formula for calculating the average entropy value Ave_Entropy(e) is as follows: Where H i (e) represents the classifier C. i The corresponding discriminant information entropy, and the number of classifiers n. The entropy variance var_Entropy(e) is further calculated using the average entropy value and multiple discriminant information entropies, with the specific calculation formula as follows:

[0165]

[0166] Step S23: Determine multiple pseudo-labeled sample pairs and multiple unlabeled sample pairs based on the uncertainty operator of each unlabeled sample pair.

[0167] It should be noted that pseudo-labeled sample pairs are the most certain sample pairs selected from unlabeled sample pairs; and undecided sample pairs are the least certain sample pairs selected from unlabeled sample pairs. In this embodiment, the number of pseudo-labeled sample pairs and the number of undecided sample pairs are kept consistent, with K pseudo-labeled sample pairs and K undecided sample pairs selected from the unlabeled sample pairs, respectively.

[0168] It is understandable that by using the uncertainty operator of each unlabeled sample pair, the most certain unlabeled sample pairs and the most uncertain unlabeled sample pairs can be selected. During the selection, only the average entropy value of each unlabeled sample pair can be used. The higher the average entropy value of the unlabeled sample pair, the more uncertain it is. Alternatively, only the entropy variance of each unlabeled sample pair can be used. The larger the entropy variance of the unlabeled sample pair, the more uncertain it is. Alternatively, both the average entropy value and the entropy variance of each unlabeled sample pair can be used. The higher the average entropy value and the larger the entropy variance of the unlabeled sample pair, the more uncertain it is. In addition to the above methods, other methods can also be used to select sample pairs. This embodiment does not limit this.

[0169] In one feasible implementation, the initial discriminator comprises multiple classifiers, and step S21 may further include steps C11 to C13:

[0170] Step C11: Input the feature vectors of multiple unlabeled sample pairs in the dataset to be judged into multiple classifiers respectively, and obtain the discrimination probability of each unlabeled sample pair output by each classifier.

[0171] It should be noted that since the discriminator is an ensemble of multiple classifiers, after training the discriminator on the training dataset, the multiple classifiers in the resulting initial discriminator are also trained on the training dataset. The feature vectors of multiple unlabeled sample pairs in the dataset to be discriminated against are input into the initial discriminator. Each classifier in the initial discriminator outputs the probability of a match and the probability of a mismatch for each unlabeled sample pair. Therefore, each unlabeled sample pair will have multiple matching probabilities and corresponding mismatch probabilities. In this embodiment, for each unlabeled sample pair: each discrimination probability includes the P output by the same classifier. match Match probability and non-match probability P dismatch .

[0172] Step C12: Calculate the information entropy based on the multiple discrimination probabilities of each unlabeled sample pair to obtain the multiple discrimination information entropies of each unlabeled sample pair.

[0173] It should be noted that for each unlabeled sample pair: based on the matching probability and non-match probability output by the same classifier, the discriminative information entropy corresponding to that classifier is calculated. The specific formula for calculating the discriminative information entropy is as follows: H(e) = -P match logP match -P dismatch logP dismatch .

[0174] Step C13: Determine the information entropy data of unlabeled sample pairs based on the multiple discriminative information entropies of each labeled sample pair.

[0175] It should be noted that for each unlabeled sample pair: the discriminative information entropy of each classifier is calculated based on the probability distribution of whether the output of each classifier is a match for the unlabeled sample pair, and these are summarized to obtain the information entropy data of the unlabeled sample pair.

[0176] In one feasible implementation, step S23 may further include steps D11 to D13:

[0177] Step D11: Determine the average entropy value and the entropy variance of each unlabeled sample pair based on the uncertainty operator of each unlabeled sample pair, and obtain the probability variance of each unlabeled sample pair.

[0178] It should be noted that in steps D11 to D13, the example is taken where multiple classifiers are integrated in the initial discriminator. At this time, for each unlabeled sample pair: the mean of the matching probabilities is calculated based on the matching probabilities existing among the multiple discrimination probabilities. By using the matching probabilities among multiple discrimination probabilities and the calculated mean probability, the probability variance var_Prob(e) can be further calculated. The specific calculation formula is as follows: Where P i ,match represents the classifier C i Output the matching probability.

[0179] Step D12: Perform a weighted calculation based on the probability variance of each unlabeled sample pair, the average entropy value of each unlabeled sample pair, the entropy variance of each unlabeled sample pair, the weighting coefficient corresponding to the probability variance, the weighting coefficient corresponding to the average entropy value, and the weighting coefficient corresponding to the entropy variance to determine the uncertainty score of each unlabeled sample pair.

[0180] It should be noted that uncertainty is used to describe the degree to which the initial discriminator can clearly identify sample pairs. A low uncertainty score indicates that the initial discriminator can accurately identify the samples; a high uncertainty score indicates that the initial discriminator cannot accurately identify the samples.

[0181] It should be understood that the average entropy, entropy variance, and probability variance each correspond to a weighting coefficient α, β, and γ, respectively, and the sum of the three is 1. The specific value of each weighting coefficient can be adjusted according to the actual situation. For each unlabeled sample pair: the uncertainty score Uncertainty_score(e) is obtained by weighting the probability variance var_Prob(e), average entropy Ave_Entropy(e), entropy variance var_Entropy(e), and their corresponding weighting coefficients. The specific calculation formula is as follows: Uncertainty_score(e) = αAve_Entropy(e) + βvar_Entropy(e) + γvar_Prob(e).

[0182] Understandably, the above method of calculating uncertainty scores avoids several problems: 1. Considering only the average entropy: It's possible that the discriminant information entropy calculated in one classifier is very high, while the discriminant information entropy calculated in another classifier is very low; another sample pair may have moderate discriminant information entropy in both classifiers. The average entropy of the two unlabeled sample pairs is considered, but theoretically, the former should have higher uncertainty due to the significant differences in results obtained from different classifiers. 2. Considering only the entropy variance: Entropy can describe the certainty of the model's prediction, but there may be two classifiers predicting the same entropy for a sample pair, but one classifier predicts a match while the other predicts a mismatch. Such a sample pair may have low calculated results in both average entropy uncertainty and entropy variance uncertainty, but due to the different prediction results in different classifiers, the actual uncertainty of this sample pair should be higher.

[0183] Step D13: Sort the uncertain scores of each unlabeled sample pair, and determine multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs based on the sorting results.

[0184] It should be noted that the unlabeled sample pairs are sorted according to their uncertainty scores. The sorting method can be either descending or ascending. This embodiment does not restrict this. The K most certain unlabeled sample pairs are selected as pseudo-labeled sample pairs based on the sorting results, and the K most uncertain unlabeled pairs are selected as sample pairs to be judged.

[0185] In this implementation, a semi-supervised active learning strategy is adopted. Sample pairs with annotation value are selected and handed over to humans for judgment, maximizing the value of manual labor in improving the performance of the discrimination model. Simultaneously, based on the iteration stopping condition, during the iterative training of the discrimination model, the optimal discrimination model is output when the obtained model meets the iteration stopping condition. Using the trained discrimination model to judge the remaining sample pairs can significantly reduce the cost of manual annotation.

[0186] This embodiment inputs the feature vectors of multiple unlabeled sample pairs in the dataset to be judged into an initial discriminator to obtain the information entropy data of each unlabeled sample pair. Based on the information entropy data of each unlabeled sample pair, an uncertainty operator is determined for each unlabeled sample pair. The uncertainty operator includes at least one of the average entropy value and the entropy variance. Based on the uncertainty operator of each unlabeled sample pair, multiple pseudo-labeled sample pairs and multiple unlabeled sample pairs are determined. Through this method, the information entropy data of each unlabeled sample pair can be accurately utilized to select the most valuable unlabeled sample pair for true labeling.

[0187] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data matching method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0188] This application also provides a data matching device, please refer to... Figure 9 The data matching device includes:

[0189] Module 10 is used to construct a dataset to be judged and a training dataset based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true label sample pairs with real labels.

[0190] The processing module 20 is used to determine multiple pseudo-label sample pairs and multiple sample pairs to be discriminated based on the initial discriminator and the dataset to be discriminated.

[0191] The annotation module 30 is used to annotate multiple pairs of samples to be judged with real labels to obtain multiple labeled sample pairs, and to update the training dataset with multiple labeled sample pairs and multiple pseudo-label sample pairs to obtain the updated training dataset.

[0192] The prediction module 40 is used to iteratively train the initial discriminator using the updated training dataset. During the iterative training process, if the initial discriminator meets the iteration stopping condition, it classifies the unlabeled sample pairs in the dataset to be discriminated based on the iteratively trained initial discriminator to obtain the sample labels of each unlabeled sample pair.

[0193] The matching module 50 is used to perform data matching on the data samples existing in each dataset to be matched based on the sample labels of each unlabeled sample pair, and to store the matched data samples in the target database.

[0194] Optionally, building module 10 is also used for:

[0195] Based on the data attributes of data samples in multiple datasets to be matched, sample pairs are constructed to determine multiple sample pairs. Similarity is calculated for each sample pair based on the data attributes of the data samples in each pair and the corresponding similarity evaluation method. The sample pairs are then filtered based on their similarity and a similarity threshold to determine multiple initial sample pairs and multiple unlabeled sample pairs. Real labels are applied to the initial sample pairs to obtain multiple true-label sample pairs. A training dataset is constructed based on the multiple true-label sample pairs, and a dataset to be judged is constructed based on the multiple unlabeled sample pairs.

[0196] Optionally, the processing module 20 is also used for:

[0197] The feature vectors of multiple unlabeled sample pairs in the dataset to be judged are input into the initial discriminator to obtain the information entropy data of each unlabeled sample pair; the uncertainty operator of each unlabeled sample pair is determined based on the information entropy data of each unlabeled sample pair, and the uncertainty operator includes at least one of the average entropy value and the entropy variance; multiple pseudo-labeled sample pairs and multiple sample pairs to be judged are determined based on the uncertainty operator of each unlabeled sample pair.

[0198] Optionally, the processing module 20 is also used for:

[0199] The feature vectors of multiple unlabeled sample pairs in the dataset to be discriminated are input into multiple classifiers to obtain the discrimination probability of each unlabeled sample pair output by each classifier; the information entropy is calculated based on the multiple discrimination probabilities of each unlabeled sample pair to obtain the multiple discrimination information entropy of each unlabeled sample pair; the information entropy data of the unlabeled sample pair is determined based on the multiple discrimination information entropy of each labeled sample pair.

[0200] Optionally, the processing module 20 is also used for:

[0201] The average entropy and entropy variance of each unlabeled sample pair are determined based on the uncertainty operator of each pair, and the probability variance of each pair is obtained. The uncertainty score of each unlabeled sample pair is determined by weighting the probability variance, average entropy, entropy variance, weighting coefficients corresponding to the probability variance, average entropy, and entropy variance. The uncertainty scores of each pair are then sorted, and multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs are determined based on the sorting results.

[0202] Optionally, the prediction module 40 is also used for:

[0203] The target classifier is determined based on the performance metrics of the classifiers trained in each iteration at the target iteration round; multiple labeled sample pairs are used to filter the unlabeled sample pairs in the dataset to be judged, resulting in multiple filtered unlabeled sample pairs; the target classifier is used to classify the multiple filtered unlabeled sample pairs to obtain the sample labels of the multiple filtered unlabeled sample pairs.

[0204] Optionally, the matching module 50 is also used for:

[0205] Based on the sample labels of each unlabeled sample pair, the real labels of each true labeled sample pair in the training dataset, and the real labels of each labeled sample pair, the source status of multiple sample pairs corresponding to each dataset to be matched is determined. If the source status of each sample pair is the same, the data samples in each sample pair are merged, and the merged data samples are stored in the target database. If the source status of each sample pair is different, the data samples in each sample pair are stored in the target database.

[0206] The data matching apparatus provided in this application, employing the data matching method described in the above embodiments, can solve the technical problems of low labeled sample value, imbalanced data samples, and low data matching efficiency in entity matching tasks. Compared with the prior art, the beneficial effects of the data matching apparatus provided in this application are the same as those of the data matching method provided in the above embodiments, and other technical features in the data matching apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0207] This application provides a wearable device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the data matching method in Embodiment 1 above.

[0208] The following is for reference. Figure 10 The diagram illustrates a structural schematic suitable for implementing the embodiments of this application. The wearable devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The wearable device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0209] like Figure 10As shown, the wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the wearable device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the wearable device to communicate wirelessly or wiredly with other devices to exchange data. While wearable devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0210] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0211] The wearable device provided in this application, employing the data matching method described in the above embodiments, can solve the technical problems of low labeled sample value, imbalanced data samples, and low data matching efficiency in entity matching tasks. Compared with the prior art, the beneficial effects of the wearable device provided in this application are the same as those of the data matching method provided in the above embodiments, and other technical features of the wearable device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0212] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0213] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0214] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data matching method in the above embodiments.

[0215] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0216] The aforementioned computer-readable storage medium may be included in the wearable device; or it may exist independently and not assembled into the wearable device.

[0217] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a wearable device, the wearable device: constructs a dataset to be judged and a training dataset based on multiple sample pairs corresponding to multiple datasets to be matched, wherein the training dataset includes multiple pairs of sample pairs with real labels; determines multiple pairs of pseudo-labeled sample pairs and multiple pairs of sample pairs to be judged based on an initial discriminator and the datasets to be judged; labels the multiple pairs of sample pairs to be judged with real labels to obtain multiple labeled sample pairs, and updates the training dataset using the multiple labeled sample pairs and multiple pseudo-labeled sample pairs to obtain an updated training dataset; iteratively trains the initial discriminator using the updated training dataset, and if the initial discriminator meets the iteration stopping condition during the iterative training process, classifies the unlabeled sample pairs in the datasets to be judged based on the iteratively trained initial discriminator to obtain sample labels for each unlabeled sample pair; performs data matching on the data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and stores the matched data samples in the target database.

[0218] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0219] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0220] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0221] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data matching method. This solves the technical problems of low labeled sample value, imbalanced data samples, and low data matching efficiency in entity matching tasks. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data matching method provided in the above embodiments, and will not be repeated here.

[0222] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data matching method described above.

[0223] The computer program product provided in this application can solve the technical problems of low labeled sample value, imbalanced data samples, and low data matching efficiency in entity matching tasks. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data matching method provided in the above embodiments, and will not be repeated here.

[0224] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A data matching method, characterized in that, The method includes: A dataset to be judged and a training dataset are constructed based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true label sample pairs with real labels. Based on the initial discriminator and the dataset to be discriminated, multiple pseudo-label sample pairs and multiple sample pairs to be discriminated are determined; Multiple pairs of samples to be judged are labeled with real labels to obtain multiple labeled sample pairs. The training dataset is then updated using the multiple labeled sample pairs and multiple pseudo-label sample pairs to obtain the updated training dataset. The initial discriminator is iteratively trained using the updated training dataset. If the initial discriminator meets the iteration stopping condition during the iterative training process, the unlabeled sample pairs in the dataset to be discriminated are classified according to the iteratively trained initial discriminator to obtain the sample labels of each unlabeled sample pair. Based on the sample labels of each unlabeled sample pair, the data samples in each dataset to be matched are matched, and the matched data samples are stored in the target database.

2. The method as described in claim 1, characterized in that, The steps of constructing the discriminant dataset and the training dataset based on multiple sample pairs corresponding to multiple datasets to be matched include: Based on the data attributes of data samples existing in multiple datasets to be matched, sample pairs are constructed to determine multiple sample pairs; The similarity is calculated based on the data attributes of the target database data samples in each sample pair and the similarity evaluation method corresponding to each data attribute, and the sample pair similarity is determined. Based on the sample pair similarity and similarity threshold, each sample pair is screened to determine multiple initial sample pairs and multiple unlabeled sample pairs; Multiple initial sample pairs are labeled with real labels to obtain multiple real label sample pairs; A training dataset is constructed based on multiple pairs of true labeled samples, and a dataset to be judged is constructed based on multiple pairs of unlabeled samples.

3. The method as described in claim 1, characterized in that, The step of determining multiple pseudo-label sample pairs and multiple undetermined sample pairs based on the initial discriminator and the dataset to be discriminated includes: The feature vectors of multiple unlabeled sample pairs in the dataset to be judged are input into the initial discriminator to obtain the information entropy data of each unlabeled sample pair. The uncertainty operator for each unlabeled sample pair is determined based on the information entropy data of each unlabeled sample pair. The uncertainty operator includes at least one of the average entropy value and the entropy variance. Multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs are determined based on the uncertainty operator of each unlabeled sample pair.

4. The method as described in claim 3, characterized in that, The initial discriminator contains multiple classifiers; The step of inputting the feature vectors of multiple unlabeled sample pairs in the dataset to be judged into the initial discriminator to obtain the information entropy data of each unlabeled sample pair includes: The feature vectors of multiple unlabeled sample pairs in the dataset to be judged are respectively input into multiple classifiers to obtain the discrimination probability of each unlabeled sample pair output by each classifier. Information entropy is calculated based on multiple discrimination probabilities of each unlabeled sample pair to obtain multiple discrimination information entropies for each unlabeled sample pair; The information entropy data of unlabeled sample pairs are determined based on the multiple discriminative information entropies of each labeled sample pair.

5. The method as described in claim 3, characterized in that, The step of determining multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs based on the uncertainty operator of each unlabeled sample pair includes: The average entropy value and the entropy variance of each unlabeled sample pair are determined based on the uncertainty operator of each unlabeled sample pair, and the probability variance of each unlabeled sample pair is obtained. The uncertainty score of each unlabeled sample pair is determined by weighting the probability variance, the average entropy value, the entropy variance, the weighting coefficient corresponding to the probability variance, the weighting coefficient corresponding to the average entropy value, and the weighting coefficient corresponding to the entropy variance. The uncertain scores of each unlabeled sample pair are sorted, and multiple pseudo-labeled sample pairs and multiple undiscriminated sample pairs are determined based on the sorting results.

6. The method as described in claim 1, characterized in that, The iteration stopping condition includes at least one of the following: The total number of iterations has reached the preset number of iterations; During iterative training, the performance index of the initial discriminator in the current iteration round is less than or equal to the performance index in the previous iteration round, and the state duration reaches a preset round and the performance index in the current iteration round is greater than or equal to the index threshold.

7. The method as described in claim 1, characterized in that, The initial discriminator after iterative training comprises multiple classifiers after iterative training. The step of classifying the unlabeled sample pairs in the dataset to be discriminated based on the initial discriminator after iterative training to obtain the sample labels for each unlabeled sample pair includes: The target classifier is determined based on the performance metrics of the classifiers trained in each iteration at the target iteration round. The unlabeled sample pairs in the dataset to be judged are filtered based on multiple labeled sample pairs to obtain multiple filtered unlabeled sample pairs. The target classifier is used to classify the filtered unlabeled sample pairs to obtain the sample labels of the filtered unlabeled sample pairs.

8. The method according to any one of claims 1 to 7, characterized in that, The step of matching data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and storing the matched data samples in the target database includes: Based on the sample labels of each unlabeled sample pair, the real labels of each true labeled sample pair in the training dataset, and the real labels of each labeled sample pair, determine the source status of multiple sample pairs corresponding to each dataset to be matched. When the data sources of each sample pair are of the same origin, the data samples in each sample pair are merged, and the merged data samples are stored in the target database. When the data sources of each sample pair are different, the data samples in each sample pair are stored in the target database.

9. A data matching device, characterized in that, The data matching device includes: The construction module is used to construct a dataset to be judged and a training dataset based on multiple sample pairs corresponding to multiple datasets to be matched. The training dataset includes multiple true label sample pairs with real labels. The processing module is used to determine multiple pseudo-label sample pairs and multiple sample pairs to be discriminated based on the initial discriminator and the dataset to be discriminated; The annotation module is used to annotate multiple pairs of samples to be judged with real labels to obtain multiple labeled sample pairs, and to update the training dataset with multiple labeled sample pairs and multiple pseudo-label sample pairs to obtain the updated training dataset. The prediction module is used to iteratively train the initial discriminator using the updated training dataset. If the initial discriminator meets the iteration stopping condition during the iterative training process, it classifies the unlabeled sample pairs in the dataset to be discriminated based on the iteratively trained initial discriminator to obtain the sample labels of each unlabeled sample pair. The matching module is used to match the data samples in each dataset to be matched based on the sample labels of each unlabeled sample pair, and store the matched data samples in the target database.

10. A wearable device, characterized in that, The device includes: a memory, a processor, and a data matching program stored in the memory and executable on the processor, the data matching program being configured to implement the steps of the data matching method as described in any one of claims 1 to 8.

11. A storage medium, characterized in that, The storage medium stores a data matching program, which, when executed by a processor, implements the steps of the data matching method as described in any one of claims 1 to 8.

Images