Determination device, determination method, and determination program
The determination device addresses the challenge of matching records across databases by calculating feature importance and setting parameters without data aggregation, ensuring accurate matching and reducing resource and privacy concerns.
Patent Information
- Application Number
- JP2021123734
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-28
- Publication Date
- 2024-11-11
- Estimated Expiration
- 2041-07-28
AI Technical Summary
Existing technologies face challenges in accurately matching records across multiple databases while addressing privacy concerns and resource requirements, particularly when databases are managed by different organizations.
A determination device and method that calculates the importance of features in each database and sets matching parameters without aggregating records, using the degree of importance in each database to determine the relationship between records based on weighted feature values.
Accurately matches records across databases without acquiring or aggregating data from multiple organizations, reducing processing load and privacy issues, while maintaining high matching accuracy.
Smart Images

Figure 0007580080000004 
Figure 0007580080000005 
Figure 0007580080000006
Abstract
Description
[Technical field]
[0001] The present invention relates to a determination device, a determination method, and a determination program for determining whether records in a plurality of databases (DBs) relate to the same item. [Background technology]
[0002] Conventionally, there are technologies for matching records from multiple databases. When there are hundreds of features in each database record, it is not clear how much weighting should be given to each feature to increase the accuracy of matching, so there are technologies that use machine learning to match records. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] V.Meduri, L.Popa, P.Sen, and M.Sarwat, “A Comprehensive Benchmark Framework for Active Learning Methods in Entity Matching,” Mar. 2020. Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the above technology, in order to perform machine learning, it is necessary to acquire and aggregate records from each database to be matched. Therefore, for example, if the databases to be matched are managed by different organizations, privacy issues may arise. In addition, machine learning takes time and requires a lot of hardware resources.
[0005] Therefore, an object of the present invention is to solve the above-mentioned problems, to set up matching parameters without acquiring records from databases of multiple organizations as learning data, and to match records from each database with high accuracy, without aggregating the records from each database in one place. [Means for solving the problem]
[0006] In order to solve the above-mentioned problems, the present invention provides a determination device that determines whether a first record stored in a first DB held by a first organization and a second record stored in a second DB held by a second organization are records relating to the same matter, and is characterized by comprising: an acquisition unit that acquires the first record and the second record; a score calculation unit that calculates a score indicating the degree to which the first record and the second record are records relating to the same matter, for each pair of a feature that constitutes the first record and a feature that constitutes the second record, by using the importance in the first DB of a feature that constitutes the first record and the importance in the second DB of a feature that constitutes the second record to weight a value to be added when the values of the feature quantities in the pair are the same; a determination unit that determines whether the first record and the second record are records relating to the same matter based on the magnitude of the score; and a transmission unit that transmits a result of the determination to the first organization or the second organization. Effect of the Invention
[0007] According to the present invention, matching parameters can be set up without acquiring database records of multiple organizations as learning data and without aggregating the records of each database in one place, making it possible to match the records of each database with high accuracy. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram for explaining an outline of the operation of the matching device. [Diagram 2] FIG. 2 is a diagram showing an outline of a processing procedure of the matching system. [Diagram 3] FIG. 3 is a diagram showing an outline of a processing procedure of the matching system. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a matching system. [Diagram 5] FIG. 5 is a sequence diagram illustrating an example of a processing procedure of the advance preparation phase of the matching system. [Figure 6] FIG. 6 is a sequence diagram illustrating an example of a processing procedure of the advance preparation phase of the matching system. [Figure 7] FIG. 7 is a diagram illustrating an example of the configuration of a computer that executes a matching program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, a mode (embodiment) for carrying out the present invention will be described with reference to the drawings. The present invention is not limited to the embodiment described below.
[0010] [overview] First, an outline of the operation of the matching device (determination device) 10 of this embodiment will be described with reference to Fig. 1. Here, an example will be described in which the matching device 10 performs data matching between an in-house sales DB (database) and an external company's DB. In the following description, the in-house sales DB will be referred to as DB_A, and the external company's DB will be referred to as DB_B.
[0011] First, as a preliminary step, the DB device that holds DB_A obtains the importance in DB_A for each feature included in DB_A. For example, the importance is calculated by using mutual information with other features in DB_A (details will be described later). Then, the DB device creates a weighted feature list 101 that indicates the feature names of features with high importance, the importance (weight) of the feature, and the data type (type) of the feature. Similarly, the DB device that holds DB_B obtains the importance in DB_B for each feature included in DB_B, and creates a weighted feature list 102. At this time, for the weighted feature list 101 and the weighted feature list 102, it is also possible to probabilistically define the possibility that a certain feature is of each type.
[0012] Thereafter, the matching device 10 acquires a weighted feature list 101 from the DB device that holds DB_A, and acquires a weighted feature list 102 from the DB device that holds DB_B. Then, the matching device 10 creates information (reference numeral 103) indicating the sum of weights of the feature pairs of the same type among the feature amounts shown in each list.
[0013] For example, in the information indicated by the reference numeral 103, the feature "company name" of DB_A and the feature "company name" of DB_B are a pair of type "company type", and the sum of the weights of the feature of the pair is "5.5+5.4=10.9". Also, the feature "address" of DB_A and the feature "location" of DB_B are a pair of type "address type", and the sum of the weights of the feature of the pair is "3.5+4.0=7.5". Furthermore, the feature "sales volume" of DB_A and the feature "capital" of DB_B are a pair of type "number type", and the sum of the weights of the feature of the pair is "4.0+2.0=6.0".
[0014] Thereafter, the matching device 10 performs matching between the records of DB_A and the records of DB_B based on the information indicated by the reference numeral 103 above.
[0015] For example, when the feature values of the pair indicated by reference numeral 103 are the same for a record 104 in DB_A and a record 105 in DB_B, the matching device 10 calculates a matching score by adding weighting values.
[0016] For example, for records 104 and 105, the matching device 10 does not add a weight because the value of "company name" in record 104 is not the same as the value of "business name" in record 105. On the other hand, if the value of "address" in record 104 is the same as the value of "location" in record 105, the matching device 10 adds a weight of "7.5". Furthermore, if the value of "sales volume" in record 104 is the same as the value of "capital" in record 105, the matching device 10 adds a weight of "6.0". The matching device 10 performs such processing for each feature of records 104 and 105.
[0017] This allows the matching device 10 to calculate a score indicating the degree to which the records 104 and 105 are records relating to the same item. Then, based on the calculated score value, the matching device 10 determines whether or not both records are records relating to the same item.
[0018] For example, if the scores of the records 104 and 105 exceed a predetermined value, the matching device 10 determines that the records 104 and 105 are records relating to the same subject matter (for example, records relating to the same company).
[0019] In addition, when the weighted feature lists 101 and 102 probabilistically define the possibility that a feature is of each type, for example, the matching device 10 may calculate the sum of the weights of the features for each pair of features of the same type as follows.
[0020] For example, if 90% of the pattern "company name" is "company name type" and 10% is "address type", the matching device 10 divides it into "company name of company name type" and "company name of address type". Then, the matching device 10 may divide the weight of 5.5 for "company name" in the weighted feature list 101 into 5.5 x 90% = 4.95 for the former and 5.5 x 10% = 0.55 for the latter, and calculate the sum of the weights of the features for each pair of features of the same type.
[0021] According to the matching device 10, it is possible to accurately match the records of each DB without acquiring records of DBs of a plurality of organizations as learning data.
[0022] [Processing procedure overview] Next, an overview of the processing procedure of a matching system including the matching device 10 will be described with reference to Fig. 2 and Fig. 3. As shown in Fig. 2, the matching system 1 includes the matching device 10, a DB device 20A, and a DB device 20B. The DB device 20A and the DB device 20B are held by different organizations.
[0023] [Preparation Phase] An example of a processing procedure in the advance preparation phase will be described with reference to Fig. 2. The advance preparation phase is a phase in which the matching system 1 makes advance preparations for matching records.
[0024] First, the matching device 10 acquires publicly available data such as a list of company names and addresses, and creates rules (A) for estimating the types of features such as telephone numbers and postal codes based on the acquired data ((1)).
[0025] Similarly to the above (1), the matching device 10 uses the acquired data as learning data and uses machine learning to build a model (type estimation model) (B) for estimating the type (e.g., company name, address, etc.) of the input feature ((2)).The matching device 10 then provides the above A and B to the DB device 20A ((3)).
[0026] Thereafter, the DB device 20A uses a group of records in its own DB to create information for selecting features to be used in matching ((4)). For example, the DB device 20A performs the following process. A program for creating this information can also be provided in advance from the matching device 10 to the DB devices 20A and 20B.
[0027] First, the DB device 20A uses the mutual information of all pairs of features in the DB to select features to be used in matching and calculates weights for using the features in matching ((4-1)).The DB device 20A also estimates the type of the feature selected in (4-1) using the above A and B ((4-2)).
[0028] After that, the DB device 20A adds the information (4-2) to the information (4-1) and transmits the information to the matching device 10 ((5)).
[0029] The matching device 10 also provides the above A and B to the DB device 20B ((6)). After that, the DB device 20B executes the same processes as (4) and (5) above using the records in its own DB.
[0030] This allows the matching device 10 to receive information (weighted feature list) from each of the DB devices 20A and 20B indicating features used in matching of records, estimated results of the types of the features, and weights when the features are used in matching. The matching device 10 then creates information (reference numeral 103 in FIG. 1) indicating the sum of weights of the features of a pair of features of the same type, for each pair of features indicated in the weighted feature list received from the DB devices 20A and 20B. Based on the above information, the matching device 10 also creates an equation (see equation (3) described later) used to calculate the matching score of the records received from the DB devices 20A and 20B.
[0031] [Matching Phase] Next, an example of a processing procedure in the matching phase will be described with reference to Fig. 3. The matching device 10 performs record matching as follows, using the weighted feature lists received from the DB devices 20A and 20B in the advance preparation phase.
[0032] First, the DB device 20A transmits record R to be matched to the matching device 10 ((1)). After that, the matching device 10 transmits a search query for records to be matched with record R to the DB device 20B ((2)). Next, the DB device 20B searches the DB using the search query, and transmits a group of records to be matched to the matching device 10 ((3)).
[0033] After that, the matching device 10 performs matching between the record R and each record in the record group received in (3) ((4) Perform Matching).
[0034] For example, the matching device 10 creates record pairs between the record R and each record in the record group received in (3). Then, the matching device 10 scores the degree to which each record pair matches using the weighted feature list received from the DB devices 20A and 20B.
[0035] After that, the matching device 10 transmits the result of the matching in (4) to the DB device 20A ((5)). For example, the matching device 10 transmits a pair of record R and the record with the highest matching score with record R to the DB device 20A.
[0036] In this way, the matching device 10 can match the records of each DB without acquiring records of DBs of a plurality of organizations as learning data.
[0037] If there are only records with scores equal to or less than a predetermined value, the matching device 10 may transmit to the DB device 20A a message indicating that there are no matching records. If there are multiple records with the same score, the matching device 10 may transmit all of those records to the DB device 20A, or may transmit to the DB device 20A the first or random record whose score exceeds a predetermined value.
[0038] In addition, it is assumed that the matching device 10 is managed by a trusted third party, but is not limited thereto. For example, the matching device 10 may be managed by an organization that manages the DB device 20A, or may be managed by an organization that manages the DB device 20B.
[0039] [Configuration example] Next, a configuration example of the matching system 1 will be described with reference to Fig. 4. The matching system 1 includes a matching device 10 and a DB device 20. First, the DB device 20 will be described.
[0040] [DB device] The DB device 20 includes a communication unit 21, an input / output unit 22, a storage unit 23, and a control unit 24. The communication unit 21 is an interface for performing data communication with an external device. The input / output unit 22 is an interface for performing various inputs and outputs to the DB device 20.
[0041] The storage unit 23 stores data for the control unit 24 to execute various processes. The storage unit 23 has a DB in a predetermined area. The DB stores multiple records. Each record is composed of multiple feature amounts. The storage unit 23 also stores type estimation rules (A) or type estimation models (B) transmitted from the matching device 10.
[0042] The control unit 24 controls the entire DB device 20. The control unit 24 includes a feature creation unit 241, a normalization unit 242, a feature reduction unit 243, a weight calculation unit 244, a feature selection unit 245, a type estimation unit 246, a weighted feature list transmission unit 247, a record transmission unit 248, and a matching result output unit 249.
[0043] The feature creation unit 241 creates candidates for features for matching using all records in the DB. Here, the feature creation unit 241 may incorporate, for example, behavior-based features as candidates for features for matching. For example, when a record in the DB is linked to other data (for example, a log group), the behavior-based features are features extracted from the linked data. Note that this process is optional.
[0044] The normalization unit 242 normalizes spelling variations in DB records. For example, when DB records are written in Japanese, the normalization unit 242 normalizes spellings of company names, addresses, and the like included in the records. The normalization unit 242 also standardizes spellings of hiragana / katakana included in the records to katakana, and standardizes spellings such as Kabushiki Kaisha / (Kabushiki Kaisha) to "Kabushiki Kaisha." This process is optional.
[0045] The feature amount reduction unit 243 deletes feature amounts that are not suitable for use in matching from the candidate feature amounts for matching created by the feature amount creation unit 241. For example, the feature amount reduction unit 243 deletes feature amounts whose value missing rate is equal to or greater than a predetermined value, feature amounts whose number of unique elements is equal to or less than a predetermined value, etc., from the candidate feature amounts for matching. Then, the feature amount reduction unit 243 outputs the candidate feature amounts for matching after the deletion to the weight calculation unit 244. Note that this process is optional.
[0046] The weight calculation unit 244 calculates the weight of each candidate feature for matching. The weight of the candidate feature for matching is calculated based on, for example, the importance of the candidate feature in the DB. Here, for example, the mutual information between the features in the DB is used as the importance.
[0047] Here, the reason why the mutual information between features in the DB is used as the importance is as follows. In matching, a process terminating at each DB is required without comparison with features in other DBs. In this case, it is possible to use a plurality of ways of expressing information (e.g., entropy, autoentropy, etc.), but if the mutual information of a pair of features is used from the viewpoint of comparing the relative strength with other features, the information amount of the pair of features can be defined. Therefore, the weight calculation unit 244 expresses a measure of interdependence of the pair of features using the mutual information of the pair of features as the importance. This is because the weight calculation unit 244 compares whether a certain feature is important with respect to other features, and a feature with a higher degree of interdependence is considered to be an important feature in each DB as a whole.
[0048] The weight calculation unit 244 calculates the mutual information between the feature amounts, for example, in the following manner.
[0049] (1) First, the weight calculation unit 244 takes all pairs (X, Y) of features. (2) Next, the weight calculation unit 244 calculates the mutual information I(X;Y) of each pair (X, Y) based on the records in the DB. For example, the weight calculation unit 244 calculates the mutual information I(X;Y) by the formula (1).
[0050]
number
[0051] Note that p(x,y) is the joint distribution function of X and Y. p(x,y) is calculated based on the probability that x and y appear simultaneously when a pair is created from a record set (all records may be used) randomly sampled from X and Y, respectively.
[0052] Also, p(x), p(y): marginal probability distribution functions. p(x) is the occurrence probability of x in a record set randomly sampled from X (all records may be used). Also, p(y) is calculated by the occurrence probability of y in a record set randomly sampled from Y (all records may be used).
[0053] If there are many records in the DB, the weight calculation unit 244 may sample a predetermined number of records from the DB and calculate the mutual information from the sampled records. The weight calculation unit 244 may also perform normalization processing such as dividing the calculated mutual information by the total number of records.
[0054] (3) Next, the weight calculation unit 244 calculates, for each feature candidate, the sum of mutual information of all pairs including the feature candidate. Then, the weight calculation unit 244 outputs the calculated sum of mutual information for each feature candidate to the feature selection unit 245.
[0055] The feature quantity selection unit 245 selects a feature quantity to be used for matching, using the weight (e.g., the sum of the above-mentioned mutual information) of each feature quantity candidate calculated by the weight calculation unit 244. For example, the feature quantity selection unit 245 preferentially selects, from among the feature quantity candidates output from the weight calculation unit 244, those with larger sums of mutual information.
[0056] As an example, the feature selection unit 245 arranges the feature candidates in descending order of the value of the sum of mutual information and selects feature candidates that are at or above a predetermined rank set in advance. Also, the feature selection unit 245 may select feature candidates whose sum of mutual information output from the weight calculation unit 244 exceeds a predetermined threshold.
[0057] The feature selection unit 245 selects the feature to be used for matching for the following reason. For example, when there are a large number of feature candidates, if the matching device 10 performs matching using all of the feature candidates, the amount of calculation required for matching will be large. Here, it is considered that a feature with a small sum (weight) of mutual information will not have a significant effect on the matching result even if it is used for matching. Therefore, the feature selection unit 245 selects a feature candidate whose sum of mutual features is at or above a predetermined rank, or a feature candidate whose sum is at or above a threshold, as a feature to be used for matching. This allows the matching device 10 to reduce the amount of calculation required for matching.
[0058] The feature amount selection unit 245 may use all the feature amounts output from the weight calculation unit 244 .
[0059] The type estimation unit 246 estimates the type of the feature quantity selected by the feature quantity selection unit 245. For example, the type estimation unit 246 estimates the type of each feature quantity candidate by applying a type estimation rule (A) to the record of each feature quantity candidate selected by the feature quantity selection unit 245. Alternatively, the type estimation unit 246 may input the record of each feature quantity candidate described above and estimate the type of each feature quantity candidate using a type estimation model (B). Here, it is also possible to probabilistically define multiple types for one feature quantity.
[0060] In addition, the type estimation unit 246 may sample records of each of the above feature candidate values from the DB, and if the type of the feature candidate value cannot be estimated using the above type estimation rule (A), it may estimate the type by applying the above type estimation model (B).
[0061] The feature list transmission unit 247 creates a weighted feature list indicating the features (feature candidates) selected by the feature selection unit 245, the estimation results of the types of the features estimated by the type estimation unit 246, and the weights of the features calculated by the weight calculation unit 244, and transmits the weighted feature list to the matching device 10.
[0062] The record sending unit 248 sends records to be matched to the matching device 10. For example, when the record sending unit 248 receives an instruction to send records to be matched via the input / output unit 22, the record sending unit 248 reads the records to be matched from the DB and sends them to the matching device 10. In addition, when the record sending unit 248 receives a search query for records from the matching device 10, the record sending unit 248 searches for records in the DB based on the search query and sends them to the matching device 10.
[0063] The matching result output unit 249 receives the matching result for the matching target record from the matching device 10. Then, the matching result output unit 249 outputs the received matching result via the input / output unit 22. For example, the matching result output unit 249 displays the received matching result on a display device (not shown) via the input / output unit 22. This allows the user of the DB device 20 to check the matching result for the matching target record.
[0064] [Matching device] Next, the matching device 10 will be described. The matching device 10 includes a communication unit 11, an input / output unit 12, a storage unit 13, and a control unit 14. The communication unit 11 is an interface for performing data communication with an external device. The input / output unit 12 is an interface for performing various inputs and outputs to the matching device 10.
[0065] The storage unit 13 stores data for the control unit 14 to execute various processes. For example, the storage unit 13 stores a weighted feature list received from each DB device 20. The storage unit 13 also stores a score calculation formula (details of which will be described later) created by the control unit 14.
[0066] The control unit 14 is responsible for overall control of the matching device 10. The control unit 14 includes a pattern estimation rule transmission unit 141, a weighted feature list acquisition unit 142, a feature information acquisition unit 143, a filtering unit 144, a score calculation formula creation unit 145, a record acquisition unit 146, a score calculation unit 147, a determination unit 148, and a transmission unit 149.
[0067] The type estimation rule transmission unit 141 creates type estimation rules and transmits them to the DB device 20. For example, the type estimation rule transmission unit 141 creates type estimation rules for telephone numbers, postal codes, and the like based on data such as a publicly available company name list and address list, and transmits them to each DB device 20.
[0068] Furthermore, the type estimation rule transmission unit 141 may use publicly available data such as a company name list and an address list as learning data, create a type estimation model using machine learning, and transmit the model to each DB device 20. For example, the type estimation rule transmission unit 141 creates a model that estimates the type (company name, address, etc.) of an input record using a machine learning algorithm (Random Forest, Linear Regression, etc.), and transmits the model to each DB device 20. For example, data analysis automation technology such as RakuDA may be used for tuning the accuracy of the model (parameter setting, etc.) here.
[0069] The type estimation rule transmitting unit 141 may transmit either the above-mentioned type estimation rule or type estimation model, or may transmit both of them.
[0070] The weighted feature list acquisition unit 142 acquires (receives) a weighted feature list from each DB device 20 that holds the record to be matched. The weighted feature list acquired from each DB device 20 is stored in the storage unit 13.
[0071] The feature amount information acquisition unit 143 acquires information on the feature amounts included in the weighted feature amount lists acquired from each DB device 20. For example, the feature amount information acquisition unit 143 acquires, from the weighted feature amount list of the DB device 20A, a feature amount group A (feature amount A1, feature amount A2, ..., feature amount An) included in the weighted feature amount list. Also, the feature amount information acquisition unit 143 acquires, from the weighted feature amount list of the DB device 20B, a feature amount group B (feature amount B1, feature amount B2, ..., feature amount Bm) included in the weighted feature amount list.
[0072] The filtering unit 144 removes pairs of features having different types from all pairs of feature groups of each DB device 20 acquired by the feature information acquisition unit 143. Whether or not the types of features constituting a pair are different is determined using the estimation results of the type of each feature shown in the weighted feature list of each DB device 20. Here, if the type is defined probabilistically for each feature, the weight can be distributed according to the magnitude of the probability of the type.
[0073] For example, the filtering unit 144 first creates pairs of the above feature group A (feature A1, feature A2, ..., feature An) and feature group B (feature B1, feature B2, ..., feature Bm), such as [feature A1, feature B1], [feature A1, feature B2], .... Next, the filtering unit 144 removes pairs with different feature type estimation results from among the above pairs of [feature A1, feature B1], [feature A1, feature B2], .... This leaves pairs of features with the same feature type estimation results. Note that this process is optional.
[0074] The score calculation formula creation unit 145 uses the pairs of features filtered by the filtering unit 144 and the weights of each feature shown in the weighted feature list to create a score calculation formula (score calculation formula) that indicates the degree to which records are records relating to the same item.
[0075] For example, the above calculation formula is a formula that adds a score when the feature values of a pair of records to be matched are the same. Here, the score calculation formula creation unit 145 weights the score to be added when the feature values of the above record pair are the same. The weighting value here is the sum of the weights of each feature shown in the weighted feature list. For example, the score calculation formula creation unit 145 creates the following formula (2) as the score calculation formula for a record pair.
[0076]
number
[0077] In formula (2), W[feature amount A1, feature amount B1] is the sum of the weight of feature amount A1 and the weight of feature amount B1 in the weighted feature amount list. For example, if the weight of feature amount A1 “company name” in the weighted feature amount list transmitted from DB device 20A is “5.5” and the weight of feature amount B1 “company name” in the weighted feature amount list transmitted from DB device 20B is “5.4”, W[feature amount A1, feature amount B1] is “5.5+5.4=10.9”.
[0078] In other words, the score calculation formula creation unit 145 creates a score calculation formula S(x, y) for a pair of records as shown in the following formula (3).
[0079]
number
[0080] The score calculation formula creation unit 145 stores the created score calculation formula in the storage unit 13.
[0081] The record acquisition unit 146 acquires records to be matched from each DB device 20. For example, the record acquisition unit 146 acquires records to be matched from the DB device 20A. After that, the record acquisition unit 146 transmits a search query for records to be matched with the acquired records to the DB device 20B, and acquires search results (searched records) for the search query from the DB device 20B.
[0082] For example, when the matching target records acquired from the DB device 20A by the record acquisition unit 146 are records related to companies in a certain region, a search query for records related to the region is transmitted to the DB device 20B. This allows the matching device 10 to narrow down the matching target records.
[0083] The score calculation unit 147 calculates a score for each pair of records for matching targets acquired by the record acquisition unit 146.
[0084] For example, the score calculation unit 147 creates record pairs, such as a pair of record R and record R1, a pair of record R and record R2, etc., for record R to be matched acquired from the DB device 20A and a group of records to be matched acquired from the DB device 20B (records R1 to Rn). Then, the score calculation unit 147 calculates a score for each of the record pairs using the score calculation formula created by the score calculation formula creation unit 145.
[0085] The determination unit 148 determines whether or not the record pair is a record pair relating to the same matter, based on the magnitude of each score of the record pair calculated by the score calculation unit 147. For example, the determination unit 148 checks the score of each record pair calculated by the score calculation unit 147, and if there is a record pair whose score is equal to or greater than a predetermined threshold, the determination unit 148 determines that the record pair is a record pair relating to the same matter.
[0086] Furthermore, the determination unit 148 may determine that the pair of records having the highest score is a pair of records relating to the same item, based on the magnitude of each score of the pair of records calculated by the score calculation unit 147. Furthermore, the determination unit 148 may determine that there is no pair of records relating to the same item, when there is no pair of records having a score equal to or greater than a predetermined threshold.
[0087] The transmission unit 149 transmits the result of the determination by the determination unit 148 (the result of matching) to the DB device 20. For example, the transmission unit 149 transmits the result of matching to the DB device 20A or the DB device 20B that holds the record to be matched. The transmission unit 149 may transmit the result of matching to the organization that manages the DB device 20A or the organization that manages the DB device 20B.
[0088] [Example of processing procedure] Next, a description will be given of an example of a processing procedure of the matching system 1. First, a description will be given of an example of a processing procedure of the advance preparation phase of the matching system 1 with reference to FIG.
[0089] First, the type estimation rule transmission unit 141 of the matching device 10 creates a type estimation rule (A) (S1) and also creates a type estimation model (B) (S2). Then, the type estimation rule transmission unit 141 transmits the above type estimation rule (A) and type estimation model (B) to the DB devices 20A and 20B (S3).
[0090] After S3, the DB device 20A stores the type estimation rule (A) and the type estimation model (B) in the storage unit 23 (S10). Then, the feature creation unit 241 creates feature candidates for matching using all records in the DB (S11: feature creation). In addition, the normalization unit 242 normalizes the spelling variations of the DB records (S12: normalization).
[0091] After S12, the feature reduction unit 243 removes from the candidate features for matching created in S11, features in the DB whose value missing rate is greater than or equal to a predetermined value, features whose number of unique elements is less than or equal to a predetermined value, etc. (S13: feature reduction).
[0092] After S13, the weight calculation unit 244 calculates a weight for each candidate feature amount for matching (e.g., the sum of mutual information for each candidate feature amount) (S14: Calculate weight). Thereafter, the feature selection unit 245 uses the weight for each candidate feature amount calculated in S14 to select a feature amount to be used for matching (S15: Select feature amount). For example, the feature selection unit 245 sorts the sum of mutual information for each candidate feature amount calculated in S14 in descending order, and selects feature amount candidates that are ranked higher than a predetermined rank.
[0093] After S15, the type estimation unit 246 estimates the type of the feature selected in S15 (S16). For the type estimation here, the type estimation rule (A) and the type estimation model (B) received in S10 are used. Then, the weighted feature list transmission unit 247 creates a weighted feature list indicating the feature selected in S15, the estimation result of the type of the feature estimated in S16, and the weight of the feature calculated in S14, and transmits the weighted feature list to the matching device 10 (S17).
[0094] After S3, the DB device 20B executes the processes of S20 to S27. The processes of S20 to S27 are similar to the processes of S10 to S17 described above, and therefore will not be described.
[0095] Thereafter, when the weighted feature quantity list acquisition unit 142 of the matching device 10 receives the weighted feature quantity lists from each of the DB devices 20A and 20B, it stores each weighted feature quantity list in the storage unit 13 (S31). Then, the feature quantity information acquisition unit 143 acquires feature quantity information from each of the weighted feature quantity lists of the DB devices 20A and 20B (S32). For example, the feature quantity information acquisition unit 143 acquires feature quantity group A (feature quantity A1, feature quantity A2, ..., feature quantity An) from the weighted feature quantity list of the DB device 20A, and acquires feature quantity group B (feature quantity B1, feature quantity B2, ..., feature quantity Bm) from the weighted feature quantity list of the DB device 20B.
[0096] After S32, the filtering unit 144 excludes pairs of features whose estimated types are different from the information of the features acquired in S32 (S33: filtering of feature pairs). Then, the score calculation formula creation unit 145 creates a score calculation formula using the feature pairs filtered in S33 and the weights of each feature shown in the weighted feature lists of the DB devices 20A and 20B (S34). Then, the score calculation formula creation unit 145 stores the created score calculation formula in the storage unit 13.
[0097] Next, an example of a processing procedure of the matching phase of the matching system 1 will be described with reference to FIG.
[0098] First, the record transmission unit 248 of the DB device 20A reads out a record to be matched from the DB and transmits it to the matching device 10 (S41). The record acquisition unit 146 of the matching device 10 stores in the storage unit 13 the record to be matched transmitted in S41.
[0099] After S41, the record acquisition unit 146 of the matching device 10 transmits a search query for records to be matched to the DB device 20B (S42). After that, the record transmission unit 248 of the DB device 20B searches for records to be matched from the DB based on the search query and transmits them to the matching device 10 (S43). After that, the record acquisition unit 146 of the matching device 10 stores the records to be matched transmitted in S43 in the storage unit 13.
[0100] After S43, the score calculation unit 147 of the matching device 10 calculates the score for each pair of records for matching targets transmitted from the DB devices 20A and 20B (S44). Note that the score calculation formula created in S34 of FIG. 5 is used to calculate the score.
[0101] After S44, the determination unit 148 of the matching device 10 performs a determination based on the magnitude of the score for each pair of records to be matched calculated in S44 (S45). For example, the determination unit 148 checks the score for each pair of records calculated in S44, and determines the pair of records with the highest score as a pair of records related to the same item. After that, the transmission unit 149 transmits the result of the determination in S45 (the result of matching) to the DB device 20A (S46).
[0102] Thereafter, the matching result output unit 249 of the DB device 20A outputs the matching result to a display device or the like (S47).
[0103] In this way, the matching system 1 can accurately match the records of each DB without acquiring the records of the DBs of multiple organizations as learning data. This allows the matching system 1 to accurately perform, for example, name matching of the records of the DBs of multiple organizations.
[0104] Furthermore, the matching system 1 does not require machine learning using DB records of multiple organizations as learning data, so the processing load required for matching is small. Furthermore, the matching system 1 does not need to acquire records other than those to be matched from each DB device 20 that holds the records to be matched, so there is no privacy issue even if each DB device 20 is managed by a different organization.
[0105] Furthermore, since the matching system 1 performs matching by narrowing down the feature quantities appropriate for record matching, it is also possible to match records composed of a large number of feature quantities. Furthermore, when selecting feature quantities to be used for record matching, the matching system 1 deletes feature quantities with a missing value rate equal to or greater than a predetermined value, feature quantities with a number of unique elements equal to or less than a predetermined value, and the like from feature quantity candidates for matching. Furthermore, the matching system 1 performs filtering using the feature quantity type estimation result for pairs of feature quantities to be used for record matching. In this way, the matching system 1 can perform matching even if, for example, each DB contains records in which various types are mixed for one feature quantity.
[0106] For example, a feature that constitutes a record may contain multiple types of data due to, for example, human error or differences in organizational operations. For example, when there are multiple records of the same type, such as a phone number type, it may be difficult for a human to determine which feature should be used for matching. Even in such cases, the matching system 1 can determine the feature appropriate for matching and perform matching. Furthermore, even if there is a bias in the number of columns of records between DBs, the matching system 1 can match records.
[0107] [Other embodiments] In the above embodiment, the matching device 10 performs matching of the records of the DBs of two DB devices 20, but the present invention is not limited to this. For example, the matching device 10 may perform matching of the records of the DBs of three or more DB devices 20.
[0108] Furthermore, although the matching device 10 is described as performing record matching using the estimation results of the type of each feature transmitted from each DB device 20, matching may be performed without using the estimation results of the type of each feature.
[0109] In addition, the DB device 20 may incorporate behavior-based features when creating candidates for features for matching, may perform normalization processing of spelling variations in DB records, or may not perform these processes.
[0110] [Example of application] Next, we will explain an application example of the above-mentioned matching system 1. When the matching system 1 was used to match in-house data with external data, the matching rate was 95.3%.
[0111] From the above results, it was confirmed that matching system 1 can accurately match DB records without performing machine learning using the records of each DB.
[0112] [System configuration, etc.] In addition, each component of each unit shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.
[0113] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.
[0114] [program] The above-mentioned matching device 10 can be implemented by installing a program (matching program) as package software or online software on a desired computer. For example, by executing the above-mentioned program on an information processing device, the information processing device can function as the matching device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), as well as terminals such as PDAs (Personal Digital Assistants).
[0115] 7 is a diagram showing an example of a computer that executes a matching program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0116] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.
[0117] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process executed by the above-mentioned matching device 10 is implemented as a program module 1093 in which a computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration of the matching device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).
[0118] Data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.
[0119] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]
[0120] 10 Matching device (judgment device) 20 DB device 11,21 Communications Department 12,22 Input / output section 13,23 Storage section 14,24 Control section 141 Type estimation rule sending part 142 Weighted feature list acquisition unit 143 Feature information acquisition unit 144 Filtering section 145 Score calculation formula creation section 146 Record Acquisition Section 147 Score Calculation Section 148 Judgment section 149 Transmitter 241 Feature Creation Unit 242 Normalization section 243 Feature Reduction Unit 244 Weight Calculation Unit 245 Feature Selection Unit 246 Type Estimator 247 Weighted feature list transmission unit 248 Record Transmission Unit 249 Matching result output section
Claims
1. 1. A determination device for determining whether a first record stored in a first DB owned by a first organization and a second record stored in a second DB owned by a second organization are records relating to the same item, comprising: an acquisition unit that acquires the first record and the second record; a score calculation unit that calculates a score indicating a degree to which the first record and the second record are records relating to the same subject by weighting, for each pair of a feature amount constituting the first record and a feature amount constituting the second record, a value to be added when the feature amounts in the pair are the same, using the importance in the first DB of the feature amount constituting the first record and the importance in the second DB of the feature amount constituting the second record; a determination unit that determines whether the first record and the second record are records relating to the same item based on the magnitude of the score; a transmitting unit that transmits a result of the determination to the first organization or the second organization; A determination device comprising:
2. The first record or the second record includes a behavior-based feature.
2. The determination device according to claim 1.
3. The feature amount constituting the first record is Among the feature amounts included in the first DB, the feature amount is preferentially selected from the feature amounts having a high mutual information with other feature amounts included in the first DB, The feature amount constituting the second record is Among the features contained in the second DB, the feature is preferentially selected from those having high mutual information with other features contained in the second DB.
2. The determination device according to claim 1.
4. The importance of the feature amount constituting the first record in the first DB is a value calculated based on mutual information between the feature and another feature included in the first DB, The importance of the feature amount constituting the second record in the second DB is a value calculated based on mutual information between the feature and another feature included in the second DB; 4. The determination device according to claim 3.
5. A pair of a feature amount constituting the first record and a feature amount constituting the second record is A pair of features with the same data type 2. The determination device according to claim 1, wherein
6. A determination method executed by a determination device that determines whether a first record stored in a first DB owned by a first organization and a second record stored in a second DB owned by a second organization are records relating to the same item, the method comprising: obtaining the first record and the second record; calculating a score indicating the degree to which the first record and the second record are records relating to the same subject by weighting, for each pair of a feature amount constituting the first record and a feature amount constituting the second record, a value to be added when the feature amounts in the pair are the same, using the importance in the first DB of the feature amount constituting the first record and the importance in the second DB of the feature amount constituting the second record; determining whether the first record and the second record are records relating to the same item based on the magnitude of the score; transmitting a result of the determination to the first organization or the second organization; A method for determining whether or not a particular
7. 1. A determination program for determining whether a first record stored in a first DB owned by a first organization and a second record stored in a second DB owned by a second organization are records relating to the same item, comprising: obtaining the first record and the second record; calculating a score indicating the degree to which the first record and the second record are records relating to the same subject by weighting, for each pair of a feature amount constituting the first record and a feature amount constituting the second record, a value to be added when the feature amounts in the pair are the same, using the importance in the first DB of the feature amount constituting the first record and the importance in the second DB of the feature amount constituting the second record; determining whether the first record and the second record are records relating to the same item based on the magnitude of the score; transmitting a result of the determination to the first organization or the second organization; A judgment program for causing a computer to execute the above.
Citation Information
Patent Citations
Company name matching methods, devices, computer equipment, and storage media
CN112597284B
Method and system for schema matching of web database
JP2006004411A
Data tabulation processing method and system
JP2010122880A
Extraction program, extraction method and extraction apparatus
JP2013029891A
Database binding apparatus, database binding method, and database binding program
JP2019159837A
