Data fusion system and data fusion method

Abstract

To provide a data fusion system and a data fusion method for enabling improvement of data fusion accuracy and reduction in calculation cost.SOLUTION: In a data fusion system 100, an arithmetic unit 101 includes: a covariate acquisition unit 111 that acquires a plurality of covariates included in common in first data and second data which are two pieces of data to be received by an input unit 102 and to be fused; a matching record number calculation unit 113 that calculates, from the first data and the second data, the number of records in which values of respective covariates of the plurality of covariates match; a matching record number comparison unit 114 that compares the matching record number for each combination of the respective covariates of the plurality of covariates; a covariate selection unit 115 that based on a matching record number comparison result, selects at least one covariate from among the plurality of covariates; and a post-fusion data analysis unit 117 that fuses the first data and the second data and analyzes data thereof using the at least one covariate selected by the covariate selection unit 115.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a data fusion system and a data fusion method for concatenating a plurality of data.

Background Art

[0002] By concatenating and analyzing a plurality of data, it is possible to make decisions based on more diverse analyses and highly accurate inferences. When there is a common personal key (ID assigned to each individual) between the data to be concatenated, the data can be concatenated by linking records with the same value of the personal key. However, when there is no personal key, it is difficult to concatenate the data. Therefore, a data fusion technique for concatenating data by using a plurality of common variables (covariates) between data that are difficult to concatenate as a clue has been studied.

[0003] As conventional examples of using data fusion technology to concatenate data, there are Patent Document 1 and Patent Document 2. In Patent Document 1, a computer fuses the data of a first questionnaire and a second questionnaire respectively implemented by different investigators for different monitors, using the degree of similarity of the answers to common questions included in both questionnaires as a key. At this time, the computer assigns, to the monitor of the first questionnaire, the same answer content as the answer content of the monitor to the questions included only in the second questionnaire, according to an assignment pattern set based on the degree of similarity. The assignment pattern is set according to a statistical solution so that the aggregation result regarding the questions included only in the first questionnaire does not change before and after fusion, and the aggregation result regarding the questions included only in the second questionnaire does not change before and after fusion. And it is disclosed that the computer distributes the fusion data to the terminal that issues a data distribution request.

[0004] Furthermore, Patent Document 2 discloses the following steps: Identify new data from the first survey data. Apply a function to common items of both the new first survey data and the second survey data to calculate a group of scores for distance calculation. Compare the groups of scores and perform a distance calculation to determine the degree of similarity between each monitor A' of the new first survey data and each monitor B of the second survey data. For each monitor A' of the new first survey data, identify the monitor with the closest total distance from the monitor B of the second survey data, treat them as the same monitor, and merge them. Store the merged data. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2015-32199 [Patent Document 2] Japanese Patent Publication No. 2018-156299 [Overview of the project] [Problems that the invention aims to solve]

[0006] To improve the accuracy of data fusion, it is necessary to select the optimal covariates from all available covariates. Furthermore, by selecting the appropriate covariates and reducing the number of covariates used in data fusion, it is possible to shorten the computation time for data fusion. However, neither Patent Document 1 nor Patent Document 2 includes a selection of covariates. Therefore, the following problems exist. (1) Degradation of data fusion accuracy (2) Increased computational cost when applied to large-scale data or data with many covariates Therefore, the present invention provides a data fusion system and a data fusion method that can improve data fusion accuracy and reduce computational costs. [Means for solving the problem]

[0007] To solve the above problems, the data fusion system according to the present invention is a data fusion system for linking two sets of data, comprising: a computing device that performs predetermined processing; an input unit into which data is input; and a storage device accessible by the computing device, wherein the input unit receives input of two sets of data to be fused, namely first data and second data; the computing device comprises: a covariate acquisition unit that acquires a plurality of covariates that are commonly included in the first data and the second data; a matching record count calculation unit that calculates the number of records in which the values ​​of each of the plurality of covariates match from the first data and the second data; a matching record count comparison unit that compares the matching record count for each combination of the plurality of covariates; a covariate selection unit that selects at least one covariate from the plurality of covariates based on the matching record count comparison result; a data fusion implementation unit that fuses the first data and the second data using at least one covariate selected by the covariate selection unit; and a post-fusion data analysis unit that analyzes the data after data fusion has been implemented. The matching record count calculation unit then determines the number of records in the second data set that match the values ​​of each covariate in the first data set, and calculates the average. It is characterized by doing so.

[0008] Furthermore, the data fusion method according to the present invention comprises a computing device that performs predetermined processing, an input unit into which data is input, and a storage device accessible by the computing device, and is a data fusion method for linking two sets of data, wherein the covariate acquisition unit of the computing device The 1 data and The The process includes: obtaining multiple covariates that are commonly included in the two sets of data; the matching record count calculation unit of the computing device calculating the number of records in which the values ​​of each of the multiple covariates match from the first set of data and the second set of data; the matching record count comparison unit of the computing device comparing the matching record count for each combination of the multiple covariates; the covariate selection unit of the computing device selecting at least one covariate from the multiple covariates based on the matching record count comparison result; the data fusion implementation unit of the computing device fusing the first set of data and the second set of data using at least one covariate selected by the covariate selection unit; and the post-fusion data analysis unit of the computing device analyzing the data after data fusion. The matching record count comparison unit then determines the number of records in the second data set that match the values ​​of each covariate in the first data set, and calculates the average. It is characterized by doing the following.

Advantages of the Invention

[0009] According to the present invention, it is possible to provide a data fusion system and a data fusion method that can improve data fusion accuracy and reduce calculation costs. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0010] [Figure 1] It is a block diagram showing the configuration of a data fusion system according to Embodiment 1 of the present invention. [Figure 2] It is a diagram showing an example of data A (first data). [Figure 3] It is a diagram showing an example of data B (second data). [Figure 4] It is a diagram showing an example of covariance. [Figure 5] It is a diagram showing an example of a covariance setting screen. [Figure 6] It is a diagram showing an example of a set of covariance names. [Figure 7] It is a diagram showing an example of the number of matching records. [Figure 8] It is a diagram showing an example of a covariance selection screen. [Figure 9] It is a diagram showing an example of data after data fusion. [Figure 10] It is a diagram showing an example of a data fusion result display / analysis screen. [Figure 11] It is a diagram showing a flowchart of data fusion processing. [Figure 12] It is a diagram showing a flowchart of matching record number calculation processing. [Figure 13] It is a diagram showing an example of matching record number calculation when the number of covariances K = 1. [Figure 14] It is a diagram showing an example of matching record number calculation when the number of covariances K = 2.

Modes for Carrying Out the Invention

[0011] In this specification, the first data will be referred to as Data A, and the second data will be referred to as Data B. Furthermore, in an embodiment of the present invention, a data fusion system is described in which covariates are selected for two data sets to be linked (data A and data B), and the data are linked using the selected covariates. The following describes embodiments of the present invention with reference to the drawings. [Examples]

[0012] This embodiment describes an example of a data fusion system that obtains covariates from data A (first data) and data B (second data), selects covariates based on the number of matching records, and concatenates the data using the selected covariates.

[0013] Figure 1 is a block diagram showing the configuration of the data fusion system according to this embodiment. As shown in Figure 1, the data fusion system 100 according to this embodiment includes a data fusion terminal 101 and a database 120.

[0014] The data fusion terminal 101 is a computer having an input unit 102, an output unit 103, a processor (CPU) 104, a memory 105, and a storage medium 106.

[0015] The input unit 102 is a human interface such as a mouse or keyboard, which receives input to the data fusion terminal 101. The output unit 103 is a display or printer that outputs the calculation results from the data fusion terminal 101. The storage medium 106 is a storage device that stores various programs that realize data fusion processing by the data fusion terminal 101, as well as the execution results of the data fusion processing, and is composed of, for example, a non-volatile storage medium (magnetic disk drive, non-volatile memory, etc.).

[0016] Memory 105 includes a non-volatile memory element called ROM and a volatile memory element called RAM. ROM stores immutable programs (e.g., BIOS). RAM is a high-speed, volatile memory element, such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 104 and data used during program execution. In other words, programs stored in the storage medium 106 are loaded into memory 105.

[0017] The processor 104 is an arithmetic unit that executes the program loaded into memory 105, and is, for example, a CPU or GPU. The processes and calculations described below are executed by the processor 104. Note that some of the processes that the processor 104 executes by running the program may be executed by other arithmetic units (for example, hardware such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit)).

[0018] The data fusion system 100 according to this embodiment is a computer system that operates on a single computer or on multiple logically or physically configured computers, and may operate in separate threads on the same computer, or on a virtual computer built on multiple physical computer resources.

[0019] The program executed by processor 104 is provided to each server via removable media (such as CD-ROM or flash memory) or a network, and stored in a non-volatile storage device, which is a non-temporary storage medium. For this reason, the computer system may be equipped with an interface for reading removable media.

[0020] The storage medium 106 stores programs for implementing the covariate acquisition unit 111, the matching record count calculation unit 112, the matching record count comparison unit 113, the covariate addition unit 114, the covariate selection unit 115, the data fusion implementation unit 116, and the post-fusion data analysis unit 117.

[0021] The covariate acquisition unit 111 acquires data A (Figure 2) and data B (Figure 3) input to the input unit 102, and then acquires variables whose variable names match or whose variable names or contents are similar in data A (first data) and data B (second data) as covariates (Figure 4), and displays the variable names (covariate names) of the acquired covariates on the output unit 103 (Figure 5).

[0022] The covariate addition unit 112 creates a set of covariate names (Figure 6) by adding one covariate name that is not included in the covariate list (Figure 7) to the covariate names obtained by the covariate acquisition unit 111. This is done using all covariate names that are not included in the covariate list. Here, the covariate list is a list that stores the selected covariates, and its initial value is empty.

[0023] The matching record count calculation unit 113 calculates the "matching record count" (Figure 7) using covariates whose covariate names match each element of the set of covariate names created by the covariate addition unit 112, from among the covariates acquired by the covariate acquisition unit 111. This is done for all sets created by the covariate addition unit 112. Here, the matching record count is an index indicating the degree to which people with similar characteristics can be identified, and is characterized by finding the number of records of the covariates in data B that match the value of each record of the covariates in data A, and calculating the average value of these record counts. The detailed calculation method will be described later with reference to Figures 12, 13, and 14.

[0024] The matching record count comparison unit 114 compares the matching record counts calculated by the record count calculation unit 113 and obtains the set of covariate names (combinations of covariates) that maximizes the reciprocal of the matching record count, and the maximum value of the reciprocal of the matching record count. Alternatively, it may obtain the set of covariate names (combinations of covariates) that minimizes the matching record count, and the minimum value of the matching record count.

[0025] If the maximum value of the reciprocal of the number of matching records obtained by the matching record count comparison unit 114 is greater than the value already stored in the maximum value storage variable, the covariate selection unit 115 stores the set of covariate names (combinations of covariates) obtained by the matching record count comparison unit 114 in the covariate list, and the maximum value of the reciprocal of the number of matching records obtained by the matching record count comparison unit 114 in the maximum value storage variable, and then proceeds to the covariate addition unit 112. Otherwise, it obtains the covariate list and displays the covariate names already stored in the covariate list in the output unit 103 (Figure 8). Here, the maximum value storage variable is a variable that stores the maximum value of the reciprocal of the number of matching records, and its initial value is 0. In other words, the covariate selection unit 115 compares the maximum value of the reciprocal of the number of matching records obtained by the matching record count comparison unit when there are K-1 covariates and when there are K covariates. If the maximum value of the reciprocal of the number of matching records when there are K covariates is larger, it proceeds to the calculation and comparison of the number of matching records when there are K+1 covariates. Otherwise, it outputs the combination of covariates obtained by the matching record count comparison unit when there are K covariates (displayed on the output unit 103). If the minimum number of matching records is obtained by the matching record count comparison unit 114, the reverse processing is performed. That is, if the obtained minimum value is smaller than the minimum value storage variable, the set of covariate names (combinations of covariates) obtained by the matching record count comparison unit 114 is stored in the covariate list, and the minimum number of matching records obtained by the matching record count comparison unit 114 is stored in the minimum value storage variable, and the process proceeds to the covariate addition unit 112. Otherwise, the covariate list is obtained, and the covariate names already stored in the covariate list are displayed in the output unit 103 (Figure 8). Here, the minimum value storage variable is a variable that stores the minimum number of matching records, and its initial value is set to a very large value. In other words, the covariate selection unit 115 compares the minimum value of the number of matching records obtained by the matching record count comparison unit when there are K-1 covariates and when there are K covariates. If the minimum value of the number of matching records when there are K covariates is smaller, it proceeds to the calculation and comparison of the number of matching records when there are K+1 covariates. Otherwise, it outputs the combination of covariates obtained by the matching record count comparison unit when there are K covariates (displayed on the output unit 103).

[0026] The data fusion implementation unit 116 uses covariates whose names match those included in the covariate list obtained by the covariate selection unit 115 from among all covariates of data A and data B obtained by the covariate acquisition unit 111, and performs data fusion by matching the records with the smallest distance between the covariates of data A and data B, thereby creating the data after data fusion (Figure 9).

[0027] The post-fusion data analysis unit 117 analyzes the data after data fusion acquired by the data fusion implementation unit 116. It also displays the data after data fusion and the analysis results of the data after data fusion on the output unit 103 (Figure 10).

[0028] As shown in Figure 1, the database 120 consists of a data A storage unit 121, a data B storage unit 122, a covariate storage unit 123, a matching record count storage unit 124, a covariate selection result storage unit 125, a data fusion result storage unit 126, and a post-fusion data analysis result storage unit 125.

[0029] The data A storage unit 121 stores one of the two data to be merged that are input to the input unit 102. The data B storage unit 122 stores the data that is not stored in the data A storage unit from the two data to be merged that were input to the input unit 102. In other words, it stores the other data from the two data to be merged that were input to the input unit 102.

[0030] The covariate storage unit 123 stores the covariates acquired by the covariate acquisition unit 111. The matching record count storage unit 124 stores the set of covariate names created by the covariate addition unit 112 and the number of matching records calculated by the matching record count calculation unit 113 corresponding to the set of covariate names.

[0031] The covariate selection result storage unit 125 stores the covariate list acquired by the covariate selection unit 115. The data fusion result storage unit 126 stores the data after data fusion created by the data fusion implementation unit 116. The post-fusion data analysis result storage unit 127 stores the analysis results of the data after data fusion created by the post-fusion data analysis unit 117.

[0032] Figure 2 shows an example of data A200 stored in the data A storage unit 121. Data A200 contains multiple variables that store numerical values. In the example shown, Data A200 contains a total of 10 variables, from Variable 1 to Variable 10.

[0033] Figure 3 shows an example of data B300 stored in the data B storage unit 122. Data B300 contains multiple variables that store numerical values. In the example shown, Data B300 contains a total of 10 variables, from Variable 3 to Variable 12.

[0034] Figure 4 shows an example of covariates 400 acquired by the covariate acquisition unit 111. As shown in Figure 4, covariate 400 includes covariate 401 from data A and covariate 402 from data B.

[0035] Covariate 401 of data A is a covariate included in data A from the covariate acquisition unit 111's acquired covariate acquisition unit 111's acquired covariate acquisition unit 111's acquired covariate acquisition unit 111's acquired covariate acquisition unit 111's acquired covariate acquisition unit 400's covariate included in data B.

[0036] Figure 5 shows an example of the covariate setting screen 500 that the covariate acquisition unit 111 outputs to the output unit 103. The covariate setting screen 500 includes a covariate acquisition method setting field 510, a covariate candidate display field 520, and a covariate display field 530.

[0037] The covariate acquisition method setting field 510 allows the user to select how to acquire covariates. In the example shown, "Variable names match" is selected as the covariate acquisition method. In this case, variables with the same name from data A200 and data B300 are acquired as covariates. If "Manual" is selected, the covariates are manually set in the covariate candidate display field 520, which will be explained below.

[0038] The Covariate Candidate Display Area 520 is a field that displays candidate variables to be used as covariates. In the example shown, since "Variable names match" is selected in the Covariate Acquisition Method Setting Area 510, the Covariate Candidate Display Area 520 displays variables from data A200 and data B300 that have the same variable name. The Covariate Candidate Display Area 520 includes a Covariate Candidate Deselection Button 521 and a Covariate Candidate Add Button 522. Variables for which the Covariate Candidate Deselection Button 521 is pressed are removed from the list of candidate covariates. In addition, candidate covariates can be manually added using the Covariate Candidate Add Button 522. For example, by selecting the variables "Age" from data A and "age" from data B, the variables "Age" from data A and "age" from data B are associated and displayed in the Covariate Candidate Display Area 520.

[0039] The covariate display area 530 displays the covariate names of the covariates set in the covariate candidate display area 520. If variables with different names are set as covariates in Data A and Data B, the covariate name of Data A (in the example above, "age") will be displayed.

[0040] Figure 6 shows an example of a set of covariate names 600 created by the covariate addition unit 112.

[0041] The set of covariate names 600 is the set of covariate names included in the covariate list 720 plus one covariate name not included in the covariate list 720. The illustrated example shows the set of covariate names 600 for loop 0, loop 1, and loop 3. In loop 0, since the covariate list 720 is empty, a set of 1 element is created for each covariate name. In loop 1, since "variable 5" is included in the covariate list 720, a set of 2 elements is created by adding one covariate name other than "variable 5" to "variable 5". In loop 2, since "variable 5" and "variable 3" are included in the covariate list 720, a set of 3 elements is created by adding one covariate name other than "variable 5" and "variable 3" to "variable 5" and "variable 3". The set of covariates 600 is created similarly in loops 3, 4, 5, etc.

[0042] Figure 7 shows an example of the number of matching records 710 and the covariate list 720 calculated by the matching record count calculation unit 113. The number of matching records, 710, is calculated for each set of covariate names, 600, created by the covariate addition unit 112 (Figure 1). In the illustrated example, in the 0th loop iteration, the number of matching records for {"Variable 5"} is smallest (the reciprocal of the number of matching records is largest), so "Variable 5" is stored in the covariate list 720. Similarly, in the 1st, 2nd, 3rd...th iterations, the number of matching records, 710, is calculated for each set of covariate names, 600, created by the covariate addition unit 112, and the set of covariate names, 600, with the smallest number of matching records, 710 (the reciprocal of the number of matching records is largest), is stored in the covariate list 720. If the minimum number of matching records (the maximum value of the reciprocal of the number of matching records) does not change before and after the loop, the subsequent loops are stopped, and the set of covariate names, 600, already stored in the covariate list 720 is retrieved.

[0043] Figure 8 shows an example of a covariate selection screen 800 that the covariate selection unit 115 outputs to the output unit 103. As shown in Figure 8, the covariate selection screen 800 includes a covariate selection result display field 810, a matching record count calculation result display field 820, and a selected covariate display field 830.

[0044] The covariate selection result display field 810 displays the names of covariates stored in the covariate list 720 acquired by the covariate selection unit 115. In the illustrated example, the names of four covariates stored in the covariate list 720, "Variable 5", "Variable 3", "Variable 7", and "Variable 4", are displayed. The covariate selection result display field 810 includes a covariate selection deselection button 811 and a covariate addition button 812. Covariates for which the covariate selection deselection button 811 is pressed are excluded from the selection results and will not be used for data fusion. In addition, the covariate addition button 812 allows manual addition of covariates to be used for data fusion from the covariate names acquired by the covariate acquisition unit 111.

[0045] The matching record count calculation result display field 820 displays the set of covariate names 600 that have the maximum reciprocal of the matching record count 710 obtained by the matching record count comparison unit 114, and the corresponding reciprocals of the matching record count (821-824). Alternatively, the set of covariate names 600 that have the minimum matching record count 710 and the corresponding matching record count 710 may be displayed. The selected covariates display field 830 displays the names of the covariates set in the covariate selection results display field 810.

[0046] Figure 9 shows an example of the data 900 after data fusion stored in the data fusion result storage unit 126. As shown in Figure 9, the fused data 900 includes the selected covariates 901, the variable 902 that is present only in data A, and the variable 903 that is present only in data B.

[0047] The selected covariate 901 is a covariate from the covariates 401 of data A whose covariate name matches the covariate name obtained by the covariate selection unit 115. The variable 902, which is included only in data A, is a variable from data A200 that is not included in the covariates 401 of data A. The variable 903, which is included only in data B, is a variable from data B300 that is not included in the covariates 402 of data B, and stores the value corresponding to each record of the selected covariate 901 that has been linked by data fusion.

[0048] Figure 10 shows an example of a data fusion result display / analysis screen 1000 that the post-fusion data analysis unit 117 outputs to the output unit 103. As shown in Figure 10, the data fusion result display and analysis screen 1000 includes a data fusion result display area 1010 and a post-fusion data analysis settings and result display area 1020.

[0049] The data fusion result display area 1010 displays a portion of the data 900 after data fusion. The post-fusion data analysis settings / results display area 1020 displays the analysis results of the data after data fusion and includes the analysis target variable selection area 1021, the analysis target variable selection area 1022, the analysis method selection area 1023, and the analysis result display area 1024. In the analysis target variable selection area 1021, select the variables to be analyzed from the variables 902 that are included only in data A of the post-fusion data 900. In the analysis target variable selection area 1022, select the variables to be analyzed from the variables 903 that are included only in data B of the post-fusion data 900. In the analysis method selection area 1023, select the analysis method for the post-fusion data 900. In the illustrated example, "cross-tabulation" is selected, which performs a cross-tabulation on the selected analysis target variables. Other analysis methods include "correlation coefficient," which calculates the correlation coefficient between the selected analysis target variables. The analysis results display area 1024 displays the results of analyzing the data 900 after data fusion, based on the conditions set in the analysis target variable selection area 1021, the analysis target variable selection area 1022, and the analysis method selection area 1023. In the example shown, the cross-tabulation results of the data 900 after data fusion are displayed when "Variable 1" and "Variable 11" are selected as the analysis targets. Specifically, it displays that there are 2713 records where the value of Variable 1 is 0 and the value of Variable 11 is 0, and 224 records where the value of Variable 1 is 0 and the value of Variable 11 is 1, and so on.

[0050] Next, we will explain the data fusion process using the flowchart in Figure 11. When the process shown in Figure 11 is started, the first step is the data input step 1101. In the data input step S1101, the input unit 102 of the data fusion terminal 101 receives data A200 (Figure 2) and data B300 (Figure 3). The input data A200 is stored in the data A storage unit 121, and the input data B300 is stored in the data B storage unit 122.

[0051] Next, in the covariate acquisition step S1102, the covariate acquisition unit 111 acquires variables from the data A200 (Figure 2) and data B300 (Figure 3) input in the data input step S1101 as covariates 400 (Figure 4) where the variable names match or the variable names or variable contents are similar between data A and data B, and displays the names of the acquired covariates 400 on the output unit 103. As shown in Figure 5, covariates can also be acquired manually. The acquired covariates 400 are stored in the covariate storage unit 123.

[0052] Next, in the initialization step S1103, the covariate addition unit 112 initializes the value of the maximum value storage variable max, which stores the maximum value of the reciprocal of the number of matching records 710, to 0. It also initializes the covariate list list 720, which stores the variable names (covariate names) of the selected covariates, to empty.

[0053] Next, in the covariate addition step S1104, the covariate addition unit 112 creates all possible sets of covariate names 600 (Figure 6) by adding one covariate name that is not included in the list to the covariate names included in the list from the covariates 400 acquired in the covariate acquisition step S1102.

[0054] Next, in the matching record count calculation step 1105, the matching record count calculation unit 113 calculates the matching record count 710 (Figure 7) using the covariates from the covariates 400 acquired in the covariate acquisition step S1102 whose covariate names match each element of the set of covariate names 600 created in the covariate addition step S1103. This is done for all sets created in the covariate addition step S1103. The sets of covariate names created in the covariate addition step S1104 and the corresponding matching record counts 710 are stored in the matching record count storage unit 124. The detailed calculation method for the matching record count will be described later using Figures 12, 13, and 14.

[0055] Next, in the matching record count comparison step S1106, the matching record count comparison unit 114 compares the reciprocals of the matching record count 710 calculated in the matching record count calculation step S1105, and obtains the set of covariate names that maximizes the reciprocal of the matching record count 710, and the maximum value of the reciprocal of the matching record count 710.

[0056] Next, in the maximum value comparison step S1107, the covariate selection unit 115 compares the maximum value of the number of matching records (710) obtained in the matching record count comparison step S1106. If max is smaller, the process proceeds to the covariate selection step S1108; otherwise, it proceeds to the covariate selection result acquisition step S1109.

[0057] In the covariate selection step S1108, the covariate selection unit 115 assigns the maximum value of the reciprocal of the number of matching records (710) obtained in the matching record count comparison step S1106 to max. It also assigns the set of covariate names that maximizes the reciprocal of the number of matching records (710) obtained in the matching record count comparison step S1106 to list. After that, the process proceeds to the covariate addition step S1104.

[0058] In the covariate selection result acquisition step S1109, the covariate selection unit 115 acquires covariates from the covariates 400 acquired in the covariate acquisition step S1102 whose covariate names match those included in the list, and displays the acquired covariate names on the output unit 103. The covariate names included in the list are stored in the covariate selection result storage unit 125.

[0059] Next, in the data fusion execution step S1110, the data fusion execution unit 116 uses the covariates obtained in the covariate selection result acquisition step S1109 to perform data fusion by matching the records with the smallest distance between the covariates of data A and data B, thereby creating the data 900 after data fusion (Figure 9). The data 900 after data fusion is stored in the data fusion result storage unit 126.

[0060] Finally, in the post-fusion data analysis step S1111, the post-fusion data analysis unit 117 performs data analysis processing, such as cross-tabulation and calculation of correlation coefficients, on the variables 902 that are included only in data A and the variables 903 that are included only in data B of the post-fusion data 900 created in the data fusion execution step S1110. The analysis results are displayed in the output unit 103. The analysis results are also stored in the post-fusion data analysis result storage unit 127. The user makes inferences and decisions based on the analysis results. The data fusion process is now complete.

[0061] Next, the process for calculating the number of matching records will be explained using the flowchart in Figure 12. The number of matching records is characterized by finding the number of records in Data B that match the record values ​​in Data A, and then calculating the average of these two values.

[0062] When the process shown in Figure 12 is started, the matching record count calculation unit 113 first executes the data acquisition step S1201. In the data acquisition step S1201, K covariates are acquired from the covariates 400 acquired in the covariate acquisition step S1102, the covariate names of which match each element of the set of covariate names 600 created in the covariate addition step S1103.

[0063] Next, in the initialization step S1202, the matching record count calculation unit 113 initializes the variable `count`, which stores the number of records in Data B that match the record values ​​in Data A, to 0. It also initializes the variable `i`, which identifies the record number of Data A, and the variable `j`, which identifies the record number of Data B, to 1.

[0064] Next, in the record value matching determination step S1203, the matching record count calculation unit 113 determines whether the values ​​of the K covariates in record i of data A match the values ​​of the K covariates in record j of data B. If they match, the process proceeds to the variable count increment step S1204; otherwise, the process proceeds to the variable j increment step S1205.

[0065] In the variable count increment step S1204, the matching record count calculation unit 113 performs a process to increase the value of the variable count by 1.

[0066] In the variable j increment step S1205, the matching record count calculation unit 113 performs a process to increase the value of variable j by 1.

[0067] Next, in the variable j upper limit determination step S1206, the matching record count calculation unit 113 calculates the total number of records N in data B. B Compare the magnitude of the variable j. Total number of records in data B: N B If the variable j is larger, the process proceeds to step S1207, which involves incrementing variable i and initializing variable j; otherwise, the process proceeds to step S1203, which determines whether the record value matches.

[0068] In step S1207, which involves incrementing variable i and initializing variable j, the matching record count calculation unit 113 performs the process of increasing the value of variable i by 1 and initializing variable j to 1.

[0069] Next, in the variable i upper limit determination step S1208, the matching record count calculation unit 113 calculates the total number of records N in data A. A The magnitude of the variable i is compared. Total number of records in data A: N AIf the variable i is larger, the process proceeds to the step S1209 for calculating the number of matching records; otherwise, the process proceeds to the step S1203 for determining whether the record value matches.

[0070] In the matching record count calculation step S1209, the matching record count calculation unit 113 calculates the value of the variable count to the total number of records N in data A. A Dividing by this value calculates the average number of matching records for data A and data B. This value is called the number of matching records. The process of calculating the number of matching records is now complete.

[0071] Next, we will explain examples of calculating the number of matching records using Figures 13 and 14. Figure 13 shows an example of calculating the number of matching records 1300 when the number of covariates K=1, and Figure 14 shows an example of calculating the number of matching records 1400 when the number of covariates K=2. First, we will explain an example of calculating the number of matching records using Example 1300 (Figure 13) for the case where the number of covariates K=1.

[0072] As shown in Figure 13, Example 1300 of calculating the number of matching records when the number of covariates K=1 shows Data A and Data B, which contain one covariate and have a total of 6 records. To calculate the number of matching records, first, calculate the number of records in Data B that match the value of record 10 in Data A. From the figure, it can be confirmed that there are 3 matching records in Data B. Next, calculate the number of records in Data B that match the value of record 20 in Data A. From the figure, it can be confirmed that there is 1 matching record in Data B. Repeat this process up to record 6 of Data A. After calculating the number of matching records in Data B for records 1 to 6 of Data A, calculate their average. That is, calculate the following equation (1).

[0073]

number

[0074] This value represents the number of matching records.

[0075] Next, we will explain an example of calculating the number of matching records using Example 1400 (Figure 14) for the case where the number of covariates K=2. As shown in Figure 14, Example 1400 of calculating the number of matching records when covariate K=2 shows data A and data B, which contain two covariates and have a total of 6 records. To calculate the number of matching records, first, calculate the number of records in data B that match the values ​​of covariate 1 (10) and covariate 2 (1) in record 1 of data A. From Figure 14, it can be confirmed that there are 2 matching records in data B. Next, calculate the number of records in data B that match the values ​​of covariate 1 (20) and covariate 2 (0) in record 2 of data A. From Figure 14, it can be confirmed that there is 1 matching record in record B. Repeat this process up to record 6 of data A. After calculating the number of matching records in data B for records 1 through 6 of data A, calculate their average. That is, calculate equation (1). This value is the number of matching records. Alternatively, the number of matching records may be calculated using representative values ​​such as the median or mode of the number of records in Data B that match the values ​​of the records in Data A.

[0076] Furthermore, when calculating the number of matching records using covariates that include continuous values, a matching threshold ε may be set, and records whose distance from each other is less than ε may be considered matching.

[0077] Furthermore, in this embodiment, a matching method was used as the data fusion method, which links records with the smallest distance between covariates. However, other data fusion methods, such as regression methods that estimate the values ​​of the data to be fused using a regression model, may also be used.

[0078] As described above, the data fusion system in the embodiment can fuse data using covariates optimized by a covariate selection algorithm used for concatenation. Therefore, it enables data fusion with higher accuracy and lower computational cost compared to conventional data fusion methods that use all covariates.

[0079] In this embodiment of the data fusion system, the matching record count calculation unit 113 calculates the "matching record count" using covariates whose covariate names match each element of the set of covariate names created by the covariate addition unit 112, from among the covariates acquired by the covariate acquisition unit 111 (Figures 12, 13, and 14). Therefore, the "matching record count" allows for the measurement of the degree to which people with similar characteristics can be identified for each combination of covariates, and the optimal covariates to be used for data fusion can be selected.

[0080] Furthermore, the matching record count comparison unit 114 compares the matching record counts calculated by the record count calculation unit 113 and obtains the set of covariate names that maximize the reciprocal of the matching record count, as well as the maximum value of the reciprocal of the matching record count. Then, the covariate selection unit 115 compares the maximum value of the reciprocal of the matching record count obtained by the matching record count comparison unit 114 with the previous value to select the covariate that maximizes the reciprocal of the matching record count. This allows for the selection of the optimal combination of covariates to be used for data fusion.

[0081] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all of the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Also, configurations of other embodiments may be added to the configuration of one embodiment. Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0082] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or other storage media such as IC cards, SD cards, and DVDs.

[0083] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0084] 100...Data fusion system 101...Data fusion terminal 102...Input section 103…Output section 104…CPU 105...Memory 106...Storage medium 111... Covariate acquisition unit 112... Covariate addition section 113... Matching Record Count Calculation Unit 114... Matching record count comparison section 115... Covariate Selection Department 116...Data Fusion Implementation Department 117…Post-Fusion Data Analysis Department 120... Database 121...Data A Recording Section 122...Data B storage unit 123... Covariate memory section 124... Matching record count storage unit 125... Covariate selection result storage unit 126...Data fusion result storage unit 127...Data analysis result storage unit after fusion 200...Data A 300...Data B 401... Covariates of Data A 402... Covariates of Data B 500... Covariate setting screen 510... Covariate acquisition method setting field 520... Covariate candidate display field 521... Deselect Covariate Candidates button 522... Add Covariate Candidate Button 530... Covariate display column 800... Covariate selection screen 810... Covariate selection result display field 811... Covariate Deselection Button 812... Add Covariate button 820... Display field for the calculation result of the number of matching records. 830...Selected covariates display column 1000...Data fusion result display / analysis screen 1010...Data fusion result display field 1020...Data analysis settings and results display field after fusion 1021...Selection field for variable to be analyzed 1022...Selection field for variable to be analyzed 1023...Selection field for analysis method

Claims

1. A data fusion system that connects two sets of data, It comprises a computing unit that performs predetermined processing, an input unit into which data is input, and a storage device accessible by the computing unit, The input unit receives input of two data to be merged, namely the first data and the second data. The aforementioned computing device is A covariate acquisition unit that acquires a plurality of covariates that are common to the first data and the second data, A matching record count calculation unit calculates the number of records in which the values ​​of each of the multiple covariates match, based on the first data and the second data. A matching record count comparison unit compares the number of matching records for each combination of the plurality of covariates, A covariate selection unit selects at least one covariate from the plurality of covariates based on the results of the comparison of the number of matching records, A data fusion implementation unit that fuses the first data and the second data using at least one covariate selected by the covariate selection unit, It has a post-integration data analysis unit that analyzes the data after data fusion has been performed, The data fusion system is characterized in that the matching record count calculation unit determines the number of records in the second data that match the values ​​of each covariate in the first data, and calculates the average thereof.

2. A data fusion system for linking two sets of data, It comprises a computing unit that performs predetermined processing, an input unit into which data is input, and a storage device accessible by the computing unit, The input unit receives input of two data to be merged, namely the first data and the second data. The aforementioned computing device is A covariate acquisition unit that acquires a plurality of covariates that are common to the first data and the second data, A matching record count calculation unit calculates the number of records in which the values ​​of each of the multiple covariates match, based on the first data and the second data. A matching record count comparison unit compares the number of matching records for each combination of the plurality of covariates, A covariate selection unit selects at least one covariate from the plurality of covariates based on the results of the comparison of the number of matching records, A data fusion implementation unit that fuses the first data and the second data using at least one covariate selected by the covariate selection unit, It has a post-integration data analysis unit that analyzes the data after data fusion has been performed, The matching record count comparison unit compares the reciprocals of the matching record count calculated by the matching record count calculation unit for each combination of K covariates (a subset of the plurality of covariates), and obtains the combination of covariates that maximizes the reciprocal of the matching record count, as well as the maximum value of the reciprocal of the matching record count.

3. A data fusion system for linking two sets of data, It comprises a computing unit that performs predetermined processing, an input unit into which data is input, and a storage device accessible by the computing unit, The input unit receives input of two data to be merged, namely the first data and the second data. The aforementioned computing device is A covariate acquisition unit that acquires a plurality of covariates that are common to the first data and the second data, A matching record count calculation unit calculates the number of records in which the values ​​of each of the multiple covariates match, based on the first data and the second data. A matching record count comparison unit compares the number of matching records for each combination of the plurality of covariates, A covariate selection unit selects at least one covariate from the plurality of covariates based on the results of the comparison of the number of matching records, A data fusion implementation unit that fuses the first data and the second data using at least one covariate selected by the covariate selection unit, It has a post-integration data analysis unit that analyzes the data after data fusion has been performed, The matching record count comparison unit compares the reciprocals of the matching record count calculated by the matching record count calculation unit for each combination of K covariates (a subset of the plurality of covariates), and obtains the combination of covariates that minimizes the matching record count, as well as the minimum value of the matching record count. This data fusion system is characterized by this comparison unit performing a comparison of the reciprocals of the matching record count calculated by the matching record count calculation unit, and obtaining the combination of covariates that minimizes the matching record count, as well as the minimum value of the matching record count.

4. A data fusion system according to claim 2, The data fusion system is characterized in that the covariate selection unit compares the maximum value of the reciprocal of the number of matching records obtained by the matching record count comparison unit when there are K-1 covariates and when there are K covariates, and if the maximum value of the reciprocal of the number of matching records when there are K covariates is larger, it proceeds to the calculation and comparison of the number of matching records when there are K+1 covariates, and otherwise outputs the combination of covariates obtained by the matching record count comparison unit when there are K covariates.

5. A data fusion system according to claim 3, The data fusion system is characterized in that the covariate selection unit compares the minimum value of the number of matching records obtained by the matching record count comparison unit when there are K-1 covariates and when there are K covariates, and if the minimum value of the number of matching records when there are K covariates is smaller, it proceeds to the calculation and comparison of the number of matching records when there are K+1 covariates, and otherwise outputs the combination of covariates obtained by the matching record count comparison unit when there are K covariates.

6. A data fusion method for linking two sets of data, comprising: an arithmetic unit that performs a predetermined process; an input unit into which data is input; and a storage device accessible by the arithmetic unit, The covariate acquisition unit of the calculation device performs the steps of acquiring a plurality of covariates that are included in common in the first data and the second data, The matching record count calculation unit of the calculation device performs the step of calculating the number of records in which the values ​​of each of the plurality of covariates match from the first data and the second data, The matching record count comparison unit of the calculation device performs the step of comparing the matching record count for each combination of the plurality of covariates, The covariate selection unit of the calculation device performs the step of selecting at least one covariate from the plurality of covariates based on the matching record count comparison result, The data fusion unit of the computing device performs the step of fusing the first data and the second data using at least one covariate selected by the covariate selection unit, The post-fusion data analysis unit of the aforementioned computing device includes a step of analyzing the data after data fusion has been performed, A data fusion method characterized in that the matching record count comparison unit determines the number of records in the second data that match the values ​​of each covariate in the first data, and calculates the average thereof.

7. A data fusion method comprising a computing device that performs predetermined processing, an input unit into which data is input, and a storage device accessible by the computing device, wherein two sets of data are linked together, The covariate acquisition unit of the calculation device performs the steps of acquiring a plurality of covariates that are included in common in the first data and the second data, The matching record count calculation unit of the calculation device performs the step of calculating the number of records in which the values ​​of each of the plurality of covariates match from the first data and the second data, The matching record count comparison unit of the calculation device performs the step of comparing the matching record count for each combination of the plurality of covariates, The covariate selection unit of the calculation device performs the step of selecting at least one covariate from the plurality of covariates based on the matching record count comparison result, The data fusion unit of the computing device performs the step of fusing the first data and the second data using at least one covariate selected by the covariate selection unit, The post-fusion data analysis unit of the aforementioned computing device includes a step of analyzing the data after data fusion has been performed, A data fusion method characterized in that the matching record count comparison unit compares the magnitudes of the reciprocals of the matching record count calculated by the matching record count calculation unit for each combination of K covariates (a subset of the plurality of covariates), and obtains the combination of covariates that maximizes the reciprocal of the matching record count, and the maximum value of the reciprocal of the matching record count.

8. A data fusion method for concatenating two sets of data, comprising: an arithmetic unit that performs a predetermined process; an input unit into which data is input; and a storage device accessible by the arithmetic unit, The covariate acquisition unit of the calculation device performs the steps of acquiring a plurality of covariates that are included in common in the first data and the second data, The matching record count calculation unit of the calculation device performs the step of calculating the number of records in which the values ​​of each of the plurality of covariates match from the first data and the second data, The matching record count comparison unit of the calculation device performs the step of comparing the matching record count for each combination of the plurality of covariates, The covariate selection unit of the calculation device performs the step of selecting at least one covariate from the plurality of covariates based on the matching record count comparison result, The data fusion unit of the computing device performs the step of fusing the first data and the second data using at least one covariate selected by the covariate selection unit, The post-fusion data analysis unit of the aforementioned computing device includes a step of analyzing the data after data fusion has been performed, A data fusion method characterized in that the matching record count comparison unit compares the reciprocals of the matching record count calculated by the matching record count calculation unit for each combination of K covariates (a subset of the plurality of covariates), and obtains the combination of covariates that minimizes the matching record count, and the minimum value of the matching record count.

9. A data fusion method according to claim 7, A data fusion method characterized in that the covariate selection unit compares the maximum value of the reciprocal of the number of matching records obtained by the matching record count comparison unit when there are K-1 covariates and when there are K covariates, and if the maximum value of the reciprocal of the number of matching records when there are K covariates is larger, it proceeds to the calculation and comparison of the number of matching records when there are K+1 covariates, and otherwise outputs the combination of covariates obtained by the matching record count comparison unit when there are K covariates.

10. A data fusion method according to claim 8, A data fusion method characterized in that the covariate selection unit compares the minimum value of the number of matching records obtained by the matching record count comparison unit when there are K-1 covariates and when there are K covariates, and if the minimum value of the number of matching records when there are K covariates is smaller, it proceeds to the calculation and comparison of the number of matching records when there are K+1 covariates, and otherwise outputs the combination of covariates obtained by the matching record count comparison unit when there are K covariates.

Images