Information processing device and method

Abstract

An information processing device according to one embodiment has an extraction unit that extracts candidate information that is a single item of data that is stored at a first database and a plurality of candidates for data that is stored at a second database that represents the same matters as the single item of data that is stored at the first database and a determination unit that generates a query that asks which of the plurality of candidates is data that represents the same matters as the single item of data that is stored at the first database on the basis of the information extracted by the extraction unit, inputs the query into a model, and determines which of the plurality of candidates is data that represents the same matters as the single item of data that is stored at the first database on the basis of a response to the query that is outputted from the model.

Description

Information processing device and method

[0001] FIELD Embodiments of the present invention relate to an information processing apparatus and method.

[0002] Entity matching, which uses a large-scale language model as a pre-trained base model, is used as a so-called name matching method, which is robust against data contamination such as differences in format or spelling, and is used to identify data that refer to the same entity across different data sources.

[0003] For example, Non-Patent Documents 1 and 2 disclose a technology for determining entity matching by asking a base model whether or not 1:1 pair candidate information consisting of a combination of one piece of data from one data source and one piece of data from another data source points to the same entity.

[0004] Narayan, Avanika, Ines Chami, Laurel Orr, and Christopher Re (2022). “Can Foundation Models Wrangle Your Data?” In: International Conference on Very Large Data Bases 16.4, pp. 738-746. Peeters, R., Bizer, C. (2023). Using ChatGPT for Entity Matching. In: Abello, A., et al. New Trends in Database and Information Systems. ADBIS 2023. Communications in Computer and Information Science, vol 1850. Springer, Cham.

[0005] However, in existing methods, entity matching determination is performed on 1:1 pair candidate information, so when one piece of data in one of the above-mentioned data sources has multiple pieces of data that can be pair candidates in the other of the above-mentioned data sources, the number of determinations made by the basic model increases according to the number of pieces of data that can be pair candidates, and the processing volume and processing time related to the determination increase according to this number.

[0006] Furthermore, existing methods cannot determine entity matching taking other pair candidates into consideration, which can result in reduced entity matching accuracy. Furthermore, existing methods perform entity matching determination for 1:1 pair candidate information, so it is necessary to extract pair candidate information to be input from multiple data sources in advance.

[0007] The present invention has been made in light of the above circumstances, and its purpose is to provide an information processing device and method that can appropriately identify information indicating the same item.

[0008] An information processing device according to one aspect of the present invention includes an extraction unit that extracts candidate information consisting of one piece of data stored in a first database and multiple candidates of data stored in a second database that represent the same item as the item represented in the one piece of data stored in the first database; and a determination unit that generates a question inquiring about which of the multiple candidates represents the same item as the item represented in the one piece of data stored in the first database based on the information extracted by the extraction unit, inputs the question into a model, and determines which of the multiple candidates represents the same item as the item represented in the one piece of data stored in the first database based on the answer to the question output from the model.

[0009] An information processing method according to one aspect of the present invention is a method performed by an information processing device, and includes: a unit of the information processing device extracting candidate information consisting of one piece of data stored in a first database and multiple candidates of data stored in a second database and representing the same item as the item represented in the one piece of data stored in the first database; a determination unit of the information processing device generating a question inquiring about which of the multiple candidates represents the same item as the item represented in the one piece of data stored in the first database based on the information extracted by the extraction unit, and inputting the question into a model; and determining which of the multiple candidates represents the same item as the item represented in the one piece of data stored in the first database based on the answer to the question output from the model.

[0010] According to the present invention, information indicating the same item can be appropriately identified.

[0011] FIG. 1 is a diagram illustrating an application example of an information processing device according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating an example of processing operations of an information processing device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of processing by a candidate extraction unit. FIG. 4 is a diagram illustrating an example of a table for managing data stored in data source B. FIG. 5 is a diagram illustrating an example of processing by a matching determination unit. FIG. 6 is a diagram illustrating an example of processing by the candidate extraction unit when candidate data stored in multiple data sources is used. FIG. 7 is a diagram illustrating an example of entity matching determination by a matching determination unit when 1:N pair candidate information is extracted by the candidate extraction unit. FIG. 8 is a diagram illustrating an example of pair candidate information extraction processing by the candidate extraction unit and entity matching determination processing by a matching determination unit when candidate data stored in multiple data sources is used. FIG. 9 is a block diagram illustrating an example of the hardware configuration of an information processing device according to an embodiment of the present invention.

[0012] An embodiment of the present invention will be described below. Fig. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. As shown in Fig. 1, an information processing device 100 according to an embodiment of the present invention includes a candidate extraction unit 10 and a matching determination unit 20.

[0013] The candidate extraction unit 10 inputs data stored in multiple data sources, which are multiple databases (here, data source A and data source B), extracts one piece of query data derived from data source A, and extracts multiple pieces of candidate data (here, N pieces of candidate data) derived from data source B that are pair candidates that may match the query data, based on, for example, vector similarity between the query data and the query data. That is, the candidate extraction unit 10 extracts one piece of data derived from data source A and multiple candidates of data derived from data source B that represent the same item as the item represented in the above-mentioned one piece of data derived from data source A. Based on the result of this extraction, the candidate extraction unit 10 outputs 1:N pair candidate information consisting of one piece of query data derived from data source A and N pieces of candidate data derived from data source B.

[0014] The matching determination unit 20 inputs the pair candidate information into a pre-trained base model, for example, a large-scale language model, and determines a match between data derived from data source A and data derived from data source B based on the context. That is, based on the information extracted by the candidate extraction unit 10, the matching determination unit 20 generates a question inquiring about data representing the same item as the item represented in the one piece of data derived from data source A from among the above-mentioned multiple candidates, inputs the question to the base model, and determines data representing the same item as the item represented in the one piece of data stored in data source A from among the above-mentioned multiple candidates based on the answer to the question output from the base model. In this embodiment, appropriate entity matching is achieved by having the matching determination unit 20 query the base model about 1:N pair candidates to determine a match, and by having the candidate extraction unit 10 extract 1:N pair candidates from multiple data sources based on vector similarity.

[0015] In addition, in this embodiment, the candidate extraction unit 10 creates the above-mentioned 1:N pair candidate information between a plurality of data sources, so that matching results between a plurality of data sources can be output all at once.

[0016] In this embodiment, by querying the base model based on information on 1:N pair candidates, the number of entity matching determinations can be reduced when there are multiple data that can be pair candidates for one piece of data, and entity matching determination can be performed taking other pair candidates into consideration.

[0017] For example, when the data derived from data source A shown in Figure 1 includes query data a and the data derived from data source B includes candidate data b, c, d, and e, and the pair candidate information is configured as the above-mentioned 1:N pair candidate information as in this embodiment, the pair candidate information consisting of the query data a and the candidate data b, c, d, and e is input to the base model of the matching determination unit 20.

[0018] On the other hand, when the data derived from data source A shown in FIG. 1 includes query data a and the data derived from data source B includes candidate data b, c, d, and e, and the pair candidate information is configured as the above-mentioned 1:1 pair candidate information as in the existing method, first, the first pair candidate information consisting of the query data a and the candidate data b is input to the base model. Second, the second pair candidate information consisting of the query data a and the candidate data c is input to the base model. Next, third, the third pair candidate information consisting of the query data a and the candidate data d is input to the base model. Fourth, the fourth pair candidate information consisting of the query data a and the candidate data e is input to the base model.

[0019] In this way, when the pair candidate information is configured as 1:1 pair candidate information, query data overlaps between each input to the base model. That is, when the pair candidate information is configured as 1:N pair candidate information as in the present embodiment, the number of entity matching determinations can be reduced compared to when the pair candidate information is configured as 1:1 pair candidate information, and therefore the overall amount of query data input to the base model can be reduced.

[0020] Furthermore, when the data derived from data source A shown in Figure 1 includes query data a and the data derived from data source B includes candidate data b, c, d, and e, and the pair candidate information is configured as the above-mentioned 1:N pair candidate information as in this embodiment, in the entity matching determination by the matching determination unit 20, candidate data that matches the query data a in the pair candidate information can be determined from among the candidate data b, c, d, and e, taking into account the relationship between the candidate data b, c, d, and e.

[0021] On the other hand, when the data derived from data source A shown in FIG. 1 includes query data a and the data derived from data source B includes candidate data b, c, d, and e, and the pair candidate information is configured as the above-described 1:1 pair candidate information as in the existing method, first, it is determined whether or not the query data a and the candidate data b in the first pair candidate information consisting of the query data a and the candidate data b match. Second, it is determined whether or not the query data a and the candidate data c in the second pair candidate information consisting of the query data a and the candidate data c match. Next, third, it is determined whether or not the query data a and the candidate data d in the third pair candidate information consisting of the query data a and the candidate data d match. Fourth, it is determined whether or not the query data a and the candidate data e in the fourth pair candidate information consisting of the query data a and the candidate data e match.

[0022] In this way, when the pair candidate information is configured as 1:1 pair candidate information, the relationship between the plurality of candidate data is not taken into consideration in determining entity matching. In other words, when the pair candidate information is configured as 1:N pair candidate information as in this embodiment, improved accuracy in entity matching determination can be expected compared to when the pair candidate information is configured as 1:1 pair candidate information.

[0023] Furthermore, in this embodiment, the candidate extraction unit 10 automatically extracts 1:N pair candidate information, so that entity matching results between multiple data sources can be output without extracting pair candidates in advance.

[0024] Next, extraction of 1:N pair candidate information by the candidate extraction unit 10 will be described. FIG. 2 is a flowchart showing an example of the processing operation of an information processing device according to an embodiment of the present invention. FIG. 3 is a diagram explaining an example of processing by the candidate extraction unit. As shown in FIG. 3, the candidate extraction unit 10 has a data conversion unit 11, a vector DB (database) 12, and a candidate search unit 13.

[0025] The data conversion unit 11 inputs data stored in data source A and managed in a table, and data stored in data source B and managed in a table (S11). The data conversion unit 11 extracts all data in the table that manages the data stored in data source B as candidate data, converts this candidate data into text, and converts each text into a d-dimensional vector using a pre-trained text embedding model or the like, and stores the converted text in the vector DB 12 (S12).

[0026] 4 is a diagram showing an example of a table for managing data stored in data source B. FIG. 4 shows an example in which data source B stores multiple pieces of data, each of which has multiple types of attributes and is assigned an ID, which is identification information. When these pieces of data are converted into text in S12, the result of this conversion is expressed, for example, as follows: (Text of data with ID=1) "Title: Apple. Price: $2" (Text of data with ID=2) "Title: Orange. Price: $3" (Text of data with ID=3) "Title: Banana. Price: $4" The information vectorized in S12 and stored in vector DB 12 can also be used for entity matching determination with data stored in other data sources, for example, data source A'.

[0027] The data conversion unit 11 selects one piece of data derived from data source A, for example, one row of data in a table, as query data, converts this query data into text, and converts this text into a d-dimensional vector using the above-mentioned text embedding model or the like, and outputs it to the candidate search unit 13 (S13).

[0028] The candidate search unit 13 searches the vector DB 12 using the vector converted from the query data in S13, and extracts, as search results for candidate data, the top N candidate data corresponding to vectors having a relatively high degree of vector similarity with the vector converted from the query data derived from data source A in S13, among the vectors converted from the plurality of candidate data derived from data source B in S12 (S14). This extraction is repeated for all data derived from data source A (symbol a in FIG. 3 ).

[0029] N is a parameter. Note that a threshold value may be set for the vector similarity, and data derived from data source B that does not meet this threshold value may be excluded from the candidate data extracted by the candidate search unit 13. By the processes of S11 to S13, 1:N pair candidate information is generated for each query data derived from data source A.

[0030] 5 is a diagram illustrating an example of processing by the matching determination unit 20. As shown in FIG. 5, the matching determination unit 20 includes an instruction generation unit 21, an answer generation unit 22, and an answer determination unit 23.

[0031] The instruction generation unit 21 generates an input sentence based on pair candidate information consisting of a combination of one piece of query data and multiple pieces of candidate data. The answer generation unit 22 generates an output sentence for the input sentence using a base model. The answer determination unit 23 determines entity matching by determining data that point to the same entity from the 1:N pair candidate information based on the output sentence. In the example shown in FIG. 5 , the entity matching determination result is indicated by "◯" or "X" for each combination of query data and candidate data in the pair candidate information.

[0032] A judgment result of "o" indicates that the query data and the candidate data to be judged are the same entity, and a judgment result of "x" indicates that the query data and the candidate data to be judged are not the same entity. The answer judgment unit 23 then outputs the final judgment result of entity matching by listing combinations of query data and candidate data for which the judgment result is "o".

[0033] Next, a specific example will be described of entity matching determination related to 1:N pair candidate information by the matching determination unit 20. The instruction generation unit 21 of the matching determination unit 20 uses the 1:N pair candidate information output from the candidate extraction unit 10 to generate an instruction sentence, which is a question sentence in the base model, and inputs this instruction sentence to the base model in the answer generation unit 22.

[0034] An instruction sentence is expressed as one of the following first to fifth examples: (First example of instruction sentence) "Product A is (query data). Product B is (candidate data 1). Product C is (candidate data 2) ... Which of product B, product C, ... is the same as product A?"

[0035] (Second example of an instruction) "Product A is (query data). Product B is (candidate data 1). Product C is (candidate data 2)... Which of product B, product C, etc. is the same as product A?"

[0036] (Third example of an instruction) "Product A is (query data). Product B is (candidate data 1). Product C is (candidate data 2)... Select at most one product identical to Product A from Product B, Product C, etc."

[0037] (Fourth example of an instruction) "Select at most one product identical to product A from the following products B, C, .... Product A is (query data). Product B is (candidate data 1). Product C is (candidate data 2) ..."

[0038] (Fifth example of an instruction) "Select at most one product identical to Product A from the following: Product B, Product C, ... Product A (query data) Product B (candidate data 1) Product C (candidate data 2) ..."

[0039] The answer generation unit 22 generates an output sentence for the input sentence, that is, an answer sentence for the question sentence, using the base model.

[0040] The answer sentence is expressed as follows, for example: (Answer sentence) "Product C" The answer determination unit 23 outputs a determination result based on the answer sentence from the base model.

[0041] When a corresponding identifier, for example, "product C," is included in the response sentence from the response generation unit 22, the response determination unit 23 identifies candidate data that includes the corresponding identifier from among the candidate data included in the pair candidate information, and determines that this candidate data and the query data are the same entity. Furthermore, when the candidate data included in the pair candidate information does not include the corresponding identifier, the response determination unit 23 determines that the candidate data and the query data are not the same entity.

[0042] 6 is a diagram illustrating an example of entity matching determination by the matching determination unit when 1:N pair candidate information is extracted by the candidate extraction unit. In the example shown in Fig. 3 above, the number of data sources from which candidate data is extracted by the candidate extraction unit 10 is one, namely, data source B, but in the example shown in Fig. 6, the number of data sources from which candidate data is extracted by the candidate extraction unit 10 is multiple, including data source B and data source C.

[0043] In this way, when multiple data sources are provided as sources from which candidate data is extracted, for each data source from which candidate data is extracted, the data is converted into a vector by the data conversion unit 11, stored in the vector DB 12, and searched for by the candidate search unit 13, and each pair candidate information is output.

[0044] In the example shown in Figure 6, first, based on the query data derived from data source A and the candidate data derived from the first data source B as the source from which the candidate data is extracted, 1:N pair candidate information, i.e., pair candidate information consisting of one query data derived from data source A and multiple candidate data derived from data source B, is extracted for each query data derived from data source A.

[0045] Next, based on the query data derived from data source A and the candidate data derived from a second data source C as the source from which the candidate data is extracted, 1:N pair candidate information, i.e., pair candidate information consisting of one query data derived from data source A and multiple candidate data derived from data source C, is extracted for each query data derived from data source A, and similar processing is performed for other data sources as the source from which the candidate data is extracted.

[0046] 7 is a diagram illustrating an example of entity matching determination by the matching determination unit when 1:N pair candidate information is extracted by the candidate extraction unit. In the example shown in Fig. 7, the candidate extraction unit 10 extracts 1:N pair candidate information consisting of one query data derived from data source A and N candidate data derived from data source B, and the matching determination unit 20 lists data combinations for which the entity matching determination result between the data stored in data source A and the data stored in data source B is "OK" based on this pair candidate information.

[0047] 8 is a diagram illustrating an example of a process of extracting pair candidate information by the candidate extraction unit and a process of determining entity matching by the matching determination unit when candidate data stored in multiple data sources are used. In the example shown in Fig. 8, the candidate extraction unit 10 extracts first pair candidate information that is 1:N pair candidate information consisting of one query data derived from data source A and N pieces of candidate data derived from data source B, and second pair candidate information that is 1:N pair candidate information consisting of one query data derived from data source A and N pieces of candidate data derived from data source C, and similar processes are performed for other data sources from which candidate data is extracted.

[0048] Next, the matching determination unit 20 lists data combinations for which the entity matching determination result between the data stored in data source A and the data stored in data source B is "OK" based on the first pair candidate information, and lists data combinations for which the entity matching determination result between the data stored in data source A and the data stored in data source C is "OK" based on the second pair candidate information. Similar processing is performed for other data sources from which candidate data is extracted.

[0049] 9 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. In the example shown in FIG. 9, the information processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0050] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0051] An input device 200 and an output device 300 attached to the information processing device 100 and used by a user or the like are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 200 such as a keyboard, touch panel, touchpad, or mouse, and can output output data to an output device 300 including a display device using a liquid crystal or organic electroluminescence (EL) display, etc., for display. The input device 200 and the output device 300 may be devices built into the information processing device 100, or may be input devices and output devices of other information terminals that can communicate with the information processing device 100 via a network.

[0052] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0053] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processing steps.

[0054] An information processing apparatus 100 according to one embodiment of the present invention can be configured as a data processing apparatus having the units shown in FIG. 1 as software-based processing function units.

[0055] The information storage unit used as a work memory or the like by each unit of the information processing device 100 can be configured by using the data memory 112 shown in Fig. 9. However, these configured storage areas are not essential components within the information processing device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0056] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0057] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0058] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0059] REFERENCE SIGNS LIST 100: information processing device 10: candidate extraction unit 11: data conversion unit 12: vector DB 13: candidate search unit 20: matching determination unit 21: instruction generation unit 22: answer generation unit 23: answer determination unit

Claims

an extraction unit that extracts one piece of data stored in a first database and candidate information that is stored in a second database and includes a plurality of candidates for data that represent the same item as the item represented in the one piece of data stored in the first database; a determination unit that generates a question inquiring about data from the plurality of candidates that represents the same item as the item represented in the one piece of data stored in the first database based on the information extracted by the extraction unit, inputs the question to a model, and determines data from the plurality of candidates that represents the same item as the item represented in the one piece of data stored in the first database based on an answer to the question output from the model; and An information processing device comprising: 　 The extraction unit converting the one piece of data stored in the first database and the plurality of pieces of data stored in the second database into vectors, and extracting a plurality of candidates for data stored in the second database that represent the same item as the item represented in the one piece of data stored in the first database, based on the similarity between the vector based on the one piece of data stored in the first database and the vector based on the plurality of pieces of data stored in the second database; The information processing device according to claim 1 . 　 The determination unit as the question, based on the information extracted by the extraction unit, a question inquiring about data from the plurality of candidates that represents the same item as that represented in the one piece of data stored in the first database, or a question inquiring about selection of data from the plurality of candidates that represents the same item as that represented in the one piece of data stored in the first database, based on the information extracted by the extraction unit, is generated and input to the model; The information processing device according to claim 1 . 　 A method performed by an information processing device, extracting, by an extraction unit of the information processing device, one piece of data stored in a first database and candidate information consisting of a plurality of candidates of data stored in a second database and expressing the same item as the item expressed in the one piece of data stored in the first database; a determination unit of the information processing device generates a question inquiring about data among the plurality of candidates that represents the same item as the item represented in the one piece of data stored in the first database based on the information extracted by the extraction unit, and inputs the question to a model; and determines, based on an answer to the question output from the model, data among the plurality of candidates that represents the same item as the item represented in the one piece of data stored in the first database; An information processing method comprising:

Images