Information processing device, information processing method, and program

Abstract

To provide an information processing technique to allow two tables to be coupled to a large-scale data without requiring high calculation cost.SOLUTION: An information processing apparatus acquires target data, and converts the target data into an embedding vector indicating the amount of potential feature of the target data. The information processing apparatus searches for pieces of data similar to the amount of potential feature of the target data as pieces of candidate data, applies predetermined processing on the pieces of candidate data, and gives a ranking coupled to the target data for every candidate data. The information processing apparatus outputs a result of the coupling of the candidate data to the target data based on the given ranking.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to information processing related to data joining. [Background Art]

[0002] A technique for integrating various databases having different attributes (heterogeneous databases) is known. Non-Patent Document 1 discloses a technique for determining whether or not to join two tables included in different databases by using a supervised machine learning technique. [Prior Art Documents] [Non-Patent Documents]

[0003] [Non-Patent Document 1] Javier Flores, et.al, 'Scalable Data Discovery Using Profiles', [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, with the technique described in Non-Patent Document 1, since joining determination is performed in units of columns included in a table, there has been a problem that the calculation cost is large when performing joining determination on large-scale data.

[0005] The present disclosure has been made in view of the above problem, and an example of the object thereof is to provide an information processing technique capable of joining two tables without requiring high calculation cost even for large-scale data. [Means for Solving the Problem]

[0006] One aspect of the present disclosure is an information processing apparatus, A data acquisition method for acquiring target data, A data transformation means that transforms the target data into an embedding vector by inputting the feature vector extracted from the target data into an embedding model, which is a model representing the distribution of latent features in a vector space. A candidate search means that uses a search index that associates the target data with the embedding vector of the target data, and searches for the target data whose embedding vector is similar to the embedding vector of the target data as candidate data. A candidate ranking means that assigns a ranking to each of the candidate data to be combined with the target data by inputting the feature vector extracted from the target data and the feature vector extracted from the candidate data into a ranking model, which is a model for evaluating the ease of combining the target data and the candidate data. A result output means that outputs the result of combining the target data and the candidate data based on the assigned rank, comprising.

[0007] Another aspect of the present disclosure is The computer executes an information processing method, comprising Obtain the target data, The feature vectors extracted from the aforementioned target data are input into an embedding model, which is a model representing the distribution of latent features in a vector space, thereby converting the target data into embedding vectors. Using a search index that associates the target data with the embedding vector of the target data, the system searches for the target data whose embedding vector is similar to that of the target data, and identifies the target data as candidate data. By inputting the feature vector extracted from the target data and the feature vector extracted from the candidate data into a ranking model, which is a model for evaluating the ease of combining the target data and the candidate data, a ranking is assigned to each of the candidate data to determine its suitability for combining with the target data. Based on the assigned ranking, the result of combining the target data and the candidate data is output.

[0008] Still another aspect of the present disclosure is program , which is Obtain the target data, The feature vectors extracted from the aforementioned target data are input into an embedding model, which is a model representing the distribution of latent features in a vector space, thereby converting the target data into embedding vectors. Using a search index that associates the target data with the embedding vector of the target data, the system searches for the target data whose embedding vector is similar to that of the target data, and identifies the target data as candidate data. By inputting the feature vector extracted from the target data and the feature vector extracted from the candidate data into a ranking model, which is a model for evaluating the ease of combining the target data and the candidate data, a ranking is assigned to each of the candidate data to determine its suitability for combining with the target data. The computer is instructed to perform a process that outputs the result of combining the target data and the candidate data based on the assigned ranking. [Advantageous Effects of Invention]

[0012] According to the present disclosure, an effect is achieved in that two tables can be joined without requiring high computational cost even for large-scale data. [Brief Description of Drawings]

[0013] [Figure 1] It is a block diagram showing the configuration of the information processing apparatus according to the first embodiment. [Figure 2] It is a diagram showing an outline of processing for converting a query column into an embedded vector. [Figure 3] It is a diagram showing an outline of processing for searching for neighboring vectors using a search index. [Figure 4] It is a diagram showing an outline of processing performed to obtain a ranking result. [Figure 5] It is a flow chart showing the flow of information processing according to the first embodiment. [Figure 6]It is a block diagram showing the configuration of a computer that functions as the information processing apparatus according to the first embodiment. [Figure 7] It is a block diagram showing the configuration of an example to which the information processing apparatus is applied. [Figure 8] It is a block diagram showing the configuration of another information processing apparatus used for constructing the information processing apparatus according to the first embodiment. [Figure 9] It is a diagram outlining the learning of an embedding model. [Figure 10] It is a diagram outlining the process of converting index target columns into embedding vectors. [Figure 11] It is a diagram outlining the process of constructing a search index. [Figure 12] It is a diagram outlining the learning of a ranking model. [Figure 13] It is a flow diagram showing the processing flow in another information processing apparatus used for constructing the information processing apparatus according to the first embodiment. [Figure 14] It is a block diagram showing the configuration of an information processing apparatus according to the second embodiment. [Figure 15] It is a flowchart for explaining processing performed in the information processing apparatus according to the second embodiment. [Figure 16] It is a block diagram showing the configuration of an information processing apparatus according to the third embodiment. [Figure 17] It is a flowchart for explaining processing performed in the information processing apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, the first embodiment of the present disclosure will be described in detail with reference to the drawings.

[0015] [First Embodiment] <Configuration of Information Processing Apparatus> First, the configuration of the information processing device 1 according to this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 is, for example, a data retrieval device that integrates data, or a retrieval device that retrieves data. The information processing device 1 includes a data acquisition unit 11, a data conversion unit 12, a candidate search unit 13, a candidate ranking unit 14, and a result output unit 15.

[0016] In this embodiment, the data acquisition unit 11 is configured to implement data acquisition means, and the data conversion unit 12 is configured to implement data conversion means. Furthermore, the candidate search unit 13 is configured to implement candidate search means, the candidate ranking unit 14 is configured to implement candidate ranking means, and the result output unit 15 is configured to implement result output means.

[0017] The data acquisition unit 11 acquires target data. Here, the target data is data to which a predetermined process is applied, and as an example, is a database containing one or more records. However, the target data is not limited to the example described above and may be other data. The target data includes one or more attributes. The attributes included in the target data indicate the characteristics of the target data or the characteristics of the data included in the target data, and as an example, are fields included in the database that is the target data. However, the attributes included in the target data are not limited to the example described above and may be other attributes. The data acquisition unit 11 outputs the acquired target data to the data conversion unit 12.

[0018] The data transformation unit 12 applies predetermined processing to the target data to convert it into an embedding vector. Here, predetermined processing is processing applied to the target data. For example, it is a process that converts records contained in the database, which is the target data, into embedding vectors that represent the latent features of the target data using the embedding model contained in the embedding model storage unit 20. An embedding model is a model that represents arbitrary data in a vector space, and is a model in which the similarity of data is represented as distance in space. For example, the data transformation unit 12 obtains features from the target data. Then, the data transformation unit 12 calculates the vector value of the target data using the obtained features and the embedding model, and outputs it to the candidate search unit 13.

[0019] The data transformation unit 12 may use features calculated by a language model as strings representing the values ​​of the columns in the target data, as features of the target data. Alternatively, the data transformation unit 12 may use statistical quantities such as the number of words or characters as features of the target data. The method for training the embedding model and the method for acquiring features are not limited to any particular method, and general machine learning techniques may be used. For example, the data transformation unit 12 may use a model trained by a learning algorithm using a multilayer neural network as the embedding model.

[0020] The candidate search unit 13 searches for candidate data that is similar to the embedding vector of the target data and that can be combined with the target data. As an example, the candidate search unit 13 acquires data as candidate data whose distance (similarity) in vector space to the embedding vector of the target data satisfies predetermined conditions. More specifically, the candidate search unit 13 uses a search index contained in the index storage unit 30, which associates candidate data with the embedding vector of that data, to perform a nearest neighbor search and acquire data with a small distance to the embedding vector of the target data as candidate data. The candidate search unit 13 then outputs the acquired candidate data to the candidate ranking unit 14.

[0021] The number of candidate data points acquired by the candidate search unit 13 is not particularly limited. For example, the candidate search unit 13 may use a predetermined value (K) as the number of candidate data points.

[0022] The candidate ranking unit 14 applies predetermined processing to the candidate data to rank the candidate data to be combined with the target data. Here, the predetermined processing is the processing applied to the candidate data. As an example, the candidate ranking unit 14 uses the target data, the candidate data, and the ranking model contained in the ranking model storage unit 40 to assign a priority order to each candidate data to be combined with the target data. The candidate ranking unit 14 then outputs the ranking result indicating the priority order to the result output unit 15.

[0023] The ranking model is a model for evaluating the ease of combining (connectivity) between input target data and candidate data. The ranking model is a pre-prepared model and is stored in the ranking model memory unit 40. The algorithm for generating the ranking model is not particularly limited. The ranking model may be a model that evaluates connectivity using predetermined rules. Alternatively, the ranking model may be a model learned by a learning algorithm that utilizes a multilayer neural network.

[0024] The result output unit 15 combines the target data and candidate data based on the ranking results of the candidate data and outputs it as combined data.

[0025] The number of candidate data that the result output unit 15 combines is not particularly limited. For example, the result output unit 15 may output a result in which the top M candidates and the target data are combined according to a predetermined value (M). Specifically, for example, the ranking result may include 10 × M (= K) candidate data, and combined data showing the result of combining the top M candidate data from those 10 × M candidates with the target data may be output.

[0026] <Specific examples related to the configuration of information processing equipment> The data transformation unit 12 extracts feature vectors from the query column using the same method as when training the embedding model, inputs the extracted feature vectors into the embedding model, and outputs the vectors calculated in the intermediate layers of the embedding model as embedding vectors. For example, as shown in Figure 2, when the target data includes a query column containing strings such as "Tokyo," "Yokohama," and "Tsukuba," the data transformation unit 12 extracts feature vectors that represent these strings numerically, inputs these feature vectors into the embedding model, and transforms the query column into an embedding vector. Figure 2 is a diagram illustrating the overview of the process of transforming a query column into an embedding vector.

[0027] The candidate search unit 13 uses a search index to search for neighboring vectors corresponding to the set of embedding vectors in the vicinity of the embedding vector output from the data conversion unit 12. It then obtains a set of index target columns associated with the searched neighboring vectors as candidate data and outputs the obtained candidate data to the candidate ranking unit 14. For example, the candidate search unit 13 obtains the search results for neighboring vectors as shown in Figure 3 by performing a search using the search index. Figure 3 is a diagram illustrating the overview of the process of searching for neighboring vectors using the search index. Through this process, the candidate search unit 13 can obtain K index target columns corresponding to each of the K neighboring vectors as candidate data.

[0028] The candidate ranking unit 14 uses a ranking model to assign relatively higher ranks to index target columns that are similar to the query column, and relatively lower ranks to index target columns that are not similar to the query column, thereby obtaining a ranking result. For example, as shown in Figure 4, the candidate ranking unit 14 inputs feature vectors extracted from the query column containing strings such as "Tokyo," "Yokohama," and "Tsukuba," and feature vectors extracted from each index target column included in the candidate data into the ranking model, thereby obtaining a ranking result for the top M index target columns among those index target columns. In the example in Figure 4, by inputting the feature vectors extracted from the query column and the candidate data into the ranking model, a ranking result including a join score, which is a value indicating the inference result regarding the degree of joinability (similarity) to the query column, is obtained. Also, according to the example in Figure 4, the index target column containing strings such as "Tokyo," "Tsukuba," "Nagoya," and "Kawasaki" is assigned the rank of 1st. Furthermore, as shown in the example in Figure 4, a value of 0.8 is obtained as the join score for indexed columns containing strings such as "Tokyo," "Tsukuba," "Nagoya," and "Kawasaki." Also, as shown in the example in Figure 4, a ranking result can be obtained in which indexed columns with relatively high join scores are assigned relatively higher ranks, and indexed columns with relatively low join scores are assigned relatively lower ranks. Note that the candidate ranking unit 14 may obtain a ranking result sorted in descending order of join scores, or it may obtain a ranking result corresponding to a score corrected by a predetermined method. Figure 4 is a diagram showing an overview of the process performed to obtain the ranking result.

[0029] <Information Processing> Next, the information processing flow according to this embodiment will be explained with reference to Figure 5. Figure 5 is a flowchart showing the information processing flow.

[0030] First, in step S11, the data acquisition unit 11 acquires the target data. In step S12, the data conversion unit 12 applies a predetermined process to the target data to convert it into an embedding vector and outputs it to the candidate search unit 13. In step S13, the candidate search unit 13 searches the index storage unit 30 for candidate data that is similar to the embedding vector of the target data and that can be combined with the target data, and outputs it to the candidate ranking unit 14. In step S14, the candidate ranking unit 14 applies a predetermined process to the multiple candidate data input from the candidate search unit 13 to rank the multiple candidate data that can be combined with the target data, and outputs the ranking result to the result output unit 15. In step S15, the result output unit 15 combines the target data and the candidate data based on the ranking result of the candidate data and outputs it as combined data.

[0031] <Examples of implementation using software> Some or all of the functions of the information processing device 1 may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0032] When implemented by software, the information processing device 1 is implemented, for example, by a computer that executes instructions for a program, which is software that implements each function. Figure 6 shows an example of such a computer. The computer 50 includes at least one processor 51 and at least one memory 52. ​​The memory 52 stores a program 53 that causes the computer 50 to operate as the information processing device 1. In the computer 50, each function of the information processing device 1 is realized when the processor 51 reads the program 53 from the memory 52 and executes it.

[0033] For the processor 51, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), microcontroller, or a combination thereof can be used. For the memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0034] Furthermore, the computer 50 may also be equipped with RAM (Random Access Memory) for deploying program 53 at runtime and for temporarily storing various data. The computer 50 may also be equipped with a communication interface for sending and receiving data with other devices. Furthermore, the computer 50 may also be equipped with an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0035] Furthermore, the program 53 can be recorded on a non-temporary, tangible recording medium 54 that is readable by the computer 50. Such a recording medium 54 could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. The computer 50 can acquire the program 53 via such a recording medium 54. The program 53 can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. The computer 50 can also acquire the program 53 via such a transmission medium.

[0036] <Effects of this embodiment> As described above, in this embodiment, the information processing device 1 acquires target data, converts it into an embedding vector obtained by applying a predetermined process to the target data, and searches for candidate data that is similar to the embedding vector of the target data and that can be joined with the target data. Furthermore, in this embodiment, the information processing device 1 ranks the searched candidate data using a ranking model and joins the target data and the candidate data based on the rank. As described above, this embodiment has the effect of being able to join two tables without requiring high computational costs even for large amounts of data by using the similarity of the embedding vector and a ranking model.

[0037] <Examples> Figure 7 shows the configuration of an embodiment in which the information processing device according to the above embodiment is applied. In this embodiment, the information processing device searches a large external table for tables that can be joined with the target table (hereinafter also referred to as the "target table") and joins them with the target table.

[0038] In Figure 7, the information processing device 1 has the same configuration as the information processing device 1 shown in Figure 1 and is connected to the embedded model storage unit 20, the index storage unit 30, and the ranking model storage unit 40. The target table is input to the information processing device 1 as target data. In this embodiment, the information processing device 1 extracts query columns from the input target table and converts the query columns into embedded vectors using a pre-prepared embedded model.

[0039] On the other hand, a search index is generated in advance from the external tables to be searched and stored in the index storage unit 30. Specifically, in this embodiment, first, columns to be indexed are selected from the external tables. These columns correspond to the candidate data mentioned above. Next, the selected columns to be indexed are converted into embedding vectors. This conversion process is performed using the same embedding model as the one used to convert the query columns extracted from the target table. Then, a search index is generated that associates the columns to be indexed with their embedding vectors and stored in the index storage unit 30. In this way, the index storage unit 30 stores search indexes for the tables that can be joined with the target table.

[0040] The information processing device 1 uses the search index stored in the index storage unit 30 to obtain multiple index target columns as candidate data, which have a small distance from the embedding vector of the query column. Next, the information processing device 1 ranks the candidate data using a ranking model and generates a ranking result indicating the priority for joining the multiple candidate data with the target data. Then, based on the ranking result, the information processing device 1 joins, for example, the top M candidate data with the highest priority with the target data and outputs the joined data.

[0041] According to this embodiment, it is possible to obtain tables that can be joined with the query columns of the target table from a large group of external tables without requiring high computational costs, and then join them with the target table.

[0042] <Configuration for building an information processing device> Next, the configuration of other information processing devices used in the construction of the information processing device 1 will be described. Figure 8 is a block diagram showing the configuration of other information processing devices used in the construction of the information processing device according to the first embodiment. The information processing device 100 is a device used in the construction of the information processing device 1 and has the same hardware configuration as the information processing device 1. As shown in Figure 8, the information processing device 100 includes a data acquisition unit 101, an embedded model learning unit 102, an index target column acquisition unit 103, an embedded vector conversion unit 104, an index construction unit 105, and a ranking model construction unit 106.

[0043] The data acquisition unit 101 acquires training data and test data to be used as training data for a machine learning model or deep learning model.

[0044] The embedding model learning unit 102 has an embedding model, which is a model representing the distribution of latent features in a vector space. Specifically, the embedding model is composed of a deep learning model, such as a three-layer neural network.

[0045] The embedded model learning unit 102 extracts feature vectors from the column values ​​of the training data obtained by the data acquisition unit 101, and learns the embedded model based on the output results when these feature vectors are input into the embedded model. For example, as shown in Figure 9, if the training data columns contain strings such as "Tokyo," "Tsukuba," "Nagoya," and "Kawasaki," the embedded model learning unit 102 extracts feature vectors that represent these strings numerically, and learns the embedded model by inputting these extracted feature vectors into the embedded model. The embedded model learning unit 102 also extracts feature vectors from the column values ​​of the test data obtained by the data acquisition unit 101, and adjusts the parameters of the embedded model based on the output results when these feature vectors are input into the embedded model. Figure 9 is a diagram illustrating the overview of the embedded model learning process.

[0046] Each feature included in the feature vector used to train the embedding model may be calculated based on a predetermined language model, for example, when both the values ​​in the training data column and the test data column are strings, or it may be calculated based on a word count or character count. Furthermore, for example, when both the values ​​in the training data column and the test data column are strings, the feature vector used to train the embedding model can be obtained by performing a conversion process using word embedding.

[0047] The training of the embedding model can be performed, for example, based on the relationship between the values ​​of the columns that formed the basis of the feature vector and the embedding vectors output in response to the input of the feature vector. Alternatively, the training of the embedding model can be performed, for example, as self-supervised learning using output data output in response to the input of the feature vector. Furthermore, the training of the embedding model can be repeated until the termination condition is met. The termination condition can be set, for example, when multiple feature vectors extracted from multiple similar column values ​​are input, multiple embedding vectors are output that are close to each other in the vector space and have a dimension smaller than the dimension of the input feature vector.

[0048] The embedding model learning unit 102 stores the trained embedding model, which has undergone training as described in the specific example above, in the embedding model storage unit 20. Therefore, the data transformation unit 12 can convert the target data into embedding vectors by inputting the feature vectors extracted from the target data into the trained embedding model obtained from the embedding model storage unit 20. Furthermore, the data transformation unit 12 can use the trained embedding model obtained from the embedding model storage unit 20 to convert multiple similar target data into multiple embedding vectors that are close to each other in the vector space.

[0049] The index target column acquisition unit 103 acquires the index target columns to be used for constructing the search index from the group of external tables stored in the table storage unit 500. The index target columns only need to include at least one or more columns selected from one table belonging to the group of external tables. The index target columns may also include multiple columns selected from multiple tables belonging to the group of external tables. Furthermore, the index target column acquisition unit 103 may selectively acquire the index target columns according to predetermined rules, or it may selectively acquire the index target columns using a machine learning inference method. Specifically, for example, the index target column acquisition unit 103 may selectively acquire as index target columns columns whose values ​​are strings and which have the most unique values ​​within their respective tables.

[0050] The embedding vector conversion unit 104 uses the trained embedding model stored in the embedding model storage unit 20 to convert the set of index target columns obtained by the index target column acquisition unit 103 into a set of embedding vectors. For example, as shown in Figure 10, if the index target columns contain strings such as "Tokyo", "Tsukuba", "Nagoya", and "Kawasaki", the embedding vector conversion unit 104 extracts a feature vector that quantifies these strings and inputs the extracted feature vector into the embedding model to convert the index target columns into embedding vectors. The method for extracting feature vectors from the index target columns can be the same as the method used when extracting feature vectors from the training data (training data and test data) of the embedding model. Figure 10 is a diagram illustrating the overview of the process of converting index target columns into embedding vectors.

[0051] As shown in Figure 11, the index building unit 105 constructs a search index by applying a predetermined algorithm to the set of embedding vectors obtained by transforming the set of columns to be indexed. Specifically, the index building unit 105 constructs a search index by applying, for example, the algorithm disclosed at https: / / github.com / spotify / annoy or the algorithm disclosed at https: / / github.com / facebookresearch / faiss to the set of embedding vectors obtained by transforming the set of columns to be indexed. Figure 11 is a diagram illustrating the overview of the process for constructing the search index.

[0052] The index construction unit 105 stores the search index constructed by the method described in the above specific example in the index storage unit 30.

[0053] According to the methods described in the specific examples above, a search index can be constructed in which the index target column obtained by the index target column acquisition unit 103 is associated with the embedding vector of the index target column. Furthermore, the candidate search unit 13 can use the search index constructed in the manner described above to search for (acquire) search target columns as candidate data in which embedding vectors similar to the embedding vector of the target data are associated.

[0054] Furthermore, using the methods described in the specific examples above, it is possible to construct a search index that includes representative vectors associated with multiple embedding vectors that are close to each other in the vector space. In such cases, the candidate search unit 13 can, for example, search for representative vectors similar to the embedding vectors of the target data, identify multiple embedding vectors associated with those representative vectors, and thereby search (obtain) the index target columns associated with each of those multiple embedding vectors as candidate data.

[0055] The ranking model construction unit 106 has a ranking model composed of either a model conforming to predetermined rules, a machine learning model, or a deep learning model. In this specific example, unless otherwise specified, the case where the ranking model is a machine learning model or a deep learning model will be described.

[0056] The ranking model construction unit 106 extracts feature vectors from the column values ​​of the training data obtained by the data acquisition unit 101, and trains the ranking model based on the output results when these feature vectors are input into the ranking model. For example, as shown in Figure 12, if the training data columns contain strings such as "Tokyo," "Tsukuba," "Nagoya," and "Kawasaki," the ranking model construction unit 106 extracts feature vectors that represent these strings numerically, and trains the ranking model by inputting these extracted feature vectors into the ranking model. The ranking model construction unit 106 also extracts feature vectors from the column values ​​of the test data obtained by the data acquisition unit 101, and adjusts the parameters of the ranking model based on the output results when these feature vectors are input into the ranking model. Figure 12 shows an overview of the ranking model training.

[0057] Each feature included in the feature vector used to train the ranking model may be calculated based on a predetermined language model, for example, when both the values ​​in the training data column and the test data column are strings, or it may be calculated based on a word count or character count. Furthermore, for example, when both the values ​​in the training data column and the test data column are strings, the feature vector used to train the ranking model can be obtained by performing a conversion process using word embedding.

[0058] The ranking model construction unit 106 stores the ranking model constructed in the manner described in the above specific example in the ranking model storage unit 40. Therefore, the candidate ranking unit 14 can obtain a ranking result by inputting the feature vectors extracted from the target data and candidate data into the ranking model obtained from the ranking model storage unit 40. Furthermore, the candidate ranking unit 14 can obtain a ranking result in which candidate data similar to the target data are given a relatively higher rank, and candidate data that is not similar to the target data are given a relatively lower rank.

[0059] Next, the processing flow of the information processing device 100 described above will be explained. Figure 13 is a flowchart showing the processing flow of another information processing device used in the construction of the information processing device according to the first embodiment.

[0060] First, in step S101, the data acquisition unit 101 acquires training data for a machine learning model or a deep learning model. Next, in step S102, the embedding model learning unit 102 trains the embedding model using the training data obtained in step S101, and then stores the trained embedding model in the embedding model storage unit 20. Subsequently, in step S103, the index target column acquisition unit 103 acquires the index target column from the group of external tables stored in the table storage unit 500. Subsequently, in step S104, the embedding vector conversion unit 104 converts the index target column obtained in step S103 into an embedding vector. Subsequently, in step S105, the index construction unit 105 constructs a search index by applying a predetermined algorithm to the embedding vector obtained in step S104, and stores the constructed search index in the index storage unit 30. Finally, in step S106, the ranking model construction unit 106 trains the ranking model using the training data obtained in step S101, and then stores the trained ranking model in the ranking model storage unit 40.

[0061] [Second Embodiment] Figure 14 is a block diagram showing the configuration of the information processing device according to the second embodiment.

[0062] The information processing device 200 according to this embodiment has the same hardware configuration as the information processing device 1. The information processing device 200 also includes a data acquisition means 201, a data conversion means 202, a candidate search means 203, a candidate ranking means 204, and a result output means 205.

[0063] Figure 15 is a flowchart illustrating the processing performed in the information processing device according to the second embodiment.

[0064] The data acquisition means 201 acquires the target data (step S201).

[0065] The data conversion means 202 converts the target data into an embedding vector that represents the latent features of the target data (step S202).

[0066] The candidate search means 203 searches for data similar to the latent features of the target data as candidate data (step S203).

[0067] The candidate ranking means 204 applies a predetermined process to the candidate data and assigns a ranking to each candidate data to be combined with the target data (step S204).

[0068] The result output means 205 outputs the result of combining the target data and candidate data based on the assigned rank (step S205).

[0069] According to this embodiment, two tables can be joined without requiring high computational costs, even for large amounts of data.

[0070] [Third Embodiment] Figure 16 is a block diagram showing the configuration of an information processing device according to the third embodiment.

[0071] The information processing device 300 according to this embodiment has the same hardware configuration as the information processing device 100. The information processing device 300 also includes a data acquisition means 301, a data conversion means 302, a candidate search means 303, a candidate ranking means 304, and a result output means 305.

[0072] Figure 17 is a flowchart illustrating the processing performed in the information processing device according to the third embodiment.

[0073] The data acquisition means 301 acquires the target data (step S301).

[0074] The data conversion means 302 converts the target data into an embedding vector by inputting the feature vector extracted from the target data into an embedding model, which is a model representing the distribution of latent features in a vector space (step S302).

[0075] The candidate search means 303 uses a search index that associates the search target data with the embedding vector of the search target data, and searches for the search target data to which an embedding vector similar to the embedding vector of the target data is associated, as candidate data (step S303).

[0076] The candidate ranking means 304 applies a predetermined process to the candidate data and assigns a ranking to each candidate data to be combined with the target data (step S304).

[0077] The result output means 305 outputs the result of combining the target data and candidate data based on the assigned rank (step S305).

[0078] According to this embodiment, two tables can be joined without requiring high computational costs, even for large amounts of data.

[0079] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0080] (Note 1) A data acquisition method for acquiring target data, A data transformation means for transforming the target data into an embedding vector representing the latent features of the target data, A candidate search means for searching for data similar to the latent features of the aforementioned target data as candidate data, A candidate ranking means that applies a predetermined process to the candidate data and assigns a ranking to each of the candidate data to be combined with the target data, A result output means that outputs the result of combining the target data and the candidate data based on the assigned rank, An information processing device equipped with the following features.

[0081] (Note 2) The candidate search means is an information processing device described in Appendix 1 that searches for candidate data by performing a nearest neighbor search based on the embedding vector of the target data.

[0082] (Note 3) The aforementioned embedding vector is obtained by the information processing device described in Appendix 1, which is trained using a deep learning model that utilizes a multilayer neural network.

[0083] (Note 4) Obtain the target data, The aforementioned target data is converted into an embedding vector that represents the latent features of the aforementioned target data. Search for data similar to the latent features of the aforementioned target data as candidate data. A predetermined process is applied to the candidate data, and a ranking is assigned to each of the candidate data to be combined with the target data. An information processing method that outputs the result of combining the target data and the candidate data based on the assigned rank.

[0084] (Note 5) Obtain the target data, The aforementioned target data is converted into an embedding vector that represents the latent features of the aforementioned target data. Search for data similar to the latent features of the aforementioned target data as candidate data. A predetermined process is applied to the candidate data, and a ranking is assigned to each of the candidate data to be combined with the target data. A program that causes a computer to perform a process to output the result of combining the target data and the candidate data based on the assigned rank.

[0085] (Note 6) A data acquisition method for acquiring target data, A data transformation means that transforms the target data into an embedding vector by inputting the feature vector extracted from the target data into an embedding model, which is a model representing the distribution of latent features in a vector space. A candidate search means that uses a search index that associates the target data with the embedding vector of the target data, and searches for the target data whose embedding vector is similar to the embedding vector of the target data as candidate data. A candidate ranking means that applies a predetermined process to the candidate data and assigns a ranking to each of the candidate data to be combined with the target data, A result output means that outputs the result of combining the target data and the candidate data based on the assigned rank, An information processing device equipped with the following features.

[0086] (Note 7) The embedding model is an information processing device as described in Appendix 6, which is a model that, when multiple feature vectors extracted from multiple similar data are input, outputs multiple embedding vectors whose distances in the vector space are close to each other and which have a number of dimensions smaller than the number of dimensions of the input feature vectors.

[0087] (Note 8) The information processing device described in Appendix 7 is obtained by transforming the embedding vector of the search target data included in the search index by inputting the feature vector extracted from the search target data into the learned embedding model.

[0088] (Note 9) Obtain the target data, The feature vectors extracted from the aforementioned target data are input into an embedding model, which is a model representing the distribution of latent features in a vector space, thereby converting the target data into embedding vectors. Using a search index that associates the target data with the embedding vector of the target data, the system searches for the target data whose embedding vector is similar to that of the target data, and identifies the target data as candidate data. A predetermined process is applied to the candidate data, and a ranking is assigned to each of the candidate data to be combined with the target data. An information processing method that outputs the result of combining the target data and the candidate data based on the assigned rank.

[0089] (Note 10) Obtain the target data, The feature vectors extracted from the aforementioned target data are input into an embedding model, which is a model representing the distribution of latent features in a vector space, thereby converting the target data into embedding vectors. Using a search index that associates the target data with the embedding vector of the target data, the system searches for the target data whose embedding vector is similar to that of the target data, and identifies the target data as candidate data. A predetermined process is applied to the candidate data, and a ranking is assigned to each of the candidate data to be combined with the target data. A program that causes a computer to perform a process to output the result of combining the target data and the candidate data based on the assigned rank.

[0090] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the present invention. That is, the present invention naturally includes the full disclosure, including the claims, and various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, each disclosure of the above-mentioned patent documents and other references is incorporated herein by reference. [Explanation of symbols]

[0091] 1. Information Processing Device 11 Data Acquisition Unit 12 Data Conversion Unit 13 Candidate Search Section 14 Candidate Ranking Section 15 Result Output Section 20 Embedded Model Memory Unit 30 Index storage 40 Ranking Model Memory Unit 50 Computers

Claims

1. A data acquisition means for acquiring target data, A data transformation means that transforms the target data into an embedding vector by inputting the feature vector extracted from the target data into an embedding model, which is a model representing the distribution of latent features in a vector space. A candidate search means that uses a search index that associates the target data with the embedding vector of the target data, and searches for the target data whose embedding vector is similar to the embedding vector of the target data as candidate data. A candidate ranking means that assigns a ranking to each of the candidate data to be combined with the target data by inputting the feature vector extracted from the target data and the feature vector extracted from the candidate data into a ranking model, which is a model for evaluating the ease of combining the target data and the candidate data. A result output means that outputs the result of combining the target data and the candidate data based on the assigned rank, An information processing device equipped with the following features.

2. The information processing apparatus according to Claim 1, wherein the embedding model is a model that has been trained to output a plurality of embedding vectors having a number of dimensions smaller than the number of dimensions of the input feature vectors, when a plurality of feature vectors extracted from a plurality of similar data are input.

3. The information processing apparatus according to claim 2, wherein the embedding vector of the search target data included in the search index is obtained by transforming the feature vector extracted from the search target data by inputting it into the learned embedding model.

4. A computer-based information processing method, Obtain the target data, The feature vectors extracted from the aforementioned target data are input into an embedding model, which is a model representing the distribution of latent features in a vector space, thereby converting the target data into embedding vectors. Using a search index that associates the target data with the embedding vector of the target data, the system searches for the target data whose embedding vector is similar to that of the target data, and identifies the target data as candidate data. By inputting the feature vector extracted from the target data and the feature vector extracted from the candidate data into a ranking model, which is a model for evaluating the ease of combining the target data and the candidate data, a ranking is assigned to each of the candidate data to determine its suitability for combining with the target data. An information processing method that outputs the result of combining the target data and the candidate data based on the assigned rank.

5. Acquire target data, The feature vectors extracted from the aforementioned target data are input into an embedding model, which is a model representing the distribution of latent features in a vector space, thereby converting the target data into embedding vectors. Using a search index that associates the target data with the embedding vector of the target data, the system searches for the target data whose embedding vector is similar to that of the target data, and identifies the target data as candidate data. By inputting the feature vector extracted from the target data and the feature vector extracted from the candidate data into a ranking model, which is a model for evaluating the ease of combining the target data and the candidate data, a ranking is assigned to each of the candidate data to determine its suitability for combining with the target data. A program that causes a computer to perform a process to output the result of combining the target data and the candidate data based on the assigned rank.

Images