Dataset exploration system

Abstract

When collecting datasets for a particular purpose, it is possible to easily collect multiple datasets, including not only those directly related to that purpose, but also datasets that are not directly related but may be useful. [Solution] The system for searching for datasets calculates the vector values ​​of each of the multiple datasets stored in the database, calculates the degree of similarity of the vector values ​​between the multiple datasets (12), accepts a keyword (20), searches for datasets related to the keyword from among the datasets stored in the database based on the degree of similarity of the vector values ​​(30), and presents the searched dataset as a search result (50).

Description

Technical Field

[0001] The present invention relates to a system for searching for a data set in a group of data sets that are electronically stored in a database device or a storage device, and more particularly, to a system for searching for a data set related to any keyword. In this specification, a data set is a set of data in which numerical values or texts are listed and recorded item by item, such as in CSV format or JSON format.

Background Art

[0002] Conventionally, various configurations of systems for retrieving and extracting certain data from a group of electronic data have been proposed. For example, in Patent Document 1, in keyword search using an associative memory, in order to reduce the ambiguity of input information, ambiguous keywords such as telephone numbers and product numbers are held in the associative memory, and a configuration for detecting a portion that matches an ambiguous keyword in an input query has been proposed. In Patent Document 2, a configuration has been proposed in which a user is made to input a search keyword (e.g., rental real estate in Tokyo) and a concept (e.g., medical institution), and a keyword conceptually close to the keyword and the concept (e.g., ~ general hospital) is obtained and output on a network. In Non-Patent Document 1, a configuration has been proposed in which open data published on the www (World Wide Web) is collected, its metadata (title, author, description, etc.) is organized, and a keyword search is performed on the metadata to obtain a data set.

Prior Art Documents

[0005] In today's big data era, many datasets have become open source and are available for use as data lakes within companies, making a vast number of datasets accessible. As the available datasets become larger in scale, the scope of dataset exploration also expands. Therefore, methods for understanding and exploring the locations of various datasets are useful in order to efficiently find the necessary datasets. Furthermore, in cases where alternative data is referenced, it is sometimes preferable to collect related datasets, including those from other fields, rather than just a single dataset, in order to achieve a specific objective or effect. Therefore, when collecting datasets for a particular purpose, it is advantageous to prepare multiple datasets simultaneously, including those that are not directly related, in order to better achieve that objective.

[0006] In this regard, with conventional dataset search techniques, while it is typically possible to search for datasets using keywords, it requires that metadata be properly organized. Furthermore, because it is a keyword search, the search is limited to the range of the input keyword, and even if the keyword is expanded with similar concepts or ambiguous terms, it often only returns a single dataset related to the initially entered keyword. Collecting data that is not directly related to the input keyword but can be combined with datasets related to that keyword has been difficult. This is because dataset searches were carried out using the same techniques as web searches and document searches. More specifically, the search techniques for datasets are still in the trial-and-error stage, and existing web and document search techniques are simply being applied. However, datasets themselves contain a mixture of words, IDs, numbers, etc. When applying current web and document search techniques, it is only possible to create keyword matching targets through metadata organization and then search against them. Moreover, with existing web search techniques, the search ends when the correct site is reached, making it unsuitable for collecting data from multiple datasets.

[0007] In view of the above circumstances, the main object of the present invention is to provide a system for searching for datasets that enables the easy collection of multiple datasets, including not only datasets directly related to a particular purpose but also datasets that are not directly related but may be useful, when collecting datasets for a particular purpose.

[0008] Currently, there are known techniques that use machine learning algorithms to vectorize arbitrary text data (which may also include numerical data) (vectorization is the process of converting text into multi-dimensional numerical components). In such text vectorization techniques, generally speaking, a multi-dimensional vector is generated from arbitrary text data by extracting numerical values ​​from multiple neurons in an appropriate intermediate layer of a language processing model, such as a model that generates a summary of the content of text data or a natural language processing model that predicts the text following a given text. The numerical values ​​of each component of the vector thus generated from the text data reflect the content of the text data. Therefore, in the vector space, vectors generated from text data with similar content will be located close to each other, and it is possible to measure the degree of similarity between the content of multiple text data using an index value that represents the closeness of vectors, such as cosine similarity. In other words, by using text vectorization techniques to vectorize multiple datasets and referring to the degree of similarity between the vectors of any multiple datasets, it becomes possible to evaluate the degree of relevance of the content between those datasets, even if the multiple datasets do not contain common keywords. By using the high degree of relevance evaluated in this way, it is expected that it will be possible to explore multiple datasets, including datasets that are not directly related but may be useful, such as those that share a common purpose and keywords. This finding is utilized in the present invention. [Means for solving the problem]

[0009] According to the present invention, the above problem is solved by a system for searching a dataset, A database means configured to store multiple datasets, A dataset vectorization means configured to calculate the vector values ​​of the dataset, A vector value similarity calculation means configured to calculate the degree of similarity of vector values ​​between multiple datasets, A keyword receiving means configured to receive keywords, A dataset search means configured to search for datasets related to the keyword from among the datasets stored in the database means based on the degree of similarity of the vector values, A dataset presentation means configured to present the datasets searched by the dataset selection means as search results. This is achieved by a system including the above. Furthermore, the system of the present invention is realized by operation according to a program of any computer device.

[0010] In the system of the present invention described above, "dataset" refers to a set of data recorded by listing texts, each containing numerical values ​​as data for each item, as previously stated. The content of the data may relate to any matter, such as stock prices, weather forecasts, factory operations, personnel matters, and sales performance. "Database means" may be a device that electromagnetically stores the above-mentioned dataset. "Vector values" refer to a set of numerical values ​​with an appropriately set number of dimensions, and "dataset vectorization means" refers to means that convert the dataset into vector values ​​according to an arbitrary machine learning algorithm, as previously mentioned. "Similarity of vector values" may be any index value that represents the degree of closeness between two vectors in a vector space, and is typically cosine similarity. "Keywords" are words that are initially set appropriately for searching for datasets, and may be words, phrases, numerical values, etc., and may be set appropriately according to the type of data, content, and purpose of data analysis that the system user wants to search for. "Dataset search means" is a means configured to search for datasets related to the set keywords. In this process of searching for datasets related to a keyword, the relevance to the keyword is evaluated based on the degree of similarity of the vector values ​​between each dataset. This means that datasets related to the keyword may include not only datasets that are found through keyword searches (for example, datasets that contain the keyword or its synonyms in data items or data more than a predetermined number of times), but also datasets that are not found through keyword searches (for example, datasets that do not explicitly contain the keyword or its synonyms in data items or data more than a predetermined number of times). The dataset presentation means then presents the datasets determined to be related to the keyword in any manner.

[0011] In the above configuration, first, the datasets stored in the database are vectorized, and the degree of similarity of the vector values ​​between these datasets is calculated. Then, in searching for datasets based on keywords, datasets other than those directly related to the keyword (i.e., datasets hit by keyword search) among the datasets stored in the database are also judged to be included in the datasets related to the keyword based on the degree of similarity of the vector values ​​with the datasets directly related to the keyword. Here, as already mentioned, the degree of similarity of the vector values ​​between datasets increases as the degree of closeness or relevance of the data content within the dataset increases. Therefore, as in the above configuration, by evaluating the relevance to the keyword through the degree of similarity of the vector values ​​between datasets, it is expected that not only datasets directly related to the keyword but also datasets that may be useful but for which no direct relevance to the keyword is detected will be included in the group of datasets presented.

[0012] In the system of the present invention described above, more specifically, the dataset search means may be configured to select a starting dataset from datasets stored in a database means based on its relevance to a keyword, and to search for datasets related to the keyword based on the degree of similarity of the vector values ​​between each of the datasets stored in the database means and the starting dataset. Here, the degree of relevance between a dataset and a keyword may be estimated in several ways. For example, for a given dataset, the higher the number of occurrences of the keyword or its synonyms (which may be set by any algorithm) within it, or the higher the ratio of occurrences to the length of the dataset, the higher the relevance. Alternatively, the closer the distance between the keyword and the dataset, determined using any language processing algorithm, the higher the relevance. The starting dataset is selected based on the relevance to the keyword, for example, a number of datasets that are equal to or greater than a predetermined relevance value, or datasets that are appropriately set in descending order of relevance, and the search for datasets is performed based on the degree of similarity of the vector values ​​between the starting dataset and the datasets. Here, datasets with a high degree of similarity of vector values ​​between them and the starting dataset may be selected as datasets related to the keyword. With this configuration, datasets directly related to the keywords are first selected as the starting dataset, and even if a dataset has a low degree of relevance to the keywords, if it is similar in content to the starting dataset, it can be included in the group of datasets related to the keywords.

[0013] Furthermore, in selecting datasets with a high degree of similarity in vector values ​​to the starting dataset, the dataset search means may be configured to select datasets whose "degree of association" with the starting dataset exceeds a predetermined value. Here, the "degree of association" with the starting dataset is, simply put, an index calculated to search for datasets that have content that can be associated with the content of the starting dataset.

[0014] More specifically, as already mentioned, since the degree of similarity of vector values ​​is calculated for multiple datasets stored in the database, by setting up associations between one dataset and other datasets according to the degree of similarity of vector values, and then repeating the process of setting up associations between datasets associated with one dataset and other datasets according to the degree of similarity of vector values, it is possible to construct a network of associations between datasets in the database based on the degree of similarity of vector values. In such a network of datasets, by setting a dataset with a high degree of relevance to the keyword as the starting dataset and exploring datasets by following associations according to the degree of similarity of vector values ​​from that starting dataset, it becomes possible to reach datasets that do not have a high direct relevance to the keyword but whose content is related to the starting dataset, that is, datasets that contain content that can be associated with the content of the starting dataset. Furthermore, datasets with a high degree of similarity in vector values ​​to the source dataset and that are directly associated will contain more content that can be associated with the content of the source dataset. On the other hand, the more datasets one passes through to reach a particular dataset by following the network of associations from the source dataset, the less content that can be associated with the content of the source dataset is expected to be in that particular dataset.

[0015] Therefore, in this invention, an "association score" is calculated as an indicator of the amount of content that can be associated with the content of the starting dataset that is expected to be included in each dataset. The higher the degree of similarity of vector values ​​with adjacent datasets and the shorter the path to the starting dataset, the greater the value. Datasets with such an association score exceeding a predetermined value may be selected as datasets that contain content associated with the content of the starting dataset. Specifically, for example, the association score As may be defined as follows. As=α N ·X …(1) Here, α is a positive number less than 1, which is set as appropriate; N is the number of datasets traversed in the association network to reach the starting dataset (distance to the starting dataset); and X is the degree of similarity of vector values ​​between directly associated datasets in the association network to the starting dataset. A predetermined value for the degree of association may be set as appropriate. A dataset whose degree of association calculated as described above exceeds the predetermined value is presumed to contain a certain amount of content that can be associated with the content of the starting dataset.

[0016] Furthermore, in the configuration of the present invention described above, multiple starting datasets with a high degree of relevance to the keyword may be set, and for each starting dataset, a group of datasets with a degree of association exceeding a predetermined value may be selected. In that case, for each group of datasets, the higher the degree of relevance between the keyword and the starting dataset, and the higher the degree of association of the datasets included in the group, the more preferentially the datasets related to the keyword may be presented. To this end, for each group of datasets, a "diversity" value is calculated, which increases as the degree of relevance between the keyword and the starting dataset, and as the sum of the degrees of association of the datasets included in the group increases, and the dataset presentation means may be configured to present the groups of datasets in descending order of diversity as a search result. Specifically, the diversity V may be calculated, for example, as follows. V = Rs · (ΣAsi + 1) …(2) Here, Rs is the degree of relevance between a keyword and the starting dataset, Asi is the degree of relevance of each dataset in a group of selected datasets to a single starting dataset, and Sum is the sum of all datasets within the group.

[0017] Incidentally, in each dataset, it may be possible to extract information about the region, time of day, or period during which the data was collected. In such cases, when the regions, times, or periods during which the data was collected differ among multiple datasets, or when the degree of overlap in such regions, times, or periods is small, it may be reasonable to assume that the correlation between them is low, regardless of the degree of similarity of the vector values ​​between them. Therefore, in the configuration of the present invention, the dataset search means may be configured to select a dataset to be searched where the degree of overlap between the data collection region or collection period of the dataset and the starting dataset exceeds a predetermined value, and the degree of association with the starting dataset also exceeds a predetermined value. Regarding the degree of overlap in data collection regions between datasets, for example, the degree of overlap in data collection regions of one dataset with respect to other datasets may be the ratio of the area of ​​overlapping regions between one dataset and other datasets to the area of ​​data collection regions of one dataset. Furthermore, regarding the degree of overlap in data collection periods between datasets, it may be the ratio of the length of overlapping time between one dataset and other datasets to the length of data collection period of one dataset. [Effects of the Invention]

[0018] Thus, according to the system of the present invention, by simply inputting a single keyword, it becomes possible to search not only for datasets directly related to the input keyword, but also for datasets that may be usable even if no direct association with the keyword is detected. Therefore, it is expected that by inputting keywords related to the purpose, it will be possible to collect a group of potentially usable datasets all at once, and it is advantageous that the presented datasets may include datasets that the searcher did not initially anticipate but which could be used due to associative imagery. According to the system of the present invention, when using datasets for various purposes, it is expected that it will be possible to reach the necessary datasets with less prior knowledge regarding the location and search methods of each dataset.

[0019] Other objects and advantages of the present invention will become apparent from the following description of the preferred embodiments of the present invention.

Brief Description of the Drawings

[0020] [Figure 1] FIG. 1 is a diagram schematically showing a computer in which a dataset search system according to this embodiment is implemented. [Figure 2] FIG. 2 is a diagram showing the configuration of the dataset search system according to this embodiment in the form of a block diagram. [Figure 3] FIG. 3 is a diagram showing the dataset search process in the system according to this embodiment in the form of a flowchart. [Figure 4] FIG. 4 is a schematic diagram explaining the association based on the vector similarity between datasets stored in the database according to this embodiment. [[ID=第二十一条]] [[ID=第二十二条]] [Figure 5] [[ID=第二十三条]]FIG. 5 is a schematic diagram explaining the link between datasets based on the association degree from the starting dataset (starting node) according to this embodiment. [[ID=第二十四条]] [[ID=第二十五条]] [[ID=第二十六条]]

Explanation of Reference Numerals

[0021] [[ID=第三十条]] [[ID=第三十一条]]1... computer main body, 2... computer terminal, 3... monitor, 4... keyboard, mouse (input device), 10... dataset storage unit (search target dataset database), 11... feature extraction unit, 12... vectorization processing and similarity calculation unit of dataset, 1... extraction unit of data collection period of dataset, 14... extraction unit of data collection area of dataset, 15... feature storage unit, 20... keyword input unit, 30... search network generation unit, 31... starting node identification unit, 32... associated link assignment unit, 40... priority presentation component determination unit, 41... diversity calculation unit, 42... component ranking unit, 50... search result display unit [[ID=第三十二条]] [[ID=第三十三条]]

Best Mode for Carrying Out the Invention

[0022] [[ID=第三十七条]] The present invention will be described in detail below with reference to the attached figures, with reference to several preferred embodiments. In the figures, the same reference numerals indicate the same parts.

[0023] Computer device configuration The dataset search system according to this embodiment may be implemented by operation in accordance with a program on a computer device 1 of a type commonly used in this field, as illustrated in Figure 1. The computer device 1 is equipped with a CPU, storage device, and input / output devices (I / O) interconnected by a bidirectional common bus in a typical configuration. The storage device includes memory PM, which stores each program that executes the arithmetic processing used in the calculations of this embodiment, and work memory WM and data memory DM(5) used during calculations. Instructions to the computer device 1 by the searcher, as well as the display and output of search results and other information, are made through a computer terminal device 2 connected to the computer device 1. The computer terminal device 2 is equipped with a monitor 3 and input devices 4 such as a keyboard and mouse in a typical configuration. When the program is started, the searcher can give various instructions and inputs to the computer device 1 using the input devices 4 according to the program's procedure and the display on the monitor 3, and can also visually confirm the calculation status and calculation results from the computer device 1 on the monitor 3. The dataset to be searched may be stored in the data memory DM(5) equipped on the computer device 1, or it may be stored in a cloud system accessible via any communication network.

[0024] Configuration of the data exploration system Referring to Figure 2, the dataset search system according to this embodiment generally consists of a dataset storage unit 10, a feature extraction unit 11, a feature storage unit 15, a keyword input unit 20, a search network generation unit 30, a priority presentation component determination unit 40, and a search result display unit 50.

[0025] The dataset storage unit 10 is a database that stores the datasets to be searched. As already mentioned, a dataset is a set of data recorded in a format such as CSV or JSON, where numerical or text data is listed for each item. The content of the data may be anything, such as stock prices, weather forecasts, factory operations, personnel, or sales performance. The dataset storage unit 10 may be located on a server accessible via a communication network. In addition, the dataset storage unit 10 may store the datasets themselves, but it may also store information that serves as a pointer to each dataset, such as www or a URL within the local area network.

[0026] The feature extraction unit 11 is configured to extract features from the datasets stored in the dataset storage unit 10. Features are vectors that represent the features of each dataset extracted from each dataset, or arbitrary index quantities such as the degree of similarity of vectors or the distance between datasets. More specifically, the feature extraction unit 11 is first provided with a dataset vectorization and similarity calculation unit 12, where the vectorization of each dataset stored in the dataset storage unit 10 is performed, and a vector similarity index, which is an index value of the degree of similarity of the vector values ​​between each dataset, is calculated. As already mentioned, the vectorization of datasets can be achieved by any machine learning algorithm that converts arbitrary text data or natural language data into vectors, which are sets of multidimensional numbers, for example, by machine learning models constructed according to TaBERT, TURL, etc. Since the vectors obtained from a dataset reflect the content of the dataset, the higher the degree of similarity between the content of any two datasets, the closer the vectors of those datasets will be in the vector space. By referring to the degree of proximity of the vectors, that is, the degree of similarity between the vectors of any two datasets, it becomes possible to estimate the degree of similarity or relationship between the content of those two datasets. Therefore, in this embodiment, the vectorization processing and similarity calculation unit 12 calculates the degree of similarity of the respective vector values ​​(hereinafter referred to as "vector similarity") for all combinations of two datasets in the dataset stored in the dataset storage unit 10. Typically, the cosine similarity of the vectors of the two datasets may be used as the "vector similarity" between two datasets.

[0027] Furthermore, the feature extraction unit 11 may be provided with a data collection period extraction unit 13 and a data collection region extraction unit 14, which extract information about the time period or duration of data collection and information about the region where data was collected for each dataset, respectively. The time period or duration information and the region information extracted by the data collection period extraction unit 13 and the data collection region extraction unit 14 may be used when performing associations between datasets, as will be explained later.

[0028] In the feature storage unit 15, the features calculated or extracted by the feature extraction unit 11 are stored and referenced when searching for a dataset based on keywords, as described later.

[0029] The keyword input section 20 is configured to accept keywords that the explorer initially sets appropriately according to the type and content of the data they want to explore and the purpose of the data analysis. Keywords may be words, phrases, numbers, etc. Multiple keywords may be entered at once.

[0030] The search network generation unit 30 consists of a starting node identification unit 31 that receives a keyword input in the keyword input unit 20 and identifies a dataset with a high degree of relevance to the keyword as a starting node, and an associative link assignment unit 32 that generates a group of datasets (hereinafter referred to as "components") containing data with content associated with the identified starting node. As will be described in detail later, the starting node identification unit 31 calculates the degree of relevance between each dataset to be searched and the keyword, detects datasets that are directly related to the keyword based on the degree of relevance, and sets such datasets as starting nodes to be used as the starting point for searching datasets. Note that there may be only one starting node, but usually two or more are set. As will be described in detail later, the associative linking unit 32 calculates the degree of association from a starting node for each of the datasets to be searched, and sequentially associates (associative links) datasets whose degree of association exceeds a predetermined value with datasets that significantly contain content associated with the content of the starting node, thereby forming a group of datasets to which these associative links have been added, i.e., a component.

[0031] The priority component determination unit 40 may include a diversity calculation unit 41 and a component ranking unit 42. When multiple starting nodes are set and multiple components are formed in the search network generation unit 30, the priority component determination unit 40 determines the order of components to be presented preferentially to the searcher. To this end, the diversity calculation unit 41 calculates the diversity of each component, and the component ranking unit 42 determines the order of components to be presented as search results based on the diversity. Note that when only one starting node is set and only one component is generated, diversity calculation and component presentation prioritization are not performed.

[0032] The search result display unit 50 presents a group of datasets to the searcher according to the priority given to the components by the priority presentation component determination unit 40. If only one component has been generated, it may be presented as is.

[0033] System operation (1) Overview In the system of this embodiment, as already mentioned, first, in the preparation stage for searching for datasets, each of the datasets to be searched is vectorized, and the vector similarity is calculated in advance for all combinations of two datasets in the dataset to be searched. Then, when searching for datasets, as shown in the flowchart of Figure 3, when a keyword appropriately determined by the searcher is input, datasets with a high degree of relevance to the keyword are identified as starting nodes (S1), and associative links are added from each identified starting node to datasets whose degree of relevance exceeds a predetermined value (S2), and components are formed (S3). After that, if multiple components are formed, the diversity of each component is calculated (S4), and the components are ranked in order of diversity to be presented, and the search results are presented (S5). The series of processing operations of this embodiment will be explained in order below.

[0034] (2) Calculation of vector similarity in the target dataset As already described, in this embodiment, the vectorization processing and similarity calculation unit 12 vectorizes all of the target datasets stored in the dataset storage unit 10, and calculates the vector similarity of the vector values ​​obtained by vectorization for all combinations of two datasets in the target dataset, and records it in the vector similarity storage unit 15. With this configuration, as schematically depicted in Figure 4, each of the target datasets (d1, d2, ...) is associated with other datasets according to their vector similarity, that is, according to the closeness of their contents. As a result, by following sequentially from one dataset to another based on the level of vector similarity, it becomes possible to sequentially detect datasets with similar contents. In the search for datasets based on the following keywords, this configuration that enables the sequential detection of datasets with similar contents is utilized.

[0035] (3) Keyword input and setting of starting node In the system of this embodiment, the search for a dataset is initiated when the searcher enters an arbitrary keyword into the keyword input unit 20. As already mentioned, the keyword may be a word, phrase, or numerical value appropriate to the type of data to be searched, its content, the purpose of data analysis, etc. Once a keyword is entered, the starting node identification unit 31 first selects a dataset to be the starting node from the datasets stored in the dataset storage unit 10.

[0036] In selecting the starting node, the degree of relevance between the keyword and each dataset may be calculated, and the starting node may be determined based on that degree of relevance. The degree of relevance between the keyword and each dataset is, in short, an indicator of the degree of relevance between the keyword and the content of each dataset, and as already mentioned, may be calculated in several ways. Specifically, for example, in one way, the degree of relevance may be determined such that the more times the keyword or its similar word (which can be determined based on the keyword using any method) is included in the data item name or the data itself, the higher the value. Alternatively, in another way, the closer the distance representing the closeness of the content between the keyword and the dataset, determined using any language processing algorithm, the higher the degree of relevance. Therefore, for each dataset, the higher the degree of relevance, the higher the direct relevance of the keyword. Datasets with a high degree of relevance are typically datasets that are hit by a normal keyword search. The starting node may be set to a dataset whose degree of relevance exceeds a predetermined value that is set as appropriate, or a dataset that is included in a number set as appropriate, counting from the highest degree of relevance. In other words, there may be multiple starting nodes. Furthermore, since the datasets set in the above starting nodes are datasets directly related to the keywords, the above process has resulted in the detection of datasets directly related to the keywords.

[0037] (4) Formation of components centered around the starting node (associative linking) As described above, once a starting node—a dataset directly related to a keyword—is determined, a search for datasets associated with the starting node is performed. This allows for the discovery of datasets that, while not directly related to a keyword, are associated with datasets directly related to the keyword and may be useful for data analysis or other purposes.

[0038] In searching for datasets associated with a starting node, specifically, as described in the section on the summary of the invention, first, an index is calculated of the degree of association between each dataset and the starting node, that is, an index of the amount of content that can be associated with the content of the starting node that is expected to be included in each dataset. In this regard, as already mentioned, the datasets to be searched stored in the dataset storage unit 10 have a vector similarity calculated between them, and it is possible to trace from one dataset to datasets with high relevance in order, relying on the vector similarity. Therefore, the degree of association may be set so that it is possible to trace datasets with high relevance to the starting node based on such vector similarity. Thus, in this embodiment, specifically, the degree of association may be calculated by, for example, the above formula (1). The degree of association according to formula (1) is set so that the value decreases as the path from the starting node to datasets with high vector similarity in a group of datasets to be searched as shown in Figure 4 becomes longer (as the number of datasets passed through increases), and as the vector similarity between datasets decreases. As a result, datasets whose degree of association exceeds a predetermined value, which can be set as appropriate, are expected to have a significantly higher correlation with the content of the starting node and a significantly larger amount of content associated with the content of the starting node. Thus, by detecting datasets whose degree of association exceeds a predetermined value, it becomes possible to identify datasets that do not have a direct correlation with the keyword, but are related to datasets that have a direct correlation with the keyword and may be usable for purposes such as data analysis.

[0039] The process of searching for datasets whose degree of association from the starting node exceeds a predetermined value may be performed in any manner. For example, first, datasets whose vector similarity with a starting node is greater than or equal to a predetermined value (which may be set as appropriate) are selected and an associative link is assigned to them (the degree of association is equal to the vector similarity with the starting node). Next, datasets whose vector similarity with each dataset to which an associative link has been assigned is greater than or equal to a predetermined value (which may be set as appropriate), the degree of association for each of those datasets is calculated, each dataset with a degree of association greater than or equal to the predetermined value is selected and an associative link is assigned to it. Furthermore, datasets whose vector similarity with each of those datasets to which an associative link has been assigned is greater than or equal to a predetermined value (which may be set as appropriate), the degree of association for each of those datasets is calculated, each dataset with a degree of association greater than or equal to the predetermined value is selected and an associative link is assigned to it. This process is repeated until a group of datasets whose degree of association from the starting node exceeds a predetermined value, i.e., a "component," is formed.

[0040] Figure 5 schematically illustrates the process from the above keywords to the formation of components. Referring to this figure, first, starting nodes T1 and T2 with a relevance r of a predetermined value or higher are selected from the keyword KW. Next, the degree of association with other datasets is calculated for each starting node. For example, at starting node T1, first, datasets D1, D2, and D5 with a vector similarity X of a predetermined value or higher are selected, and associative links are assigned. Here, the degree of association As of datasets with a vector similarity X of a predetermined value or higher with starting node T1 is equal to the vector similarity X. Next, for each of the datasets D1, D2, and D5 to which associative links have been assigned, datasets D3, D6, etc., with a vector similarity X of a predetermined value or higher with them, and whose degree of association As of starting node T1 is a predetermined value or higher are selected, and associative links are assigned. Furthermore, the same process is performed on datasets D3 and D6 to which associative links have been assigned, and dataset D4, whose degree of association As from starting node T1 is a predetermined value or higher, is selected, and an associative link is assigned. Thus, when all datasets whose relevance from the starting node T1 is greater than or equal to a predetermined value are selected, they become a single component C1. A similar process may be performed at another starting node T2 to form another component C2.

[0041] (5) Determining the order in which to present the components According to the above process, if multiple starting nodes are set, a component is formed for each starting node. While all of these components may be presented as search results, it is convenient to present them in order from the most useful to the most useful. Therefore, in this embodiment, the order in which components are presented may be determined based on the degree of relevance to the keyword and the degree of relevance of the datasets contained in each component.

[0042] Specifically, diversity is calculated such that the higher the correlation between the keywords of each component formed in the previous processing and the starting dataset, and the higher the sum of the associative degrees of the datasets included in the group of datasets, the greater the value. The components may be presented in order of decreasing diversity. Diversity may be calculated, for example, by formula (2) described in the section on the summary of the invention.

[0043] (6) Selection of datasets based on data collection region and data collection period As already mentioned, each dataset may explicitly state the region, time of day, or period during which the data was collected. When using datasets for any purpose, if the regions, times, or periods of data collection differ among multiple datasets, or if the degree of overlap in such regions, times, or periods is small, it may be reasonable to assume that the correlation between them will be low, regardless of the similarity of their vector values. Therefore, in this embodiment, when extracting the data collection regions or data collection periods recorded in the datasets and forming components, datasets with a low degree of overlap in data collection regions or data collection periods may be excluded from the components.

[0044] Specifically, first, regarding the data collection period, the data collection period extraction unit 13 of the feature extraction unit 11 extracts and records information about the data collection time period or duration as a feature of the dataset stored in the dataset storage unit 10. Then, when the degree of associativity is calculated during component formation, if the degree of time overlap with the starting node (or adjacent dataset) is lower than a predetermined value set appropriately, an associative link will not be assigned, and the component may be excluded, regardless of the value of the degree of associativity. The degree of time overlap may be any index value that measures the degree of overlap of the data collection periods, for example, the ratio of the length of time that overlaps in data collection time with the starting node (or adjacent dataset) to the length of the data collection period of each dataset. If information about the data collection period of a dataset cannot be obtained, it does not need to be excluded from the component.

[0045] Similarly, regarding the data collection area, the data collection area extraction unit 14 of the feature extraction unit 11 extracts and records information about the data collection area as a feature of the dataset stored in the dataset storage unit 10. When the degree of association is calculated during component formation, if the degree of regional overlap between the starting node (or adjacent dataset) is lower than a predetermined value set appropriately, an associative link will not be assigned, and the component may be excluded, regardless of the value of the degree of association. The degree of regional overlap may be any index value that measures the degree of overlap of the data collection areas. For example, the degree of regional overlap may be the ratio of the area of ​​the data collection area of ​​each dataset to the area of ​​the data collection area of ​​the starting node (or adjacent dataset). If information about the data collection area of ​​a dataset cannot be obtained, it does not need to be excluded from the component.

[0046] Thus, according to the system of this embodiment, by inputting keywords related to the objective, it becomes possible to collect a set of potentially usable datasets. Among the collected and presented datasets, there may be datasets that the explorer did not initially anticipate but can be used, based on associative imagery. One application of the system of this embodiment is to improve the efficiency of data utilization within a company. A system that acquires datasets associatively, like the one of this embodiment, is an effective means of utilizing data in various activities and is important in reducing unnecessary expenses and expanding the methods of utilization.

[0047] While the above description is made in relation to embodiments of the present invention, many modifications and changes are readily possible for those skilled in the art, and it will be clear that the present invention is not limited to the embodiments illustrated above, but can be applied to various devices without departing from the concept of the present invention.

Claims

1. A system for exploring datasets, A database means configured to store multiple datasets, A dataset vectorization means configured to calculate the vector values ​​of the dataset, A vector value similarity calculation means configured to calculate the degree of similarity of vector values ​​between multiple datasets, A keyword receiving means configured to receive keywords, A dataset search means configured to search for datasets related to the keyword from among the datasets stored in the database means based on the degree of similarity of the vector values, A dataset presentation means configured to present the datasets searched by the dataset selection means as search results. A system that includes this.

2. A system according to claim 1, wherein the dataset search means is configured to select a starting dataset from datasets stored in the database means based on the degree of relevance to the keyword, and to search for datasets related to the keyword based on the degree of similarity of the vector values ​​between each of the datasets stored in the database means and the starting dataset.

3. A system according to claim 2, wherein the dataset search means is configured to select a dataset to be searched whose degree of association with the starting dataset exceeds a predetermined value.

4. A system according to claim 3, wherein the dataset search means is configured to select a plurality of datasets as the starting dataset, select a group of datasets for each of the starting datasets in which the degree of association exceeds a predetermined value, calculate the degree of diversity for each group of datasets, and the dataset presentation means is configured to present the groups of datasets as search results in descending order of the degree of diversity.

5. A system according to claim 2, wherein the dataset search means is configured to select a dataset to be searched in which the degree of overlap between the data collection region or collection period of the dataset and the starting dataset exceeds a predetermined value, and the degree of association with the starting dataset also exceeds a predetermined value.

Images