Data set search system

Abstract

The present invention addresses the problem of, when collecting a data set for a certain purpose, easily collecting not only a data set directly related to the purpose, but also a plurality of data sets including a data set that may be utilized even though the data set is directly independent of the purpose. A system for searching for a data set, which performs the following processes: calculating a vector value for each of a plurality of data sets stored in a database, and calculating the similarity of the vector values between the plurality of data sets; receiving a keyword; searching a data set related to the keyword from a data set stored in a database according to the similarity of the vector values; and taking the searched data set as a search result for prompting.

Description

Technical Field

[0001] This invention relates to a system for searching datasets within a dataset group that is electronically and electromagnetically stored in a database device or storage device. More specifically, it relates to a system for searching datasets that are relevant to any keyword. Furthermore, in this specification, a dataset refers to a data group recorded in CSV, JSON, or other formats, where numerical or textual data is listed as items. Background Technology

[0002] In the past, various system architectures have been proposed to search for and extract specific data from digitized datasets. For example, Patent Document 1 proposes a structure for keyword searches based on associative memory, which, to reduce the ambiguity of input information, retains vague keywords such as phone numbers and product numbers in the associative memory and detects the portion of the input query that matches the vague keywords. Patent Document 2 proposes a structure that allows users to input search keywords (e.g., rental real estate in Tokyo) and concepts (e.g., medical institutions), obtains keywords conceptually close to the keywords and concepts (e.g., ~general hospital), and outputs them to the network. Non-Patent Document 1 proposes a structure that collects open data published on the World Wide Web, organizes its metadata (title, creator, description, etc.), and performs keyword searches on this metadata to obtain the dataset.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2013-33473

[0004] Patent Document 2: Japanese Patent Application Publication No. 2007-12039

[0005] Non-patent document 1: "Google Dataset Search: Building a search engine for datasets in an open Web ecosystem", Natasha Noy, Matthew Burgess, DanBrickley, 28th Web Conference (WebConf 2019), ACM https: / / datasetsearch.research.google.com / https: / / research.google / pubs / google-dataset-search-building-a-search-engine-for-datasets-in-an-openweb-ecosystem / Summary of the Invention

[0006] In recent years, we have entered the era of big data. Many datasets have become open, and data lakes can be formed and utilized within companies, allowing for the use of even more datasets. If the datasets that can be utilized in this way become large, the search scope also expands. Therefore, to efficiently obtain the required datasets, it is beneficial to have methods for locating or searching various datasets. Furthermore, as with the case of reference data, to achieve a certain purpose or effect, it is sometimes preferable not only to collect a single dataset but also to collect related datasets, including those from other fields. Therefore, when collecting datasets for a specific purpose, it is advantageous to prepare multiple datasets simultaneously, including those that are not directly related, to better achieve the objective.

[0007] Regarding this point, in the case of previous dataset search techniques, it is typically possible to search datasets by keywords, but this requires proper metadata preparation. Because it is a keyword search, the search is performed within the scope of the input keywords. Even if the keywords are expanded with similar concepts or vague terms, it usually only yields a single dataset related to the initially input keywords. It cannot collect data that is not directly related to the input keywords but can be combined with datasets related to those keywords. This is because dataset search is implemented using the same techniques as web / document search. More specifically, dataset search techniques are still in the trial-and-error stage, only applicable to existing web / document search techniques. Datasets themselves contain a mixture of words, IDs, numbers, etc. When using existing web / document search techniques, keyword matching destinations can only be created through metadata preparation, and then the search is performed on those matching destinations. Furthermore, with existing web search techniques, the search ends once the correct website is reached, making it unsuitable for collecting multiple datasets.

[0008] In view of the above, the main objective of the present invention is to provide a system for searching datasets that, when collecting datasets for a certain purpose, can not only easily collect datasets directly related to that purpose, but also easily collect multiple datasets, including datasets that are not directly related but may be usable.

[0009] Currently, there is a known technique that uses machine learning algorithms to vectorize arbitrary text data (which may also include numerical data). Vectorization refers to converting data into multidimensional numerical components. In this text vectorization technique, generally speaking, multidimensional vectors are generated by extracting the values ​​of multiple neurons in appropriate intermediate layers of a language processing model, such as a model that extracts a summary of the content of the generated text data or a natural language processing model that predicts text that follows a given text. Furthermore, the values ​​of each component of the vector generated from the text data reflect the content of the text data. Therefore, in vector space, vectors generated from text data with similar content exist close to each other, and the similarity between multiple text data can be measured using an index representing the proximity of the vectors (such as cosine similarity). That is, if multiple datasets are vectorized using this text vectorization technique, and the similarity between the vectors of any multiple datasets is referenced, the degree of relevance between the content of these datasets can be evaluated even if the multiple datasets do not contain common keywords. Using this evaluated degree of relevance, it is expected that multiple datasets, including those that are directly unrelated but potentially usable, such as those sharing a common keyword for a specific purpose, can be searched.

[0010] This insight can be utilized in this invention.

[0011] According to the present invention, the above-mentioned problem is solved by a system for searching datasets, the system comprising:

[0012] A database unit is configured to store multiple datasets;

[0013] A dataset vectorization unit is configured to compute vector values ​​of the dataset;

[0014] The vector value similarity calculation unit is configured to calculate the similarity of vector values ​​between multiple datasets.

[0015] The keyword receiving unit is configured to receive keywords.

[0016] A dataset search unit is configured to search for datasets related to the keywords from the datasets stored in the database unit based on the similarity of the vector values; and

[0017] The dataset suggestion unit is configured to suggest the datasets retrieved by the dataset search unit as search results.

[0018] Furthermore, the system of the present invention is implemented by operating according to a program of any computer device.

[0019] In the system of the present invention described above, as stated above, a "dataset" refers to a group of data recorded as text, which may include numerical values, listed by item. The content of the data may be related to any matter such as stock prices, weather forecasts, factory operations, personnel, and business performance. A "database unit" may be a device that stores the aforementioned dataset electromagnetically. A "vector value" is a group of values ​​with an appropriately set dimension. A "dataset vectorization unit," as stated above, may be a unit that converts the dataset into vector values ​​according to any machine learning algorithm. "Vector value similarity" may be any index value representing the degree of closeness between two vectors in a vector space, typically cosine similarity, etc. "Keywords" are words appropriately set initially for searching the dataset, and may be words, phrases, numerical values, etc., and may be appropriately set in relation to the type, content, and purpose of data analysis that the system user wants to search for. A "dataset search unit" is a unit configured to search for datasets related to the set keywords. Here, when searching for datasets related to keywords, the relevance to keywords is evaluated based on the similarity of vector values ​​between datasets. This allows for the inclusion of datasets that are matched by keyword searches (e.g., datasets containing keywords or their similar words a specified number of times in data items or data) in the keyword-related datasets, as well as datasets that are not matched by keyword searches (e.g., datasets that do not explicitly contain keywords or their similar words a specified number of times in data items or data). Furthermore, the dataset suggestion unit can suggest datasets determined to be related to keywords in any manner.

[0020] In the above structure, firstly, the datasets stored in the database unit are vectorized, and the similarity of the vector values ​​between these datasets is calculated. Furthermore, in keyword-based dataset searches, datasets stored in the database unit, excluding those directly related to the keywords (i.e., datasets hit by the keyword search), are also included in the keyword-related datasets based on the similarity of their vector values ​​to those of the keyword-related datasets. Here, as mentioned above, the similarity of vector values ​​between datasets increases with the degree of similarity or relevance of the data content within the datasets. Therefore, by evaluating the relevance to keywords based on the similarity of vector values ​​between datasets, as described in the above structure, it is expected that not only datasets directly related to the keywords can be included in the suggested dataset group, but also datasets that are not directly related to the keywords but may be usable can be included in the suggested dataset group.

[0021] In the system of the present invention described above, more specifically, the dataset search unit can be configured as follows:

[0022] Based on the relevance to the keywords, a starting dataset is selected from the datasets stored in the database unit; and

[0023] Data sets related to keywords are searched based on the similarity of vector values ​​between each dataset stored in the database unit and the starting dataset. The relevance between a dataset and a keyword can be estimated in several ways. For example, for a dataset, the higher the frequency of occurrence of the keyword or its similar words (which can be set by any algorithm) or the proportion of occurrences relative to the length of the dataset, the higher the relevance. Alternatively, the closer the keyword, determined by any language processing algorithm, is to the dataset, the higher the relevance. Furthermore, regarding the starting dataset, datasets with a relevance value above a pre-defined threshold, or a number of datasets arranged in descending order of relevance, can be selected as the starting dataset based on their relevance to the keyword. The search for datasets is then performed based on the similarity of vector values ​​between the starting dataset and the starting dataset. Data sets with high similarity in vector values ​​to the starting dataset can be selected as keyword-related datasets. Based on this structure, datasets directly related to the keyword are first selected as starting datasets, and even datasets with low relevance to the keyword, as long as they are content-wise similar to the starting dataset, can be included in the keyword-related dataset group.

[0024] Furthermore, when selecting datasets with high similarity in vector values ​​to the starting dataset, the dataset search unit can be configured to select datasets whose "association" with the starting dataset exceeds a predetermined value as the searched datasets. Here, "association" with the starting dataset is, in short, a metric calculated to search for datasets that contain content associated with the content of the starting dataset.

[0025] More specifically, as described above, since the similarity of vector values ​​between multiple datasets stored within a database unit is calculated, by repeatedly establishing associations between one dataset and other datasets that correspond highly to the similarity of their vector values, and further establishing associations between datasets associated with a particular dataset and other datasets that correspond highly to the similarity of their vector values, a network of associations based on the similarity of vector values ​​can be constructed among datasets within the database unit. In such a dataset network, a dataset with a high relevance to the keyword is set as the starting dataset. By searching for datasets based on associations corresponding highly to the similarity of vector values ​​from that starting dataset, datasets that are not directly related to the keyword but are related to the content of the starting dataset can be reached—that is, datasets containing content associated with the content of the starting dataset. Furthermore, datasets with high similarity to the vector values ​​of the starting dataset and directly associated include more content associated with the content of the starting dataset. On the other hand, the more datasets traversed from the starting dataset through the association network to reach a particular dataset, the less content associated with the content of the starting dataset will be on that particular dataset; this is foreseeable.

[0026] Therefore, in this invention, as an indicator of the amount of content associated with the content of the starting dataset to be included in each dataset, a "relevance" can be calculated, where the higher the similarity of the vector values ​​of adjacent datasets and the shorter the path to the starting dataset, the larger the value. Datasets with a relevance exceeding a predetermined value can be selected as datasets containing content associated with the content of the starting dataset. Specifically, for example, the relevance As can be defined as follows.

[0027] As=α N ·X …(1)

[0028] Here, α is a positive number less than 1, N is the number of datasets traversed in the association network up to the starting dataset (distance to the starting dataset), and X is the similarity of vector values ​​between the datasets directly associated with the starting dataset on the path to the starting dataset in the association network. The specified value for the association degree can be appropriately set. It can be inferred that datasets with an association degree exceeding the specified value, as calculated above, include more content associated with the content of the starting dataset than a certain extent.

[0029] Furthermore, in the structure of the present invention described above, multiple starting datasets with high relevance to keywords can be set, and for each starting dataset, dataset groups with relevance exceeding a predetermined value are selected. In this case, for each dataset group, the higher the relevance between the keyword and the starting dataset, and the higher the relevance of the datasets included in the dataset group, the more preferentially it is presented as a dataset related to the keyword. Therefore, for each dataset group, the higher the relevance between the keyword and the starting dataset, and the higher the sum of the relevances of the datasets included in the dataset group, the greater the value of "diversity." The dataset suggestion unit can be configured to present dataset groups as search results in descending order of diversity. Specifically, regarding diversity V, for example, it can be calculated as follows.

[0030] V=Rs·(ΣAsi+1)…(2)

[0031] Here, Rs represents the relevance between the keyword and the starting dataset, Asi represents the relevance between each dataset in the dataset group selected for a starting dataset, and the sum represents the sum of all datasets in the dataset group.

[0032] However, within each dataset, it is sometimes possible to extract information about the region, time band, or period of the collected data. In such cases, when the regions, time bands, or periods of the collected data differ between multiple datasets, or when the overlap between these regions, time bands, or periods is small, it can sometimes be considered that their correlation will be low regardless of the similarity of their vector values. Therefore, in the structure of this invention, the dataset search unit can be configured to select datasets whose overlap with the starting dataset of the data collection region or collection period exceeds a predetermined value and whose correlation with the starting dataset exceeds a predetermined value as the searched datasets. Regarding the overlap of the data collection regions between datasets, for example, the overlap of the data collection regions of one dataset relative to the data collection regions of other datasets can be the ratio of the area of ​​the overlapping region between one dataset and other datasets in the data collection regions to the area of ​​the data collection region of one dataset. Furthermore, regarding the overlap of the data collection periods between datasets, it can be the ratio of the length of the time that one dataset overlaps with other datasets in the data collection periods to the length of the data collection period of one dataset.

[0033] Invention Effects

[0034] Thus, according to the system based on the present invention, by simply inputting a keyword, it is possible not only to search for datasets directly related to the input keyword, but also to search for datasets that are not directly related to the keyword but may be usable. Therefore, if a target-related keyword is input, a group of potentially usable datasets can be collected, which is advantageous because the suggested dataset may also include datasets that the searcher had not initially considered but could utilize due to associated images. According to the system of the present invention, when using datasets for various purposes, the method for determining or searching the location of each dataset is expected to allow reaching the desired dataset with less prior knowledge.

[0035] Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the invention. Attached Figure Description

[0036] Figure 1 This is a diagram schematically representing a computer that implements the dataset search system based on this embodiment.

[0037] Figure 2 This is a diagram that represents the structure of the dataset search system based on this implementation method in the form of a block diagram.

[0038] Figure 3 This is a flowchart representing the dataset search processing in the system based on this implementation.

[0039] Figure 4 This is a schematic diagram illustrating the association between datasets based on vector similarity stored in the database based on this embodiment.

[0040] Figure 5 This is a schematic diagram illustrating the links between datasets based on the correlation between the starting dataset (starting node) and the starting dataset in this implementation.

[0041] Symbol Explanation

[0042] 1-Computer main body, 2-Computer terminal, 3-Monitor, 4-Keyboard and mouse (input devices), 10-Dataset storage unit (search object dataset database), 11-Feature extraction unit, 12-Dataset vectorization processing and similarity calculation unit, 13-Data extraction unit during dataset collection, 14-Data collection area extraction unit, 15-Feature storage unit, 20-Keyword input unit, 30-Search network generation unit, 31-Starting node determination unit, 32-Associative link assignment unit, 40-Priority suggestion component determination unit, 41-Diversity calculation unit, 42-Component sorting unit, 50-Search result display unit. Detailed Implementation

[0043] Hereinafter, several preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same symbols denote the same parts.

[0044] Structure of computer devices

[0045] The dataset search system based on this embodiment can be performed according to... Figure 1 The program in the computer device 1, as illustrated in the example, is implemented as is commonly used in the art. The computer device 1 typically includes a CPU, a storage device, and input / output devices (I / O) interconnected via a bidirectional public bus. The storage device includes a memory PM storing the various programs used in the computational processing of this embodiment, a working memory WM used in the computation, and a data memory DM (5). Furthermore, the searcher's instructions to the computer device 1, the display of search results, and the output of other information are performed via a computer terminal device 2 connected to the computer device 1. The computer terminal device 2 typically includes a display 3 and input devices 4 such as a keyboard and mouse. When the program is started, the searcher can follow the program steps, according to the display on the display 3, use the input devices 4 to give various instructions and inputs to the computer device 1, and can visually confirm the computational status and results from the computer device 1 on the display 3. The dataset being searched can be stored in the data memory DM (5) provided in the computer device 1, or it can be stored in a cloud system accessible through any communication network.

[0046] Structure of a data search system

[0047] refer to Figure 2 Generally speaking, the dataset search system based on this embodiment consists of a dataset storage unit 10, a feature extraction unit 11, a feature storage unit 15, a keyword input unit 20, a search network generation unit 30, a priority suggestion component determination unit 40, and a search result display unit 50.

[0048] The dataset storage unit 10 is a database that stores datasets used as search objects. As described above, a dataset is a group of data recorded in CSV, JSON, or other formats, listing numerical or textual data by item. The content of the data can be related to any matter such as stock prices, weather forecasts, factory operations, personnel, or sales performance. Furthermore, the dataset storage unit 10 can also be located on a server accessible via a communication network. Moreover, the dataset storage unit 10 can store the datasets themselves, as well as information such as pointers to URLs within a local area network for each dataset.

[0049] The feature extraction unit 11 is configured to extract features from the dataset stored in the dataset storage unit 10. Features are vectors representing features extracted from each dataset, or arbitrary metrics such as vector similarity or distance between datasets. More specifically, in the feature extraction unit 11, a dataset vectorization processing and similarity calculation unit 12 can be first set up, in which the vectorization of each dataset stored in the dataset storage unit 10 is performed, and the similarity index value of the vector values ​​between each dataset, i.e., vector similarity, is calculated. As described above, the vectorization of the dataset can be achieved using any machine learning algorithm that converts arbitrary text data or natural language data into multidimensional numerical groups, i.e., vectors, such as TaBERT, TURL, etc. The vectors obtained from the dataset reflect the content of the dataset; therefore, the higher the similarity of the content between any two datasets, the closer their dataset vectors are in vector space. By referencing the proximity of the vectors, i.e., the similarity of the vectors between any two datasets, the degree of similarity or correlation between the content of the two datasets can be estimated. Therefore, in this embodiment, the vectorization processing and similarity calculation unit 12 calculates the similarity of vector values ​​(hereinafter referred to as "vector similarity") for all combinations of combinations of two datasets stored in the dataset storage unit 10. Furthermore, typically, the cosine similarity of the vectors of the two datasets can be used as the "vector similarity" between the two datasets.

[0050] Furthermore, the feature extraction unit 11 can be equipped with a data collection period extraction unit 13 and a data collection region extraction unit 14, which respectively extract information about the time band or period of data collection for each dataset and information about the region of the collected data. As will be explained later, the information about the time band or period and the information about the region extracted by the data collection period extraction unit 13 and the data collection region extraction unit 14 can be used when performing association between datasets.

[0051] The feature quantity storage unit 15 stores the feature quantities calculated or extracted by the feature quantity extraction unit 11, and refers to them when searching the keyword-based dataset described later.

[0052] The keyword input section 20 is configured to receive keywords initially set by the searcher in relation to the type, content, and purpose of the data analysis, etc., of the data they wish to search for. Keywords can be words, phrases, numbers, etc. Multiple keywords can be entered at once.

[0053] The search network generation unit 30 comprises: a starting node determination unit 31, which receives keywords input by the keyword input unit 20 and determines datasets with high relevance to the keywords as starting nodes; and an associative link assignment unit 32, which generates dataset groups (hereinafter referred to as "components") including content associated from the determined starting nodes. As described in detail later, the starting node determination unit 31 calculates the relevance of each dataset of the search target dataset to the keywords, detects datasets directly related to the keywords based on the relevance, and sets such datasets as starting nodes to become the starting points for dataset search. Furthermore, there may be one starting node, but usually two or more are set. As described in detail later, the associative link assignment unit 32 calculates the relevance of each dataset of the search target dataset from a starting node, and sequentially associates datasets with relevance exceeding a predetermined value with datasets including content associated from the content of the starting nodes (assigning associative links), forming these dataset groups, i.e., components, that have been assigned associative links.

[0054] The priority suggestion component determination unit 40 may include a diversity calculation unit 41 and a component sorting unit 42. In the search network generation unit 30, when multiple starting nodes are set and multiple components are formed, the priority suggestion component determination unit 40 determines the order in which components are preferentially suggested to the searcher. Therefore, the diversity calculation unit 41 calculates the diversity of each component, and the component sorting unit 42 determines the order in which components are suggested as search results based on the diversity. Furthermore, when only one starting node is set and only one component is generated, diversity calculation and priority setting of component suggestions are not performed.

[0055] The search results display unit 50 displays dataset groups to the searcher according to the priority order of the components in the priority suggestion component decision unit 40. Alternatively, when only one component is generated, it can be displayed directly.

[0056] System operation

[0057] (1) Summary

[0058] In the system of this embodiment, as described above, firstly, during the preparation phase of dataset search, each dataset of the dataset to be searched is vectorized, and all vector similarities of combinations of two datasets in the dataset to be searched are pre-calculated. Furthermore, when searching the dataset, as... Figure 3As shown in the flowchart, if the input keyword is appropriately determined by the searcher, the dataset with a high relevance to the keyword is identified as the starting node (S1), and associative links are assigned to datasets whose relevance to each of the identified starting nodes exceeds a predetermined value (S2), forming a component (S3). Then, if multiple components are formed, the diversity of each component is calculated (S4), and the components are sorted according to their diversity level, and search results are displayed (S5). Hereinafter, a series of processing operations in this embodiment will be described in sequence.

[0059] (2) Calculation of vector similarity in the dataset of the search object

[0060] As described above, in this embodiment, through the vectorization processing and similarity calculation unit 12, the datasets of all search objects stored in the dataset storage unit 10 are vectorized, and the vector similarity of the vector values ​​obtained through vectorization is calculated. All combinations of combinations of two datasets within the search object datasets are recorded in the vector similarity storage unit 15. With this structure, as... Figure 4 The diagram illustrates how the datasets (d1, d2, ...) of the search target dataset are correlated with each other based on their vector similarity to other datasets, i.e., their content proximity. Thus, by tracing back to other datasets based on the height of vector similarity of one dataset, datasets with similar content can be detected sequentially. The following keyword-based dataset search utilizes this structure, which allows for the sequential detection of datasets with similar content.

[0061] (3) Keyword input and starting node setting

[0062] In this embodiment, dataset searching begins by the searcher inputting arbitrary keywords into the keyword input unit 20. As described above, keywords can be appropriately set to words, phrases, numerical values, etc., corresponding to the type, content, and purpose of the data to be searched. If keywords are input, firstly, in the starting node determination unit 31, a dataset to be selected as the starting node is selected from the datasets stored in the dataset storage unit 10.

[0063] In selecting the starting node, the relevance between the keyword and each dataset can be calculated, and the starting node can be determined based on this relevance. In short, the relevance between the keyword and each dataset is a metric indicating the degree of correlation between the keyword and the content of each dataset. As mentioned above, it can be calculated in several ways. Specifically, for example, in one approach, the relevance can be set as the more times the keyword or its similar words (which can be determined by any method based on the keyword) are included in the data item name or the data itself, the higher the value. Alternatively, in another approach, the relevance can be set as the closer the keyword, determined by any language processing algorithm, is to the content of the dataset, the higher the relevance. Therefore, for each dataset, the higher the relevance, the higher the direct relevance of the keyword. Datasets with high relevance are typically datasets hit through typical keyword searches. Furthermore, the starting node can be set to datasets with relevance exceeding a predefined value or datasets included from appropriately defined numbers within datasets with high relevance. That is, multiple starting nodes may be set. And, the datasets set at the aforementioned starting nodes are datasets directly related to the keyword; therefore, through the above processing, datasets directly related to the keyword can be detected.

[0064] (4) The formation of components centered on the starting node (associating them with associative links)

[0065] As described above, if a dataset directly related to the keyword is determined, i.e., the starting node, then a search for datasets related to the starting node is performed. Thus, it is possible to search for datasets that are not directly related to the keyword but are directly related to it, and that can be used for data analysis and other purposes.

[0066] In the search of datasets related to the starting node, specifically, firstly, as explained in the summary of the invention, the correlation between each dataset and the starting node is calculated, which is an indicator of the amount of content associated with the expected starting node included in each dataset. Regarding this, as described above, the datasets of search objects stored in the dataset storage unit 10 calculate vector similarity with each other. Based on the vector similarity, highly correlated datasets can be traced sequentially from one dataset. Therefore, the correlation can be set such that datasets highly correlated with the starting node can be traced based on this vector similarity. Thus, in this embodiment, specifically, the correlation can be calculated, for example, using the above equation (1). The correlation based on equation (1) is set as follows: Figure 4In the dataset group of the search objects shown, the longer the path tracing back from the starting node to the datasets with high vector similarity (the more datasets traversed), and the lower the vector similarity between datasets, the smaller the value. Therefore, it is expected that the content of datasets with a correlation exceeding a predefined value will have a significantly higher correlation with the content of the starting node, and will generate significantly more related content from the content of the starting node. Thus, by detecting datasets with a correlation exceeding the predefined value, it is possible to detect datasets that are directly related to the keywords but not directly correlated with them, and which can be used for data analysis and other purposes.

[0067] The search process for datasets whose relevance to the starting node exceeds a specified value can be performed in any manner. For example, firstly, the following process can be repeated: select datasets whose vector similarity to a starting node is above a reasonably set specified value and assign them associative links (the relevance is equal to the vector similarity to the starting node). Next, select datasets whose vector similarity to each dataset with associative links is above a reasonably set specified value, calculate the relevance of these datasets, select datasets whose relevance is above the specified value and assign them associative links. Further, select datasets whose vector similarity to each of these datasets with associative links is above a reasonably set specified value, calculate the relevance of these datasets, select datasets whose relevance is above the specified value and assign them associative links, thus forming a dataset group, or component, whose relevance to the starting node exceeds the specified value.

[0068] Figure 5 This is a schematic diagram illustrating the process from the formation of the component from the aforementioned keywords. (Reference) Figure 5 First, starting nodes T1 and T2 with a relevance value r of at least a specified value are selected from the keyword KW. Next, the relevance to other datasets is calculated for each starting node. For example, in starting node T1, datasets D1, D2, and D5 with a vector similarity X of at least a specified value are selected and associated with each other. Here, the relevance As of the datasets with a vector similarity X of at least a specified value to starting node T1 is equal to the vector similarity X. Next, among datasets D1, D2, and D5 that have been associated with each other, datasets D3, D6, etc., with a relevance As of at least a specified value to starting node T1 in the datasets with a vector similarity X of at least a specified value are selected and associated with each other. Further, the same process is performed on datasets D3 and D6 that have been associated with each other, and dataset D4 with a relevance As of at least a specified value from starting node T1 is selected and associated with each other. Thus, if all datasets with a relevance of at least a specified value from starting node T1 are selected, they form a group of components C1. The same process can also be performed in other starting node T2 to form another component C2.

[0069] (5) Determining the order of component prompts

[0070] Based on the above processing, when multiple starting nodes are set, components are formed for each starting node. All components of these components can be suggested as search results, but it is convenient to suggest components sequentially from those that seem more useful. Therefore, in this embodiment, the suggestion order of components can be determined based on the degree of relevance to the keywords and the degree of relevance to the datasets included in each component.

[0071] Specifically, it can be configured to calculate the diversity of each component formed in the previous processing, which is more related to the keywords and the starting dataset, and the higher the sum of the relatedness of the datasets included in the dataset group, the greater the value of diversity, and then indicate the components in descending order of diversity. Regarding diversity, for example, it can be calculated by equation (2) as described in the invention summary column.

[0072] (6) Selection of datasets based on data collection area and data collection period

[0073] As mentioned above, the data collection area, time band, or period is sometimes explicitly recorded in each dataset. Furthermore, when a dataset is used for various purposes, sometimes the areas, time bands, or periods of data collected from multiple datasets are different, or when the overlap between these areas, time bands, or periods is small, it can be assumed that their correlation will be low regardless of the similarity of their vector values. Therefore, in this embodiment, the data collection area or data collection period recorded in the dataset can be extracted, and datasets with low overlap between their data collection areas or data collection periods can be excluded from the component when forming the component.

[0074] Specifically, firstly, regarding the data collection period, the data collection period extraction unit 13 of the feature extraction unit 11 extracts and records information about the data collection time band or period as a feature quantity of the dataset stored in the dataset storage unit 10. Furthermore, when forming a component, when calculating the correlation degree, if the time overlap with the starting node (or adjacent datasets) is lower than a properly set predetermined value, an associative link may not be assigned regardless of the correlation degree value, and the component may be excluded. The degree of time overlap can be any index value that measures the degree of overlap during the data collection period; for example, it can be the ratio of the length of time overlapping with the starting node (or adjacent datasets) in the data collection period to the length of the data collection period for each dataset. When information about the data collection period of a dataset is unavailable, it may not be excluded from the component.

[0075] Similarly, regarding the data collection area, the data collection area information extracted and recorded by the data collection area extraction unit 14 of the feature extraction unit 11 is stored as a feature of the dataset in the dataset storage unit 10. Furthermore, when forming a component, when calculating the correlation degree, if the overlap between the region and the starting node (or adjacent datasets) is lower than a predefined value, an associative link may not be assigned regardless of the correlation degree value, and the component may be excluded. The degree of region overlap can be any index value that measures the overlap of the data collection areas. For example, the degree of region overlap can be the ratio of the area of ​​the data collection area of ​​each dataset and the data of the starting node (or adjacent datasets) to the area of ​​the data collection area of ​​the starting node (or adjacent datasets). When information about the data collection area of ​​a dataset cannot be obtained, it may not be excluded from the component.

[0076] Thus, based on the system of this embodiment described above, if keywords related to the objective are input, a collection of potentially usable datasets can be gathered. The collected and suggested datasets also include datasets that the searcher did not initially anticipate but can utilize through associated images. As an example of how this system can be used, it can be applied to improving the efficiency of data utilization within a company. The dataset association system of this embodiment is an important means of effectively utilizing data in various activities, reducing unnecessary costs and expanding utilization methods.

[0077] The above description relates to the embodiments of the present invention, but those skilled in the art can easily make numerous modifications and changes. The present invention is not limited to the embodiments illustrated above, and can obviously be applied to various devices without departing from the concept of the present invention.

Claims

1. A system for searching a dataset, characterized in that, include: A database unit is configured to store multiple datasets; A dataset vectorization unit is configured to compute vector values ​​of the dataset; The vector value similarity calculation unit is configured to calculate the similarity of vector values ​​between multiple datasets. The keyword receiving unit is configured to receive keywords. A dataset search unit is configured to search for datasets related to the keyword from the datasets stored in the database unit based on the similarity of the vector values. and The dataset suggestion unit is configured to suggest the datasets retrieved by the dataset search unit as search results.

2. The system according to claim 1, characterized in that, The dataset search unit is configured as follows: Based on the relevance to the keywords, a starting dataset is selected from the datasets stored in the database unit; and Based on the similarity of the vector values ​​of each dataset stored in the database unit with the starting dataset, a dataset related to the keyword is searched.

3. The system according to claim 2, characterized in that, The dataset search unit is configured to select datasets whose correlation with the starting dataset exceeds a predetermined value as the searched datasets.

4. The system according to claim 3, characterized in that, The dataset search unit is configured as follows: Select multiple datasets as the starting dataset; Select a group of datasets whose correlation with each dataset in the starting dataset exceeds the specified value; and Calculate diversity for each of the dataset groups. The dataset suggestion unit is configured to suggest the dataset group as the search result in descending order of diversity.

5. The system according to claim 2, characterized in that, The dataset search unit is configured to select datasets whose overlap with the data collection area or the starting dataset during the collection period exceeds a predetermined value and whose correlation with the starting dataset exceeds a predetermined value as the searched datasets.

Images