Data usage support system and data usage support method

Abstract

To reduce complexity of formatting for using data, thereby supporting efficient use of the data.SOLUTION: A system for supporting the use of data is configured to: store tabular-format data which is data with information described in a tabular form; sequentially select a row of the tabular-format data; determine, for each of the selected rows, a similarity between the row and a row following thereto; estimate, based on the determined similarities, a data item row in which a data item name is to be described in the tabular-format data; and generate, based on the estimated data item row, formatted data which is obtained by formatting the tabular-format data into data of a format which can be read by software that uses the tabular-format data.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a data utilization support system and a data utilization support method. [Background technology]

[0002] In recent years, organizations such as companies and government agencies have been making progress in utilizing various types of data, including open data. However, this data is often not in a format that can be read by the system that is importing it, and in such cases, the data content must be reformatted (adjusted) so that it can be read by the system.

[0003] As a mechanism for reducing the complexity of formatting data when using it, for example, Patent Document 1 describes a data reading device configured to relax requirements for the format of table data from which data can be read and to enable data reading in a flexible manner in response to table data in various formats. The data reading device acquires input item names given to a processing tool for the type of data to be processed by the processing tool, acquires information indicating the arrangement direction of data groups having the same attribute among the table data and arranged linearly for each attribute, searches for cells in which character string data indicating the input item names are arranged, and, starting from the cell found by the search, reads data arranged in each cell arranged in the arrangement direction indicated by the acquired information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-086650 Summary of the Invention [Problem to be solved by the invention]

[0005] The data reading device described in Patent Document 1 acquires input item names of a processing tool that processes table data, creates a mapping table that shows the correspondence between the acquired input item names and the header item names of the table data, and reads data from the table data using the mapping table. Therefore, for the target processing tool, the user must acquire input item names and set a mapping table in advance. Furthermore, acquisition of input item names and setting of the mapping table must be done for each processing tool, and the mechanism described in this document is not sufficiently versatile.

[0006] The present invention has been made based on this background, and aims to provide a data usage support system and a data usage support method that can reduce the complexity of formatting data when using it and support the efficient use of data. [Means for solving the problem]

[0007] One aspect of the present invention for achieving the above object is a system for supporting data utilization (data utilization support system), which is configured using an information processing device, stores tabular data, which is data in which information is described in a tabular format, selects rows of the tabular data in order, and for each selected row, calculates a similarity between the row and the row following the row, infers a data item row, which is a row in the tabular data in which a data item name is written, based on the calculated similarity, and generates formatted data, which is data obtained by formatting the tabular data into data in a format that can be read by software that utilizes the tabular data, based on the inferred data item row, Based on the similarity, a data row is estimated in the tabular data, which is a row in which data corresponding to a data item is written; based on the estimated data row, formatted data is generated, which is data obtained by formatting the tabular data into a format that can be read by software that uses the tabular data; based on the similarity, other rows in the tabular data that are neither data item rows nor data rows are estimated; and by deleting the estimated other rows, formatted data is generated, which is data obtained by formatting the tabular data into a format that can be read by software that uses the tabular data.

[0008] Other problems and solutions disclosed in the present application will be made clear in the detailed description and drawings. [Effects of the Invention]

[0009] According to the present invention, it is possible to reduce the complexity of formatting data when using it, and to support efficient use of data. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a schematic configuration of a data usage support system according to a first embodiment. [Figure 2] This is an example of an estimation standard. [Figure 3A] FIG. 10 is a diagram showing an example in which the type of the target row is estimated to be "other row." [Figure 3B] FIG. 10 is a diagram showing an example in which the type of the target row is estimated to be a “data item row.” [Figure 3C] FIG. 10 is a diagram showing an example in which the type of the target row is estimated to be a "data row." [Figure 4] FIG. 10 is a diagram illustrating another example of a method for estimating the type of a row. [Figure 5A] 10 is an example of an estimation result display screen (when a single data item is estimated). [Figure 5B] 10 is an example of an inference result display screen (when a data item row is not inferred). [Figure 5C] 10 is an example of an estimation result display screen (when multiple data item rows are estimated). [Figure 6] This is an example of data after formatting. [Figure 7] FIG. 2 is a diagram showing the main functions of each device in the data utilization support system. [Figure 8] FIG. 4 is a sequence diagram illustrating a data registration process according to the first embodiment. [Figure 9] 10 is a flowchart illustrating a row-by-row type estimation process. [Figure 10] FIG. 10 is a sequence diagram illustrating a data usage process. [Figure 11] 1 is an example of an information processing device used in the configuration of a data utilization support system. [Figure 12] FIG. 10 is a diagram illustrating a schematic configuration of a data utilization support system according to a second embodiment. [Figure 13]FIG. 10 is a sequence diagram illustrating a data registration process according to the second embodiment. [Figure 14] 10 is a flowchart illustrating a row-by-row type estimation process according to the second embodiment. [Figure 15] 10 is an example of a classification history. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings as appropriate. The following description and drawings are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0012] In the following description, the same or similar components may be denoted by the same reference numerals, and redundant explanations may be omitted. Also, in the following description, the letter "S" added before a reference numeral indicates a processing step.

[0013] In the following explanation, data in which information is written in the form of a table (data consisting of a set of description fields (hereinafter referred to as "cells") of character information (characters, character strings, numbers, number strings, etc.) specified by specifying rows and columns) will be referred to as "tabular data".

[0014] [First embodiment] FIG. 1 shows a schematic configuration of an information processing system (hereinafter referred to as "data utilization support system 1") described as a first embodiment. As shown in the figure, the data utilization support system 1 includes a data utilization support device 100, a data registration device 2, and a data utilization device 3. Each of these devices is configured using an information processing device (computer). Both the data registration device 2 and the data utilization device 3 are configured using the data utilization support device 10. 0 and the communication network 5 in a state where two-way communication is possible. The communication network 5 may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), an Internet The internet, public communication networks, dedicated lines, etc. At least two of the data utilization support device 100, data registration device 2, and data utilization device 3 may be realized by a common information processing device.

[0015] The users of the data utilization support system 1 are not necessarily limited, but may be, for example, departments in organizations such as companies and government agencies that perform various analytical tasks using data provided by third parties, such as open data. The various functions of the data utilization support device 100 may also be provided as services on the Internet (e.g., as SaaS (Software as a Service)).

[0016] The data utilization support device 100 formats tabular data (hereinafter referred to as "registered data 21") registered via the data registration device 2 into data (hereinafter referred to as "formatted data 31") that can be read by software (hereinafter referred to as "utilization software"), such as spreadsheet software, used by a user who uses the registered data 21, to identify and read rows (hereinafter referred to as "data item rows") in which names of data items (hereinafter referred to as "data item names") are stored. In response to a provision request sent from the data utilization device 3, the data utilization support device 100 provides the user with the formatted data 31 specified in the provision request. The formatted data 31 is data written, for example, in a format proprietary to the utilization software or in CSV (Comma Separated Value) format.

[0017] The registered data 21 includes the above-mentioned data item rows and one or more rows in which the values ​​of the data items are stored (hereinafter referred to as "data rows"). The registered data 21 may also include rows that are neither "data item rows" nor "data rows" (for example, rows in which descriptions are stored for the purpose of improving human readability, etc.; hereinafter referred to as "other rows").

[0018] Note that the registered data 21 may not only have one or more data items listed in a column direction, but may also have one or more data items listed in a row direction, but in the latter case, it can be easily converted into the former type and handled by performing a process of swapping rows and columns (transposition process). Therefore, in the following, we will explain the case where the registered data 21 is of the former type as an example, and will omit explanations of the case where it is of the latter type.

[0019] The data utilization support device 100 estimates the type of each row ("data item row," "data row," "other row") in the registered data 21, and formats the registered data 21 based on the estimated row type to generate formatted data 31. For example, the data utilization support device 100 sequentially selects rows from the registered data 21, starting with the first row (the top row), and for each selected row, calculates the similarity between the selected row (hereinafter referred to as the "target row") and the rows following the target row, and performs the above estimation by comparing the calculated similarity with a preset estimation standard. The above similarity may be, for example, a similarity based on the similarity of information written in the same column of the target row and the rows following the target row (hereinafter referred to as "storage value similarity"), or a similarity based on the similarity of the number of columns of the target row and the rows following the target row (hereinafter referred to as "column number similarity"). The above-mentioned "storage value similarity" is calculated based on, for example, the similarity of the characters or character strings stored in the cells of the same column in the target row and in each of the following rows of the target row, and the similarity of the parts of speech and types (characters, numbers) of the words stored in the cells.

[0020] An example of the estimation criterion is shown in Figure 2. The illustrated estimation criterion 111 estimates the target row as an "other row" when the "storage value similarity" is low and the "number of columns similarity" is low, as a "data item row" when the "storage value similarity" is low and the "number of columns similarity" is high, and as a "data row" when the "storage value similarity" is high and the "number of columns similarity" is high. The degree of similarity is determined by comparing the obtained similarity with a preset threshold (whether the similarity is equal to or greater than the threshold or less than the threshold). Therefore, by adjusting the threshold according to the state of the registered data 21, it is possible to improve the accuracy of estimating the type of the target row.

[0021] 3A to 3C show examples of estimation of the type of target row of registered data 21 based on estimation criterion 111. FIG. 3A shows a case where the first row of the exemplified registered data 21 is selected as the target row. FIG. 3B shows a case where the third row of the exemplified registered data 21 is selected as the target row. FIG. 3C shows a case where the fourth row of the exemplified registered data 21 is selected as the target row.

[0022] For example, in Figure 3A, the cell value "Title:" in column A of the target row (row 1) and the "Publication" in each subsequent row The similarity between "Opening date:", "Survey year", "2019 fiscal year", etc. is low, and the value of the cell in column B of the target row Between "Regional Income" and each subsequent line, such as "February 9, 2022," "Region," and "Hokkaido Sapporo City." The similarity between the cell value " " in column C of the target row and the values ​​" ", "Income (thousand yen)", "123456789", etc. in the subsequent rows is low. Also, the number of columns in the target row is "2". However, the number of columns in each subsequent row is "3," and the similarity between the two is low. Therefore, in this example, the data utilization support device 100 infers that the type of the target row is "other row" because the "storage value similarity" is low and the "column number similarity" is low.

[0023] For example, in Figure 3B, the similarity between the cell value "Survey Year" in column A of the target row (row 3) and each subsequent row's "2019," "2019," etc. is low, the similarity between the cell value "Region" in column B of the target row and each subsequent row's "Hokkaido Sapporo City," "Hokkaido Hakodate City," etc. is low, and the similarity between the cell value "Income (thousand yen)" in column C of the target row and each subsequent row's "1234567890," "222555888," etc. is low. The similarity is low. Also, the number of columns in the target row is "3," and the number of columns in each subsequent row is also "3," so the similarity between the two is high. Therefore, in this example, the data utilization support device 100 infers that the type of the target row is a "data item row" because the "storage value similarity" is low and the "column number similarity" is high.

[0024] For example, in Figure 3C, the similarity between the cell value "2019" in column A of the target row (row 4) and each subsequent row's "2019," "2019," etc. is high, and the similarity between the cell value "Hokkaido Sapporo City" in column B of the target row and each subsequent row's "Hokkaido Sapporo City," "Hokkaido Hakodate City," etc. is low. The similarity between the cell value "1234567890" in column C of the target row and the values ​​"1234567890", "222555888", etc. in each subsequent row is high. Also, the number of columns in the target row is "3", and the similarity between the cell value "1234567890" in column C of the target row and the values ​​"1234567890", "222555888", etc. in each subsequent row is high. The number of columns in is also "3," so the similarity between the two is high. Therefore, in this example, the data utilization support device 100 infers that the type of the target row is a "data row" because the "storage value similarity" and "column number similarity" are both high.

[0025] The above-described method of estimating row types using "storage value similarity" and "column number similarity" is merely an example, and the data utilization support device 100 may estimate row types using methods other than this. For example, the data utilization support device 100 may generate a machine learning model trained using training data in which information based on similarity calculated based on a row of tabular data (e.g., registered data 21 for which type estimation has been performed previously) and the rows following that row is used as an explanatory variable, and information indicating the type of the row is used as a target variable, and then input the description of each row of registered data 21 into the machine learning model to perform the above estimation. When a machine learning model is used, improvement in the accuracy of row type estimation can be expected as the model is repeatedly trained.

[0026] A specific example of a mechanism for estimating the type of a line using a machine learning model is shown below. In this example, the data utilization support device 100 estimates the type of a line by combining the normalized edit distance (normalized Levenshtein distance), morphological analysis, and machine learning. The type of each row of the registered data 21 is estimated by the procedure shown in (also see FIG. 4). <Procedure> 1. For each column, find the maximum similarity of the stored values ​​between the target row and each subsequent row. The similarity is found using the following method. 1-1. Normalized edit distance (high similarity if strings are similar) 1-2. Morpheme cosine similarity (high similarity if the parts of speech contained are similar) 2. For each column, find the average of the highest values ​​above. 3. Calculate the average difference between the number of columns in the "Data Item Rows" and the number of columns in the "Data Rows." 4. Generate training data based on past data (training data with the above three values ​​as explanatory variables and the estimation results as the target variable), generate a machine learning model trained using the generated training data, and use the generated machine learning model to estimate the type of each row.

[0027] The data utilization support device 100 presents the results of the row type estimation ("other row," "data item row," "data row") to the user (e.g., the user who registered the registration data 21) via the user interface, and prompts the user to confirm or correct the results.

[0028] 5A to 5C show examples of screens (hereinafter referred to as "estimated result display screens 500") that the data utilization support device 100 displays when the above confirmation and correction is performed.

[0029] 5A shows an example of an inference result presentation screen 500 displayed by the data utilization support device 100 when only one (single) "data item row" is inferred for the registered data 21. The illustrated inference result presentation screen 500 has a registered data display field 511, an inference result display field 512, an inference result correction field 513, a message display field 514, and a data registration button 515.

[0030] The registered data display field 511 displays the contents of the registered data 21. In this example, the data utilization support device 100 estimates the third line of the registered data 21 as the "data item line." The user can edit the data item names of each column of that line in the registered data display field 511.

[0031] The estimation result for each row type is displayed in the estimation result display field 512. The estimation result correction field 513 is in the form of a pull-down menu, and if the estimation result of the data utilization support device 100 is incorrect, the user can correct the type of each row by selecting a type ("Other Rows," "Data Item Rows," or "Data Rows") from the pull-down menu.

[0032] A message regarding the estimation result is displayed in the message display field 514. When the user operates the data registration button 515, the data utilization support device 100 generates and stores the shaped data 31 based on the settings made on the screen.

[0033] Fig. 5B shows an example of an inference result presentation screen 500 displayed by the data utilization support device 100 when no "data item row" is found in the registered data 21. The screen configuration is the same as in Fig. 5A.

[0034] In this example, since no "data item rows" were found, the data utilization support device 100 displays a data item name input field in the registration data display field 511 and prompts the user to set the data item name in the message display field 514. Note that on this screen, the user cannot operate the data registration button 515 unless all data item names corresponding to each column of the "data rows" are entered.

[0035] FIG. 5C shows an example of an inference result display screen 500 displayed by the data utilization support device 100 when multiple "data item rows" (two in this example) are inferred for the registered data 21. The screen configuration is the same as in FIG. 5A.

[0036] In this example, multiple "data item lines" were estimated, so the data utilization support device 100 prompts the user to set the estimation result correction field 513 so that the "data item line" becomes singular. Note that on this screen, the user cannot operate the data registration button 515 unless the "data item line" is corrected to become singular.

[0037] 6 shows an example of the reformatted data 31 generated by the data utilization support device 100 based on the results of the estimation of the type of each line as described above (results confirmed or edited by the user on the estimation result presentation screen 500). The reformatted data 31 shown in the example has the contents of the registered data 21 with "other lines" (or the description of "other lines") deleted.

[0038] 7 is a diagram showing the main functions of each device that constitutes the data utilization support system 1. Below, the functions of each device will be explained with reference to this diagram.

[0039] As shown in the figure, the data utilization support device 100 comprises the functions of a storage unit 110, a data registration unit 120, and a data provision unit .

[0040] Of these, the storage unit 110 stores the registered data 21, the shaped data 31, and the estimation criterion 111.

[0041] As shown in the figure, the data registration unit 120 has the functions of a registration data receiving unit 121, a line-to-line similarity calculation unit 122, a line-by-line type estimation unit 123, an estimation result presentation unit 124, a correction result receiving unit 125, and a formatted data generation unit 126.

[0042] Of these, the registration data receiving unit 121 receives the registration data 21 sent from the data registration device 2 .

[0043] The line similarity calculation unit 122 calculates the above-mentioned similarity for each line of the registered data 21.

[0044] The line type estimation unit 123 estimates the type of each line of the registered data 21 based on the calculated similarity.

[0045] The inference result presentation unit 124 presents the inference result of the type of each row to the user via the data registration device 2 (for example, the data registration device 2 displays the screens exemplified in FIGS. 5A to 5C).

[0046] The correction result receiving unit 125 receives from the data registration device 2 the details of the correction made by the user to the estimation result.

[0047] The shaped data generating unit 126 generates shaped data 31 based on the estimation result (or the estimated result after correction if the user corrects the estimated result).

[0048] As shown in the figure, the data providing unit 130 includes a data request receiving unit 131 and a data transmitting unit 132.

[0049] Of these, the data providing unit 130 receives a request for providing the shaped data 31 from the data utilization device 3 .

[0050] In response to the provision request, the data transmission unit 132 transmits the reformed data 31 specified in the provision request to the data utilization device 3.

[0051] As shown in the figure, the data registration device 2 has the functions of a registration data receiving unit 211, a registration data transmitting unit 212, and an estimation result editing unit 213.

[0052] Of these, the registration data receiving unit 211 receives the registration data 21 input from the user.

[0053] The registration data transmission unit 212 transmits the accepted registration data 21 to the data utilization support device 100 .

[0054] The inference result editing unit 213 presents to the user the inference results for each row type sent from the data utilization support device 100. The inference result editing unit 213 also accepts corrections to the inference results from the user (for example, by displaying the screens exemplified in FIGS. 5A to 5C to accept corrections from the user), and transmits the contents of the accepted corrections to the data utilization support device 100 as correction results.

[0055] As shown in the figure, the data utilization device 3 includes the functions of a data provision request transmission unit 311, a formatted data reception unit 312, and a data utilization unit 313.

[0056] Of these, the data provision request sending unit 311 sends a request to provide the formatted data 31 to the data utilization support device 100 .

[0057] The formatted data receiving unit 312 receives and stores the formatted data 31 sent from the data utilization support device 100.

[0058] The data utilization unit 313 is a function realized by, for example, spreadsheet software, and performs various processes using the formatted data 31 .

[0059] 8 is a sequence diagram illustrating an example of processing (hereinafter referred to as "data registration processing S800") performed in the data utilization support system 1 when a user (such as a data provider) registers registration data 21 in the data utilization support device 100 via the data registration device 2. The data registration processing S800 will be described below with reference to this diagram.

[0060] First, the user inputs (uploads) the registration data 21 into the data registration device 2 (S811).

[0061] When the registration data 21 is input, the data registration device 2 transmits the input registration data 21 to the data utilization support device 100 (S812).

[0062] The data utilization support device 100 receives and stores the transmitted registered data 21 (S813). Then, the data utilization support device 100 estimates the type of each row by calculating the aforementioned similarity of the registered data 21 and comparing it with the estimation standard 111 (S814), and transmits the estimation results to the data registration device 2 (S815).

[0063] Upon receiving the estimation results, the data registration device 2 displays the contents and prompts the user to confirm or correct them (S816), accepts the confirmation or correction results (S817), and transmits the accepted results to the data utilization support device 100 (S818).

[0064] Upon receiving the confirmation result or the correction result, the data utilization support device 100 generates and stores the reshaped data 31 (S819).

[0065] 9 is a flowchart illustrating the details of the process (hereinafter referred to as "row-by-row type inference process S814") performed by the data registration unit 120 in S814 of the data registration process S800 in Fig. 8. Below, the row-by-row type inference process S814 will be described with reference to this figure.

[0066] The processing of S911s to S911e shown in the figure is a loop processing in which the data registration unit 120 selects target rows one by one in order from the first row of the registration data 21 and repeats the processing.

[0067] First, in S912, the data registration unit 120 compares the target line with the subsequent line to find the similarity (S912).

[0068] In S913, the data registration unit 120 compares the calculated similarity with the estimation criterion 111 to estimate the type of the target row.

[0069] When the above loop processing S911s to S911e is completed, the data registration unit 120 outputs the estimation result of the type of each row of the registered data 21 (S914).

[0070] 10 is a sequence diagram illustrating an example of processing (hereinafter referred to as "data utilization processing S1000") performed in the data utilization support system 1 when a user (such as a data user) acquires and utilizes the formatted data 31 via the data utilization device 3. The data utilization processing S1000 will be described below with reference to the same diagram.

[0071] First, the user inputs information specifying the shaped data 31 to be used into the data utilization device 3 (S1011).

[0072] Upon receiving the above input, the data utilization device 3 transmits a request to provide the specified formatted data 31 to the data utilization support device 100 (S1012).

[0073] When the data utilization support device 100 receives the request for provision, it searches for the specified formatted data 31 (S1013) and transmits it to the data utilization device 3 (S1014).

[0074] When the data utilization device 3 receives the shaped data 31, it outputs the received shaped data 1015 (or passes it to the data utilization unit 313) (S1015).

[0075] As described above, in the data utilization support system 1 of the first embodiment, the data utilization support device 100 automatically infers the type of each line in the registered data 21 and generates the formatted data 31 based on the inferred result. This allows the user to efficiently utilize data that includes descriptions that are inferred to be "other lines" without having to perform tedious formatting work. Furthermore, if the inferred result of the type of each line needs to be corrected, the user can efficiently correct the data using the user interface provided by the data utilization support system 1. In this way, the data utilization support system 1 of the first embodiment reduces the complexity of formatting when using data written in various formats, and supports efficient data utilization.

[0076] <Example of information processing device> FIG. 11 shows an example of an information processing device used in the configuration of the data utilization support system 1 (data utilization support device 100, data registration device 2, data utilization device 3) described above.

[0077] The illustrated information processing device 10 includes a processor 11, a main storage device 12 (memory), an auxiliary storage device 13 (external storage device), an input device 14, an output device 15, and a communication device 16. These are communicably connected via a bus, a communication cable, or the like. Examples include personal computers, server devices, smartphones, tablets, office computers, and general-purpose computers (mainframes).

[0078] The information processing device 10 may be realized, in whole or in part, by using virtual information processing resources provided using virtualization technology, process space separation technology, or the like, such as a virtual server provided by a cloud system. Also, all or in part of the functions provided by the information processing device 10 may be realized, for example, by a service provided by a cloud system via an API (Application Programming Interface), or the like. All or part of the functions provided by the information processing device 10 may be provided as, for example, SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service), etc. It may be realized by using the Service as a Service.

[0079] The processor 11 may be, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or the like. It is composed of chips such as Gate Array (Gate Array), ASIC (Application Specific Integrated Circuit), and AI (Artificial Intelligence) chips.

[0080] The main memory device 12 is a device used by the processor 11 when executing a program, and is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), or a non-volatile memory (NVRAM (Non Volatile RAM)). The various functions realized in each component of the data utilization support system 1 are realized by each processor 11 reading out programs and data stored (memorized) in the auxiliary memory device 13 into the main memory device 12 and executing them.

[0081] The auxiliary storage device 13 is a device for storing programs and data, and may be, for example, an SSD (Solid State Drive), a hard disk drive, or an optical storage device (CD (Compact Disc) The auxiliary storage device 13 can be configured with a non-transitory storage medium read / write device such as a digital video card (Digital Versatile Disc), a DVD (Digital Versatile Disc), a storage system, an IC card, an SD card, or an optical storage medium, a non-transitory storage area of ​​a cloud server, etc. Programs and data can be read into the auxiliary storage device 13 from other information processing devices equipped with non-transitory storage media or non-transitory storage devices via a storage medium reader or a communication device 16. The programs and data stored in the auxiliary storage device 13 are read into the main storage device 12 as needed.

[0082] The input device 14 is an interface that accepts input of information from the outside, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a pen-input tablet, a voice input device, or the like.

[0083] The output device 15 is an interface that outputs various information such as the progress of processing and the results of processing to the outside. The output device 15 is, for example, a display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphic card, etc.) that visualizes the various information described above, a device that converts the various information described above into audio (audio output device (speaker, etc.)), or a device that converts the various information described above into text (printer, etc.). Note that, for example, the information processing device 10 may be configured to input and output information to and from other devices via the communication device 16.

[0084] The input device 14 and the output device 15 constitute a user interface that realizes interactive processing with the user (receiving information, providing information, etc.).

[0085] The communication device 16 is a device that realizes communication with other devices. The communication device 16 is a wired or wireless communication interface that realizes communication with other devices via the communication network 5, and may be, for example, a NIC (Network Interface Card), a wireless communication module, or the like. modules, USB modules, etc.

[0086] The information processing device 10 may be equipped with, for example, an operating system, a file system, a DBMS (DataBase Management System) (relational database, NoSQL, etc.), a KVS (Key-Value Store), etc.

[0087] [Second embodiment] In the data utilization support system 1 of the first embodiment, the data utilization support device 100 inferred the type of line by focusing on the similarity between each line of the registered data 21 and the following lines, but the data utilization support device 100 of the second embodiment further infers the type of line by using descriptions of lines previously classified as "data item lines" in the registered data 21 (results of the user confirming or correcting the inference results; hereinafter referred to as "classification results"). Note that the basic configuration of the data utilization support system 1 of the second embodiment is the same as that of the first embodiment, so the following explanation will focus on the differences from the first embodiment.

[0088] 12 is a diagram showing a schematic configuration of a data utilization support system 1 according to the second embodiment. As shown in the figure, the data utilization support system 1 according to the second embodiment differs from the first embodiment in that the storage unit 110 of the data utilization support device 100 further stores the classification results as classification history 112, and the data registration unit 120 further estimates the type of row using the classification history 112.

[0089] 13 is a sequence diagram illustrating the data registration process S800 of the second embodiment. The data registration process S800 of the second embodiment is similar to the data registration process S800 of the first embodiment, but differs from the data registration process S800 of the first embodiment in that the data registration unit 120 further estimates the type of row using the classification history 112 in S814, and in that the data registration unit 120 adds a description of the row classified as a "data item row" to the classification history 112 in S820.

[0090] Fig. 14 is a flowchart illustrating the details of the process (hereinafter referred to as "row-by-row type inference process S814") performed by the data registration unit 120 of the second embodiment in S814 of the data registration process S800 in Fig. 12. Below, the row-by-row type inference process S814 of the second embodiment will be described with reference to Fig. 14.

[0091] The processing of S1411s to S1411e shown in the figure is a loop process in which the data registration unit 120 selects target rows one by one in order from the first row of the registration data 21 and repeats the process.

[0092] First, in S1412, the data registration unit 120 compares the target line with the subsequent line to find the similarity (hereinafter referred to as "first similarity").

[0093] Next, the data registration unit 120 compares the first similarity with the estimation criterion 111 to estimate the type of the target row (S1413).

[0094] Next, the data registration unit 120 determines whether the inference result in S1413 is a "data item row" (S1414). If the inference result is a "data item row" (S1414: YES), processing of the current target row ends and processing of the next target row begins. If there are no subsequent rows, the loop is exited. On the other hand, if the inference result is not a "data item row" (S1414: NO), processing proceeds to S1415.

[0095] In S1415, the data registration unit 120 compares the target row with the classification history 112 to find the similarity (hereinafter referred to as "second similarity").

[0096] 15 shows an example of the classification history 112. The data registration unit 120 calculates the second similarity as the similarity between the description of the target row and the description of the row estimated to be a "data item row" in the classification history 112. The data registration unit 120 calculates the second similarity based on, for example, the similarity of the character information (characters, character strings, numbers, number strings, etc.) stored in the cells of each row, the similarity of the parts of speech and types (characters, numbers) of the words stored in the cells of each row, etc.

[0097] Returning to FIG. 14, in S1416, the data registration unit 120 determines whether the second similarity is equal to or greater than a preset threshold. If the second similarity is not equal to or greater than the threshold (S1416: NO), processing of the current target row ends and processing of the next target row begins. If there are no subsequent rows, the loop is exited. On the other hand, if the second similarity is equal to or greater than the threshold (S1416: YES), the data registration unit 120 estimates the target row to be a "data item row."

[0098] When the above loop processing S1411s to S1411e is completed, the data registration unit 120 outputs the estimation result of the type of each row of the registered data 21 (S1418).

[0099] As described above, in the second embodiment, the data utilization support device 100 infers rows using the classification history 112, which is the result of past row type inferences. Therefore, for example, a row that should be a "data item row" but could not be inferred as a data item row by the method of the first embodiment can be inferred as a "data item row," thereby improving the accuracy of row type inference.

[0100] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments, and various modifications are included, and the present invention is not necessarily limited to those including all of the configurations described. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0101] 1 Data usage support system, 2 Data registration device, 211 Registration data reception unit, 212 Registered data transmission unit, 213 inference result editing unit, 3 data utilization device, 311 data provision request transmission unit, 312 reformed data reception unit, 313 data utilization unit, 21 registered data, 31 reformed data, 100 data utilization support device, 110 memory unit, 111 estimation standard, 112 classification history, 120 data registration unit, 121 registered data reception unit, 122 line-to-line similarity calculation unit, 123 line-by-line type estimation unit, 124 inference result presentation unit, 125 correction result reception unit, 126 reformed data generation unit, 130 data provision unit, 131 data request reception unit, 132 data transmission unit, 500 inference result presentation screen, S800 data registration processing, S814 line-by-line type estimation processing

Claims

1. A system for supporting data utilization, The information processing device is configured using the storing tabular data, which is data in which information is described in a tabular format; selecting rows of the tabular data in order, calculating a similarity between each selected row and a row following the selected row, and estimating a data item row in the tabular data in which a data item name is written based on the calculated similarity; generating formatted data, which is data obtained by formatting the tabular data into data in a format that can be read by software that uses the tabular data, based on the estimated data item row; estimating a data row in the tabular data in which data corresponding to a data item is written based on the similarity; generating formatted data, which is data obtained by formatting the tabular data into data in a format that can be read by software that uses the tabular data, based on the estimated data rows; based on the similarity, estimating other rows in the tabular data that are neither the data item rows nor the data rows; generating reformulated data, which is data obtained by reformatting the tabular data into data in a format that can be read by software that uses the tabular data, by deleting the estimated other rows; Data usage support system.

2. 2. The data utilization support system according to claim 1, The similarity is a storage value similarity, which is a similarity based on the similarity of information described in the same column between the selected row and the row following the selected row, and a column number similarity, which is a similarity based on the similarity of the number of columns between the selected row and the row following the selected row. Data usage support system.

3. 2. The data utilization support system according to claim 1, the similarity is at least one of a storage value similarity, which is a similarity based on the similarity of information described in the same column of the selected row and the row following the selected row, and a column number similarity, which is a similarity based on the similarity of the number of columns of the selected row and the row following the selected row; determining whether the stored value similarity is high or low by comparing it with a preset threshold value; determining whether the column number similarity is high or low by comparing it with a preset threshold; estimating at least one of the data item row, the data row, and the other row in the tabular data based on the determination result of the level of the stored value similarity and the determination result of the level of the column number similarity; Data usage support system.

4. 2. The data utilization support system according to claim 1, A machine learning model is generated by training data in which information based on the similarity calculated based on a row of tabular data and the row's subsequent rows is used as an explanatory variable, and information indicating whether the row is the data item row, the data row, or the other row is used as a target variable. 、 The estimation is performed by inputting information based on a similarity calculated based on a row of the tabular data and a row following the row into the machine learning model. Data usage support system.

5. 5. The data utilization support system according to claim 4, The similarity is calculated based on at least one of a normalized edit distance and a morpheme cosine similarity between a description of a row of the tabular data and a description of a subsequent row of the row. Data usage support system.

6. 2. The data utilization support system according to claim 1, a user interface for presenting a result of estimating whether each row of the tabular data is the data item row, the data row, or the other row; Data usage support system.

7. 2. The data utilization support system according to claim 1, a user interface for editing the data item names of the data item rows and the estimated rows; Data usage support system.

8. 2. The data utilization support system according to claim 1, a user interface for setting the data item row in the tabular data when it is not possible to estimate the row that will become the data item row in the tabular data; Data usage support system.

9. 2. The data utilization support system according to claim 1, a user interface for editing the tabular data so that the number of data item rows in the tabular data becomes one when a plurality of rows are estimated to be the data item rows in the tabular data; Data usage support system.

10. 2. The data utilization support system according to claim 1, storing a classification history that is a description of one or more rows that have been previously classified as the data item rows; calculating a similarity between a description of a row of the tabular data and a description of the classification history, and if the similarity is equal to or greater than a preset threshold, estimating that the row is the data item row; Data usage support system.

11. 2. The data utilization support system according to claim 1, performing a transposition process on the tabular data, and estimating the data item row using the tabular data after the transposition process; Data usage support system.

12. A method for supporting the use of data, comprising: The information processing device storing tabular data, which is data in which information is described in a tabular format; a step of sequentially selecting rows of the tabular data, calculating a similarity between each selected row and a subsequent row of the selected row, and estimating a data item row in the tabular data where a data item name is written based on the calculated similarity; and Based on the estimated data item rows, the tabular data is formatted into data in a format that can be read by software that uses the tabular data. generating a a step of estimating a data row in the tabular data in which data corresponding to a data item is written, based on the similarity; generating formatted data, which is data obtained by formatting the tabular data based on the estimated data rows into data in a format that can be read by software that uses the tabular data; a step of estimating other rows in the tabular data that are neither the data item rows nor the data rows based on the similarity; and generating reformulated data, which is data obtained by reformatting the tabular data into data in a format that can be read by software that uses the tabular data, by deleting the estimated other rows; A data utilization support method that implements the above.

13. The data utilization support method according to claim 12, The similarity is a storage value similarity, which is a similarity based on the similarity of information described in the same column between the selected row and the row following the selected row, and a column number similarity, which is a similarity based on the similarity of the number of columns between the selected row and the row following the selected row. How to support data use.

Images