Learning device, learning method, and program

Abstract

A training device 1X mainly includes a query acquisition means 17Xa, a related table acquisition means 17Xb, an InSite document acquisition means 17Xc, and a training means 17Xd. The query acquisition means 17Xa acquires a query. The related table acquisition means 17Xb acquires a related table, which is a table related to the query. The InSite document acquisition means 17Xc acquires, as a document representing an insight useful for answering the query, a document obtained by converting aggregated data obtained by aggregating related tables into natural language. The training means 17Xd trains a machine learning model for evaluating the relation between a table and a query on the basis of the degree of relation between a document and the query.

Description

Learning device, learning method, and storage medium

[0001] The present disclosure relates to the technical fields of a learning device, a learning method, and a storage medium that perform processing related to model learning.

[0002] In data analysis, a process of discovering insights into a target object or phenomenon is performed. Regarding such data analysis, for example, Patent Literature 1 discloses a question answering device that receives a reference text and a question (query) as input and outputs an answer and evidence for the answer to an arbitrary output destination.

[0003] International Publication WO2022 / 079826

[0004] To find insights from aggregated data that can provide visualizations that humans find useful in the tables being analyzed, an evaluation model is used to evaluate the relevance of a specified query to a table. Such evaluation models are typically designed for documents, so they may not work accurately for tables.

[0005] In view of the above-mentioned problems, one of the objectives of the present disclosure is to provide a learning device, a learning method, and a storage medium that can suitably perform learning of a machine learning model for evaluating the relevance between a query and a table.

[0006] One aspect of the learning device is a learning device having: a query acquisition means for acquiring a query; an associated table acquisition means for acquiring an associated table that is a table related to the query; an insight document acquisition means for acquiring a document obtained by converting aggregated data from the associated table into natural language as a document representing an insight useful for answering the query; and a learning means for training a machine learning model that evaluates the association between the query and a table that is a candidate for the associated table based on the degree of association between the document and the query.

[0007] One aspect of the learning method is a learning method in which a computer acquires a query, acquires an associated table that is a table related to the query, acquires a document obtained by converting aggregated data from the associated table into natural language as a document that represents insights useful for answering the query, and trains a machine learning model that evaluates the relationship between the query and tables that are candidates for the associated table based on the degree of relationship between the document and the query.

[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following processes: acquire a query; acquire an associated table that is a table related to the query; acquire a document obtained by converting aggregated data from the associated table into natural language as a document that represents insights useful for answering the query; and learn a machine learning model that evaluates the relationship between the query and tables that are candidates for the associated table based on the degree of relationship between the document and the query.

[0009] One example of the effect of the present disclosure is that it is possible to effectively train a machine learning model for evaluating the relevance between a query and a table.

[0010] 1 shows the configuration of a data analysis system. FIG. 2 shows the hardware configuration of a data analysis device. FIG. 3 is an example of a functional block of a processor of a data analysis device. FIG. 4 is a diagram illustrating an overview of processing by a data analysis unit. FIG. 5 shows a functional block diagram of a data analysis unit. (A) shows an overview of a related document evaluation model. (B) shows an overview of a related document evaluation model configured using a dual encoder. (C) shows an overview of a related document evaluation model configured using a cross encoder. FIG. 6 shows an overview of processing to obtain table relevance. FIG. 7 is a diagram illustrating an overview of processing to generate aggregated data based on semantic syntactic analysis. FIG. 8 is a diagram illustrating an overview of processing to generate aggregated data based on chart recommendations. FIG. 9 is a diagram illustrating an overview of generating verbalized insights using a data-to-text model. FIG. 10 is a diagram illustrating an overview of generating verbalized insights using related vectors. FIG. 11 is a diagram illustrating an overview of generating verbalized insights based on related scores. FIG. 12 shows an example of a display of a portal screen. FIG. 13 is an example of a display of a details screen. FIG. 14 is an example of a flowchart illustrating an overview of processing executed by a data analysis device. FIG. 15 is an example of a flowchart related to additional learning of an associated table evaluation model. FIG. 16 is an example of a flowchart for processing to generate verbalized insights using related vectors. FIG. 17 shows the configuration of a data analysis system. FIG. 18 is a diagram illustrating the relationship between a user, a data analysis device, and a terminal device. (A) An example of prescription data related to national health insurance and medical assistance is shown. (B) An example of data showing the results of specific insurance guidance is shown. (C) An example of prescription data related to nursing care is shown. A database of information transmitted in SNS (Social Networking Service). A functional block diagram of a learning device. (A) A first example configuration of a learning device configured separately from a data analysis device. (B) A second example configuration of a learning device configured separately from a data analysis device. An example of a flowchart showing the processing procedure of a learning device.

[0011] Hereinafter, embodiments of a learning device, a learning method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration Fig. 1 shows the configuration of a data analysis system 100. The data analysis system 100 mainly includes a data analysis device 1, an input device 2, a display device 3, and a data warehouse DWH.

[0013] Hereinafter, "query" refers to a natural language inquiry (including questions and hypothesis statements) passed from a user to the data analysis system 100. "Answer" refers to a natural language statement (including facts, explanatory statements, Yes or No, etc.) that the data analysis system 100 returns to the user in response to the query. "Insight" refers to aggregated data that provides useful information for answering a query. "Evidence" refers to documents that provide useful information for answering a query. "Data warehouse" refers to a collection of data from which insights and evidence are derived. "Data-to-Text" refers to a general term for the task of generating text from a table (i.e., the task of generating natural language statements, which are unstructured data, using structured data as input).

[0014] The data analysis device 1 performs a cross-sectional analysis of documents and tables included in a data warehouse DWH that are related to a query specified by a user, and generates answers to the query, insights, and evidence as data analysis results. The data analysis device 1 then controls the display of information related to the data analysis results.

[0015] The data analysis device 1 performs data communication with the input device 2, the display device 3, and the data warehouse DWH via a communication network or by direct wireless or wired communication.

[0016] The input device 2 is an interface that accepts user input, which is external input, and corresponds to, for example, a touch panel, buttons, a keyboard, a voice input device, etc. The input device 2 supplies input information generated based on the user input to the data analysis apparatus 1.

[0017] The display device 3 is, for example, a display, a projector, or the like, and performs a predetermined display based on the display information supplied from the data analysis device 1 .

[0018] The data warehouse DWH is a collection of data related to documents and tables. The data warehouse DWH is held, for example, by a server device that performs data communication with the data analysis device 1. In this case, the data warehouse DWH may be held in a distributed manner by multiple server devices. In another example, the data warehouse DWH may be held by a storage device such as a hard disk connected to or built into the data analysis device 1, or may be held in a storage medium such as a flash memory.

[0019] The configuration of the data analysis system 100 shown in FIG. 1 is an example, and various modifications may be made to the configuration. For example, the input device 2 and the display device 3 may be configured as an integrated device. In this case, the input device 2 and the display device 3 may be configured as a tablet terminal integrated with the data analysis device 1. The data analysis device 1 may be connected to or have a built-in sound output device, such as a speaker, and output information by sound. The data analysis device 1 may also be configured from multiple devices. In this case, the multiple devices that make up the data analysis device 1 exchange information necessary to execute pre-assigned processing between these multiple devices.

[0020] (2) Hardware Configuration of Data Analysis Apparatus Fig. 2 shows the hardware configuration of the data analysis apparatus 1. The data analysis apparatus 1 includes, as hardware components, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 19.

[0021] The processor 11 executes predetermined processes by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0022] The memory 12 is composed of various types of volatile and non-volatile memories such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory 12 also stores programs for the data analysis device 1 to execute various processes. The memory 12 is also used as a working memory, and temporarily stores information acquired from the data warehouse DWH, generated information, and the like.

[0023] The memory 12 stores various information related to data analysis, which will be described later. For example, the memory 12 stores a query Q, an association table X, an insight I, evidence E, an answer A, and an evaluation model parameter Pe. As will be described later, the query Q is decomposed, and information about the decomposed n queries (also referred to as "subqueries") Q1, ..., Qn (n is an integer greater than or equal to 1) is stored in the memory 12. The evidence E includes related documents ED selected from documents in the data warehouse DWH and verbalized insights EI (i.e., documents representing the insights I) obtained by verbalizing the insights I. The answer A includes answers A1, ..., An corresponding to the queries Q1, ..., Qn decomposed from the query Q, respectively.

[0024] The evaluation model parameters Pe are parameters of the association table evaluation model used to derive the insight I, and are updated (also referred to as "optimized") through additional learning of the association table evaluation model. For example, if the association table evaluation model is a model based on a neural network such as a convolutional neural network, the evaluation model parameters Pe include various parameters such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weight of each element of each filter.

[0025] The memory 12 may function as the data warehouse DWH. Similarly, the data warehouse DWH may function as the memory 12 of the data analysis device 1. Furthermore, part of the information stored in the memory 12 may be stored in a storage medium other than the memory 12 (including an external device capable of data communication with the data analysis device 1).

[0026] The interface 13 is an interface for electrically connecting the data analysis device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0027] The hardware configuration of the data analysis device 1 is not limited to the configuration shown in Fig. 2. For example, the data analysis device 1 may include at least one of the input device 2 and the display device 3. Furthermore, the data analysis device 1 may be connected to or have a built-in sound output device such as a speaker.

[0028] (3) Processing Overview Fig. 3 shows an example of functional blocks of the processor 11. Functionally, the processor 11 has a data analysis unit 15, a UI (User Interface) control unit 16, and an evaluation model learning unit 17. Note that in Fig. 3, blocks that exchange data are connected by solid lines, but the combination of blocks that exchange data is not limited to Fig. 3. The same applies to other functional block diagrams described later.

[0029] The data analysis unit 15 refers to various information stored in the memory 12 and performs a cross-sectional analysis of documents and tables in the data warehouse DWH related to a query Q specified by the user. The data analysis unit 15 then generates an answer A to the query Q, an insight I, and evidence E as data analysis results. The data analysis unit 15 supplies the generated analysis results to the UI control unit 16. Details of the processing by the data analysis unit 15 will be described later.

[0030] The UI control unit 16 controls the reception of user input and the display of information to be viewed by the user. For example, the UI control unit 16 generates a query Q based on input information (i.e., external input) supplied from the input device 2, and stores the generated query Q in the memory 12. The UI control unit 16 also generates display information based on the data analysis results generated by the data analysis unit 15, and controls the display of the display device 3 by supplying the generated display information to the display device 3. Specific processing by the UI control unit 16 will be described later with reference to display examples.

[0031] The evaluation model learning unit 17 performs additional learning of the association table evaluation model used to derive the insight I, based on the data generated by the data analysis unit 15 in the data analysis. In this case, the evaluation model learning unit 17 updates the evaluation model parameters Pe, which are parameters of the association table evaluation model stored in the memory 12. The additional learning of the association table evaluation model will be described later.

[0032] The components of the data analysis unit 15, UI control unit 16, and evaluation model learning unit 17 described in FIG. 3 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least some of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0033] (4) Data Analysis Processing Next, the data analysis processing executed by the data analysis unit 15 will be described.

[0034] (4-1) Overview Fig. 4 is a diagram showing an overview of the data analysis process by the data analysis unit 15. As shown in Fig. 4, when a query Q is input, the data analysis unit 15 performs data analysis process by referring to the data warehouse DWH, and generates an answer A and evidence E. In the example of Fig. 4, the query Q is, for example, "Which company has seen the biggest increase in sales since the COVID-19 shock?"

[0035] In this case, the data analysis unit 15 performs data analysis processing including a process of acquiring an associated table X related to the query Q from the data warehouse DWH, a process of deriving an insight I from the acquired associated table X, and a process of verbalizing the derived insight I. Note that the insight I is aggregated data obtained by aggregating the associated table X, and may be a table extracted from the associated table X, or may be a visualization of the extracted table as a chart. As will be described later, the data analysis unit 15 also performs a process of acquiring documents related to the query Q from the data warehouse DWH as part of the data analysis processing. Then, the data analysis unit 15 acquires an answer A and evidence E through such data analysis processing.

[0036] 5 is a functional block diagram of the data analysis unit 15. Functionally, the data analysis unit 15 includes a query decomposition unit 51, a related document acquisition unit 52, a related table acquisition unit 53, an insight derivation unit 54, an insight verbalization unit 55, a reading unit 56, and a query correction unit 57.

[0037] The query decomposition unit 51 retrieves query Q from the memory 12 or the like and decomposes query Q into an arbitrary number (n in this example). As a result, the query decomposition unit 51 generates queries Q1 to Qn, which become subqueries. Queries Q1 to Qn have dependencies. The query decomposition unit 51 understands the dependencies between the decomposed queries and assigns an index of 1 to n to each query in order based on the dependencies. Specifically, if a first subquery and a second subquery exist and the second subquery depends on the first subquery (i.e., the second subquery is in a format in which the answer to the first subquery is substituted for the answer to the first subquery), the index of each decomposed subquery is determined so that the first subquery has a smaller index than the second subquery (i.e., the first subquery is processed before the second subquery). Then, the generation of insight I and answer A is performed in order from the subquery with the smallest index. When an answer to a subquery is generated, the query modification unit 57 modifies the subqueries that depend on the subquery, as described below.

[0038] Hereinafter, the index of the subquery to be processed in each processing unit will be represented as "k" (k = 1, ..., n), and the subquery to be processed will be represented as "Qk." The query decomposition unit 51 supplies the query Qk to the related document acquisition unit 52, the related table acquisition unit 53, the interpretation unit 56, the evaluation model learning unit 17, etc.

[0039] The related document acquisition unit 52 selects related documents ED from the documents included in the data warehouse DWH based on the degree of relevance between the documents included in the data warehouse DWH and the query Qk. The calculation of the degree of relevance will be described later. The related document acquisition unit 52 supplies the related documents ED for the query Qk to the interpretation unit 56.

[0040] The related table acquisition unit 53 selects a related table X from the tables included in the data warehouse DWH based on the degree of relevance between the tables included in the data warehouse DWH and the query Qk. The degree of relevance is calculated using an related table evaluation model to which an evaluation model parameter Pe is applied, as will be described later. The related table acquisition unit 53 supplies the related table X for the query Qk to the insight derivation unit 54 and the evaluation model learning unit 17.

[0041] The insight derivation unit 54 generates aggregated data that becomes insight I based on the query Qk and the associated table X. In this case, the insight derivation unit 54 extracts or selects aggregated data from the associated table X based on semantic syntactic analysis, chart recommendations, etc. The insight derivation unit 54 supplies the insight I corresponding to the query Qk to the insight verbalization unit 55.

[0042] The insight verbalization unit 55 converts the insight I into a verbalized insight EI, which is an explanatory sentence in natural language, based on the query Qk and the insight I so that the subsequent comprehension unit 56 can interpret it. The verbalized insight EI generated by the insight verbalization unit 55 is a document that expresses the content of the insight I (aggregated data) in natural language, and therefore can be suitably used as evidence E. The insight verbalization unit 55 supplies the verbalized insight EI corresponding to the query Qk to the comprehension unit 56 and the evaluation model learning unit 17.

[0043] The comprehension unit 56 generates an answer Ak to the query Qk based on the query Qk and evidence E (i.e., at least one of the related documents ED and the verbalized insight EI). The comprehension unit 56 supplies the generated answer Ak to the query correction unit 57. Note that if k = n, the comprehension unit 56 supplies, for example, answer An and insight I and evidence E corresponding to queries Q1 to Qn to the UI control unit 16. Note that after correction by the query correction unit 57 (described later), the query Qn is equal to the query Q, and the answer An is equal to the answer A.

[0044] The query modification unit 57 modifies the query Q generated from the query Q by the query decomposition unit 51 based on the answer Ak corresponding to the query Qk. ｋ＋１ Here, the query Q ｋ＋１ If depends on query Qk, then query Q ｋ＋１ Since the query Q is in a format that refers to the answer Ak, the query correction unit 57 ｋ＋１ By substituting the answer Ak into ｋ＋１ into a format that does not refer to the answer Ak (i.e., a format that is complete by itself). ｋ＋１ to the query decomposition unit 51, which then decomposes the modified query Q ｋ＋１ are supplied to the related document acquisition unit 52, the related table acquisition unit 53, the reading unit 56, and the evaluation model learning unit 17.

[0045] Hereinafter, we will explain in detail the processing performed by the query decomposition unit 51, the related document acquisition unit 52, the related table acquisition unit 53, the insight derivation unit 54, the insight verbalization unit 55, the interpretation unit 56, and the query correction unit 57.

[0046] (4-2) Query Decomposition Unit The decomposition of query Q by the query decomposition unit 51 will be described. For example, the query decomposition unit 51 uses a question decomposition model to decompose query Q into any number of queries (n in this case). When query Q needs to be decomposed into two or more queries, the query decomposition unit 51 can decompose query Q into any number of queries by recursively executing the question decomposition model. Note that if query Q or its subqueries cannot be decomposed into queries with dependent relationships, the query decomposition unit 51 does not perform further decomposition of query Q or its subqueries. Therefore, if query Q cannot be decomposed into queries with dependent relationships, n=1.

[0047] Here, the question decomposition model may be, for example, a learning model based on deep learning, and may be any model that decomposes a question (query). Examples of such a question decomposition model include DecompRC. The question decomposition model undergoes machine learning in advance so as to output a query having a dependency relationship (the pair of the first query and the second query described above) when a query is input, and the learned parameters are pre-stored in the memory 12 or the like. Note that the question decomposition model undergoes machine learning using learning data including multiple records that form pairs of an input sample (here, a query) to the question decomposition model and a correct answer (here, a decomposed query) that the score conversion model should output when the sample is input. In this case, the parameters of the question decomposition model are determined so as to minimize the error (loss) between the inference result output by the question decomposition model when the sample is input to the question decomposition model and the correct answer. The algorithm that determines the above parameters so as to minimize the loss may be any learning algorithm used in machine learning, such as gradient descent or backpropagation. Similarly, for other machine learning models described below, learned parameters are obtained by performing learning using learning data including multiple records that are pairs of input samples and correct answers to be output, as described above.The query decomposition unit 51 then inputs the query Q or the decomposed query to a question decomposition model constructed by referencing the learned parameters from the memory 12, etc., and obtains the decomposition result of the input query output by the question decomposition model.

[0048] Here, a specific example of decomposition of query Q will be described. For example, if query Q is "Which company has increased its sales the most since the coronavirus shock?", the query decomposition unit 51 generates the following two queries Q1 and Q2. Q1: When did the coronavirus shock occur? Q2: Which company has increased its sales the most since [answer A1]?

[0049] In this case, query Q2 is in the form where answer A1 to query Q1 is substituted for "[answer A1]," making it a subquery that depends on query Q1. In this way, an index is assigned to each subquery so that the dependent subquery (dependent query) has a smaller index than the subquery that depends on the dependent query (dependent query), and the answer is obtained first. As a result, query Q2 is modified by the query modification unit 57 after answer A1 to query Q1 is obtained so that the dependency is resolved, and the related document acquisition unit 52 and the related table acquisition unit 53 use query Q2 to appropriately acquire related documents ED and related table X, respectively.

[0050] As another example, if the query Q is "What news is there about companies whose stock prices have recently plummeted?", the query decomposition unit 51 generates the following two queries Q1 and Q2. Q1: What company has its stock price recently plummeted? Q2: What news is there about [Answer A1]? In this case, the query Q2 is in a format in which the answer A1 to the query Q1 is substituted for "[Answer A1]", and is a subquery that depends on the query Q1.

[0051] As yet another example, if the query Q is "Are the stock prices of industries where sales are increasing this year above the market average?", the query decomposition unit 51 generates the following two queries Q1 and Q2. Q1: Which industries are seeing an increase in sales this year? Q2: Is the stock price of [Answer A1] above the market average? In this case, the query Q2 is in a format where the answer A1 of the query Q1 is substituted for "[Answer A1]", and is a subquery that depends on the query Q1.

[0052] (4-3) Related Document Acquisition Unit Next, the acquisition of related documents ED by the related document acquisition unit 52 will be described. The related document acquisition unit 52 calculates the degree of relevance (also referred to as "document relevance") between the documents included in the data warehouse DWH and the query Qk using a related document evaluation model, and selects documents in the data warehouse DWH to be related documents ED based on the document relevance. For example, the related document acquisition unit 52 calculates the document relevance for each document in the data warehouse DWH (i.e., each document that is a candidate for the related document ED), and selects documents in the data warehouse DWH whose calculated document relevance is equal to or greater than a predetermined threshold as related documents ED. The threshold is, for example, a value stored in advance in the memory 12 or the like.

[0053] Instead of using the document relevance, the related document acquisition unit 52 may use an index value that decreases as the degree of relevance between the document and the query Qk increases. In this case, for example, the related document acquisition unit 52 selects, as the related document ED, documents in the data warehouse DWH whose index values ​​are less than a predetermined threshold. The threshold is stored in advance in, for example, the memory 12.

[0054] Here, the related document evaluation model will be described. FIG. 6A shows an overview of the related document evaluation model. As shown in FIG. 6A, the related document evaluation model is a machine-learned model of the relationship between the query, candidate documents for the related document ED, and the document relevance between them. Specifically, the related document evaluation model is, for example, a learning model based on deep learning. When a query Qk and an arbitrary document in the data warehouse DWH are input, machine learning is performed in advance so that the model outputs an evaluation result of the relevance between the input query and the document (here, an inference result of document relevance). The trained parameters of the related document evaluation model are pre-stored in the memory 12 or the like. The query decomposition unit 51 inputs the query Qk and the documents in the data warehouse DWH into the related document evaluation model, to which the trained parameters are applied by referencing the memory 12 or the like, and obtains the inference result of document relevance output by the related document evaluation model.

[0055] FIG. 6B is a diagram illustrating an overview of a related document evaluation model configured using a dual encoder. In this case, the related document evaluation model includes a query encoder to which a query Qk is input and a document encoder to which a document in a data warehouse DWH is input. When a query is input, the query encoder converts the input query into a query vector, which is a feature vector represented in a feature space with a predetermined number of dimensions. When a document is input, the document encoder converts the input document into a document vector, which is a feature vector represented in a feature space with a predetermined number of dimensions. The related document evaluation model then compares the query vector output by the query encoder with the document vector output by the document encoder, and outputs a document relevance score corresponding to the comparison result (e.g., similarity). An example of such a dual encoder includes Dense Passage Retrieval (DPR).

[0056] 6(C) is a diagram showing an overview of a related document evaluation model configured using a cross encoder. In this case, the related document evaluation model is a cross encoder to which a query Qk and a document in the data warehouse DWH are input. When the query Qk and a document in the data warehouse DWH are input, the cross encoder outputs a document relevance, which is the relevance between the query Qk and the document in the data warehouse DWH. Note that an example of such a cross encoder includes a BERT re-ranker (re-ranking).

[0057] (4-4) Related Table Acquisition Unit Next, the acquisition of related table X by the related table acquisition unit 53 will be described. The related table acquisition unit 53 calculates the degree of relevance (also referred to as "table relevance") between each table included in the data warehouse DWH and the query Qk using a related table evaluation model. Then, the related table acquisition unit 53 selects a table in the data warehouse DWH to be the related table X based on the table relevance. For example, the related table acquisition unit 53 calculates the table relevance for each table in the data warehouse DWH (i.e., each table that is a candidate for the related table X), and selects as the related table X a table in the data warehouse DWH whose calculated table relevance is equal to or greater than a predetermined threshold. The above-mentioned threshold is stored in advance in, for example, the memory 12.

[0058] Instead of the table relevance, the related table acquisition unit 53 may use an index value that decreases as the degree of relevance between the table and the query Qk increases. In this case, for example, the related table acquisition unit 53 selects, as the related table X, a table in the data warehouse DWH whose index value is less than a predetermined threshold. The threshold is stored in advance in the memory 12, for example.

[0059] 7 shows an overview of the process by which the association table acquisition unit 53 acquires table relevance using the association table evaluation model. The association table acquisition unit 53 converts a table into sequential data, inputs the sequential data and a query Qk to the association table evaluation model, and acquires a table relevance corresponding to the evaluation result of the relevance between the query Qk and the sequential data of the table output by the association table evaluation model.

[0060] The related table evaluation model is a model that has been machine-learned to determine the relationship between a query, candidate tables for the related table X, and the table relevance between them. Specifically, the related table evaluation model is, for example, a learning model based on deep learning, and machine learning is performed in advance so that, when a query Qk and an arbitrary table in the data warehouse DWH are input, an evaluation result of the relevance between the input query and the table (here, an inference result of the table relevance) is output. The memory 12 pre-stores trained evaluation model parameters Pe for the related table evaluation model. The related table acquisition unit 53 acquires the table relevance by inputting the query Qk and the sequence data of the table into the related table evaluation model configured with reference to the evaluation model parameters Pe stored in the memory 12, etc.

[0061] Here, the sequence data is a string of characters in which table elements are linked in series, and may be in any format. In Fig. 7, as an example, sequence data is generated in which the title (financial results information) is listed after [TITLE], attributes (year, company, revenue, etc.) are listed after [COL], and fields for each record are listed after [ROW] provided for each record.

[0062] By converting the table into sequence data in this way, the related document evaluation model can also be used as the related table evaluation model. In this case, the related table evaluation model may be configured as a dual encoder as shown in Figure 6(B) or as a cross encoder as shown in Figure 6(C). Note that when additional learning of the related table evaluation model is performed by the evaluation model learning unit 17 (described later), the evaluation model parameter Pe is updated by the additional learning.

[0063] (4-5) Insight Derivation Unit Next, a process will be described in which the insight derivation unit 54 generates aggregated data that becomes the insight I based on the query Qk and the related table X. Hereinafter, a method based on semantic syntactic analysis and a method based on chart recommendation will be described, respectively.

[0064] In the semantic analysis-based method, the insight deriving unit 54 converts the query Qk into a data query language such as SQL or a visualization query language such as Vega-Zero or Vega-Lite, and uses the converted query language to acquire the aggregate data intended by the user from the related table X. In this case, the insight deriving unit 54 may perform post-processing to replace character strings in the query language to ensure that the converted query language includes the column names of the target related table X and returns valid results.

[0065] FIG. 8 is a diagram illustrating an overview of the process of generating aggregated data (i.e., insight I) based on semantic syntactic analysis. The insight derivation unit 54 uses a semantic syntactic analysis model to generate a query language corresponding to query Qk. In the example of FIG. 8, the insight derivation unit 54 inputs query Qk (here, "Which company has seen the largest increase in sales since 2020?") into the semantic syntactic analysis model, thereby obtaining the query language (here, SQL or Vega-Lite) output by the semantic syntactic analysis model. The insight derivation unit 54 then performs aggregation processing on the associated table X using the generated query language, and obtains the aggregated data obtained by the aggregation processing as insight I. In this example, a table with column names of "Year," "BBB," and "BAC" (here, BBB and BAC are company names) is generated as aggregated data.

[0066] The semantic parsing model may be a rule-based model or a learning model based on deep learning. An example of the semantic parsing model is ncNet. Parameters and the like for configuring the semantic parsing model are stored in advance in the memory 12, and the insight derivation unit 54 generates a query language from the query Qk using the semantic parsing model configured by referring to the parameters and the like.

[0067] In the chart recommendation-based method, the insight derivation unit 54 identifies table columns and insight types associated with the query Qk, and selects aggregated data (i.e., insight I) that is deemed useful based on the identified information and a chart recommendation model that recommends charts deemed useful to humans. In this case, the insight derivation unit 54, for example, calculates the relevance between the query Q and each column of the table using any calculation method (for example, based on the degree of string coincidence or semantic similarity), and identifies columns whose relevance is equal to or greater than a predetermined threshold.

[0068] FIG. 9 is a diagram showing an overview of the process of generating aggregated data (i.e., insight I) based on chart recommendations. In the example of FIG. 9, the insight derivation unit 54 identifies the table columns "Year," "Revenue," and "Company" that are highly associated with the partial character strings "2020 or later," "Sales," and "Company," respectively, of the query Qk. The insight derivation unit 54 also identifies the insight type "Trend" from the partial character string "Most Extending" of the query Qk. The insight derivation unit 54 then inputs the identified table columns and insight type into the chart recommendation model and obtains the aggregated data (i.e., insight I) output by the chart recommendation model.

[0069] The chart recommendation model is, for example, a learning model based on deep learning, and an example of a chart recommendation model is QuickInsights. Parameters and the like for configuring the chart recommendation model are pre-stored in the memory 12, etc., and the insight derivation unit 54 configures the chart recommendation model by referring to the parameters and the like. Note that the chart recommendation model is not limited to a model that takes a table column and an insight type as input. For example, the chart recommendation model may be any model that outputs aggregated data (i.e., insight I) when a query and a table are input.

[0070] (4-6) Insight Verbalization Unit The insight verbalization unit 55 generates a verbalized insight EI by converting the insight I based on the query Qk and the insight I. In this case, the insight verbalization unit 55 uses a context-aware Data-to-Text model (document conversion model) to generate a verbalized insight EI that is an explanatory sentence according to the content of the query Qk.

[0071] FIG. 10 is a schematic diagram of generating a verbalized insight EI using a data-to-text model. The insight verbalization unit 55 converts the aggregated data, which is the insight I, into sequence data and combines a query Qk with the sequence data of the aggregated data as information representing the context. If the aggregated data includes a chart, the insight verbalization unit 55 may add visual information obtained from the chart to the sequence data of the aggregated data. For example, if the chart indicates an upward trend, the insight verbalization unit 55 adds the character string "[Type] up-trend" to the sequence data.

[0072] The insight verbalization unit 55 then inputs data obtained by combining the sequence data of the aggregated data with the query Qk into the Data-to-Text model, and obtains the verbalized insight EI output by the Data-to-Text model. Fig. 10 shows, as an example, an explanatory sentence that becomes the verbalized insight EI when the aggregated data shown in Fig. 8 or 9 is obtained.

[0073] Here, the Data-to-Text model is a model that learns the relationship between the series data of insight I and query Qk when query Qk is used as context, and verbalized insight EI (i.e., document that serves as evidence). The Data-to-Text model is, for example, a learning model based on deep learning, and examples of Data-to-Text models include T5 (Text-to-Text Transfer Transformer). Parameters and the like for configuring the Data-to-Text model are stored in advance in the memory 12, etc., and the insight verbalization unit 55 configures the Data-to-Text model by referring to the parameters and the like. The Data-to-Text model functions, for example, as an encoder that vectorizes input text and a decoder that converts the data vectorized by the encoder back into text.

[0074] Preferably, the insight verbalization unit 55 generates the verbalized insight EI by taking into account an evaluation of the relevance between the query Qk and the cell of the aggregated data and highlighting and verbalizing a part related to the query Qk. In this case, for example, the insight verbalization unit 55 generates a feature vector (also referred to as an "relevance vector") representing the relevance between the query Qk and the cell of the aggregated data using a natural language understanding model, and adds the relevance vector to the input of the Data-to-Text model as additional information.

[0075] Here, the natural language understanding model is, for example, a learning model based on deep learning, and examples of natural language understanding models include BERT (Bidirectional Encoder Representations from Transformers). Parameters and the like for configuring the natural language understanding model are stored in advance in the memory 12, etc., and the insight verbalization unit 55 configures the natural language understanding model by referring to the parameters and the like.

[0076] FIG. 11 is a diagram showing an outline of generating verbalized insights EI using relevance vectors.

[0077] The insight verbalization unit 55 first converts each cell of the aggregated data (or each group of cells; in FIG. 11 , each row and each column) into sequence data, and then combines the sequence data of each cell of the aggregated data with the query Qk. In this case, for example, when combining the sequence data corresponding to the first row of the aggregated data with the query Qk, the query Qk is placed after [CLS], and a separator [SEP] is further placed after the query Qk, and the sequence data of the first row of the aggregated data is placed after [SEP]. Similarly, when combining the sequence data of the first column of the aggregated data with the query Qk, the query Qk is placed after [CLS], and a separator [SEP] is further placed after the query Qk, and the sequence data of the first column of the aggregated data is placed after [SEP].

[0078] Next, the insight verbalization unit 55 inputs the sequential data (see dotted line) for each cell (here, for each group in rows and columns) of the aggregated data combined with the query Qk into the natural language understanding model and acquires the vector output by the natural language understanding model as an association vector. For example, the insight verbalization unit 55 inputs sequential data obtained by combining the query Qk with the sequential data corresponding to the cell in the first row of the aggregated data (see dashed line) into the natural language understanding model, thereby acquiring an association vector that evaluates the association between the query Qk and the cell in the first row of the aggregated data. In another example, the insight verbalization unit 55 inputs sequential data obtained by combining the query Qk with the sequential data corresponding to the cell in the first column of the aggregated data (see dash-dot line) into the natural language understanding model, thereby acquiring an association vector (see dotted line) that evaluates the association between the query Qk and the cell in the first column of the aggregated data. In this way, the insight verbalization unit 55 acquires an association vector that evaluates the association between the query Qk and each cell (each group of cells) of the aggregated data.

[0079] The insight verbalization unit 55 then inputs the acquired association vectors into the Data-to-Text model as additional information, and identifies the explanatory text output by the Data-to-Text model as a portion to be emphasized (highlighted) in the verbalized insight EI or the verbalized insight EI generated by the procedure shown in Fig. 10. This is expected to generate a verbalized insight EI that is an explanatory text that takes into account the association vectors and focuses on highly relevant cells in a self-determined manner.

[0080] For example, as described in FIG. 10 , the insight verbalization unit 55 inputs data combining the sequence data of the aggregated data and the query Qk to the Data-to-Text model (the input layer of the Data-to-Text model), and inputs the associated vector to the intermediate layer of the Data-to-Text model. The intermediate layer corresponds to the decoder layer if the Data-to-Text model has an encoder layer that vectorizes the text input to the input layer and a decoder layer that converts the vectors output from the encoder layer back into text. That is, in this case, the insight verbalization unit 55 inputs the vectors output from the encoder layer and the associated vectors to the decoder layer. In this way, the insight verbalization unit 55 additionally inputs the associated vector to the intermediate layer of the document transformation model, which receives as input vectors representing the sequence data for each cell of the aggregated data and the query Qk.

[0081] Instead of using the relevance vectors as additional information to be input to the Data-to-Text model, the insight verbalization unit 55 may select data to be input to the Data-to-Text model based on the relevance vectors. In this case, for example, the insight verbalization unit 55 converts the relevance vector for each cell into a score (also referred to as a "relevance score") indicating the degree of relevance with the query Qk, and excludes the text of cells corresponding to the relevance vectors whose relevance scores do not satisfy a predetermined condition from the sequence data to be input to the Data-to-Text model. This can also be expected to generate a verbalized insight EI that is an explanatory text that takes into account the relevance vectors and focuses on highly relevant cells in a self-determined manner. The relevance score is an example of an index value indicating the degree of relevance between a cell and a query.

[0082] FIG. 12 is a diagram showing an overview of generation of verbalized insights EI based on relevance scores.

[0083] The insight verbalization unit 55 first converts each cell of the aggregated data (here, each group in rows and columns) into sequential data, then inputs data (see dotted line) combining the sequential data of the aggregated data for each cell with the query Qk into the natural language understanding model, and acquires the association vector (see dotted line) output by the natural language understanding model. Note that the method for acquiring the association vector is the same as the method for generating the verbalized insight EI shown in FIG. 11.

[0084] Next, the insight verbalization unit 55 inputs the associated vectors into a score conversion model, which is a machine learning model having a neural network architecture such as MLP (Multi-Layer Perceptron), and acquires the scores output by the score conversion model as associated scores (see dotted lines) corresponding to the input associated vectors. The insight verbalization unit 55 acquires associated scores for each cell of the aggregated data (here, for each row and column) by sequentially inputting the associated vectors for each cell of the aggregated data into the score conversion model.

[0085] The insight verbalization unit 55 then selects cells to input in the aggregated data based on the relevance scores, and inputs data obtained by linking the sequence data of the selected aggregated data with the query Qk into the Data-to-Text model.The insight verbalization unit 55 then identifies the explanatory sentence output by the Data-to-Text model as a part to be emphasized (highlighted) in the verbalized insight EI or the verbalized insight EI generated by the procedure shown in FIG.

[0086] In selecting the aggregated data, for example, if the relevance score represents the presence or absence of relevance using a binary value, the insight verbalization unit 55 retains cells corresponding to relevance scores indicating "relevance" as data to be input to the Data-to-Text model, and excludes cells corresponding to relevance scores indicating "no relevance" from the data to be input to the Data-to-Text model. Even when the relevance score represents a score other than a binary value, the insight verbalization unit 55 compares the relevance score with a predetermined threshold and selects the aggregated data. For example, if the relevance score is a score that increases as the degree of relevance increases, the insight verbalization unit 55 retains cells corresponding to the relevance score as data to be input to the Data-to-Text model when the relevance score is equal to or greater than a predetermined threshold, and excludes cells corresponding to the relevance score from the data to be input to the Data-to-Text model when the relevance score is less than the predetermined threshold.

[0087] Here, a supplementary explanation will be given about the score conversion model described above. The score conversion model is a machine learning model trained to output an associated score when an associated vector is input. The score conversion model undergoes machine learning using training data including a plurality of records each of which is a pair of an associated vector that is a sample input to the score conversion model and a correct associated score that the score conversion model should output when the associated vector is input to the score conversion model. In this case, the parameters of the score conversion model are determined so that the error (loss) between the inference result output by the score conversion model when the associated vector is input and the correct associated vector is minimized. The trained parameters of the score conversion model are stored in advance in the memory 12 or the like.

[0088] (4-7) Reading Unit The reading unit 56 inputs at least one of the related document ED and the verbalized insight EI, which are evidence E, and the query Qk to the question reading model, and obtains the answer output by the question reading model as the answer Ak.

[0089] The question comprehension model may be any text generation model based on machine learning. An example of a question comprehension model is FiD (Fusion-In-Decoder). Trained parameters of the question comprehension model are stored in advance in the memory 12 or the like, and the comprehension unit 56 configures the question comprehension model by referring to the parameters. Alternatively, the question comprehension model may be trained according to the type of query, and the trained parameters may be stored in the memory 12 or the like.

[0090] (5) Learning of Association Table Evaluation Model Next, learning of the association table evaluation model executed by the evaluation model learning unit 17 will be described.

[0091] In a first example, when training data (i.e., a teacher dataset) is stored in the memory 12 or the like, the evaluation model training unit 17 performs additional training of the association table evaluation model based on the training data. In this case, the training data includes, for example, a plurality of records each forming a pair of a query and a table to be extracted as the association table X when the query is given. For example, when the evaluation model training unit 17 inputs the series data of the above-described pair of query and table to the association table evaluation model, the evaluation model training unit 17 updates the evaluation model parameter Pe so that the table relevance becomes a predetermined value (e.g., a maximum value or a value equal to or greater than a predetermined threshold). The above-described predetermined value is an example of a "predetermined evaluation." In this case, the evaluation model parameter Pe is updated so that the error (loss) between the table relevance output by the association table evaluation model and a desired value is minimized. The algorithm for determining the evaluation model parameter Pe so as to minimize the loss may be any learning algorithm used in machine learning, such as gradient descent or backpropagation.

[0092] According to the first example, when a pair of a query and a table exists as learning data, the evaluation model learning unit 17 can preferably perform additional learning of the related table evaluation model so as to improve the performance of the related table evaluation model.

[0093] In the second example, the evaluation model learning unit 17 performs additional learning of the association table evaluation model based on the relationship between the evidence E (i.e., the verbalized insight EI) derived based on the insight I and the query Qk. Specifically, the evaluation model learning unit 17 updates the evaluation model parameter Pe so as to increase the table relevance of the table from which the insight I highly relevant to the query Qk is derived. In this case, the evaluation model learning unit 17 performs additional learning of the association table evaluation model based on the query Qk supplied from the query decomposition unit 51, the association table X supplied from the association table acquisition unit 53, and the verbalized insight EI supplied from the insight verbalization unit 55. The flow of data supplied to the evaluation model learning unit 17 is shown in FIG. 5 .

[0094] FIG. 13 shows an overview of the additional learning of the association table evaluation model in the second example.

[0095] As described above, the data analysis unit 15 extracts the associated table X from the data warehouse DWH based on the query Qk, generates an insight I (i.e., aggregated data) from the associated table X, and converts the aggregated data into a verbalized insight EI. The evaluation model training unit 17 then calculates the relevance between the query Qk and the verbalized insight EI. The relevance between the query Qk and the verbalized insight EI may be, for example, a document relevance calculated using a related document evaluation model. In this case, the evaluation model training unit 17 acquires the relevance output by the related document evaluation model when the query Qk and the verbalized insight EI are input to the related document evaluation model. The evaluation model training unit 17 may calculate the relevance between the query Qk and the verbalized insight EI using a model that has been trained separately from the related document evaluation model, or may calculate the relevance between the query Qk and the verbalized insight EI using any other method.

[0096] Then, when the degree of relevance between the query Qk and the verbalized insight EI is higher than a predetermined threshold (i.e., when the relevance between the query Qk and the verbalized insight EI is considered to be high), the evaluation model learning unit 17 performs additional learning of the association table evaluation model in the same manner as in the first example, based on the pair of the query Qk and the association table X corresponding to the verbalized insight EI. For example, when the evaluation model learning unit 17 inputs the series data of the query Qk and the association table X forming the above-mentioned pair into the association table evaluation model, it updates the evaluation model parameter Pe so that the table relevance output by the association table evaluation model becomes a desired value (for example, a maximum value or a value equal to or greater than a threshold).

[0097] According to the second example, even when there is no teacher data set of queries and tables as in the first example, the evaluation model learning unit 17 can preferably perform additional learning of the association table evaluation model. Through such additional learning, the association table evaluation model is learned so that the table relevance output by the association table evaluation model becomes high when the relevance between the query Qk and the verbalized insight EI is high, and the association table evaluation model is learned so that the relevance between the query Qk and the verbalized insight EI becomes high.

[0098] (6) Display Example Fig. 14 is a display example of a portal screen that the UI control unit 16 causes the display device 3 to display. The UI control unit 16 generates display information for displaying the display screen based on the processing results generated by the data analysis unit 15, and transmits the generated display information to the display device 3 via the interface 13, thereby displaying the portal screen on the display device 3. Here, the UI control unit 16 mainly sets a query input field 60, an insight display specification field 61, a display button 62, an insight general display field 63, an answer display field 64, and an insight individual display field 65 on the display screen.

[0099] The query input field 60 is an input field that accepts user input of a query Q, and when the UI control unit 16 detects that the display button 62 has been selected, it stores the text entered in the query input field 60 in the memory 12 as a query Q and instructs the data analysis unit 15 to start data analysis.

[0100] The insight display specification field 61 is a field where the user specifies the display mode of the insight general display field 63 described later, and mainly includes a layout selection field 61a, a filter selection field 61b, and a sort type selection field 61c.

[0101] The layout selection field 61a is a field in which the user selects the display format of the insight general display field 63. For example, when "table" is selected in the layout selection field 61a, the UI control unit 16 displays, in table format, in the insight general display field 63, the verbalized insights EI (which may include related documents ED; the same applies hereinafter in the description of display examples) derived based on the query Q specified in the query input field 60. On the other hand, when "graph" is selected in the layout selection field 61a, the UI control unit 16 displays, in graph format, in the insight general display field 63, the verbalized insights EI derived based on the query Q specified in the query input field 60. As will be described later, the insight general display field 63 clearly shows the relationships between the verbalized insights EI.

[0102] The filter selection field 61b is a field where the user selects the type of relationship (detail, cause, inclusion, etc.) between the verbalized insights EI (and the related documents ED) to be displayed in the insight general display field 63. The UI control unit 16 displays, in the insight general display field 63, the relationships between the verbalized insights EI (and the related documents ED) that correspond to the type of relationship selected in the filter selection field 61b, together with the verbalized insights EI.

[0103] The sort type selection field 61c is a field in which the user selects a sort type for the verbalized insights EI when "table" is selected in the layout selection field 61a. For example, when "query similarity" is selected in the sort type selection field 61c, the UI control unit 16 determines the display order of each verbalized insight EI based on the similarity between each verbalized insight EI and the query Q. This similarity may be, for example, a document relevance obtained by inputting each verbalized insight EI and the query Q into a related document evaluation model. Note that the sort type selectable in the sort type selection field 61c may be various sort types other than "query similarity."

[0104] The insight general display field 63 displays the verbalized insight EI derived based on the query Q specified in the query input field 60 based on the content specified in the insight display specification field 61.

[0105] In the state shown in FIG. 14 , because "Table" is selected in the layout selection field 61 a, the UI control unit 16 displays the verbalized insights EI in table format in the insight general display field 63. Specifically, the table shown in the insight general display field 63 includes an "ID" assigned to each verbalized insight EI, an "Insight" indicating the corresponding verbalized insight EI, and a "Link (To)" and a "Link (From)" indicating a relationship with other verbalized insights EI. Here, because "Query Similarity" is selected in the sort type selection field 61 c, IDs are assigned to the verbalized insights EI in descending order of similarity with the query Q. Note that instead of displaying the entire text of the corresponding verbalized insight EI, only a portion of the corresponding verbalized insight EI (for example, a highlighted portion on a detailed screen, which will be described later) may be displayed in "Insight."

[0106] Furthermore, the "Link (To)" and "Link (From)" fields contain information for clearly indicating pairs of verbalized insights EI that have a relationship corresponding to the type of relationship selected in the filter selection field 61b. For example, a verbalized insight EI with an "ID" of "1" is in a causal relationship with a verbalized insight EI with an "ID" of "2" (i.e., there is a "cause" relationship from "1" to "2"). Therefore, based on a predetermined rule, the UI control unit 16 enters <ID1, cause, ID2> in the "Link (To)" field for the "ID" of "1" and <ID1, cause, ID2> in the "Link (From)" field for the "ID" of "2." Note that the method of describing the relationship is not limited to the method shown in FIG. 14 , and various methods may be used to describe the relationship. In this way, the UI control unit 16 can appropriately present the relationship between the verbalized insights EI to the user.

[0107] In addition, each display column for "Insight" in the table can be selected by the user, and the UI control unit 16 highlights the selected display column (here, the display column with "ID" "2") by framing it, and displays individual information in the insight individual display column 65 described later.

[0108] When "Graph" is selected in the layout selection field 61a, the UI control unit 16 displays, for example, a graph in the insight general display field 63 in which each verbalized insight EI is treated as a node (entity) and the relationships between the verbalized insights EI are represented by edges. In this case, not only the verbalized insights EI but also keywords included in the query Q (e.g., "women in their 20s to 40s," "hot milk") may be treated as nodes (entities) and the relationships with the verbalized insights EI may be represented by edges.

[0109] Here, a method for identifying relationships between verbalized insights EI will be described in more detail. The UI control unit 16 identifies relationships between verbalized insights EI, for example, based on a relationship discovery model. The relationship discovery model is, for example, a learning model based on a neural network, and is machine-learned so that when two documents are input, it outputs an inference result about the relationship between the input documents. The relationship discovery model is trained based on training data having multiple records that form pairs of two documents and correct relationships related to the documents, and the trained parameters of the relationship discovery model are stored in advance in the memory 12, etc. Then, the UI control unit 16 sequentially inputs all pairs of verbalized insights EI (which may include related documents ED) obtained from the specified query Q into the relationship discovery model, and identifies the relationship between the pairs based on the inference result output by the relationship discovery model.

[0110] The answer display field 64 is a display field that displays an answer to the query entered in the query input field 60. When detecting that the display button 62 has been selected, the UI control unit 16 displays the answer A to the query Q generated by the data analysis unit 15.

[0111] The individual insight display field 65 is a display field for individual information of the verbalized insight EI (here, the "ID" is "2") selected in the general insight display field 63. The UI control unit 16 displays the target verbalized insight EI in the insight display field 65a. Furthermore, when the UI control unit 16 detects that the details button 65b has been selected, it displays a details screen, which will be described later in FIG. 15 . Furthermore, the UI control unit 16 displays the identification name of the data (table) from which the verbalized insight EI was generated in the source data display field 65c. The identification name of the data displayed in the source data display field 65c is selectable, and the UI control unit 16 excerpts a portion of the chart related to the data with the selected identification name and displays it in the excerpt display field 65d. Furthermore, when the UI control unit 16 detects that the original data confirmation button 65e has been selected, it displays a screen showing details of the data with the selected identification name (e.g., multiple charts) in the source data display field 65c.

[0112] 15 is a display example of a details screen that the UI control unit 16 causes the display device 3 to display. For example, when the UI control unit 16 detects that the details button 65b on the portal screen shown in FIG. 14 has been selected, the UI control unit 16 causes the display device 3 to display a details screen related to the verbalized insight EI that is the display target in the individual insight display field 65. The UI control unit 16 mainly provides an associated table display field 66, a verbalized insight display field 67, and a chart display field 68 on the details screen.

[0113] The UI control unit 16 displays the related table X used to generate the target verbalized insight EI in the related table display field 66. Here, the UI control unit 16 displays a table titled "2021 Beverage Survey Results" as the related table X in the related table display field 66. The UI control unit 16 also highlights cells extracted as insight I (i.e., aggregated data) from the displayed related table X by outlining them. Instead of or in addition to highlighting the cells extracted as insight I, the UI control unit 16 may further identify cells highly relevant to the query Q based on the relevance score (e.g., cells whose relevance score is equal to or greater than a predetermined value) from the cells extracted as insight I (i.e., aggregated data) and highlight the identified cells. The UI control unit 16 may also highlight cells corresponding to the highlighted portion in the verbalized insight display field 67, which will be described later.

[0114] Furthermore, instead of or in addition to displaying the relevance table X, the UI control unit 16 may display aggregated data that becomes insight I. In this case, too, the UI control unit 16 may identify a cell of aggregated data that is highly relevant to the query Q based on the relevance score, for example, and highlight the identified cell. Furthermore, the UI control unit 16 may highlight a cell of aggregated data that corresponds to a highlighted portion in a verbalized insight display field 67, which will be described later.

[0115] Furthermore, the UI control unit 16 displays the entire text of the target verbalized insight EI in the verbalized insight display field 67, and highlights (here, highlights by underlining and bolding, etc.) parts that are particularly highly relevant to the query. In this case, the UI control unit 16 displays, for example, the verbalized insight EI generated by the method shown in Fig. 10 (i.e., a method that does not take relevance vectors into consideration), and highlights parts of the verbalized insight EI that overlap with sentences generated by the method shown in Fig. 11 or 12 (i.e., a method that uses relevance vectors). In this case, for example, character strings corresponding to cells of aggregated data whose relevance scores are equal to or greater than a predetermined value are highlighted.

[0116] Furthermore, the UI control unit 16 displays a chart corresponding to the target verbalized insight EI in the chart display field 68. In this case, for example, the UI control unit 16 displays a chart that visualizes aggregated data extracted from the related table X as insight I in the chart display field 68. Here, the UI control unit 16 displays a pie chart showing the percentage of age groups that like hot milk and a pie chart showing the percentage of favorite drinks for women in their 20s to 40s in the chart display field 68. In this case, the UI control unit 16 may generate the chart using any method for generating a chart from a table. Note that the UI control unit 16 may visualize insight I using any type of chart, not limited to a pie chart.

[0117] In this way, the data analysis device 1 introduces analysis technology using natural language, performs cross-sectional analysis of documents and tables related to a query, and then generates and displays answers and related insights, thereby reducing the effort and time required for data analysis on the user's side. In this case, the user can obtain a direct answer to the query without having to interpret the results output by the analysis system. Furthermore, because answers and insights can be obtained appropriately for queries that require cross-sectional analysis of multiple information sources, the user no longer needs to interpret the relationships between the results output by the analysis system.

[0118] (6) Processing Flow FIG. 16 is an example of a flowchart showing an outline of processing executed by the data analysis device 1.

[0119] First, the data analysis apparatus 1 decomposes the query Q identified based on the input information supplied by the input device 2 into n queries Q1, ..., Qn, and identifies dependencies between the queries (step S11). In this case, the indexes of the decomposed queries are determined so that a dependent query that depends on another query has a higher index than the other query that is a dependent query. The processing of step S11 corresponds to the processing executed by the query decomposition unit 51.

[0120] Next, the data analysis device 1 sets the index k of the query representing the processing target to "k=0" (step S12), and then increments the index k by 1 (step S13).

[0121] After executing step S13, the data analysis apparatus 1 acquires the related documents ED of the query Qk (step S14). The process of step S14 corresponds to the process executed by the related document acquisition unit 52.

[0122] Furthermore, after executing step S13, the data analysis apparatus 1 acquires an associated table X of the query Qk (step S15). The processing of step S15 corresponds to the processing executed by the associated table acquisition unit 53. Then, the data analysis apparatus 1 derives an insight I from the associated table X acquired in step S15 (step S16). In this case, the data analysis apparatus 1 generates an insight I, which is aggregated data obtained by aggregating the insights I. The processing of step S16 corresponds to the processing executed by the insight derivation unit 54. Then, the data analysis apparatus 1 verbalizes the insight I derived in step S16 (step S17). As a result, the data analysis apparatus 1 generates a verbalized insight EI. A detailed flow of the processing of step S17 will be described later with reference to FIG. 18. The processing of step S17 corresponds to the processing executed by the insight verbalization unit 55.

[0123] After executing steps S14 and S17, the data analysis device 1 generates an answer Ak to the query Qk based on the related documents ED and the verbalized insights EI, which serve as evidence E (step S18). The processing of step S18 corresponds to the processing executed by the comprehension unit 56.

[0124] Then, the data analysis apparatus 1 determines whether the index k is less than n (step S19). If the index k is less than n (step S19; Yes), the data analysis apparatus 1 calculates the query Q based on the answer Ak. ｋ＋１ (Step S20). The process of Step S20 corresponds to the process executed by the query modification unit 57. Thereafter, the data analysis device 1 returns the process to Step S13, increments the index k by 1, and executes the processes of Steps S14 to S19 again.

[0125] On the other hand, if index k has reached n (step S19; No), the data analysis apparatus 1 regards query Qk as query Q, and displays information about the query data analysis results, including the obtained answer A, insight I, and evidence E, on the display device 3 (step S21). In this case, for example, the data analysis apparatus 1 displays various screens such as those shown in FIGS. 14 and 15 on the display device 3 based on the processing results obtained in steps S14 to S18. The processing of step S21 corresponds to the processing executed by the UI control unit 16.

[0126] 17 is an example of a flowchart related to additional learning of the association table evaluation model executed by the evaluation model learning unit 17 of the data analysis device 1. The evaluation model learning unit 17 executes the processing of the flowchart shown in FIG. 17 at any timing during or after the processing of the flowchart in FIG.

[0127] First, the evaluation model learning unit 17 acquires a query Qk, a verbalized insight EI corresponding to the query Qk, and an association table X (step S31). The evaluation model learning unit 17 acquires the above information from the data analysis unit 15 or the memory 12.

[0128] Next, the evaluation model learning unit 17 calculates the relevance between the query Qk acquired in step S31 and the verbalized insight EI (step S32). Then, the evaluation model learning unit 17 determines whether the relevance calculated in step S32 is equal to or greater than a threshold (step S33). The threshold is stored in advance in, for example, the memory 12.

[0129] If the relevance is greater than or equal to the threshold (Step S33; Yes), the evaluation model training unit 17 trains the association table evaluation model based on the corresponding query Qk and the association table X (Step S34). In this case, the evaluation model training unit 17 updates the evaluation model parameter Pe so that the table relevance between the query Qk and the series data in the association table X becomes higher. As a result, the evaluation model training unit 17 additionally trains the association table evaluation model so as to output an inference result that increases the relevance between the query Qk and the verbalized insight EI. On the other hand, if the relevance is less than the threshold (Step S33; No), the evaluation model training unit 17 skips Step S34 and proceeds to Step S35. In this case, the evaluation model training unit 17 may update the evaluation model parameter Pe so that the table relevance between the query Qk and the series data in the association table X becomes lower.

[0130] The evaluation model training unit 17 then determines whether or not to terminate the training of the association table evaluation model (step S35). For example, the evaluation model training unit 17 determines that the training should be terminated if there are no other queries Qk, verbalized insights EI, and associated tables X that can be used as training data in additional training of the association table evaluation model. In another example, the evaluation model training unit 17 may determine whether or not to terminate the training based on any criteria, such as the number of times step S34 is executed. If the evaluation model training unit 17 determines that the training should be terminated (step S35; Yes), it terminates the processing of the flowchart. On the other hand, if the evaluation model training unit 17 determines that the training should not be terminated (step S35; No), it returns the processing to step S31, acquires a combination of a query Qk, verbalized insight EI, and associated table X that has not been used in training the association table evaluation model, and executes the processing of steps S32 to S35 again.

[0131] 18 is an example of a flowchart of a process for generating a verbalized insight EI using an association vector. The insight verbalization unit 55 executes the flowchart of FIG. 18, for example, in step S17 of FIG.

[0132] First, the insight verbalization unit 55 acquires the series data of the aggregated data that will become the insight I and the query Qk (step S41). Then, the insight verbalization unit 55 generates an association vector, which is a feature vector relating to the association between the cell and the query Qk, for each cell of the aggregated data (or each group of rows or columns) (step S42).

[0133] Next, the insight verbalization unit 55 determines additional information to be input to the Data-to-Text model based on the relevance vector, or selects cells of aggregated data to be input to the Data-to-Text model (step S43). In the former case, the insight verbalization unit 55 uses the relevance vector as additional information to be input to the intermediate layer of the Data-to-Text model, as shown in Figure 11. On the other hand, in the latter case, the insight verbalization unit 55 selects cells of aggregated data to be input to the Data-to-Text model based on the relevance score calculated from the relevance vector, as shown in Figure 12.

[0134] Then, the insight verbalization unit 55 acquires a verbalized insight EI by applying the Data-to-Text model (step S44). In this case, the insight verbalization unit 55 acquires, as the verbalized insight EI, an explanatory sentence output by the Data-to-Text model when the sequence data of the aggregated data and the query Qk are input to the Data-to-Text model.

[0135] 19 shows the configuration of a data analysis system 100A. The data analysis system 100A mainly includes a data analysis device 1A and a terminal device 5. The data analysis device 1A and the terminal device 5 perform data communication via a network 6.

[0136] The data analysis device 1A is one or more devices that function as a server (including a cloud server) and performs processing related to the data analysis executed by the data analysis device 1 in the first embodiment. In this case, the data analysis device 1A receives input information from the terminal device 5 via the network 6, which the data analysis device 1 receives from the input device 2 in the first embodiment. Furthermore, the data analysis device 1A transmits display information that the data analysis device 1 transmitted to the display device 3 in the first embodiment to the terminal device 5 via the network 6. Furthermore, the data analysis device 1A stores the data warehouse DWH of the first embodiment, or references the data warehouse DWH via the network 6.

[0137] The terminal device 5 is a terminal having an input function, a display function, and a communication function, and functions as the input device 2 and the display device 3 in the first embodiment. The terminal device 5 may be, for example, a personal computer, a tablet terminal, a PDA (Personal Digital Assistant), or the like. The terminal device 5 transmits input information generated based on the received user input to the data analysis device 1A via the network 6. Furthermore, when the terminal device 5 receives display information from the data analysis device 1A, it displays information based on the display information.

[0138] The data analysis device 1A according to the second embodiment can preferably perform the input process and output process that the data analysis device 1 according to the first embodiment performs on the user of the terminal device 5 .

[0139] 20 is a diagram showing the relationship between a user, a data analysis device 1A, and a terminal device 5. In this case, the data analysis device 1A functions as a server that executes an algorithm, and the terminal device 5 functions as a user terminal that accepts input of parameters and the like necessary for the algorithm. The terminal device 5 exchanges information with the data analysis device 1A to present the user with display screens such as those shown in FIGS. 14 and 15, etc. This can favorably encourage the user to make a decision.

[0140] The present disclosure is suitably applied to evidence-based policy making (EBPM) in the medical field. For example, while each local government is required to formulate effective medical policies, there are cases where only a few people are considering measures for a local government with tens of thousands of residents, and due to factors such as a lack of a large budget, formulating effective medical policies can be difficult. In particular, medical and healthcare-related data is rarely integrated and managed. In such cases, by using the data analysis system 100A, the data analysis system 100A can support evidence-based policy formulation in fields such as the medical and healthcare field where databases are not integrated, allowing users to formulate policies based on hypotheses without relying on experts.

[0141] An application example of EBPM in the medical field will be described below. In this application example, a user is a healthcare policy planner using a terminal device 5. The user accesses a data analysis device 1A, which functions as a server device of a data analysis system 100A that performs cross-sectional data verification, through the terminal device 5, and performs operations such as creating an account and logging in. The healthcare policy planner then inputs a hypothesis as a query on a portal screen, such as that shown in FIG. 14 , on which the data analysis device 1A generates display information. The data analysis device 1A then transmits display information to the terminal device 5, including an answer to the input query and visual data such as supporting documents and charts, as shown in FIGS. 14 and 15 . In this case, the healthcare policy planner checks the analysis results for the input query on the screen displayed by the terminal device 5 and investigates existing information and case studies related to home medical support for the elderly, health classes, and exercise programs, as well as local healthcare needs. Healthcare policymakers will also review the information provided on the screen and decide whether it is appropriate to provide new healthcare services (such as home medical support, health classes, and exercise programs) that can extend the healthy lifespan of the elderly and address local medical needs.

[0142] In this case, the data warehouse DWH referred to by the data analysis device 1A may include various databases related to medical databases, welfare policies, and the like.

[0143] FIG. 21(A) shows an example of medical receipt data related to national health insurance and medical assistance. FIG. 21(B) shows an example of data showing the results of specific insurance guidance. FIG. 21(C) shows an example of medical receipt data related to nursing care. FIG. 22 is a database of information posted (tweets) on social media by citizens. The data analysis device 1A cross-sectionally collects and references various databases (including medical databases and databases related to welfare policies) such as those shown in FIGS. 21(A) to 21(C) and 22 as elements of a data warehouse DWH, and performs processing to extract related documents ED and related tables X, etc. The data analysis device 1A may also analyze the collected information using techniques such as data mining and statistical analysis, and convert or extract useful information.

[0144] According to this application example, the data analysis system 100A can effectively support policy making by medical policy makers.

[0145] 23A is a functional block diagram of a study device 1X according to a third embodiment. The study device 1X mainly includes a query acquisition unit 17Xa, an association table acquisition unit 17Xb, an insight document acquisition unit 17Xc, and a study unit 17Xd. The study device 1X may be composed of multiple devices.

[0146] The query acquisition unit 17Xa acquires a query. The associated table acquisition unit 17Xb acquires an associated table, which is a table related to the query. The insight document acquisition unit 17Xc acquires a document obtained by converting aggregated data obtained by aggregating the associated table into natural language, as a document representing an insight useful for answering the query. The "document obtained by converting aggregated data into natural language" and the "document representing an insight" are, for example, the verbalized insight EI in the first or second embodiment. The learning unit 17Xd trains a machine learning model that evaluates the association between a table and a query based on the degree of association between the document and the query. The query acquisition unit 17Xa, the associated table acquisition unit 17Xb, the insight document acquisition unit 17Xc, and the learning unit 17Xd can be, for example, the evaluation model learning unit 17 in the first or second embodiment.

[0147] The learning device 1X can be the data analysis device 1 in the first embodiment or the data analysis device 1A in the second embodiment. In another example, the learning device 1X may be configured separately from a device that performs data analysis based on the data warehouse DWH.

[0148] FIG. 24(A) shows a first configuration example of a learning device 1X configured separately from the data analysis device 1B, and FIG. 24(B) shows a second configuration example of a learning device 1X configured separately from the data analysis device 1B. The data analysis device 1B shown in FIGS. 24(A) and 24(B) corresponds to the data analysis device 1 of the first embodiment or the data analysis device 1A of the second embodiment, excluding the function corresponding to the evaluation model learning unit 17. In the first configuration example shown in FIG. 24(A), the learning device 1X acquires a query, an association table, and verbalized insights (documents representing the insights) as part of the processing results of the data analysis by the data analysis device 1B, and performs learning of a machine learning model. In the second configuration example shown in FIG. 24(B), the data analysis device 1B stores processing results including the query, the association table, and the verbalized insights (documents representing the insights) in the storage device 1Y, and the learning device 1X acquires the query, the association table, and the verbalized insights (documents representing the insights) from the storage device 1Y, and performs learning of a machine learning model.

[0149] 25 is an example of a flowchart executed by the learning device 1X in the third embodiment. The query acquisition unit 17Xa acquires a query (step S51). The association table acquisition unit 17Xb acquires an association table, which is a table related to the query (step S52). The insight document acquisition unit 17Xc converts aggregated data from the association table into natural language and acquires the document as a document representing an insight useful for answering the query (step S53). The learning unit 17Xd trains a machine learning model that evaluates the association between a table and a query based on the degree of association between the document and the query (step S54).

[0150] The learning device 1X according to the third embodiment can suitably execute learning of a machine learning model that evaluates the association between a table and a query.

[0151] In addition, part or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0152] [Supplementary Note 1] A learning device comprising: a query acquisition means for acquiring a query; an association table acquisition means for acquiring an association table that is a table related to the query; an insight document acquisition means for acquiring a document obtained by converting aggregated data obtained by aggregating the association table into natural language, as a document expressing insight useful for answering the query; and a learning means for training a machine learning model that evaluates the association between the query and a table that is a candidate for the association table based on the degree of association between the document and the query. [Supplementary Note 2] The learning device described in Supplementary Note 1, wherein the learning means optimizes parameters of the machine learning model so as to increase the degree of association. [Supplementary Note 3] The learning device described in Supplementary Note 2, wherein the learning means optimizes parameters of the machine learning model based on the query and the association table used to generate the document whose degree of association is equal to or greater than a threshold. [Supplementary Note 4] The learning device described in Supplementary Note 3, wherein the learning means optimizes parameters of the machine learning model so that the evaluation output by the machine learning model is a predetermined evaluation when data based on the query and the association table used to generate the document whose degree of association is equal to or greater than the threshold is input to the machine learning model. [Supplementary Note 5] The learning device according to Supplementary Note 1, wherein the machine learning model includes one or more encoders to which the query and sequence data converted from the table are input. [Supplementary Note 6] The learning device according to Supplementary Note 1, wherein the associated table is selected from the candidate tables based on an evaluation output by the machine learning model when data based on each of the candidate tables and the query is input to the machine learning model. [Supplementary Note 7] The learning device according to Supplementary Note 1, wherein the learning means trains the machine learning model based on the associated table, the insight, and the document acquired or generated by data analysis based on the query and the table. [Supplementary Note 8] The learning device according to Supplementary Note 7, further comprising data analysis means for performing the data analysis, wherein the data analysis means generates an answer to the query based on the document. [Supplementary Note 9] The learning device according to Supplementary Note 8, further comprising display control means for displaying information related to the answer and the document on a display device.[Supplementary Note 10] A learning method in which a computer receives a query, receives an association table that is a table related to the query, receives a document obtained by converting aggregated data obtained by aggregating the association table into natural language as a document representing insights useful in answering the query, and trains a machine learning model that evaluates a relationship between the query and a table that is a candidate for the association table, based on the degree of relationship between the document and the query. [Supplementary Note 11] A storage medium storing a program that causes a computer to execute the processes of receiving a query, receives an association table that is a table related to the query, receives a document obtained by converting aggregated data obtained by aggregating the association table into natural language as a document representing insights useful in answering the query, and trains a machine learning model that evaluates a relationship between the query and a table that is a candidate for the association table, based on the degree of relationship between the document and the query.

[0153] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0154] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0155] 1, 1A, 1B Data analysis device 1X Learning device 2 Input device 3 Display device 5 Terminal device 100, 100A Data analysis system

Claims

1. A query retrieval method for obtaining queries, Related table acquisition means for acquiring related tables which are tables related to the aforementioned query, An insight document acquisition means obtains a document that converts aggregated data obtained by aggregating the aforementioned related tables into natural language, and uses this document to obtain insights that are useful for answering the query. A learning means for training a machine learning model that evaluates the relationship between candidate tables for the related table and the query based on the degree of relationship between the document and the query, A learning device having the following features.

2. The learning device according to claim 1, wherein the learning means optimizes the parameters of the machine learning model so that the degree of association increases.

3. The learning device according to claim 2, wherein the learning means optimizes the parameters of the machine learning model based on the association table and the query used to generate the documents whose degree of association is above a threshold.

4. The learning device according to claim 3, wherein the learning means optimizes the parameters of the machine learning model so that when data based on the association table and the query used to generate the document whose degree of association is equal to or greater than the threshold is input to the machine learning model, the evaluation output by the machine learning model becomes a predetermined evaluation.

5. The learning device according to claim 1, wherein the machine learning model includes one or more encoders into which the sequence data obtained by transforming the table and the query are input.

6. The learning device according to claim 1, wherein the associated table is selected from the candidate tables based on an evaluation output by the machine learning model when data based on each of the candidate tables and the query is input to the machine learning model.

7. The learning device according to claim 1, wherein the learning means learns the machine learning model based on the related tables, insights, and documents obtained or generated by data analysis based on the queries and the tables.

8. The system further includes data analysis means for performing the aforementioned data analysis, The learning device according to claim 7, wherein the data analysis means generates answers to the queries based on the documents.

9. Computers Get the query, Retrieve the related tables, which are tables related to the aforementioned query. The aggregated data obtained by summarizing the aforementioned related tables is converted into a document in natural language, which is then obtained as a document representing insights useful for answering the query. Based on the degree of relevance between the document and the query, a machine learning model is trained to evaluate the relationship between candidate tables for the related table and the query. Learning methods.

10. Get the query, Retrieve the related tables, which are tables related to the aforementioned query. The aggregated data obtained by summarizing the aforementioned related tables is converted into a document in natural language, which is then obtained as a document representing insights useful for answering the query. A program that causes a computer to perform a process of training a machine learning model that evaluates the relationship between candidate tables for the related table and the query, based on the degree of relationship between the document and the query.