Platform for data observation in large-scale database construction

Abstract

To provide an information processor, an information processing method and an information processing program, allowing observation of accuracy and behavior of data, and improving the accuracy and availability of the data, when constructing and updating a database by aggregating data from a plurality of data sources.SOLUTION: The data observation apparatus 1 includes a data set extraction unit 12 that extracts a data set from a plurality of databases respectively belonging to a plurality of platforms, a data quality analysis unit 14 that analyzes quality of each data set of the data set by using a first metric for the data set, a data verification unit 15 that verifies a data value of each item of the data set by using a second metric for the data set, and a metric construction unit 13 that dynamically constructs at least a part of the first metric and the second metric to be used for the data set based on a processing history of the data set.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a data observation platform for building large-scale databases, and more particularly to a technique for observing data to be stored in a database in a workflow for extracting data from a data source and building and updating the database. [Background technology]

[0002] Electronic commerce (EC) platforms accumulate huge amounts of data on customers who use each EC site provided on the platform every day. This customer data includes each customer's attributes, behavioral history on the EC site, etc., and is scattered across multiple data platforms located in the cloud and on-premise, with data being added and updated continuously.

[0003] It is possible to aggregate data from multiple customers that exists across these multiple data platforms, extract data sets from the aggregated data, and build an integrated customer database. By referencing the integrated customer database constructed in this way, it is possible to, for example, model a large number of customers by cluster and use the modeled cluster's behavior and persona for analysis and prediction. As an example of a use case, if personalized advertisements are provided to clusters modeled in this way, it is possible to expect an improvement in advertising effectiveness, as indicated by indicators such as conversion rate (CVR).

[0004] Patent Document 1 discloses a data analysis device that analyzes records of customer behavior history and extracts potential customers through a comprehensive trial and error process. Specifically, the data analysis device disclosed in Patent Document 1 has a comparison unit that comprehensively compares data on customers in a desirable state with customer data on customers in an undesirable state based on customer behavior parameters, which are factors that influence whether a customer is in a desirable state or not, and a potential customer / basis extraction unit that, based on the comparison results from the comparison unit, finds at least one of promising potential customers from among the customers in an undesirable state and potential basis for the parameters.

[0005] However, such a customer's integrated database contains a huge amount of data, includes many interrelated tables each with many columns, and has a complex data structure. Furthermore, data is frequently added and updated in each of the multiple data platforms that serve as data sources, and as a result, the amount and range of data in the datasets extracted from the data sources fluctuates frequently in response to changes in the data sources. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-135616 Summary of the Invention [Problem to be solved by the invention]

[0007] However, data sets extracted from data sources scattered across multiple data platforms inevitably contain errors, such as data that partially contains incorrect values ​​or data types, missing or duplicated data, etc. To avoid a decrease in the accuracy of data stored in the integrated database or a system downtime caused by such errors, a scheme for filtering out errors that should not be stored in the integrated database is required when processing datasets.

[0008] However, as mentioned above, the amount and range of data in the datasets extracted from the data sources fluctuate frequently in response to changes in the data sources. Therefore, if the evaluation criteria for filtering errors are uniform, it will not be possible to keep up with these changes in the datasets, and the accuracy of the data stored in the customer's integrated database will decrease. On the other hand, if the error evaluation criteria for each of the multiple data platforms that are the data sources are updated each time, the system becomes complicated and delays occur. In either case, it is not easy to monitor and maintain the accuracy of the data across the entire integrated database.

[0009] The present disclosure has been made to solve the above-mentioned problems, and its purpose is to provide an information processing device, an information processing method, and a program that enable the accuracy and behavior of data to be observed when aggregating data from multiple data sources to build and update a database, and that can improve the accuracy and availability of the data. [Means for solving the problem]

[0010] In order to solve the above problem, one aspect of an information processing device according to the present invention comprises a dataset extraction unit that extracts datasets from multiple databases that belong to multiple platforms, a quality analysis unit that analyzes the quality of each dataset of the dataset using first metrics for the dataset, a data verification unit that verifies data values ​​for each item of the dataset using second metrics for the dataset, and a metrics construction unit that dynamically constructs at least a portion of the first metrics and the second metrics to be used for the dataset based on the processing history of the dataset.

[0011] One aspect of the information processing method of the present invention is an information processing method executed by an information processing device, comprising the steps of extracting datasets from multiple databases belonging to multiple platforms, analyzing the quality of each dataset of the datasets using first metrics for the datasets, verifying data values ​​for each item of the datasets using second metrics for the datasets, and dynamically constructing at least a portion of the first metrics and the second metrics to be used for the datasets based on the processing history of the datasets.

[0012] One aspect of the information processing program of the present invention is an information processing program for causing a computer to execute information processing, the program causing the computer to execute processes including: a dataset extraction process for extracting datasets from multiple databases each belonging to multiple platforms; a quality analysis process for analyzing the quality of each dataset of the dataset using first metrics for the dataset; a data verification process for verifying data values ​​for each item of the dataset using second metrics for the dataset; and a metrics construction process for dynamically constructing at least a portion of the first metrics and second metrics to be used for the dataset based on the processing history of the dataset. [Effects of the Invention]

[0013] According to the present invention, when a database is constructed and updated by aggregating data from multiple data sources, the accuracy and behavior of the data can be observed, and the accuracy and availability of the data can be improved. The above-mentioned objects, aspects, and advantages of the present invention, as well as other objects, aspects, and advantages of the present invention not described above, will be understood by those skilled in the art from the following detailed description of the invention by referring to the accompanying drawings and the claims. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a data observation device according to each embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a workflow configuration when the data observation device according to this embodiment is implemented in an integrated database construction workflow. [Figure 3] FIG. 3 is a flowchart showing an example of detailed processing procedures of the data quality check processing executed by the data observation device according to this embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a screen displaying the results of the freshness check output to the client device via the UI in step S32 of FIG. [Figure 5] FIG. 5 is a diagram showing an example of a screen displaying the execution result of the volume check output to the client device via the UI in step S33 of FIG. [Figure 6A] FIG. 6A is a diagram showing an example of a screen displaying the execution result of the data distribution check output to the client device via the UI in step S34 of FIG. [Figure 6B] FIG. 6B is a diagram showing an example of a screen displaying another execution result of the data distribution check output to the client device via the UI in step S34 of FIG. [Figure 6C] FIG. 6C is a diagram showing an example of a screen displaying another execution result of the data distribution check output to the client device via the UI in step S34 of FIG. [Figure 7] FIG. 7 is a flowchart showing an example of detailed processing procedures of the data verification processing executed by the data observation device according to this embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of detailed processing steps of the metrics construction processing executed by the data observation device according to this embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of a detailed processing procedure of the error processing executed by the data observation device according to this embodiment. [Figure 10]FIG. 10 is a diagram illustrating an example of definitions of data quality analysis metrics. [Figure 11] FIG. 11 is a diagram illustrating an example of the definition of data verification metrics. [Figure 12] FIG. 12 is a block diagram showing an example of the hardware configuration of the data observation device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, with reference to the accompanying drawings, an embodiment for carrying out the present invention will be described in detail. Among the components disclosed below, those having the same function will be given the same reference numerals, and their description will be omitted. Note that the embodiment disclosed below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0016] The data observation device according to this embodiment extracts data sets from multiple customer databases belonging to multiple data platforms (hereinafter also simply referred to as "platforms"), applies first metrics to the extracted data sets to analyze the quality of each data set, and applies second metrics to the extracted data sets to verify the data values ​​of each item in the data sets. The data observer also dynamically constructs at least a portion of the first metric and the second metric to be applied to the dataset based on a processing history of the dataset.

[0017] Below, we will explain an example in which the data observation device of this embodiment extracts a dataset from customer data such as customer behavior history on e-commerce sites and customer attributes accumulated across multiple platforms, builds an integrated customer database from the extracted dataset, and sequentially updates the integrated customer database, but this embodiment is not limited to this. This embodiment is applicable not only to platform customer data, but also to any data that is aggregated and used for any purpose. In this embodiment, a data set is extracted from the customer data, and if necessary, the data set is converted into an appropriate format such as a hierarchical table, and stored in an integrated database. The integrated database thus constructed may be updated at any frequency and may differ depending on the data source and the type of data set.

[0018] <Functional configuration of data observation device> FIG. 1 is a block diagram showing an example of the functional configuration of a data observation device 1 according to this embodiment. The data observation device 1 shown in FIG. 1 includes a data acquisition unit 11, a data set extraction unit 12, a metrics construction unit 13, a data quality analysis unit 14, a data verification unit 15, a data collection unit 16, a database output unit 17, and an orchestrator 18.

[0019] The data observation device 1 is configured to be able to access the data sources 2 distributed across multiple platforms, the integrated database 4, and the data set processing history 5, as well as the respective storage devices. The data observation device 1 connects to data sources 2 distributed across multiple platforms, communicates with the connected data sources to read and aggregate customer data stored in the storage devices of the data sources, extracts a dataset from the aggregated customer data, and builds an integrated database 4 from the extracted dataset. The data observation device 1 also stores the processing results obtained by processing the dataset in a dataset processing history 5. The data source 2, the integrated database 4, and the data set processing history 5 are provided with a storage device, which may be configured as a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0020] The data observation device 1 may also include one or more client devices 3, such as PCs (Personal Computers), or may be communicably connected to the client devices 3 via a network. In this case, the data observation device 1 is implemented in a server, and the client devices 3 may provide a user interface for the data observation device 1 to input and output information from and to the outside, and may also include some or all of the components 11 to 18 of the data observation device 1. In this embodiment, the client device 3 may provide a user interface for setting and editing various parameters and templates in the data quality check processing and data verification processing performed by the data observation device 1, and a user interface for outputting search results and processing results of the data quality check processing and data verification processing.

[0021] The data acquisition unit 11 connects to data sources 2 distributed across multiple platforms, acquires customer data from the connected data sources 2 via a communication I / F, and supplies the acquired customer data to the data set extraction unit 12. The data acquisition unit 11 may receive customer data from each of the data sources 2 of the multiple platforms periodically or when an update occurs. Alternatively, the data acquisition unit 11 may periodically issue queries to each of the multiple data sources 2 and receive customer data in response to the queries.

[0022] The data acquisition unit 11 may acquire customer data by directly connecting to the data source 2 and reading out customer data pre-stored in the storage device of the data source 2, or may receive customer data via the same or a different counterpart device that manages the data source 2. The data acquisition unit 11 also accepts input of various parameters necessary for executing the data observation process in the data observation device 1. The data acquisition unit 11 may accept input of various parameters via a user interface of a client device 3 connected to the data observation device 1.

[0023] The customer data acquired by the data acquisition unit 11 may typically include any information related to the customer's use of e-commerce sites, etc. on the platform, including the customer's behavioral history such as searching, browsing, purchasing, etc. on e-commerce sites, etc., as well as the customer's personal or segment attributes, etc.

[0024] The data set extraction unit 12 extracts a data set from the customer data supplied from the data acquisition unit 11 and supplies it to the metrics construction unit 13, data quality analysis unit 14, and data verification unit 15 at the subsequent stages. The dataset extracted by the dataset extraction unit 12 is a unit for storing or processing data extracted from the customer data acquired by the data acquisition unit 11 as information to be stored in the integrated database 4. In this embodiment, the dataset includes data describing the attributes, characteristics, structure, meaning, and relationships that represent the customer data. In this embodiment, a case will be described as an example in which the dataset extracted by the dataset extraction unit 12 includes demographic data as demographic attributes of customers who use an EC site, etc. This demographic data is demographic attributes used as indices for analyzing customer data, and specifically includes gender, age, residential area, occupation, annual income, educational background, family structure, etc.

[0025] The metrics construction unit 13 constructs metrics for evaluating the data set extracted by the data set extraction unit 12 . Specifically, the metrics construction unit 13 dynamically constructs metrics for the data quality analysis unit 14 to analyze the data quality of a dataset, and metrics for the data verification unit 15 to verify the dataset. The former are called data quality analysis metrics, and the latter are called data verification metrics. The metrics construction unit 13 may construct multiple data quality analysis metrics and data verification metrics. The metrics constructed by the metrics construction unit 13 are indices set according to the conditions (parameters) to be applied in the data quality check performed by the data quality analysis unit 14 or the data verification performed by the data verification unit 15. In this embodiment, the metrics construction unit 13 dynamically constructs metrics to be applied to the next dataset to be processed by referring to the processing history 5 of the dataset collected by the data collection unit 16. Details of this metrics construction process will be described later with reference to Figure 8.

[0026] The data quality analysis unit 14 applies the data quality analysis metrics constructed by the metrics construction unit 13 to the data set extracted by the data set extraction unit 12, thereby analyzing the quality of the data set. The data quality analysis metrics applied to the datasets by the data quality analysis unit 14 are metrics for analyzing the quality of each dataset in the dataset over time. Therefore, the data quality analysis unit 14 analyzes the quality of each dataset in the dataset. The data quality analysis unit 14 may execute a process of applying multiple data quality analysis metrics to each dataset in parallel in a pipeline. Details of the data quality check process for analyzing the data quality will be described later with reference to FIG. 3.

[0027] The data verification unit 15 verifies the data set extracted by the data set extraction unit 12 by applying the data verification metrics constructed by the metrics construction unit 13 to the data set. The data verification metrics applied to a dataset by the data verification unit 15 are metrics for verifying data for each item of the dataset. Therefore, the data verification unit 15 verifies data for each item of the dataset. The data verification unit 15 may execute a process of applying multiple data verification metrics to each dataset in parallel in a pipeline. Details of this data verification process will be described later with reference to FIG. 7.

[0028] The data collection unit 16 acquires the results of the data quality analysis processing of the dataset output from the data quality analysis unit 14 and the results of the data verification processing of the dataset output from the data verification unit 15, and stores the processing results of these datasets in the dataset processing history 5. The data collection unit 16 further provides the dataset processing history 5 to the metrics construction unit 13. The dataset processing history 5 collected by the data collection unit 16 is referenced by the metrics construction unit 13 and used as parameters for dynamically reconstructing data quality analysis metrics and data verification metrics.

[0029] The database output unit 17 outputs the data set extracted by the metadata extraction unit 12 to the integrated database 4. The database output unit 17 may convert the extracted data set into a desired format according to the requirements of downstream applications that process the integrated database 4, such as customer analysis and prediction, for example, into a format of multiple tables that are interrelated in a hierarchical structure. The database output unit 17 may exclude, as error data, the whole or part of a dataset or the data items of a dataset that fall outside the thresholds set for each applied metric from the processing results of the data quality check processing by the data quality analysis unit 14 and the processing results of the data verification processing by the data verification unit 15, from those to be output to the integrated database 4. In this case, the error data does not constitute the integrated database 4, but is instead output to the dataset processing history 5 as error data.

[0030] The orchestrator 18 executes a workflow for constructing the integrated database 4, and performs various settings, management, and adjustments in the workflow execution. In this embodiment, a data observation workflow executed by the data observation device 1 is added to this integrated database construction workflow, and the orchestrator 18 also performs various settings, management, and adjustments in the data observation workflow. Specifically, the orchestrator 18 performs scheduling for the workflow execution of the data observation device 1, issues queries using each metric in the data quality check processing and data verification processing, and performs various error processing based on the error data output from the data quality check processing and data verification processing. The orchestrator 18 also controls pipeline processing in the workflow. Specifically, the orchestrator 18 controls the data quality check processing performed by the quality analysis unit 14 and the data verification processing performed by the data verification unit 15 so that they are processed in parallel in the pipeline. Furthermore, the orchestrator 18 controls the application of multiple quality analysis metrics in the data quality check processing performed by the quality analysis unit 14 and the application of multiple data verification metrics in the data verification processing performed by the data verification unit 15 so that they are processed in parallel in the pipeline.

[0031] <Implementation of data observation in the integrated database construction workflow> FIG. 2 is a diagram showing an example of the configuration when the data observation device 1 according to this embodiment is implemented in an integrated database construction workflow. Referring to FIG. 2 , the integrated database construction workflow incorporates a data source connector 201, a dataset extractor 202, a metrics configurator 203, a data quality checker 204, a data validator 205, a data observation client 206, a data collector 207, a data store 208, and a user interface 209.

[0032] The integrated database construction workflow executes, in a pipeline, an extraction process that extracts datasets to be processed from multiple data sources 2, a conversion process that converts the extracted datasets into a format for storage in the integrated database 4, and a loading process that loads the converted data into the integrated database 4. 2 may be incorporated at the source code level as part of the pipeline processing executed by this integrated database construction workflow. The data observation functions executed by the components 201 to 209 constitute part of the pipeline of the integrated database construction workflow, so there is no need to install a separate virtual machine or server for data observation, and it is possible to reduce the load on the CPU (Central Processing Unit) and GPU (Graphics Processing Unit) and memory resources such as RAM (Random Access Memory) when executing data observation.

[0033] Referring to FIG. 2, at F1, a data source connector 201 connects an integrated database construction workflow to multiple data sources 210 distributed across multiple platforms. At F2, the dataset extractor 202 extracts datasets to be processed from the databases of multiple data sources 210 connected to the workflow by the data source connector 201, and stores the extracted target datasets in a dataset storage unit 211 consisting of a storage device. In this embodiment, the dataset extracted by F2 includes demographic data of customers who use the e-commerce site, specifically including gender, age, residential area, occupation, annual income, educational background, family structure, etc. In F2, the dataset extractor 202 may extract the table names of multiple tables that make up this demographic data, the data types of all columns in each table, the location of the data, etc., together with the demographic data via an API (Application Programming Interface).

[0034] At F3, the metrics configurator 203 dynamically constructs data quality analysis metrics for analyzing the data quality of the dataset extracted at F2, and data validation metrics for validating the data of the dataset extracted at F2. Specifically, the metrics configurator 203 stores the constructed data quality analysis metrics and data verification metrics in a metrics storage unit 212, which is configured as a storage device. The metrics configurator 203 supplies the constructed data quality analysis metrics together with a data set to the data quality checker 204. In parallel with this, the metrics configurator 203 supplies the constructed data verification metrics together with the data set to the data validator 205.

[0035] In F4, the data quality checker 204 applies (uses) the data quality analysis metrics supplied from the metrics configurator 203 to the datasets to analyze the data quality of each dataset, and in F5 outputs a report of the analysis results of the data quality analysis of the datasets. Note that a dataset may be composed of a single table, or may be composed of multiple interrelated tables. The data quality checker 204 may apply multiple data quality analysis metrics, each of different types, to the data sets to analyze the data quality of each of the data sets, F4. Specifically, the data quality checker 204 determines that a dataset that is within the threshold set for each data quality analysis metric is a dataset that meets the predetermined data quality standard for constructing the integrated database 213. On the other hand, the data quality checker 204 determines that a dataset that is outside the threshold set for each data quality analysis metric is error data.

[0036] In F6, the data validator 205 applies (uses) the data validation metrics supplied from the metrics configurator 203 to the dataset to validate the data for each item in the dataset, and in F7 outputs a report of the data validation results for the dataset. In F6, the data validator 205 may apply user-defined data validation metrics to the data set instead of or in addition to the data validation metrics provided by the metrics configurator 203. In addition, in F6, the data validator 205 may apply multiple data validation metrics, each of different types, to the data set to perform data validation for each data item of the data set.

[0037] Specifically, the data validator 205 determines that data items in a dataset that are within the thresholds set for the respective data validation metrics are datasets that meet the predetermined data validation criteria for constructing the integrated database 213. On the other hand, the data validator 205 determines that data items in a dataset that are outside the thresholds set for the respective data validation metrics are error data.

[0038] The data quality checker 204 and the data validator 205 may simultaneously execute the data quality analysis of F4 and the data verification of F6 in parallel. Because the data quality check and data verification for the dataset do not interfere with or depend on each other, executing them simultaneously in the integrated database construction workflow can improve the throughput of the integrated database construction.

[0039] In F8, the data collector 206 collects the data quality check results output from the data quality checker 204 and the data verification results output from the data validator 205, and feeds them back to the metrics configurator 203. The data quality check results and data verification results collected by the data collector 206 are supplied to the metrics configurator 203 as a data set processing history, and are also stored in the dataset storage unit 211, and are used for dynamic construction of metrics executed by the metrics configurator 203. The data collector 206 also supplies to the data store 208 data sets that are determined to be within the valid range of the thresholds set for the respective metrics in the data quality check in F4 and the data validation in F6.

[0040] The data observation client 207 provides an interface for each of the multiple data quality checks in F9, and also schedules the entire integrated database construction workflow together with each of the workflow components 201 to 209 in F10. The data observation client 207 can also issue queries to the data quality checker 204 and the data validator 205 via the user interface to observe the time-series progress and statistical output of the data quality check using data quality analysis metrics, and can inquire about the data quality check results for a specific data set or data item.

[0041] In F11, the data storage device 208 stores the data set supplied from the data collector 206 in the integrated database 213 at the back end. The data set stored in the integrated database 213 in F11 is a data set that has cleared the standards set for the respective metrics in the data quality check in F4 and the data validation in F6. In addition, the data set stored in the integrated database 213 in F11 is data that has been converted into a data structure and data format for the integrated database 213 in the workflow.

[0042] The data quality check results and data validation results collected by the data collector 206 may be stored in the metadata processing history 5 in, for example, the format of OpenTelemetry, an open-source observability framework. Similarly, the data store 208 may build the integrated database 4 in the OpenTelemetry format in F11. Furthermore, reporting in F5 and F7 may be output in the OpenTelemetry format. This provides a standard API to the dataset processing history 5 and the integrated database 4, facilitating processing by third parties and enhancing extensibility.

[0043] In F12, the user interface 209 visualizes the processing results, error occurrence status, etc. of each process F1 to F11 executed by each of the workflow components 201 to 208. The user interface 209 can provide an appropriate visualization interface according to the situation that requires observation in the integrated database construction workflow, and can enhance visibility by appropriately processing the processing results of, for example, data quality checks and data verifications graphically.

[0044] <Detailed procedure for data quality check processing> FIG. 3 is a flowchart showing an example of detailed processing procedures of the data quality check processing executed by the data observation device according to this embodiment. Each step in FIG. 3 is realized by a CPU reading and executing a program stored in a storage device such as an HDD of the data observation device 1. At least a part of the flowchart shown in FIG. 3 may be realized by other hardware such as a GPU. When realized by hardware, for example, a predetermined compiler may be used to automatically generate a dedicated circuit on an FPGA (Field Programmable Gate Array) from a program for realizing each step. Alternatively, a gate array circuit may be formed in the same manner as an FPGA and realized as hardware. Alternatively, it may be realized by an ASIC (Application Specific Integrated Circuit). The same applies to the steps shown in FIGS. 7 to 9, which will be described later.

[0045] In step S31, the data quality analysis unit 14 of the data observation device 1 acquires data quality analysis metrics supplied from the metrics construction unit 13. In step S31, the data quality analysis unit 14 may acquire multiple data quality analysis metrics of different types.

[0046] In step S32, the data quality analysis unit 14 selects data quality analysis metrics for freshness check from the multiple data quality analysis metrics acquired in step S31, and applies the selected data quality analysis metrics for freshness check to the dataset to perform a freshness check of the dataset.

[0047] A freshness check is a process for analyzing the freshness of data. Specifically, in step S32, the data quality analysis unit 14 analyzes how up-to-date the data is. For example, by measuring the time that has passed since the data was last updated or integrated, the freshness of the data, i.e., whether the data has been updated when it should be updated, can be evaluated. For example, a customer demographics table may be updated daily because it serves as a data source for many downstream tables derived from it and therefore must be up-to-date. Therefore, updating the demographics table is central to data decision-making across the platform. The data quality analyzer 14 may retrieve data freshness information via the orchestrator 18 scheduler and calculate data freshness using statistical functions.

[0048] Fig. 4 is a diagram showing an example of a screen displaying the results of the freshness check output to the client device via the UI in step S32 of Fig. 3. Referring to Fig. 4, field 401 on the screen shows the timestamp of when the table "general_demography" of the dataset was last updated. For data quality analysis metrics for freshness checks, one day (24 hours) may be set as the threshold indicating the valid range of the update interval. Alternatively, by referring to the processing history of past datasets, the threshold indicating the valid range for data quality analysis metrics for freshness checks may be dynamically changed based on the update interval or update frequency of the target dataset over a predetermined period in the past. The data quality analysis unit 14 may treat a data set that has not been updated for a period exceeding the update interval set in the data quality analysis metrics for freshness check as error data and may output an alert or error message via the user interface. In this case, the data source 2 that supplies the customer's data may be notified of the error and prompted to resend the data.

[0049] In step S33, the data quality analysis unit 14 selects data quality analysis metrics for volume check from the multiple data quality analysis metrics acquired in step S31, and applies the selected data quality analysis metrics for volume check to the dataset to perform a volume check of the dataset.

[0050] A volume check is a process of analyzing the amount of data generated, integrated, or processed. Specifically, in step S33, the data quality analysis unit 14 analyzes the extent to which the amount of data has increased or decreased. For example, the data quality analysis unit 14 may track the daily increase or decrease in the amount of transaction data in the platform and apply data quality analysis metrics for volume check using the valid range of increase or decrease in data amount as a threshold. For example, it can be assumed that the total number of users in customer demographic data will not fluctuate significantly over a short period of time. Therefore, by monitoring increases or decreases in data volume using data quality analysis metrics for volume checks, if the data volume increases or decreases significantly over a short period of time, it can be assumed that there has been unintended duplication or deletion of data across databases. For example, in a volume check of a dataset, the data quality analysis unit 14 may monitor the data volume by counting the number of rows in a table.

[0051] Fig. 5 is a diagram showing an example of a screen displaying the execution result of the volume check output to the client device via the UI in step S33 of Fig. 3. In the display example shown in Fig. 5, the volume of the dataset is acquired at timestamps every second, and column 501 showing the number of rows (number of users) in the dataset shows that the number of rows in the dataset increased from 1,048,537 to 1,048,542 between 8:30:00 and 8:30:05.

[0052] A threshold indicating an acceptable range for an increase or decrease in the data volume of a dataset may be set for the data quality analysis metrics for volume checks. Alternatively, the threshold indicating an effective range for the data quality analysis metrics for volume checks may be dynamically changed based on the results of monitoring the daily increase or decrease in the volume of the dataset by referring to the processing history of past datasets 5. The data quality analysis unit 14 may treat a data set that has been increased or decreased beyond the allowable increase or decrease amount set in the data quality analysis metrics for volume check as error data and may output an alert or error message via the user interface. In this case, the data source 2, which is the supplier of the customer's data, may be notified of the error and prompted to resend the data.

[0053] In step S34, the data quality analysis unit 14 selects data quality analysis metrics for a data distribution check from the multiple data quality analysis metrics acquired in step S31, and applies the selected data quality analysis metrics for the data distribution check to the dataset to perform a data distribution check of the dataset.

[0054] The data distribution check is a process of analyzing the variance and distribution of the entire data. Specifically, in step S34, the data quality analysis unit 14 analyzes whether the data distribution is balanced or skewed in the distribution check. The data quality analysis unit 14 may also analyze whether the data distributions of segments of different categories are sufficiently different from the data distributions of other segments to adequately represent the segments. For example, customer demographic data is expected to be distributed fairly evenly across segments organized by prefecture, age, and gender, i.e., distributions are expected not to differ significantly across different segments. The data quality analysis unit 14 may periodically track the distribution curve of the data distribution and the shift in the distribution curve, and apply data quality analysis metrics for checking the data distribution, for example, using an acceptable value for the skew of the data distribution as a threshold.

[0055] 6A to 6C are diagrams showing examples of screens that display the execution results of the data distribution check output to the client device via the UI in step S34 of FIG. 6A to 6C show the skew of the data distribution, and in each figure, the mean, median, and mode of the data are plotted. Figure 6B shows an unskewed data distribution, with the mean, median, and mode of the data nearly aligned, whereas Figure 6A shows a negatively skewed data distribution, with the mean and median shifted negatively from the mode, and Figure 6C shows a positively skewed data distribution, with the median and mean shifted positively from the mode.

[0056] By using data quality analysis metrics for checking data distribution, it is possible to monitor the degree of skew in data distribution for the same data set, and also to monitor differences in data distribution skew between different data sets in different segments, thereby identifying data whose data distribution differs significantly due to the presence of outliers, etc.

[0057] A threshold indicating an acceptable range of skew in the data distribution of a dataset may be set for the data quality analysis metrics for data distribution checks. Alternatively, the threshold indicating an effective range for the data quality analysis metrics for distribution checks may be dynamically changed based on the results of monitoring daily fluctuations in the data distribution of the dataset by referring to the processing history of past datasets 5. The data quality analysis unit 14 may treat a data set that has been increased or decreased beyond the allowable increase or decrease amount set in the data quality analysis metrics for checking data distribution as error data and may output an alert or error message via the user interface. In this case, the data source 2, which is the supplier of the customer's data, may be notified of the error and prompted to resend the data.

[0058] In step S35, the data quality analysis unit 14 selects data quality analysis metrics for schema check from the multiple data quality analysis metrics acquired in step S31, and applies the selected data quality analysis metrics for schema check to the dataset to perform a schema check of the dataset.

[0059] Schema checking is a process that analyzes the structure and consistency of data. Specifically, in step S35, the data quality analysis unit 14 analyzes whether the data conforms to a predefined data model or schema. To build an integrated customer database, it is important for all customers to have the same fields and formats to maintain consistency. For example, by monitoring whether the multiple columns in a customer demographic table and the data types of each column are consistent over time, changes in the data schema can be determined. For example, if the data type of a column in a dataset changes from integer to floating-point, it can be determined that a schema change has occurred in the dataset.

[0060] The data quality analysis metrics for schema check may be set to columns of a dataset or a table within the dataset and their data types. Alternatively, the columns and data types for the data quality analysis metrics for schema check may be dynamically changed based on changes in the schema of the data over a predetermined period of time in the past, by referring to the processing history 5 of past datasets. The data quality analysis unit 14 may treat a data set whose schema differs from the columns and data types set in the data quality analysis metrics for schema check as error data, and may output an alert or error message via a user interface. In this case, the data source 2, which is the supplier of the customer's data, may be notified of the error and prompted to resend the data.

[0061] In step S36, the data quality analysis unit 14 selects data quality analysis metrics for data lineage check from the multiple data quality analysis metrics acquired in step S31, and applies the selected data quality analysis metrics for data lineage check to the dataset to perform a data lineage check of the dataset. The data lineage check is a process of tracing the path of data from a data source to a data destination. Specifically, in step S36, the data quality analysis unit 14 analyzes the data source data set or table name (e.g., the name of the parent table that generated the table) and the data destination data set or table name (e.g., the name of a child table to be generated from the table) that are described in or associated with the data set.

[0062] Maintaining predefined data lineage indicates that the data transformations and processing steps of a dataset are transparent and adhere to data governance standards. Data lineage can be obtained by referencing upstream tables that supply data and downstream tables that supply data, creating a coherent representation of the data flow. For example, a demographic table underlying an integrated database may have inputs from diverse data sources and serve as inputs to many downstream tables. For example, monitoring the coherent representation of the data flow of a customer demographic table for consistency over time can identify changes in the data lineage.

[0063] The data quality analysis metrics for data lineage checks may be set to the data lineage of a dataset or a table within a dataset, or the data lineage may be dynamically changed for the data quality analysis metrics for data lineage checks based on changes in the data lineage over a predetermined period of time in the past by referring to the processing history 5 of past datasets. The data quality analysis unit 14 may process a data set having a data sequence that differs from the coherent representation of the data sequence set in the data quality analysis metrics for data sequence check as error data, and may output an alert or error message via a user interface. In this case, the data source 2, which is the supplier of the customer's data, may be notified of the error and prompted to resend the data.

[0064] In step S37, the data quality analysis unit 14 outputs the check results of the multiple data quality checks that were executed by applying the multiple data quality analysis metrics in steps S32 to S36. The data quality checks in steps S32 to S36 shown in FIG. 3 are not necessarily performed in a specific order, and may be performed simultaneously, may be performed in any order, or may not include all of them.

[0065] By using the above multiple data quality analysis metrics, data observability in building an integrated database can be improved, and problems such as data stagnation, sudden data increase, data schema changes, data skew, and data series gaps can be visualized and identified. By applying these data quality analysis metrics to datasets to monitor their quality and behavior over time, it is possible to visually confirm that the datasets from which an integrated database is to be built are reliable, accurate, and suitable for downstream uses.

[0066] <Detailed procedure for data verification processing> FIG. 7 is a flowchart showing an example of detailed processing procedures of the data verification processing executed by the data observation device 1 according to this embodiment. In step S71, the data verification unit 15 of the data observation device 1 acquires data verification metrics supplied from the metrics construction unit 13. In step S71, the data verification unit 14 may acquire a plurality of data verification metrics of different types.

[0067] In step S72, the data verification unit 15 obtains user-defined data verification metrics. The metrics construction unit 13 may allow the user to define new customized data verification metrics for a table or table data items of a new data set via the GUI. Alternatively, the metrics construction unit 13 may allow the user to modify data verification metrics by reading and editing already constructed data verification metrics stored in the metrics storage unit 212 via the GUI.

[0068] In step S73, the data verification unit 15 performs data verification of the data set by applying the data verification metrics acquired in steps S71 and S72 to the data set. Data validation is a process that verifies the quality, accuracy, and consistency of the values ​​(data entries) for each data item in the target dataset or tables within the dataset. The data validation metrics that should be applied to this data validation verify, for each data item, whether a column exists, the data type, the range of data, the uniqueness of data, whether a blank value has been entered in a required field, the percentage of data entered, etc. This makes it possible to confirm that the datasets to be integrated into the integrated database 4 comply with the desired standards and requirements. In step S74, the data verification unit 15 outputs the result of the data verification executed in step S73.

[0069] <Detailed procedure for metrics construction process> FIG. 8 is a flowchart showing an example of detailed processing procedures of the metrics construction processing executed by the data observation device 1 according to this embodiment. In step S81, the metrics construction unit 13 of the data observation device 1 acquires the dataset processing history 5 stored in the storage device. The dataset processing history 5 includes a history of datasets previously extracted from the database of the data source 2, and data quality check and data verification results for datasets collected by the data collection unit 16.

[0070] In step S82, the metrics construction unit 13 derives parameters to be set for data quality analysis metrics to be applied to data quality check of the dataset and data verification metrics to be applied to data verification of the dataset from the dataset processing history 5 acquired in step S81. The parameters to be set for the metrics refer to conditions to be applied in data quality checks and data validation of the dataset, and are also referred to as metadata in this embodiment. Specifically, first, for the data quality analysis metrics, the metrics construction unit 13 may derive a predetermined update interval as a parameter for the data quality analysis metrics for freshness checks based on the update interval of the target dataset during a predetermined period in the past, by referring to the processing history 5 of the past dataset.

[0071] For data quality analysis metrics for volume checks, the metrics construction unit 13 may derive a predetermined threshold for increase / decrease as a parameter based on the increase / decrease in the amount of data in the target dataset (e.g., the number of columns in each table) per day over a predetermined period in the past, by referring to the processing history 5 of the dataset. For example, suppose the standard deviation of the volume of the target dataset acquired each day over the past month is σ. In this case, for example, the volume of the target dataset on the previous day ±n*σ (n is an integer equal to or greater than 1, for example, n=3) may be set as the threshold for each data quality analysis metric for volume check.

[0072] For data quality analysis metrics for data distribution checks, the metrics construction unit 13 may refer to the processing history 5 of past datasets and derive a predetermined fluctuation threshold as a parameter based on fluctuations in the data distribution of the target dataset over a predetermined period in the past. For example, suppose the standard deviations of the mean, median, mode, etc. in the data distribution of a target data set acquired daily over the past month are each σ. In this case, all or part of the values ​​of the mean, median, mode, etc. of the previous day ±n*σ (n is an integer equal to or greater than 1, for example, n=3) may be set as thresholds for the data quality analysis metrics used to check the data distribution.

[0073] The metrics construction unit 13 may derive data quality analysis metrics for schema checking using the columns and their data types of datasets or tables within datasets in past target datasets as parameters by referring to the processing history 5 of past datasets. The metrics construction unit 13 may further derive the data quality analysis metrics for the data series using the data series of the target dataset or the tables within the dataset (for example, descriptions of tables upstream and downstream of the table) as parameters by referring to the processing history 5 of past datasets.

[0074] In step S82, for the data verification metrics, the metrics construction unit 13 may refer to the past metadata processing history 5 and derive a predetermined valid range as a parameter based on the fluctuations in the values ​​of each column (each item) of the dataset of the target dataset or a table within the dataset over a predetermined period in the past. For example, suppose the standard deviation of the monthly average values ​​of the average, maximum, minimum, and standard deviation of the corresponding item of the target dataset acquired in the past month is σ. In this case, for example, the value of the previous target dataset ±n*σ (n is an integer equal to or greater than 1, for example, n=3) may be set as the threshold for each data validation metric. The above values ​​and parameters are merely examples, and any parameters may be derived from the data set processing history 5. Furthermore, default values ​​may be preset for each of the parameters of the above multiple quality analysis metrics and data verification metrics.

[0075] In step S83, the metrics construction unit 13 reflects the parameters derived in step S82 in the corresponding quality analysis metrics and data verification metrics. In step S84, the data quality analysis unit 14 applies the quality analysis metrics in which the derived parameters are reflected in step S83 to the data set. Also, the data quality analysis unit 14 applies the data verification metrics in which the derived parameters are reflected in step S83 to the data set. As a result, in the data quality check process and data verification process, metrics that dynamically reflect parameters derived from the data set processing history 5 are applied.

[0076] <Detailed error handling procedure> FIG. 9 is a flowchart showing an example of a detailed processing procedure of the error processing executed by the data observation device 1 according to this embodiment. In step S91, the metrics construction unit 13 of the data observation device 1 acquires the dataset processing history 5 stored in the storage device. The dataset processing history 5 includes the history of datasets previously extracted from the database of the data source 2, and the results of data quality checks and data verifications of datasets collected by the data collection unit 16. In step S92, the metrics construction unit 13 derives each parameter to be set for the data quality analysis metrics and data verification metrics from the processing history 5 of the dataset, and sets each derived parameter for the corresponding data quality analysis metrics or data verification metrics.

[0077] In step S93, the data quality analysis unit 14 applies the data quality analysis metrics whose parameters were set in step S92 to the data set, and analyzes the data quality of the data set from multiple aspects. At the same time, in step S93, the data validation unit 15 applies the data validation metrics parameterized in step S92 to the data set to validate the data entries of the data set. The processing up to this point corresponds to the processing in steps S81 to S84 in FIG.

[0078] In step S93, if the dataset is within the valid range set for the data quality analysis metrics and data verification metrics (step S93: N), error processing is skipped and the dataset is stored in the integrated database 4 at a later stage. On the other hand, if the data set is outside the valid range set for either the data quality analysis metrics or the data verification metrics in S93 (step S93: Y), it is determined that there is an error in the data set, and the process proceeds to step S94.

[0079] In step S94, the orchestrator 18 of the data observation device 1 determines whether or not the integrated database construction workflow needs to be completely stopped due to an error that has occurred in the dataset. If the integrated database construction workflow needs to be stopped (step S94: Y), the process proceeds to step S95, where the orchestrator 18 stops the workflow, and then to step S98. On the other hand, if the integrated database construction workflow does not need to be stopped (step S94: N), the process proceeds to step S96.

[0080] In step S94, the orchestrator 18 may decide whether to stop the workflow based on whether the component in which the error occurred is a data quality check or a data validation. For example, if an error occurs in a data quality check, the data quality of the entire dataset may be reduced, and the impact on data users is expected to be significant, so the orchestrator 18 may stop the workflow. On the other hand, if an error occurs in data validation, there is no problem with the data quality of data items other than the data item where the error occurred, and the impact on data users is expected to be relatively small, so the orchestrator may continue extracting, converting, and loading the dataset into the integrated database without stopping the workflow. In this way, the cases and scope for stopping the workflow are localized, thereby improving the availability of the workflow and the integrated database.

[0081] In step S96, orchestrator 18 determines whether or not to re-execute the integrated database construction workflow. If it is necessary to re-execute the integrated database construction workflow (step S96: Y), the process proceeds to step S97, where orchestrator 18 re-executes the workflow after a preset period of time has elapsed, and then terminates the process. On the other hand, if it is not necessary to re-execute the integrated database construction workflow (step S96: N), the process proceeds to step S98. In step S96, if the error is caused by data source 2, orchestrator 18 may re-execute the workflow to again obtain customer data from data source 2 and extract the dataset. On the other hand, if the error is not caused by data source 2, orchestrator 18 may proceed to step S98 without re-executing the workflow.

[0082] In step S98, the orchestrator 18 executes error processing according to the type and degree of the error, and then ends the processing. Specifically, the orchestrator 18 notifies the supervisor of the integrated database construction workflow of the event of the error occurrence and details of the error. Additionally, if the error is caused by the data source 2, the orchestrator 18 notifies the administrator of the data source 2 of the error. The orchestrator 18 then accepts retransmission of the data with the error corrected from the data source 2 that has been notified of the error, and extracts a data set from the retransmitted data.

[0083] FIG. 10 is a diagram illustrating an example of a description of data quality analysis metrics set in the data observation device 1 according to this embodiment. Referring to Figure 10, three types of data quality analysis metrics are defined. The metric name "size" indicates a data quality analysis metric for volume checks, the metric name "freshness" indicates a data quality analysis metric for freshness checks, and the metric name "histogram" indicates a data quality analysis metric for data distribution checks. The data quality analysis metric "histogram" for data distribution checks has two arguments set for it: "reg_gender_cd", which checks the data distribution by gender, and "reg_prefecture_cd", which checks the data distribution by place of residence (prefecture).

[0084] FIG. 11 is a diagram illustrating an example of a description of data verification metrics set in the data observation device 1 according to this embodiment. 11, data verification metrics 111 to 118 are defined. Each of these metrics 111 to 118 has the name of a data item to be checked written as an argument. The metric name "completeness" checks how much data has been entered. For example, if 90% of all customer data has been entered, the data validation result may be determined to be normal. The metric name "min" checks the minimum value of the data. The metric name "max" checks the maximum value of the data. The metric name "mean" checks the average value of the data. The metric name "stddev" checks the standard deviation of the data. For these metrics, thresholds that define the valid range and error range may be set dynamically and automatically by referring to the past processing history of the dataset stored in the dataset processing history 5. The metric name "value_set" checks that the data is a specific value. The metric name "unique" checks that all data values ​​are different across all customers. The metric name "not_null" checks that the data value is not blank.

[0085] As described above, according to this embodiment, the data observation device of this embodiment extracts datasets from multiple customer databases belonging to multiple platforms, applies first metrics to the extracted datasets to analyze the quality of each dataset in the dataset, and applies second metrics to the extracted datasets to verify the data values ​​for each item in the dataset. The data observer also dynamically constructs at least a portion of the first metric and the second metric to be applied to the metadata based on a processing history of the dataset.

[0086] This makes it possible to observe the accuracy and behavior of data when aggregating data from multiple data sources to build and update a database, and also improves the accuracy and availability of the data.

[0087] <Hardware configuration of data observation device> FIG. 12 is a diagram showing a non-limiting example of the hardware configuration of the data observation device 1 according to this embodiment. The data observation device 1 according to this embodiment can be implemented on a single or multiple computers, mobile devices, or any other processing platform. 12, an example in which the data observation device 1 is implemented in a single computer is shown, but the data observation device 1 according to this embodiment may be implemented in a computer system including multiple computers. The multiple computers may be connected to each other via a wired or wireless network so as to be able to communicate with each other.

[0088] 12, the data observation device 1 may include a CPU 121, a ROM 122, a RAM 123, an HDD 124, an input unit 125, a display unit 126, a communication I / F 127, and a system bus 128. The data observation device 1 may also include an external memory. The CPU (Central Processing Unit) 121 controls the overall operation of the data observation device 1, and controls each component (122 to 127) via a system bus 128, which is a data transmission path. Note that instead of or in addition to the CPU 121, the data observation device 1 may be equipped with a GPU (Graphics Processing Unit).

[0089] The ROM (Read Only Memory) 122 is a nonvolatile memory that stores control programs and the like necessary for the CPU 121 to execute processing. Note that the programs may be stored in a nonvolatile memory such as an HDD (Hard Disk Drive) 124 or an SSD (Solid State Drive), or in an external memory such as a removable storage medium (not shown). RAM (Random Access Memory) 123 is a volatile memory and functions as a main memory, work area, etc. of CPU 121. That is, when executing a process, CPU 121 loads necessary programs, etc. from ROM 122 into RAM 123 and executes the programs, etc. to realize various functional operations.

[0090] The HDD 124 stores, for example, various data and various information required when the CPU 121 performs processing using a program. The HDD 124 also stores, for example, various data and various information obtained when the CPU 121 performs processing using a program. The input unit 125 is configured with a keyboard and a pointing device such as a mouse. The display unit 126 is configured with a monitor such as a liquid crystal display (LCD), etc. The display unit 126 may provide a GUI (Graphical User Interface), which is a user interface for inputting various parameters used in data observation processing, communication parameters used in communication with other devices, etc., into the code documentation device 1.

[0091] The communication I / F 127 is an interface that controls communication between the data observation device 1 and an external device. The communication I / F 127 provides an interface with a network and communicates with external devices via the network. Various data, parameters, and the like are transmitted and received between the external devices and the communication I / F 127. In this embodiment, the communication I / F 127 may communicate via a wired LAN (Local Area Network) or a dedicated line conforming to a communication standard such as Ethernet (registered trademark). However, the network available in this embodiment is not limited to this and may be configured as a wireless network. This wireless network includes wireless PANs (Personal Area Networks) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). It also includes wireless LANs (Local Area Networks) such as Wi-Fi (Wireless Fidelity) (registered trademark) and wireless MANs (Metropolitan Area Networks) such as WiMAX (registered trademark). It also includes wireless WANs (Wide Area Networks) such as LTE / 3G, 4G, and 5G. Note that the network may be any network that connects devices to each other and enables communication, and the communication standard, scale, and configuration are not limited to those described above.

[0092] At least some of the functions of each element of the data observation device 1 shown in Fig. 1 can be realized by the CPU 121 executing a program. However, at least some of the functions of each element of the data observation device 1 shown in Fig. 1 may be operated as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU 121.

[0093] Although specific embodiments have been described above, these embodiments are merely examples and are not intended to limit the scope of the present invention. The devices and methods described herein may be embodied in forms other than those described above. Furthermore, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the present invention. Such omissions, substitutions, and modifications are included within the scope of the claims and their equivalents, and belong to the technical scope of the present invention.

[0094] (Embodiments of the present disclosure) The present disclosure includes the following embodiments. [1] An information processing device comprising: a dataset extraction unit that extracts datasets from multiple databases that belong to multiple platforms; a quality analysis unit that analyzes the quality of each dataset of the dataset using first metrics for the dataset; a data verification unit that verifies data values ​​for each item of the dataset using second metrics for the dataset; and a metrics construction unit that dynamically constructs at least a portion of the first metrics and the second metrics to be used for the dataset based on the processing history of the dataset.

[0095] [2] The information processing device further includes an orchestrator that controls the quality analysis unit and the data verification unit to execute the quality analysis unit and the data verification unit in parallel by pipeline processing. The information processing device according to [1].

[0096] [3] The information processing device described in [2], characterized in that the quality analysis unit applies multiple mutually different first metrics to the dataset, and the orchestrator controls the quality analysis unit to apply the multiple first metrics to the dataset in parallel using pipeline processing.

[0097] [4] The information processing device described in [2] or [3], characterized in that the data verification unit applies multiple mutually different second metrics to the dataset, and the orchestrator controls the data verification unit to apply the multiple second metrics to the dataset in parallel using pipeline processing.

[0098] [5] The information processing device described in any one of [2] to [4], characterized in that the orchestrator causes the dataset extraction unit to stop extracting the dataset if an error occurs in the quality analysis unit, and causes the dataset extraction unit to continue extracting the dataset if an error occurs in the data verification unit.

[0099] [6] The information processing device described in any one of [1] to [5], characterized in that the metrics construction unit dynamically sets thresholds for at least a portion of the first metrics and the second metrics based on a standard deviation derived from the processing history of the dataset over a predetermined period.

[0100] [7] The information processing device described in any one of [1] to [6], characterized in that the metrics construction unit dynamically constructs a plurality of the first metrics so as to observe time-series changes in the dataset on a dataset-by-dataset basis.

[0101] [8] The information processing device described in [7], characterized in that the metrics construction unit dynamically constructs the plurality of first metrics so as to observe changes in at least two or more time series of freshness, volume, distribution, schema, and data series on a dataset-by-dataset basis for the dataset.

[0102] [9] The information processing device according to any one of [1] to [8], further comprising a user interface that allows a user to query the processing results of the quality analysis unit and the data verification unit and to set thresholds for the first metrics and the second metrics.

[0103]

[10] An information processing method executed by an information processing device, comprising the steps of: extracting datasets from a plurality of databases belonging to a plurality of platforms; analyzing the quality of each dataset of the datasets using first metrics for the datasets; verifying data values ​​for each item of the datasets using second metrics for the datasets; and dynamically constructing at least a portion of the first metrics and the second metrics to be used for the datasets based on the processing history of the datasets.

[0104]

[11] An information processing program for causing a computer to execute information processing, the program causing the computer to execute processes including: a metadata extraction process for extracting datasets from multiple databases belonging to multiple platforms; a quality analysis process for applying first metrics to the datasets to analyze the quality of each dataset in the datasets; a data verification process for applying second metrics to the datasets to verify data values ​​for each item in the datasets; and a metrics construction process for dynamically constructing at least a portion of the first metrics and the second metrics to be used for the datasets based on the processing history of the datasets. [Explanation of symbols]

[0105] 1...data observation device, 2...data source, 3...client device, 4...integrated database, 5...dataset processing history, 11...data acquisition unit, 12...dataset extraction unit, 13...metrics construction unit, 14...data quality analysis unit, 15...data validation unit, 16...data collection unit, 17...database output unit, 18...orchestrator, 121...CPU, 122...ROM, 123...RAM, 124...HDD, 125...input unit, 126...display unit, 127...communication I / F, 128...system bus, 201...data source connector, 202...dataset extractor, 203...metrics configurator, 204...data quality checker, 205...data validator, 206...data collector, 207...data observation client, 208...data storage unit, 209...user interface

Claims

1. a dataset extraction unit that extracts datasets from a plurality of databases that belong to a plurality of platforms; a quality analysis unit that analyzes the quality of each of the datasets using first metrics for the datasets; a data verification unit that verifies a data value for each item of the data set using a second metric for the data set; a metrics construction unit that dynamically constructs at least a portion of the first metrics and the second metrics to be used for the dataset based on a processing history of the dataset; An information processing device comprising:

2. The information processing device further comprises: an orchestrator that controls the quality analysis unit and the data verification unit to execute them in parallel by pipeline processing; 2. The information processing apparatus according to claim 1, wherein:

3. the quality analysis unit applies a plurality of mutually different first metrics to the data set; The orchestrator controls the quality analysis unit to apply the plurality of first metrics to the dataset in parallel by pipeline processing.

3. The information processing apparatus according to claim 2, wherein:

4. the data verification unit applies a plurality of mutually different second metrics to the data set; The orchestrator controls the data verification unit to apply the second metrics to the data set in parallel by pipeline processing.

3. The information processing apparatus according to claim 2, wherein:

5. The orchestrator causes the dataset extraction unit to stop extracting the dataset when an error occurs in the quality analysis unit, and causes the dataset extraction unit to continue extracting the dataset when an error occurs in the data verification unit.

3. The information processing apparatus according to claim 2, wherein:

6. the metrics construction unit dynamically sets thresholds for at least a portion of the first metrics and the second metrics based on a standard deviation derived from a processing history of the dataset for a predetermined period; 2. The information processing apparatus according to claim 1, wherein:

7. the metrics construction unit dynamically constructs the plurality of first metrics so as to observe time-series changes in the data set on a data set basis; 2. The information processing apparatus according to claim 1, wherein:

8. the metrics construction unit dynamically constructs the plurality of first metrics so as to observe changes in at least two or more time series of freshness, volume, distribution, schema, and data series of the dataset on a dataset-by-dataset basis; 8. The information processing apparatus according to claim 7,

9. the information processing device further includes a user interface that allows a user to inquire about processing results of the quality analysis unit and the data verification unit and to set thresholds for the first metrics and the second metrics.

2. The information processing apparatus according to claim 1, wherein:

10. An information processing method executed by an information processing device, Extracting datasets from a plurality of databases belonging to a plurality of platforms; analyzing a quality of each of the datasets using a first metric for the datasets; validating data values ​​for each item of the data set using a second metric for the data set; dynamically constructing at least a portion of the first and second metrics to be used for the dataset based on a processing history of the dataset; An information processing method comprising:

11. An information processing program for causing a computer to execute information processing, the program including: a metadata extraction process for extracting datasets from multiple databases belonging to multiple platforms; a quality analysis process for analyzing the quality of each of the datasets using a first metric for the datasets; a data validation process for validating a data value for each item of the dataset using a second metric for the dataset; a metrics construction process that dynamically constructs at least a portion of the first metrics and the second metrics to be used for the dataset based on a processing history of the dataset; 2. An information processing program for executing a process including:

Images