Information processing device, information processing method, and program

Abstract

The present disclosure relates to an information processing device, an information processing method, and a program which make it possible to improve user satisfaction with respect to prediction analysis. The information processing device is provided with a leak detecting unit which, on the basis of the property of a data set used for learning a prediction model for use in prediction analysis, detects the probability of a leak of the data set. The present technology is applicable, for example, to a machine learning model or a system that generates and publish an API for using a machine learning model.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that are adapted to improve user satisfaction with predictive analysis. [Background technology]

[0002] In recent years, various data has been accumulated in business, and the importance of utilizing accumulated data in business has long been recognized. As a method for utilizing data in business, for example, a method using predictive analysis technology that predicts future results from past data using machine learning is known (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-16321 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in predictive analysis technology, even if the prediction accuracy of the prediction model used in the predictive analysis is high in advance evaluation, the expected prediction accuracy may not be obtained in actual operation, resulting in a decrease in user satisfaction. One of the main causes of this is thought to be leakage.

[0005] A leak means using data that cannot be used at the time of prediction for training a prediction model, or the data itself. For example, when predicting the number of sales of bento lunches, it is expected that the prediction accuracy of the prediction model will improve if data on "whether the bento lunches were sold out on that day" is used for training. However, the purpose of predicting the number of sales is to predict the number of bento lunches sold out several days in advance and determine the number of lunches to be manufactured, and data on whether the lunches were sold out or not, which is not known until the day of the sales, cannot actually be used. Therefore, if data on "whether the bento lunches were sold out on that day" is used for training, as described above, there is a risk that the prediction accuracy of the prediction model will diverge between the time of pre-evaluation and the time of actual operation, and user satisfaction will decrease.

[0006] In contrast, Patent Document 1 does not particularly consider leaks.

[0007] The present disclosure has been made in consideration of such circumstances, and aims to improve user satisfaction with predictive analysis. [Means for solving the problem]

[0008] The present disclosure includes a leak detection unit that detects the possibility of a leak in a dataset used to train a predictive model used in predictive analysis based on the characteristics of the dataset.

[0009] According to the present disclosure, a possible leak in a dataset is detected based on the properties of the dataset used to train a predictive model used in predictive analysis. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating the business implementation of predictive analytics. [Diagram 2] FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram illustrating an analysis process according to an embodiment of the present disclosure. [Figure 4]FIG. 1 is a diagram illustrating an example of a past case according to an embodiment of the present disclosure. [Diagram 5] FIG. 11 is a diagram illustrating an example of user data according to an embodiment of the present disclosure. [Figure 6] FIG. 13 is a diagram showing an example of an image presented to a user. [Figure 7] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a diagram illustrating an example of a past case storage unit according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a diagram illustrating a configuration example of a time series processing unit according to an embodiment of the present disclosure. [Figure 10] FIG. 4 is a diagram illustrating an example of the configuration of a leak detection and correction unit according to an embodiment of the present disclosure. [Figure 11] 1 is a flowchart illustrating a procedure of information processing according to an embodiment of the present disclosure. [Figure 12] 1 is a flowchart illustrating a procedure of time series processing according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating an example of a construction data set according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram showing an example in which the construction data set of FIG. 13 is separated for each series. [Figure 15] FIG. 14 is a diagram showing an example of a graph of a time series to be predicted for the construction data set of FIG. 13. [Figure 16] FIG. 13 is a diagram showing an example of an image presented to a user. [Figure 17] 5 is a flowchart illustrating a procedure of a leak detection process according to an embodiment of the present disclosure. [Figure 18] 5 is a flowchart illustrating a procedure of a leak detection process according to an embodiment of the present disclosure. [Figure 19] FIG. 13 is a diagram for explaining a method for verifying a prediction model. [Figure 20] FIG. 13 is a diagram showing an example of an image presented to a user. [Figure 21] FIG. 2 is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device or a terminal device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, each embodiment of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0012] The present disclosure will be described in the following order. 1. Embodiment 1-1.Background 1-2. Configuration example of information processing system according to embodiment 1-3. Overview of information processing according to the embodiment 1-4. Configuration example of information processing device according to embodiment 1-5. Configuration example of time series processing unit according to embodiment 1-6. Configuration example of leak detection correction unit according to embodiment 1-7. Information Processing Procedure According to the Embodiment 2. Variations 3. Hardware Configuration

[0013] <1. Embodiment> [1-1. Background] First, before describing the embodiments of the present disclosure in detail, a workflow for utilizing predictive analysis in business will be described as a background of the embodiments of the present disclosure.

[0014] When utilizing predictive analysis for business purposes, the user decides what type of predictive analysis to perform based on accumulated data. The user then evaluates the business effects obtained by introducing the predictive analysis by conducting a proof-of-concept experiment of the decided predictive analysis. In this way, by conducting a proof-of-concept experiment and evaluating the business effects obtained by the predictive analysis, the user can introduce highly effective predictive analysis to the business, and can make better use of predictive analysis in business.

[0015] An example of a workflow for actually utilizing predictive analysis in business is shown in Figure 1. Figure 1 is a diagram for explaining the business introduction of predictive analysis.

[0016] In step S1, the user sets a problem definition: which data from the accumulated data to use and what to predict. For example, problem definitions include "predicting whether a loan will default using data such as a customer's annual income and total assets" or "predicting future sales using data such as past sales and customer age groups." In this way, appropriate problem definitions differ depending on the business field and user. For this reason, the user sets the problem based on, for example, their own knowledge and experience.

[0017] In step S2, the user constructs a dataset from the accumulated data according to the problem setting. The user constructs the dataset, for example, by extracting data to be used for predictive analysis from the accumulated data, and interpreting and structuring the data in accordance with the predictive analysis. The construction of the dataset may also require the user's knowledge and experience, for example.

[0018] In step S3, the user generates a prediction model based on the problem definition and the data set. The prediction model is generated using general machine learning. In this case, the user can generate the prediction model using, for example, an existing information processing device.

[0019] In step S4, the user evaluates the accuracy of the generated prediction model. The accuracy of the prediction model is evaluated using a general evaluation index such as the area under the curve (AUC) or accuracy. In this case, the user can evaluate the accuracy of the prediction model using, for example, an existing information processing device.

[0020] In step S5, the user performs a demonstration experiment using the generated predictive model. For example, the user collects data within a limited range, such as a period or region, and performs predictive analysis of the data using the generated predictive model. The user introduces predictive analysis into the business on a trial basis, for example by changing product procurement or sales destinations depending on the analysis results.

[0021] In step S6, the user measures the effectiveness of the demonstration experiment. The user measures the effectiveness by comparing data before and after the experiment, for example, by comparing sales when predictive analysis is trial-introduced with sales before the introduction. After that, the user introduces predictive analysis into the actual business depending on the results of the demonstration experiment and the measured effectiveness.

[0022] As mentioned above, when introducing predictive analysis into an actual business, users need knowledge and experience in problem definition and data set construction, which can become a bottleneck in the implementation. In addition, since proof-of-concept experiments are costly, it is difficult to move on to proof-of-concept experiments unless there is certainty that the introduction of predictive analysis into the business will produce some degree of effect. As such, the transition to proof-of-concept experiments tends to be a high hurdle.

[0023] The present disclosure focuses on this point and is configured so that an information processing device performs predictive analysis including extracting problem settings and constructing a data set.

[0024] [1-2. Configuration example of information processing system according to embodiment] 2 is a diagram showing a configuration example of an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 includes a terminal device 11 and an information processing device 12. The terminal device 11 and the information processing device 12 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via a wired or wireless connection. Note that the information processing system 1 may include a plurality of terminal devices 11 and a plurality of information processing devices 12.

[0025] The terminal device 11 is an information processing device used by a user. The terminal device 11 is used to provide services related to predictive analysis. The terminal device 11 may be any device as long as it can realize the processing in the embodiment. For example, the terminal device 11 may be any device as long as it provides services related to predictive analysis to a user and has a display that displays information. For example, the terminal device 11 may be a device such as a notebook PC (Personal Computer), a desktop PC, a tablet terminal, a smartphone, a mobile phone, or a PDA (Personal Digital Assistant).

[0026] The information processing device 12 is used to provide a service related to predictive analysis to a user. The information processing device 12 is an information processing device that controls information related to problem setting and the results of predictive analysis evaluation based on user data so as to be displayed to a user. The information processing device 12 generates an image showing information related to problem setting and the results of predictive analysis evaluation, and provides the image to the terminal device 11.

[0027] Furthermore, the information processing device 12 controls the display of the terminal device 11. That is, the information processing device 12 is also a server device that provides information to be displayed on the terminal device 11. For example, the information processing device 12 controls the display of the terminal device 11 by transmitting an image including control information to the terminal device 11. Here, the control information is described, for example, in a script language such as JavaScript (registered trademark) or CSS.

[0028] The information processing device 12 may provide an application for displaying images and the like to be provided to the terminal device 11. The application itself provided from the information processing device 12 to the terminal device 11 may be regarded as control information.

[0029] [1-3. Overview of information processing according to the embodiment] An overview of the analysis process performed by the information processing device 12 will be described below with reference to FIGS.

[0030] Fig. 3 is a diagram illustrating an analysis process according to an embodiment of the present disclosure. Fig. 4 is a diagram illustrating an example of a past case according to an embodiment of the present disclosure. Fig. 5 is a diagram illustrating an example of user data according to an embodiment of the present disclosure.

[0031] In the example of FIG. 3, a case where predictive analysis is performed using user data with reference to past cases will be described. Here, the user data is, for example, data collected by a user. The user data includes various data such as customer information and product information. The user uses the user data to perform predictive analysis, for example, of next month's sales.

[0032] Generally, when performing predictive analysis using user data, the user himself / herself needs to set the problem of "what to predict using which data." Setting the problem for predictive analysis may require the user's knowledge and experience, which may be a burden for the user. Therefore, the analysis process according to the embodiment aims to reduce the burden on the user by automatically setting the problem for predictive analysis by referring to past cases.

[0033] In step S11, the information processing device 12 acquires past cases. Here, the past cases include problem settings of predictive analysis performed in the past. Specifically, the past cases include a prediction target (hereinafter also referred to as a past target) indicating what was predicted in the past, and an analysis data set (hereinafter also referred to as a past data set) used in predictive analysis of the past target indicating which data was used for the past prediction.

[0034] Here, an example of a past case will be described with reference to FIG. 4. As shown in FIG. 4, the past case includes, for example, past data set 31. Past data set 31 includes, for example, "customer ID," "loan amount," "loan type," "years of service," and "bad debt." Also, in FIG. 4, "bad debt" is indicated by a diagonal line as a past object. In this way, the past case includes past data set 31 and a past object (here, "bad debt").

[0035] Returning to FIG. 3, in step S12, the information processing device 12 acquires user data. An example of the user data will now be described with reference to FIG. 5. The user data is data generated and collected by a user, and is data used for generating a model for predictive analysis, etc. The user data 41 shown in FIG. 5 includes, for example, "customer ID," "loan amount," "loan type," "years of service," "annual income," "total account balance," and "bad debts."

[0036] Returning to FIG. 3, in step S13, the information processing device 12 extracts a prediction target based on the acquired past cases and user data 41. The information processing device 12 selects a past target related to the user from past cases, for example. The information processing device 12 selects the past target by utilizing a recommendation system using information about the user, such as the department to which the user belongs or predictive analysis performed by the user in the past. Here, it is assumed that the information processing device 12 selects "bad debt" in the past data set 31 shown in FIG. 4 from past cases as the past target.

[0037] The information processing device 12 extracts the same item as the selected past target from the user data 41 as a prediction target (hereinafter also referred to as an extracted target) for which a current predictive analysis will be performed. In the example of FIG. 4, the past target selected by the information processing device 12 is "bad debt". Therefore, the information processing device 12 extracts "bad debt" as a prediction target from the user data 41 shown in FIG. 5. In FIG. 5, the extracted target "bad debt" is indicated by diagonal lines. Details of the method for extracting the extracted target will be described later with reference to FIG. 7.

[0038] Returning to Fig. 3, in step S14, the information processing device 12 constructs a data set (hereinafter also referred to as a constructed data set) to be used for predictive analysis of the extraction target based on the user data 41. The information processing device 12 extracts, for example, items related to the extraction target as the constructed data set. For example, the information processing device 12 extracts "customer ID", "loan amount", "loan type", "years of service", and "bad debt" from the user data 41 shown in Fig. 5 to generate the constructed data set.

[0039] In this embodiment, the information processing device 12 constructs a data set including a part of the user data 41 shown in Fig. 5, but is not limited to this. A data set including all of the user data 41 may be constructed. Details of a method for constructing a data set will be described later with reference to Fig. 7.

[0040] 3, in step S15, the information processing device 12 learns a prediction model based on the extraction target and the construction data set. The information processing device 12 converts data of the construction data set excluding the prediction target (extraction target) into a feature vector. The information processing device 12 generates a prediction model by solving a classification or regression problem by machine learning based on the feature vector and the extraction target.

[0041] In step S16, the information processing device 12 evaluates the accuracy of the predictive analysis by evaluating the generated predictive model. The information processing device 12 evaluates the predictive model using the predictive model and the construction dataset. The evaluation index is selected according to the analysis method, such as AUC or Accuracy in the case of classification analysis, or MAE (Mean Absolute Error) in the case of regression analysis.

[0042] In step S17, the information processing device 12 presents extraction information including the extraction target and the evaluation result to the user. An example of the presentation of the extraction information to the user will now be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of an image presented to the user on the terminal device 11 under the control of the information processing device 12.

[0043] As shown in Fig. 6, the information processing device 12 presents a combination of question settings and evaluation results to the user. Fig. 6 shows an extraction result when the information processing device 12 extracts a plurality of question settings. In this case, the information processing device 12 displays a list of combinations of question settings and evaluation results, as in image IM1.

[0044] This allows the user to decide whether or not to perform predictive analysis with the problem setting presented by the information processing device 12, for example, by referring to the evaluation result.

[0045] The contents presented to the user by the information processing device 12 are not limited to the problem setting and the evaluation result. The information processing device 12 may present at least one of the construction data set, the extraction target, and the evaluation result to the user. Alternatively, the information processing device 12 may present reference information for the user when selecting the problem setting, such as the effect obtained when performing the predictive analysis.

[0046] As described above, by the information processing device 12 extracting problem settings, the user does not need to set problems and can more easily perform predictive analysis. Also, by the information processing device 12 evaluating the accuracy of predictive analysis, the user can select predictive analysis to be performed based on the accuracy evaluation, and can more easily perform highly accurate predictive analysis.

[0047] [1-4. Configuration example of information processing device according to embodiment] Next, a configuration example of the information processing device 12 in FIG. 2 will be described with reference to FIG.

[0048] The information processing device 12 includes a communication unit 101, a storage unit 102, and a control unit 103. The information processing device 12 may include an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 12, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0049] (Communications Department) The communication unit 101 is realized by, for example, a network interface card (NIC) etc. The communication unit 101 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the terminal device 11 and other information processing devices such as an external server.

[0050] (Storage part) The storage unit 102 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in Fig. 7, the storage unit 102 includes a past case storage unit 121, a user data storage unit 122, and a user profile storage unit 123. Although not shown in the figure, the storage unit 102 may store various information such as an image that is the basis of an image to be provided to the terminal device 11.

[0051] (Past case memory) The past case storage unit 121 stores past cases. The past cases include information on predictive analysis performed in the past. The past case storage unit 121 stores, for example, cases in which predictive analysis was introduced into a business in the past. Note that the past cases may not be held by the information processing device 12, but may be appropriately acquired from an external server or the like.

[0052] 8 is a diagram showing an example of the past case storage unit 121. The past case storage unit 121 stores, for each case, information on, for example, "problem definition," "data set," "collection cost," "prediction model," "model evaluation result," "demonstration experiment," "business effect," etc. The past case storage unit 121 stores a plurality of past cases, such as past cases A, B, ...

[0053] "Problem setting" is information that indicates what data was used and what was predicted in predictive analysis. "Problem setting" includes, for example, multiple "used items" (explanatory variables) that is, "what data was used" and one "prediction target" (objective variable) that is, "what was predicted." For example, in the example shown in Figure 4 above, the items shown with diagonal lines are the prediction target, and the remaining items are used items.

[0054] The "dataset" is a past data set used to learn a prediction model. For example, the "dataset" is a data set that includes "input data" and "correct answer data." For example, the past data set 31 shown in FIG. 4 corresponds to such a "dataset."

[0055] "Collection cost" is the cost incurred to collect the data used in the predictive analysis. "Collection cost" includes, for example, the period and cost required to collect data for each item.

[0056] A "predictive model" is a past predictive model (hereinafter also referred to as a past model) that is generated using a stored "problem setting" and "dataset." A "predictive model" is a model generated by solving a classification or regression problem, for example, by machine learning.

[0057] The "model evaluation result" is the result of the accuracy evaluation of the stored "prediction model." The "model evaluation result" includes the evaluation results based on evaluation indices such as AUC and Accuracy.

[0058] "Demonstration experiments" refers to information about the content and results of demonstration experiments conducted to introduce predictive analysis into a business. "Demonstration experiments" includes, for example, information about the period and scope of the experiment, the data used in the experiment, the effects obtained from the experiment, and the costs incurred in the experiment.

[0059] "Business benefits" refers to information about business benefits obtained after introducing predictive analysis into a business. "Business benefits" includes, for example, information about profit amounts, such as increased sales, and cost reduction amounts, such as reduced labor costs.

[0060] 8, the past case storage unit 121 stores various information on cases where predictive analysis was introduced into a business in the past, for each of a number of past cases. Note that the above-mentioned past cases are just an example, and as long as the past case storage unit 121 stores the "problem setting" and the "data set", it may not need to store some information, such as the "collection cost", "model evaluation results", and "demonstration experiment", or it may store information other than the above-mentioned information.

[0061] (User data storage unit) Returning to Fig. 7, the user data storage unit 122 will be described. User data is various data created or collected by a user. The user data may have a wide variety of data formats, for example, as listed below.

[0062] Text - words, sentences, HTML (HyperText Markup Language), etc. Media - RGB images, depth images, vector images, video, audio, etc. Compound documents - office documents, PDFs, web pages, emails, etc. Sensor data - current location, acceleration, heart rate, etc. Application data - startup logs, information on files being processed, etc. Databases - Relational databases, key-value stores, etc.

[0063] The user data may not be stored in the information processing device 12, but may be appropriately acquired from the terminal device 11, an external server, etc. The user data may be raw data acquired directly from a camera, a sensor, etc., or may be processed data obtained by performing processing such as feature extraction on raw data. Alternatively, the user data may include metadata that is a recognition result obtained by performing a recognition process on the raw data or processed data.

[0064] (User profile storage section) Next, a description will be given of the user profile storage unit 123. The user profile storage unit 123 stores profile information related to a user. The profile information includes, for example, user information and user case information.

[0065] User information is information about a user, and includes, for example, a user ID, the name of the company to which the user belongs, department, industry, etc. User information may also include information related to the user's interests and concerns, such as website and database search history, website browsing history, and keywords contained in emails and office documents.

[0066] The user case information also includes information about past predictive analyses performed by the user. The user case information includes, for example, information about predictive analyses performed by the user in the past and information about past cases in which the user was involved. Note that such predictive analyses may be performed by the user himself or by a department or company to which the user belongs.

[0067] (Control unit) The control unit 103 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), etc., executing a program (for example, a program according to the present disclosure) stored in the information processing device 12 using a RAM or the like as a working area. The control unit 103 is a controller, and is realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0068] The control unit 103 includes an acquisition unit 141, an information processing unit 142, and a display control unit 143. The information processing unit 142 includes a time prediction unit 151, an interpretation unit 152, an extraction unit 153, a time series processing unit 154, a leak detection and correction unit 155, a learning unit 156, an evaluation unit 157, a prediction unit 158, a collection determination unit 159, and a contribution degree calculation unit 160.

[0069] The internal structure of the control unit 103 is not limited to the configuration shown in Fig. 7, and may be other configurations as long as they perform information processing described later. Also, the connection relationship between the processing units included in the control unit 103 is not limited to the connection relationship shown in Fig. 7, and may be other connection relationships.

[0070] (Acquisition Department) The acquisition unit 141 acquires various information from the storage unit 102. For example, the acquisition unit 141 acquires a plurality of past cases from the past case storage unit 121. For example, the acquisition unit 141 acquires user data from the user data storage unit 122. For example, the acquisition unit 141 acquires profile information from the user profile storage unit 123. Note that the acquisition unit 141 may acquire various information from an external server, the terminal device 11, etc., instead of the past case storage unit 121, the user data storage unit 122, and the user profile storage unit 123.

[0071] (Time prediction section) The time prediction unit 151 predicts the time required for the analysis process performed by the control unit 103 from when the acquisition unit 141 starts acquiring data until the processing results such as problem setting extraction are presented to the user.

[0072] For example, the time prediction unit 151 performs analysis processing such as extraction of question settings, learning, and evaluation, using user data (hereinafter also referred to as partial data) acquired by the acquisition unit 141 in a predetermined time (e.g., 1 second). The analysis processing is processing performed by each part of the control unit 103 from when the acquisition unit 141 starts acquiring data to when the processing result is presented to the user, and will be described in detail later.

[0073] Then, the time prediction unit 151 measures the processing time of the analysis process performed using the partial data. The time prediction unit 151 predicts the time required for the analysis process (predicted processing time) based on the measured processing time. In detail, the time prediction unit 151 calculates the predicted processing time as predicted processing time=(measured processing time)*(size of user data / size of partial data).

[0074] Depending on the type and size of the user data, the analysis process may take several hours or even several days. For this reason, there is a demand from users to know the time required for the analysis process. Therefore, the time prediction unit 151 calculates the predicted processing time using a portion of the data. This makes it possible to present an estimate of the time required for the analysis process to the user. At this time, the size of the data used to calculate the predicted processing time can be limited to a size that can be acquired in, for example, one second, thereby making it possible to reduce the time required for calculating the predicted processing time.

[0075] Furthermore, the time prediction unit 151 does not simply calculate the predicted processing time from the size of the user data, but actually executes an analysis process using a portion of the data to calculate the predicted processing time. The size of the user data can be easily acquired, but the time required for predictive analysis depends not only on the size of the user data but also on the properties of the data. Therefore, the time prediction unit 151 actually executes the process to calculate the predicted processing time, thereby improving the prediction accuracy of the predicted processing time.

[0076] In this embodiment, the time prediction unit 151 calculates the predicted processing time using a portion of data acquired at a predetermined time, but the present invention is not limited to this. For example, the time prediction unit 151 may calculate the predicted processing time using a portion of data of a predetermined size (for example, 100 to 2000 rows).

[0077] Alternatively, the time prediction unit 151 may predict the predicted processing time using a trained processing time prediction model prepared in advance. In this case, the time prediction unit 151 extracts information such as the number of items (number of columns), the missing rate of each item, the data type of each item (character string / number / date, etc.), and the type of machine learning (binary classification / multiple-value classification / regression, etc.) from the part of data. The time prediction unit 151 predicts the predicted processing time using the extracted information and the trained processing time prediction model.

[0078] Furthermore, the time prediction unit 151 may update the predicted processing time at a predetermined timing, such as when a certain period of time has passed or when processing of each unit has ended. The time prediction unit 151 uses some of the data to execute processing that has not yet ended at the predetermined timing. The time prediction unit 151 updates the predicted processing time by recalculating the predicted processing time based on the time required for the executed processing.

[0079] The partial data used to update the estimated processing time may be the same as the partial data used to calculate the estimated processing time before the update, or may be user data newly acquired at the time of the current update. For example, when the interpretation unit 152 described later performs structuring processing on all user data, user data of a predetermined size may be acquired from all the user data that has been subjected to structuring processing, and used as the partial data.

[0080] (Interpretation Section) The interpretation unit 152 analyzes and structures the user data acquired by the acquisition unit 141 from the user data storage unit 122. First, the data analysis performed by the interpretation unit 152 will be described.

[0081] As described above, user data includes various data formats. The interpretation unit 152 analyzes the user data using a recognizer (not shown) for each type of data. The recognizer is assumed to be stored in the storage unit 102, for example.

[0082] Specifically, the interpretation unit 152 performs a recognition process on image data included in the user data, for example, using an image recognizer to detect faces, character strings, general objects, and the like from the image. For example, if the image data is an image of a receipt showing a product purchase, the interpretation unit 152 detects a user ID (terminal ID), a shooting location, a shooting time, and the like from the data added to the image. The interpretation unit 152 also detects character strings from the image and recognizes a phone number, a company name, a purchased product, a product price, a total amount, a payment method (cash / credit / electronic money / QR code (registered trademark) payment, and the like). The interpretation unit 152 adds the recognition result as metadata to the user data, which is raw data.

[0083] For data other than image data, the interpretation unit 152 recognizes the speaker using a voice recognizer for voice data included in the user data, and converts the spoken content into text. Alternatively, for acceleration data, the interpretation unit 152 recognizes the user's movement behavior (walking / bicycle / train, etc.) for each time. For text data, the interpretation unit 152 corrects spelling variations and adds similar expressions using a thesaurus. In this way, the interpretation unit 152 analyzes user data for each type of data and assigns metadata.

[0084] In the above example, the interpretation unit 152 recognizes one piece of data using one recognizer, but for example, the interpretation unit 152 may use multiple recognizers to recognize one piece of data. For example, when recognizing voice data, the interpretation unit 152 first converts the voice data into text data and translates the converted text data into multiple languages. Next, the interpretation unit 152 corrects notation variations in the translated text data and adds similar expressions. In this way, the interpretation unit 152 may recognize user data using multiple recognizers.

[0085] The above-described data recognition is merely an example, and the interpretation unit 152 may recognize the user data based on various known techniques.

[0086] Next, the interpretation unit 152 structures the user data based on the analysis result. The interpretation unit 152 structures the metadata attached to the user data using a template. The template is specialized for predictive analysis, and for example, a plurality of templates are stored in the storage unit 102 in advance.

[0087] When the interpretation unit 152 receives user data with metadata added thereto, it structurizes the data by fitting the data to the most suitable template.

[0088] For example, the concept of "user" has the concepts of "age" and "gender", and the concept of "product" has the concept of "price". The relationship between "user" and "product" is "purchase", and the concept of "purchase" has the concept of "purchase date and time". For example, by using a template of such a data structure, the interpretation unit 152 structures the metadata, which is unstructured data.

[0089] Furthermore, the interpretation unit 152 may add new metadata. The added metadata is used when extracting question settings. For example, the interpretation unit 152 may add a higher-level category such as "food expenses" or "miscellaneous living expenses" from the "product name" added to the receipt image as metadata.

[0090] The above-mentioned structuring is an example, and the interpretation unit 152 may structure the user data based on various known techniques. The above-mentioned templates and higher-level categories are an example, and the interpretation unit 152 may structure the user data using various templates, categories, and metadata specialized for predictive analysis. If the user data stored in the user data storage unit 122 has already been structured, the processing of the interpretation unit 152 may be omitted.

[0091] In this way, the interpretation unit 152 analyzes and structures the user data, thereby reducing the burden on the user.

[0092] (Extraction part) Next, the extraction unit 153 extracts a problem setting for predictive analysis based on the user data (hereinafter also referred to as structured data) structured by the interpretation unit 152 and the past cases acquired by the acquisition unit 141. The problem setting includes a plurality of "usage items" (explanatory variables) indicative of "what data items to use" and one "prediction target" (objective variable) indicative of "what to predict."

[0093] The extraction unit 153 extracts a "prediction target" from the structured data based on the past cases. For example, the extraction unit 153 extracts, from the structured data, an item (variable) that is the same as a past target included in the past cases as the "prediction target".

[0094] At this time, the extraction unit 153 extracts a "prediction target" that is considered to be related to the user or to be of high interest to the user, for example, based on profile information. For example, if the user is engaged in a business related to product sales, it is considered that the user is highly interested in "sales" prediction. Therefore, in this case, the extraction unit 153 extracts "sales" as the prediction target.

[0095] Specifically, the extraction unit 153 extracts candidates from past targets of past cases using a recommendation system based on, for example, profile information. The extraction unit 153 sets an item included in the user data from the extracted candidates as a "prediction target" for problem setting. The recommendation system includes, for example, ranking learning, content-based filtering, collaborative filtering, or a system that combines these.

[0096] The extraction unit 153 may extract multiple "prediction targets". For example, when multiple past targets are extracted in a ranking format, such as in ranking learning, the extraction unit 153 extracts a predetermined number of "prediction targets" from the top of the rankings. In this way, the extraction unit 153 extracts multiple "prediction targets", so that the extraction unit 153 can extract a wide range of "prediction targets" related to the user.

[0097] The extraction unit 153 extracts a plurality of "usage items" for each extracted "prediction target" (extraction target). The extraction unit 153 sets items (variables) related to the extraction target from the structured data as "usage items" (explanatory variables). The extraction unit 153 may set items that are even slightly related to the extraction target as "usage items". In this case, the information processing device 12 can improve the accuracy of learning in prediction model learning, which is processing after extraction. Alternatively, the extraction unit 153 may set a predetermined number of items in descending order of relatedness to the extraction target as "usage items". In this case, the information processing device 12 can reduce the processing load in prediction model learning.

[0098] The extraction unit 153 constructs a dataset based on the extracted "prediction target" and "usage items" (hereinafter also referred to as extracted items). The extraction unit 153 constructs a dataset by extracting data corresponding to the prediction target and the extracted items from the structured data. In this way, the extraction unit 153 constructs a dataset, so that the user does not need to construct a dataset according to the problem setting, and the burden on the user can be reduced.

[0099] As described above, the extraction unit 153 may extract, for example, a plurality of problem settings. In this case, the extraction unit 153 extracts a plurality of combinations of a "prediction target" and a plurality of "usage items" corresponding to the "prediction target". In addition, the extraction unit 153 constructs a dataset according to the extracted problem setting. Therefore, when the extraction unit 153 extracts a plurality of problem settings, it constructs a plurality of datasets corresponding to each problem setting. In this way, by the extraction unit 153 constructing a dataset, even if there are a plurality of problem settings, the user does not need to construct each corresponding dataset, and the burden on the user can be reduced.

[0100] (Time series processing unit) When the constructed data set constructed by the extraction unit 153 is a time-series data set (hereinafter referred to as a time-series data set), the time-series processing unit 154 resamples the date and time of the time-series data set. For example, the time-series processing unit 154 corrects the sampling interval (time interval) of the time-series data set, or interpolates missing values ​​of the prediction target.

[0101] Furthermore, when the constructed data set includes time-series data sets of a plurality of different series, the time-series processing unit 154 separates the time-series data sets for each series, and performs date and time resampling of the time-series data sets for each separated series.

[0102] Furthermore, when the construction data set is a time-series data set, the time-series processing unit 154 detects change points at which the trend of the value of the prediction target changes significantly, and divides the construction data set based on the change points.

[0103] A detailed configuration example of the time series processing unit 154 will be described later with reference to FIG.

[0104] (Leak detection and correction part) The leak detection and correction unit 155 detects the possibility of a leak in the constructed data set based on the properties of the constructed data set and past cases stored in the past case storage unit 121. Furthermore, the leak detection and correction unit 155 corrects the constructed data set, if necessary, based on the detected possibility of a leak.

[0105] A detailed configuration example of the leak detection and correction unit 155 will be described later with reference to FIG.

[0106] (Learning Department) The learning unit 156 learns a prediction model based on the problem setting extracted by the extraction unit 153 and the construction data set extracted by the extraction unit 153 or the construction data set corrected by the leak detection and correction unit 155. When the extraction unit 153 extracts a plurality of problem settings, the learning unit 156 learns a prediction model corresponding to each of the plurality of problem settings.

[0107] The learning unit 156 divides the construction data set into learning data and test data. The learning unit 156 converts the learning data into a feature vector. The learning unit 156 generates a prediction model by solving, for example, a classification or regression problem by machine learning based on the feature vector and the prediction target. Note that the above-mentioned machine learning is an example, and the learning unit 156 may learn the prediction model based on various known techniques.

[0108] Here, the learning unit 156 divides the construction data set, but this is just an example, and for example, the extraction unit 153 may construct each of the learning data set and the test data set.

[0109] (Evaluation Department) The evaluation unit 157 evaluates the prediction model generated by the learning unit 156. When the learning unit 156 generates a plurality of prediction models, the evaluation unit 157 evaluates each of the plurality of prediction models.

[0110] The evaluation unit 157 evaluates the prediction model using an evaluation index based on the prediction model and test data. The evaluation index is, for example, AUC for binary classification, Accuracy for multi-value classification, and MAE for regression. Note that the above-mentioned evaluation index is an example, and the evaluation unit 157 may evaluate the prediction model based on various known techniques. For example, the user may specify the evaluation index.

[0111] (Prediction section) The prediction unit 158 ​​predicts the business effect when the prediction model is introduced into a business. When the learning unit 156 generates multiple prediction models, the prediction unit 158 ​​predicts the business effect (hereinafter also referred to as the predicted effect) when each of the multiple prediction models is introduced into a business.

[0112] The prediction unit 158 ​​selects a past case having the same item as the extraction target extracted by the extraction unit 153 as the past target from the past case storage unit 121. The prediction unit 158 ​​performs prediction analysis on the "business effect" included in the selected past case as a new "prediction target" (hereinafter also referred to as "effect prediction target").

[0113] Specifically, first, the prediction unit 158 ​​sets "business effect" as "effect prediction target". Next, the prediction unit 158 ​​sets items related to the "business effect" from past cases as "usage items". Note that the prediction unit 158 ​​may set the "usage items" from items included in both the past cases and the structured user data (or the construction data set), for example.

[0114] The prediction unit 158 ​​constructs a data set (hereinafter also referred to as an effect learning data set) by extracting data corresponding to the "usage item" from past cases. The prediction unit 158 ​​generates a prediction model (hereinafter also referred to as an effect prediction model) by solving, for example, a regression problem by machine learning based on the effect prediction data set and the "effect prediction target".

[0115] Next, the prediction unit 158 ​​extracts data corresponding to the "usage items" from the structured user data to construct a data set (hereinafter also referred to as an effect prediction data set). The prediction unit 158 ​​predicts the business effect when the prediction model generated by the learning unit 156 is introduced into a business, based on the effect prediction data set and the generated effect prediction model.

[0116] The above-mentioned method is an example, and the prediction unit 158 ​​may predict the business effect based on various known techniques. Furthermore, the construction of the effect prediction data set and the learning of the effect prediction model performed by the prediction unit 158 ​​may be performed using some of the functions of the extraction unit 153 and the learning unit 156.

[0117] (Collection Decision Division) The collection determination unit 159 determines data items (hereinafter also referred to as suggested items) to be collected by the user based on past cases and user data for each extracted problem setting. When there are multiple problem settings, the collection determination unit 159 determines suggested items for each of the multiple problem settings. Note that the collection determination unit 159 may determine multiple suggested items for one problem setting.

[0118] The collection determination unit 159 compares a data set of past cases (past data set) with a data set constructed by the extraction unit 153 or a data set corrected by the leak detection correction unit 155 (constructed data set). The collection determination unit 159 extracts "used items" (hereinafter also referred to as "uncollected items") that are included in the past data set and not included in the constructed data set.

[0119] First, the collection determination unit 159 predicts the business effect when the "uncollected items" are not used in past cases. Specifically, the collection determination unit 159 learns a prediction model using a past data set excluding the "uncollected items" and evaluates the accuracy of the prediction model. The collection determination unit 159 recalculates the business effect at the evaluated prediction accuracy. Note that the learning and evaluation of the prediction model and the calculation of the business effect here are similar to the processing of the learning unit 156, the evaluation unit 157, and the prediction unit 158, and therefore will not be described.

[0120] The collection determining unit 159 determines, based on the calculated business effect, the "uncollected items" whose effect has decreased, as proposal items.

[0121] When the collection determination unit 159 extracts multiple "uncollected items", the collection determination unit 159 recalculates the business effect for each "uncollected item". Then, the collection determination unit 159 determines the "uncollected item" with the largest reduction in business effect as the proposed item. Alternatively, the collection determination unit 159 may determine the "uncollected item" with the reduction in business effect equal to or greater than a threshold as the proposed item, or may determine a predetermined number of "uncollected items" as the proposed items.

[0122] Furthermore, if the "collection costs" required for data collection are included in past cases, the collection determination unit 159 may determine the proposed items based on the newly calculated business effects and collection costs. In this case, the collection determination unit 159 calculates the difference between the introduction effect calculated by the prediction unit 158 ​​including the "uncollected items" minus the collection costs and the business effect calculated without including the "uncollected items". The collection determination unit 159 determines the "uncollected items" with the largest calculated difference as the proposed items.

[0123] In this way, the collection determination unit 159 determines the proposed items including the "collection cost" of the data, so that the information processing device 12 can preferentially propose to the user uncollected items for which the collection cost is low and data collection is easy. Alternatively, the information processing device 12 can propose to the user data collection of uncollected items for which the collection cost is high but the use of which has a large business effect.

[0124] In this embodiment, the collection determining unit 159 performs learning of a prediction model, accuracy evaluation, and calculation of business effects when "uncollected items" are not used, but is not limited to this. For example, the learning unit 156, the evaluation unit 157, and the prediction unit 158 ​​may each perform learning of a prediction model, accuracy evaluation, and calculation of business effects. In this case, the collection determining unit 159 determines the proposed items based on the results of each unit.

[0125] In addition, although the collection determination unit 159 determines the proposed items based on the business effects here, this is not limiting. The collection determination unit 159 may determine the proposed items based on, for example, the evaluation results of the prediction model. In this case, the collection determination unit 159 evaluates the accuracy of the prediction model learned without using the "uncollected items" and determines the "unused items" with the smallest decrease in evaluation as the proposed items.

[0126] (Contribution calculation section) The contribution degree calculation unit 160 calculates the contribution degree indicating how much each feature contributes to the prediction result among the features of the test data input to the prediction model learned by the learning unit 156. Specifically, the contribution degree calculation unit 160 removes the feature for which the contribution degree is to be calculated from the input to the prediction model, and calculates the contribution degree based on the change in the prediction result before and after the removal.

[0127] Here, the contribution degree calculated by the contribution degree calculation unit 160 includes positive and negative values. A positive contribution degree means that the set of features contributes positively to the prediction, i.e., improves the prediction probability predicted by the prediction model. A negative contribution degree means that the set of features contributes negatively to the prediction, i.e., decreases the prediction probability predicted by the prediction model.

[0128] Furthermore, the contribution degree calculation unit 160 calculates the ratio of the feature amount whose contribution degree has been calculated to the set (item) of feature amounts. If the calculated ratio is low, it means that the case rarely occurs even if the contribution degree is high, and the usefulness for the user is low. Therefore, in the embodiment of the present disclosure, the contribution degree calculation unit 160 calculates the ratio of the feature amount whose contribution degree has been calculated, and also presents this ratio to the user. This allows the user to check the contribution degree of the data, taking into account the occurrence rate.

[0129] Here, the prediction unit 158, the contribution degree calculation unit 160, and the collection determination unit 159 respectively calculate the business effect and the contribution degree and determine the proposed items, but it is not necessary to calculate / determine all of these. For example, the contribution degree calculation unit 160 may calculate the contribution degree, and the calculation of the business effect by the prediction unit 158 ​​and the determination of the proposed items by the collection determination unit 159 may be omitted. Alternatively, the contribution degree calculation unit 160 may calculate the contribution degree and the prediction unit 158 ​​may calculate the business effect, and the determination of the proposed items by the collection determination unit 159 may be omitted. In addition, the calculation / determination process may be selected by the user.

[0130] (Display control unit) The display control unit 143 controls the display of various information. The display control unit 143 controls the display of various information in the terminal device 11. The display control unit 143 generates an image including control information for controlling the display mode. This control information is described in a script language such as JavaScript (registered trademark) or CSS. The display control unit 143 provides the image including the above-mentioned control information to the terminal device 11, thereby causing the terminal device 11 to perform the above-mentioned display process in accordance with the control information. Note that the display control unit 143 may control the display of the terminal device 11 by appropriately using various conventional techniques, not limited to the above.

[0131] [1-5. Configuration example of time series processing unit according to embodiment] FIG. 9 shows an example of the configuration of the time series processing unit 154 in FIG.

[0132] The time series processing unit 154 includes a separating unit 201 , a resampling unit 202 , and a change point detecting unit 203 .

[0133] (separation part) When the construction data set includes multiple time-series data sets of different series, the separation unit 201 separates the time-series data sets for each series. For example, the separation unit 201 separates the construction data set into multiple time-series data sets based on the degree of overlap of values ​​of items related to date and time (hereinafter referred to as date and time items) in the construction data set.

[0134] (Resampling section) When the construction data set is a time series data set, the resampling unit 202 resamples the date and time of the time series data set. When the construction data set includes multiple time series data sets, the resampling unit 202 resamples the date and time of the time series data set for each series.

[0135] (Change point detection section) When the construction data set is a time-series data set, the change-point detection unit 203 detects a change-point at which the trend of the value to be predicted changes significantly. The change-point detection unit 203 also divides the construction data set based on the detected change-point.

[0136] [1-6. Configuration example of leak detection and correction unit according to embodiment] FIG. 10 shows an example of the configuration of the leak detection and correction unit 155 in FIG.

[0137] The leak detection and correction unit 155 includes an occurrence date and time identifying unit 221 , an acquisition date and time identifying unit 222 , a prediction execution timing setting unit 223 , a leak detection unit 224 , and a leak correction unit 225 .

[0138] (Occurrence date and time identification part) The occurrence date and time specification unit 221 specifies the occurrence date and time of each data included in the construction data set. The occurrence date and time indicates, for example, the date and time when an event related to the data actually occurred. The occurrence date and time is expressed, for example, by one or more of the year, month, day, and time.

[0139] (Acquisition date and time specification section) The acquisition date and time specification unit 222 specifies the acquisition date and time of each data included in the construction data set. The acquisition date and time indicates, for example, the date and time when the user actually became able to acquire the data, or the date and time when the user actually acquired the data. The acquisition date and time is expressed, for example, by one or more of the year, month, day, and time.

[0140] For example, the date and time of data generation for the number of shipments on December 8, 2019 will be December 8, 2019. However, the data for the number of shipments on December 8, 2019 will not necessarily be available to users on the same day. For example, it is possible that the number of shipments of a certain product is recorded at each factory, tallied at the end of the month, and actually becomes available to data analysts (users) on the 15th of the following month. In this case, the date and time of acquisition of the data for the number of shipments will be January 15, 2020.

[0141] (Prediction execution timing setting unit) The prediction execution timing setting unit 223 sets the timing for predicting the prediction target (hereinafter referred to as prediction execution timing) based on the properties of the prediction target and past cases stored in the past case storage unit 121, etc.

[0142] (Leak detection section) The leak detection unit 224 detects the possibility of a leak in the constructed data set based on the properties of the constructed data set and past cases stored in the past case storage unit 121 and the like.

[0143] The properties of the construction dataset include, for example, the properties of the entire construction dataset, the properties of each item of the construction dataset, and the properties of each piece of data included in the construction dataset.

[0144] (Leak Compensation Section) The leak correction unit 225 corrects the construction data set so as to eliminate the possibility of a leak detected by the leak detection unit 224, based on, for example, instructions from a user input via the terminal device 11, past cases stored in the past case memory unit 121, and profile information stored in the user profile memory unit 123.

[0145] [1-7. Information processing procedure according to the embodiment] Next, the procedure of information processing according to the embodiment will be described with reference to the flowchart of FIG.

[0146] In step S101, the acquisition unit 141 acquires past cases and user data from the storage unit 102 as described above.

[0147] In step S102, the time prediction unit 151 predicts the processing time (predicted processing time) required for the analysis process using the acquired part of the user data, as described above.

[0148] In step S103, the interpretation unit 152 analyzes and structures the user data to generate structured data, as described above.

[0149] In step S104, the extraction unit 153 extracts a problem setting based on the structured data and past cases, as described above.

[0150] In step S105, the extraction unit 153 constructs a data set (constructed data set) according to the extracted question setting, as described above.

[0151] In step S106, the time series processing unit 154 executes time series processing.

[0152] Here, the time series processing will be described in detail with reference to the flowchart of FIG.

[0153] In step S151, the separation unit 201 determines whether or not the constructed data set is a time-series data set. For example, if the constructed data set has a date and time item related to “date and time” among its items, the separation unit 201 determines that the constructed data set is a time-series data set, and the process proceeds to step S152.

[0154] A date and time item is an item in which data is expressed by at least one of the year, month, date, time, and day of the week, for example.

[0155] Fig. 13 shows an example of a construction data set. This construction data set includes the items of "Date", "Sales Region", "Sales Volume", "Event (Planned)", and "Advertising Cost (Planned)". In this example, "Date" is the date and time item.

[0156] In the following description, it is assumed that the "sales volume" is a prediction target, and the "sales region" is an item related to a category (hereinafter, referred to as a category item). A category is, for example, a division, a classification, a genre, or the like.

[0157] In the following description, one set of data, which is one row of data in the construction data set and includes data for each item, will be referred to as a record.

[0158] In step S152, the separation unit 201 counts the number of records with duplicated dates and times. For example, the separation unit 201 sorts the construction data set by the date and time item. Then, the separation unit 201 counts the number of records whose date and time item values ​​overlap with other records.

[0159] For example, in the construction data set of FIG. 13, the number of records whose "date" value overlaps with other records is counted.

[0160] In step S153, the separation unit 201 determines whether or not a certain ratio or more of records with overlapping dates and times exist. Specifically, the separation unit 201 calculates the ratio of the number of records found in the process of step S152 to the total number of records in the construction data set. If the calculated ratio is equal to or greater than a predetermined threshold, the separation unit 201 determines that a certain ratio or more of records with overlapping dates and times exist, and the process proceeds to step S154.

[0161] In step S154, the separation unit 201 counts the number of records whose dates and times do not overlap in each category. For example, the separation unit 201 divides the construction dataset into multiple series for each value of a category item (hereinafter referred to as a category value). Next, the separation unit 201 sorts the dataset of each series by the date and time item. Next, the separation unit 201 counts the number of records whose dates and times do not overlap with other records for each dataset of each series. Then, the separation unit 201 sums up the number of records counted for each dataset of each series.

[0162] For example, the construction dataset in FIG. 8 is divided into a dataset for the series in which the "Sales Region" value is "Region A" and a dataset for the series in which the "Region B" value is used. Next, the number of records in the dataset for the "Region A" series that have a "Date" value that does not overlap with other records is counted. Similarly, the number of records in the dataset for the "Region B" series that have a "Date" value that does not overlap with other records is counted. Then, the number of records counted in the dataset for the "Region A" series and the number of records counted in the dataset for the "Region B" series are summed up.

[0163] In step S155, the separation unit 201 determines whether the ratio of records whose dates and times do not overlap in each category is equal to or greater than a certain value. Specifically, the separation unit 201 calculates the ratio of the number of records found in the process of step S154 to the total number of records in the construction data set. If the calculated ratio is equal to or greater than a predetermined threshold, the separation unit 201 determines that the ratio of records whose dates and times do not overlap in each category is equal to or greater than a certain value, and the process proceeds to step S156.

[0164] This may be the case, for example, when the construction dataset has a high ratio of records with duplicate values ​​in the date and time field, but by dividing the construction dataset into multiple series based on category values, the ratio of records with duplicate dates and times is significantly reduced.

[0165] In step S156, the separating unit 201 determines that the constructed data set includes multiple time-series data sets. That is, the separating unit 201 determines that the constructed data set is separated into multiple time-series data sets for each category value.

[0166] For example, in the construction dataset in FIG. 13, the ratio of records whose "date" value overlaps with other records is high. On the other hand, as shown in FIG. 14, when the construction dataset is divided into a dataset for the "region A" series and a dataset for the "region B" series, the ratio of records whose "date" value overlaps with other records decreases in the dataset for each series. Therefore, the construction dataset in FIG. 13 is determined to be separated into two series of time series datasets, one for the "sales region" series of "region A" and one for the "sales region" series of "region B".

[0167] The process then proceeds to step S158.

[0168] On the other hand, if it is determined in step S155 that the ratio of records with no overlapping dates and times within each category is less than a certain value, the process proceeds to step S157.

[0169] If it is determined in step S153 that the number of records with overlapping dates and times is less than the certain ratio, the process proceeds to step S157.

[0170] In step S157, the separating unit 201 determines that the constructed data set is a data set of a single time series.

[0171] The process then proceeds to step S158.

[0172] In step S158, the resampling unit 202 performs date and time resampling of the data set.

[0173] Specifically, the resampling unit 202 divides the construction data set into each series and sorts the data set of each series by the date and time item. Note that when the construction data set is a time series data set of a single series, the entire construction data set becomes one series.

[0174] Next, the resampling unit 202 calculates the time interval between adjacent records for each series, and then determines the time interval that appears most frequently (hereinafter referred to as the most frequent time interval) in all series.

[0175] Next, the resampling unit 202 sets the time unit based on the modal time interval. For example, if the modal time interval<60 seconds, the time unit is set to "seconds". If 60 seconds≦modal time interval<60×60 seconds, the time unit is set to "minutes". If 60×60 seconds≦modal time interval<24×60×60 seconds, the time unit is set to "hours". If 24×60×60 seconds≦modal time interval<365×24×60×60 seconds, the time unit is set to "days". If 365×24×60×60 seconds≦modal time interval, the time unit is set to "years".

[0176] Then, the resampling unit 202 resamples the data set for each series in the set time unit, i.e., the value of the date and time item of the data set for each series is reset based on the set time unit.

[0177] In step S159, the resampling unit 202 interpolates missing values ​​of the prediction target. Specifically, the resampling unit 202 interpolates values ​​at the date and time when the value of the prediction target is missing for each series. For example, a method suitable for the nature of the prediction target is used as the interpolation method from among previous value inheritance, moving average, etc.

[0178] For example, in a data set acquired at one-hour intervals during business hours from 9:00 to 17:00, data is missing from 18:00 to 8:00 the following day (hereinafter referred to as the missing period). In response to this, the data to be predicted in the missing period is interpolated by the processes of steps S158 and S159.

[0179] Figure 15 is a graph showing the time series of sales volume for each series after resampling of the data set for "Sales region" being "Region A" and the data set for "Sales region" being "Region B" in the constructed data set in Figure 13. The horizontal axis shows the date, and the vertical axis shows the sales volume.

[0180] In step S160, the change-point detection unit 203 determines whether or not a change-point to be predicted exists.

[0181] Specifically, the change point detection unit 203 detects a point (date and time) at which the trend of the value of the prediction target changes significantly when the values ​​of the prediction target are arranged in a time series for each series. For example, a method suitable for the property of the prediction target is used as a change point detection method. For example, a method can be considered in which time series prediction of the prediction target is repeatedly performed in ascending order of date and time, and a point at which the prediction and the actual value deviate significantly is detected as a change point.

[0182] If a change point is detected in at least one series, the change point detection unit 203 determines that a change point to be predicted exists, and the process proceeds to step S161.

[0183] In step S161, the change-point detection unit 203 divides the data set based on the change points. Specifically, the change-point detection unit 203 divides the construction data set into a plurality of periods for each change point detected in any of the series.

[0184] Then, for example, by ignoring the data set before the last period and performing the learning process using only the data set for the last period, the prediction model is prevented from being influenced by data before the change in the prediction target occurs, thereby preventing a decrease in the prediction accuracy of the prediction model during actual operation.

[0185] The process then proceeds to step S162.

[0186] For example, if the construction data set includes multiple category items, the processes in steps S154 to S161 may be performed for each category item. As a result, separation and resampling of the construction data set are performed for each category item.

[0187] On the other hand, in step S160, if no change point is detected in any of the sequences, the change point detection unit 203 determines that no change point to be predicted exists, the process of step S161 is skipped, and the process proceeds to step S162.

[0188] In step S162, the display control unit 143 presents the result of the time-series processing to the user. For example, the display control unit 143 causes the terminal device 11 to display the image of FIG.

[0189] In the image of FIG. 16, a graph 301, a list box 302, a list box 303, a check box 304, a list box 305, a "Back" button 306, a "Cancel" button 307, and an "Execute" button 308 are displayed.

[0190] Graph 301 shows a graph of the sales volume for the "Area A" series and the sales volume for the "Area B" series of the constructed dataset in Fig. 13. In other words, the constructed dataset is separated into two time-series datasets for "Area A" and "Area B", and the results of resampling the date and time of each time-series dataset are shown.

[0191] List box 302 and list box 303 are used to set the prediction period. The prediction period is the period for which a prediction target (for example, sales volume) is predicted. List box 302 is used to set the start date and time of the prediction period, and list box 303 is used to set the end date and time of the prediction period. In other words, the prediction period is set to the date and time set in list box 302 to the date and time set in list box 303. In this example, the prediction period is set to the period from 10 days to 40 days ahead based on the prediction execution timing.

[0192] Check box 304 is used when directly inputting the forecast period without using list box 302 and list box 303. In other words, when check box 304 is checked, it becomes possible to directly input the forecast period. On the other hand, when check box 304 is unchecked, it becomes possible to input the reservation period using list box 302 and list box 303.

[0193] The list box 305 is used to select a series for separating the construction dataset into multiple time-series datasets. For example, if the construction dataset can be separated into multiple time-series datasets by multiple category items, the names of the category items are displayed as series names in the list box 305. Then, a graph of the time-series dataset separated based on the category item selected in the list box 305 is displayed as graph 301.

[0194] A "Back" button 306 is used to return to the image before transition to the image of FIG.

[0195] The "Cancel" button 307 is used to cancel the execution of learning and evaluation of the predictive model.

[0196] The "Execute" button 307 is used to execute learning and evaluation of a prediction model.

[0197] After that, the time series processing ends.

[0198] Returning to FIG. 12, on the other hand, in step S151, if there is no date and time item among the items of the constructed data set, the separation unit 201 determines that the constructed data set is a non-time series data set, the processing of steps S151 to S162 is skipped, and the time series processing ends.

[0199] Returning to FIG. 11, in step S107, the leak detection correction unit 155 executes the leak detection process.

[0200] The leak detection process will now be described in detail with reference to the flowcharts of FIGS.

[0201] In step S201, the occurrence date and time identifying unit 221 and the acquisition date and time identifying unit 222 identify the occurrence date and time and the acquisition date and time of data.

[0202] Specifically, the occurrence date and time identifying unit 221 identifies the occurrence date and time of each piece of data included in the construction data set.

[0203] In addition, the date and time of data generation is often specified by the user or is self-evident. In this case, the generation date and time identification unit 221 sets the generation date and time as the data generation date and time. For example, the generation date and time of the data "shipment quantity" of "December 8, 2019" is identified as December 8, 2019.

[0204] On the other hand, if the date and time of occurrence of the data is unknown, the date and time of occurrence identifying part 221 estimates the date and time of occurrence based on, for example, the contents of the data.

[0205] For example, the occurrence date and time specification unit 221 estimates the occurrence date and time of the data based on a database of phenomena related to the data and the occurrence times of various phenomena. For example, if a building is shown in the image data, and if the construction date of the building is known, the occurrence date and time of the image data is estimated to be after the construction date. For example, if the construction data set includes a "song title" item, the occurrence date and time of the data included in each record is estimated to be after the release date of the song with the song title included in the same record. For example, the failure data of a certain manufacturing device can be estimated from the production date and lifespan of the manufacturing device.

[0206] Furthermore, the acquisition date and time identifying unit 222 identifies the acquisition date and time of each piece of data included in the data set.

[0207] The acquisition date and time of data is, for example, strongly influenced by business workflows, and therefore it is difficult to estimate the date and time of data generation using a database. On the other hand, data used in business is under the control of business workflows, and therefore the acquisition date and time are often recorded.

[0208] Therefore, for example, the acquisition date and time identifying unit 222 identifies the acquisition date and time of each data based on the file timestamp of each data, Exif (Exchangeable image file format) information, the update time of the database, the history of the version control system, and the like.

[0209] In step S202, the prediction execution timing setting unit 223 sets the timing for executing the prediction.

[0210] The earliest timing for executing a prediction of a prediction target (prediction execution timing) is after the timing of acquiring the last data of the data set used for the prediction. However, in reality, a certain period called a gap is required between acquiring the last data and executing the prediction.

[0211] Therefore, for example, the prediction execution timing setting unit 223 sets how far in advance of the prediction period the prediction of the prediction target is to be executed based on past cases and the properties of the prediction target.

[0212] For example, when the prediction target is the "number of lunchbox shipments," prediction accuracy is improved by using the sales amount of the previous day, etc. However, for example, if the ordering deadline for ingredients for lunchboxes is at least three days before, the prediction execution timing setting unit 223 sets the prediction execution timing to three days before the prediction period.

[0213] Also, for example, in the image of Fig. 16 described above, when a prediction period is set, the prediction execution timing is set based on the prediction period. For example, in the example of Fig. 16, since the prediction period is set to a period from 10 days to 40 days ahead, the prediction execution timing is set to 40 days before the prediction period.

[0214] In step S203, the leak detection unit 224 determines whether or not there is data whose occurrence date / time or acquisition date / time is later than the predicted execution timing. For example, the leak detection unit 224 compares the occurrence date / time and acquisition date / time of each data in the construction data set with the predicted execution timing. Then, when there is data whose occurrence date / time or acquisition date / time is later than the predicted execution timing, the leak detection unit 224 determines that there is data whose occurrence date / time or acquisition date / time is later than the predicted execution timing, and the process proceeds to step S204.

[0215] In step S204, the leak detection unit 224 determines that there is a possibility of a time series leak. That is, the leak detection unit 224 determines that there is a possibility of a leak because data whose generation date / time or acquisition date / time is later than the predicted execution timing will not be generated or obtained by the predicted execution timing. In this way, the possibility of a time series leak is detected based on the relationship between the generation date / time or acquisition date / time of the data and the predicted execution timing.

[0216] The process then proceeds to step S205.

[0217] On the other hand, in step S203, if there is no data with at least one of the occurrence date and time and the acquisition date and time after the predicted execution timing, the leak detection unit 224 determines that there is no data with the occurrence date and time or the acquisition date and time after the predicted execution timing, skips the process of step S204, and proceeds to step S205. In other words, it is determined that the possibility of a time series leak is low.

[0218] In step S205, the leak detection unit 224 determines the identifiability of each item in the dataset. That is, the leak detection unit 224 determines whether each item in the dataset includes identification information (e.g., an ID, etc.) that can identify each record in the dataset.

[0219] Examples of the characteristics of distinctive items (hereinafter referred to as distinctive items) include the following:

[0220] The uniqueness rate of the data within the item is 100% or close to 100%. -The item name contains the string "ID". It is placed in the first or near-first column of the dataset. -It is not a date and time item. Data is represented as a string or an integer. The data has no units.

[0221] For example, the leak detection unit 224 determines the identifiability of each item in the data set based on the above properties and the like, on a rule basis or machine learning basis.

[0222] In step S206, the leak detection unit 224 determines whether or not an identification item exists based on the result of the process in step S205. If it is determined that an identification item exists, the process proceeds to step S207.

[0223] In step S207, the leak detection unit 224 performs clustering on the identification items. For example, when the data of the identification items (identification information) are represented by character strings, the leak detection unit 224 performs clustering on the basis of the Levenshtein distance.

[0224] As a result, the records in the construction data set are classified into a plurality of clusters based on the identification items (identification information).

[0225] In step S208, the leak detection unit 224 checks the correlation between each cluster and the prediction target. For example, the leak detection unit 224 calculates a correlation coefficient between each cluster and the value of the prediction target in the record included in each cluster.

[0226] In step S209, the leak detection unit 224 determines whether or not there is a correlation between the identification item and the prediction target. If the correlation coefficient calculated in the process of step S208 is equal to or greater than a predetermined threshold, the leak detection unit 224 determines that there is a correlation between the identification item and the prediction target, and the process proceeds to step S210.

[0227] In step S210, the leak detection unit 224 determines that there is a possibility of an identification information leak.

[0228] Usually, the value of the identification information included in the identification item has no meaning other than uniquely identifying the record. However, if there is a correlation between the identification information and the prediction target, it is assumed that there is a high possibility that important information related to the prediction target is leaked into the identification information. For example, if the identification information includes characters indicating the user's gender and the prediction target is the amount of purchase of clothes by the user, it is assumed that there is a certain degree of correlation between the gender and the amount of purchase of clothes. Therefore, in this case, it may be determined that there is a possibility of identification information leak.

[0229] The process then proceeds to step S211.

[0230] On the other hand, in step S209, if the correlation coefficient calculated in the process of step S208 is less than the predetermined threshold, the leak detection unit 224 determines that there is no correlation between the identification item and the prediction target, skips the process of step S210, and proceeds to step S211. In other words, it is determined that the possibility of identification information leak is low.

[0231] Also, if it is determined in step S206 that the identification item does not exist, the processes in steps S207 to S210 are skipped and the process proceeds to step S211.

[0232] In step S211, the leak detection unit 224 determines whether or not the data set includes a category item. If the leak detection unit 224 determines that the construction data set includes a category item, the process proceeds to step S212.

[0233] In step S212, the leak detection unit 224 performs group K-fold cross validation using groups divided according to categories.

[0234] Specifically, the leak detection unit 224 first divides the construction data set into a plurality of groups for each value of the category item (category value).Then, the leak detection unit 224 uses one of each group as test data and the remaining groups as learning data to learn and verify the prediction model.Thereby, the prediction accuracy of the prediction model when each group is used as the test data is obtained.

[0235] In step S213, the leak detection unit 224 determines whether the difference in prediction accuracy between the groups is large. Specifically, the leak detection unit 224 compares the prediction accuracy of the prediction model when each group is used as test data. For example, if there is a significant difference between the prediction accuracy of each group, the leak detection unit 224 determines that the difference in prediction accuracy between the groups is large, and the process proceeds to step S214. Alternatively, for example, if the difference between the maximum and minimum values ​​of the prediction accuracy of each group is equal to or greater than a predetermined threshold, the leak detection unit 224 determines that the difference in prediction accuracy between the groups is large, and the process proceeds to step S214.

[0236] In step S214, the leak detection unit 224 determines that there is a possibility of a category leak. That is, the data of the category item may contain information useful for predictive analysis, while a problem set may be solved with high accuracy based on the data of the category item using a simple rule base. Since such a problem is meaningless in practical terms, it is determined that there is a possibility of a leak in the category item.

[0237] The process then proceeds to step S215.

[0238] On the other hand, in step S213, for example, if there is no significant difference between the prediction accuracies of the groups, the leak detection unit 224 determines that the difference in prediction accuracies between the groups is small, the process of step S214 is skipped, and the process proceeds to step S215. Alternatively, for example, if the difference between the maximum and minimum values ​​of the prediction accuracies of the groups is less than a predetermined threshold, the leak detection unit 224 determines that the difference in prediction accuracies between the groups is small, the process of step S214 is skipped, and the process proceeds to step S215. In other words, it is determined that the possibility of a category leak is low.

[0239] When the construction data set includes a plurality of category items, for example, the processes in steps S212 to S214 are executed for each category item.

[0240] Also, if it is determined in step S211 that the data set does not include a category item, the processes in steps S212 to S214 are skipped and the process proceeds to step S215.

[0241] In step S215, it is determined whether or not the construction data set is a time-series data set, similarly to the process in step S151 in Fig. 12. If it is determined that the construction data set is a non-time-series data set, the process proceeds to step S216.

[0242] In step S216, the leak detection unit 224 performs hold-out validation and K-fold cross-validation.

[0243] Specifically, the leak detection unit 224 performs holdout verification while maintaining the order (of records) of the construction data set. For example, as shown in Fig. 19, the leak detection unit 224 sets a predetermined percentage (e.g., 80%) of records from the beginning of the construction data set as learning data (train), sets the remaining records as test data (holdout), and performs learning and verification of the prediction model. In this way, the prediction accuracy of the prediction model is obtained.

[0244] Furthermore, the leak detection unit 224 performs K-fold cross validation while maintaining the order (of records) of the construction dataset. For example, as shown in Fig. 19, the leak detection unit 224 divides the construction dataset into three groups (fold1 to fold3). Then, the leak detection unit 224 uses one of each group as test data and the remaining groups as learning data to learn and validate the prediction model. This allows the prediction accuracy of the prediction model when each group is used as test data to be obtained.

[0245] In step S217, the leak detection unit 224 determines whether there is a large difference between the result of the hold-out validation and the result of the K-fold cross validation. Specifically, the leak detection unit 224 compares the prediction accuracy in the hold-out validation with multiple prediction accuracies in the K-fold cross validation when each group is used as test data.

[0246] If at least one of the multiple prediction accuracies in the K-fold cross-validation is significantly different from the prediction accuracies in the hold-out validation, the leak detection unit 224 determines that the difference between the results of the hold-out validation and the results of the K-fold cross-validation is large, and the process proceeds to step S218.

[0247] Alternatively, for example, the leak detection unit 224 calculates the difference between each prediction accuracy in the K-fold cross validation and the prediction accuracy in the hold-out validation. If the maximum value of the calculated prediction accuracy differences is equal to or greater than a predetermined threshold, the leak detection unit 224 determines that the difference between the result of the hold-out validation and the result of the K-fold cross validation is large, and the process proceeds to step S218.

[0248] In step S218, the leak detection unit 224 determines that there is a possibility of an order leak.

[0249] If there is a large difference between the results of hold-out validation and the results of K-fold cross-validation, it is possible that, for example, there is some significance in the order of records in the construction dataset, and the accuracy of the predictive model varies significantly depending on the selection of the training data.

[0250] On the other hand, when the construction dataset is a non-time series dataset, predictive analysis is performed on the assumption that the data trends will not change between the time of pre-evaluation and the time of actual operation. However, if the accuracy of the predictive model varies significantly depending on the selection of the learning data, this assumption no longer holds true.

[0251] Therefore, it is determined that there is a possibility of a leak in the order of the construction data sets.

[0252] The process then proceeds to step S219.

[0253] On the other hand, in step S217, if there is no significant difference between the prediction accuracy in the K-fold cross validation and the prediction accuracy in the hold-out validation, the leak detection unit 224 determines that the difference between the result of the hold-out validation and the result of the K-fold cross validation is small, the process of step S218 is skipped, and the process proceeds to step S219. That is, it is determined that the possibility of a sequence leak is low.

[0254] Alternatively, for example, when the maximum value of the difference between the prediction accuracy in the K-fold cross validation and the prediction accuracy in the hold-out validation is less than a predetermined threshold, the leak detection unit 224 determines that the difference between the result of the hold-out validation and the result of the K-fold cross validation is small, the process of step S218 is skipped, and the process proceeds to step S219. In other words, it is determined that the possibility of a sequence leak is low.

[0255] Also, if it is determined in step S215 that the construction data set is a time-series data set, the processes in steps S216 to S218 are skipped and the process proceeds to step S219.

[0256] In step S219, the leak detection unit 224 detects the possibility of an item name leak based on the item names of the data set.

[0257] For example, the degree of difficulty in predicting or controlling each phenomenon is compiled into a database in advance and stored in the storage unit 102 .

[0258] On the other hand, for example, the leak detection unit 224 matches the item names of each item of the construction data set with each phenomenon based on a rule base or similarity of distributed expressions between words. Next, the leak detection unit 224 obtains the degree of difficulty of predicting or controlling the phenomenon corresponding to each item of the construction data set based on the above-mentioned database. Then, the leak detection unit 224 determines that an item corresponding to a phenomenon whose degree of difficulty is equal to or greater than a predetermined threshold may be a leak.

[0259] For example, natural phenomena are generally difficult to control and therefore difficult to predict early. Therefore, if a dataset contains item names related to natural phenomena, such as "temperature," "weather," and "seismic intensity," and the prediction execution timing is more than one month in the future, this item is determined to have a possibility of leaking.

[0260] In step S220, the leak detection unit 224 detects the possibility of a domain leak based on the domain of the data set. For example, the leak detection unit 224 detects the possibility of a leak in the constructed data set based on the domain of the entire constructed data set (e.g., sales, production management, human resources, etc.) by referring to past cases where leaks tend to occur in each domain.

[0261] The leak detection process then ends.

[0262] Returning to FIG. 11, in step S108, the learning unit 156 learns a prediction model based on the problem setting and the construction data set, as described above.

[0263] In step S109, the evaluation unit 157 divides the data set into training data and test data, as described above, and generates a prediction model using the training data. The evaluation unit 157 evaluates the prediction model using the test data.

[0264] In step S110, the prediction unit 158 ​​predicts the business effect when the prediction model is introduced into the business, as described above.

[0265] In step S111, the collection determining unit 159 determines, as proposed items, items that are likely to increase business effects when added to the data set, based on past cases as described above.

[0266] In step S112, the contribution calculation section 160 calculates the contribution of the feature amount of the test data as described above.

[0267] In step S113, the display control unit 143 presents the processing result to the user. For example, the display control unit 143 causes the terminal device 11 to display an image showing the analysis processing result of FIG.

[0268] In this image, the contribution degree indicating how much each feature amount (item) of the test data input to the prediction model trained by the training unit 156 contributes to the prediction result is displayed.

[0269] For example, the contribution rate for each item such as "number of shipments last week," "day of the week," and "date" is displayed in a bar graph. As described above, the contribution rate includes positive and negative values. Therefore, the sum of the positive values ​​and the sum of the negative values ​​are displayed in a bar graph.

[0270] Furthermore, for the items "Last week's shipment volume," "Temperature," and "Humidity" that have been determined to have a leak possibility, icons 331 to 333 indicating the possibility of a leak are displayed.

[0271] Furthermore, for the "Last week's shipments" item that has been determined to have a leak possibility, a window 334 is displayed. Within the window 334, it is displayed that "Last week's shipments" has a high possibility of a leak, that the contribution rate is very high, and the reason why it has been determined to have a high possibility of a leak. Here, the reason shown is that "Last week's shipments" cannot be obtained one month in advance (the timing for executing the prediction).

[0272] Also, for the items "Temperature" and "Humidity" that have been determined to have a leak possibility, a window 335 is displayed. In the window 335, it is displayed that "Temperature" and "Humidity" have a high possibility of leaking, and the reason why they have been determined to have a high possibility of leaking. Here, the reason shown is that it is difficult to obtain "Temperature" and "Humidity" one month in advance (the prediction execution timing).

[0273] Additionally, below the image, a window 336 is displayed, which shows the prediction accuracy before and after removing possible leak items, and the advantage of removing possible leak items in maintaining the same prediction accuracy in actual operation. Also displayed is a menu that allows the user to select whether or not to apply automatic leak correction.

[0274] Returning to FIG. 11, in step S114, the information processing unit 142 determines whether or not to correct the data set or the problem setting.

[0275] For example, if the user selects application of automatic leak correction in window 336 of the image in FIG. 20, the leak correction unit 225 determines that the construction data set is to be corrected, and the process proceeds to step S115.

[0276] Also, for example, if the construction data set contains an item whose leak probability is equal to or greater than a predetermined threshold, even if the user does not select to apply automatic leak correction in window 336 of the image in FIG. 20, the leak correction unit 225 determines to correct the construction data set, and processing proceeds to step S115.

[0277] Furthermore, for example, when a command to correct the build data set or the problem setting is input via the terminal device 11, the learning unit 156 determines to correct the build data set or the problem setting, and the process proceeds to step S115.

[0278] In step S115, the information processing unit 142 corrects the data set or the problem setting.

[0279] For example, in the image of FIG. 20, the leak correction unit 225 removes all items that may be leaks from the build data set if the user selects to apply automatic leak correction.

[0280] For example, in the image of FIG. 20, if the user does not select to apply automatic leak correction, the leak correction unit 225 removes from the construction data set any items whose leak probability is equal to or exceeds a predetermined threshold.

[0281] For example, when a command to correct the build data set or the question setting is input by the user, the learning unit 156 corrects the build data set or the question setting in accordance with the command.

[0282] Thereafter, the process returns to step S106, and the processes of steps S106 to S115 are repeatedly executed until it is determined in step S114 that the data set or the problem setting is not to be corrected. As a result, learning and evaluation of the prediction model are performed based on the corrected construction data set or problem setting.

[0283] On the other hand, if it is determined in step S114 that the data set and the problem setting are not to be corrected, the information processing ends.

[0284] For example, the time prediction unit 151 may predict the processing time at the timing when the processing of each step is completed.

[0285] Also, for example, when the extraction unit 153 extracts multiple problem settings, the information processing device 12 may perform analysis processing for all problem settings by repeatedly executing the processing of steps S105 to S115 for each problem setting.

[0286] In this way, it is possible to improve user satisfaction with predictive analysis.

[0287] For example, in a predictive analysis system like information processing system 1, which automates processes from problem definition to predictive analysis, leaks are particularly likely to occur because not only are data not checked by humans, but data collection and model tuning are also performed to increase accuracy. As a result, situations tend to arise where a high business effect is expected in a pre-evaluation and predictive analysis is introduced into a business, but the expected effect is not obtained when it is actually put into operation. This results in a decrease in user satisfaction.

[0288] On the other hand, according to the present disclosure, as described above, items and data that may be leaked in the construction data set can be automatically detected with high accuracy. Furthermore, the detected leaks can be corrected automatically or manually.

[0289] This prevents the prediction accuracy of the prediction model from fluctuating significantly between pre-evaluation and actual operation, and increases the probability of achieving the prediction accuracy and business effects expected by the user, thereby improving user satisfaction.

[0290] <2. Modifications> The above configurations are merely examples, and the information processing system 1 may have any system configuration as long as it is possible to extract problem settings and construct a data set based on past cases and user data. For example, the information processing device 12 and the terminal device 11 may be integrated.

[0291] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0292] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0293] Also, for example, feedback from each user regarding the detection result of the possibility of a leak may be accumulated in the user profile storage unit 123, and the correction method during automatic leak correction may be personalized for each user. For example, items to be removed from the construction data set during automatic leak correction may be personalized for each user.

[0294] Also, for example, the possibility of a leak may be determined by combining the possibilities of the above-mentioned multiple types of leaks.

[0295] Furthermore, even if the construction dataset does not include a date and time item, for example, it may be regarded as a time-series dataset in which records are arranged at a predetermined time interval, and time-series predictive analysis processing may be performed.

[0296] 18, for example, the construction data set may be divided into a plurality of groups based on the values ​​of items other than the category items, and group K-fold cross validation may be performed. Also, for example, the construction data set may be divided into a plurality of groups based on a combination of values ​​of two or more items, and group K-fold cross validation may be performed.

[0297] 18, only K-fold cross-validation may be performed, and if the difference in prediction accuracy between groups in the K-fold cross-validation is large, it may be determined that there is a possibility of an order leak. That is, while maintaining the order of the construction data set, the construction data set may be divided into learning data and test data in a number of different patterns, and the possibility of an order leak may be detected based on the difference in prediction accuracy obtained as a result of verifying the prediction model for each pattern.

[0298] Also, for example, items and data that may be leaked may change depending on the relationship between the prediction execution timing and the prediction period. Therefore, a prediction model may be learned using a different construction data set depending on the prediction period, and an actual prediction analysis may be performed using a different prediction model for each prediction period. For example, when performing a prediction analysis for 1 to 12 months later, the prediction model may be dynamically switched to perform the prediction analysis for 1 month later, 2 to 4 months later, 5 to 8 months later, and 9 to 12 months later.

[0299] Furthermore, for example, in an image displaying the detection results of a possible leak, in addition to the items determined to have a possible leak, for example, data determined to have a possible leak and the reason for this may be displayed.

[0300] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0301] <3. Hardware configuration> The information devices such as the information processing device 12 and the terminal device 11 according to the above-mentioned embodiments and modifications are realized by a computer 1000 having a configuration as shown in FIG. 21, for example. FIG. 21 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device such as the information processing device 12 and the terminal device 11. The information processing device 12 according to the embodiment will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0302] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0303] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0304] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a program according to the present disclosure, which is an example of program data 1450.

[0305] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0306] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0307] For example, when the computer 1000 functions as the information processing device 12 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 103, etc. Also, the HDD 1400 stores the program according to the present disclosure and data in the storage unit 102. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0308] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0309] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0310] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0311] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0312] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0313] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0314] The present technology can also be configured as follows.

[0315] (1) A leak detection unit that detects the possibility of a leak in a dataset based on the characteristics of the dataset used to train a predictive model used in predictive analysis. An information processing device. (2) The leak detection unit detects a possibility of a leak of the data set based on a relationship between a generation date / time or an acquisition date / time of data included in the data set and a timing for performing a predictive analysis. The information processing device according to (1). (3) The leak detection unit determines that data whose occurrence date and time or whose acquisition date and time is after the timing for performing the predictive analysis is likely to be leaked. The information processing device according to (2). (4) The leak detection unit detects a possibility of a leak of the dataset based on a correlation between an identifying item having discriminability of the dataset and a prediction target of the predictive analysis. The information processing device according to any one of (1) to (3). (5) The leak detection unit determines that there is a possibility of a leak in the identification item when there is a correlation between the identification item and the prediction target. The information processing device according to (4). (6) The leak detection unit divides the dataset into a plurality of groups according to a value of at least one item of the dataset, and detects a possibility of data leak of the item based on a result of group K-fold cross-validation. The information processing device according to any one of (1) to (5). (7) The leak detection unit detects a possibility of a data leak of the item based on a difference in prediction accuracy between the groups. The information processing device according to (6). (8) The leak detection unit divides the dataset into learning data and test data in a plurality of different patterns while maintaining the order of the dataset, and detects a possibility of a leak in the dataset based on a result of verifying the prediction model for each of the patterns. 8. The information processing device according to any one of (1) to (7). (9) The leak detection unit detects a possibility of a leak in the sorting order of the data set based on a difference in prediction accuracy between the patterns. The information processing device according to (8). (10) The leak detection unit detects a possibility of a leak of the data set based on the item names of the data set. The information processing device according to any one of (1) to (9). (11) The leak detector detects a possible leak of the data set based on a domain of the data set. The information processing device according to any one of (1) to (10). (12) a display control unit that controls display of the detection result of the possibility of a leak of the data set; The information processing device according to any one of (1) to (11) further comprises: (13) The display control unit controls display of items or data that may be leaked in the data set and reasons for the possible leaks. The information processing device according to (12) further comprises: (14) a leak correction unit that corrects the data set based on a result of detection of a possibility of a leak in the data set; The information processing device according to any one of (1) to (13) further comprises: (15) The leak correction unit removes potential leak items from the data set. The information processing device according to (14). (16) The leak correction unit personalizes a method of correcting the data set based on user feedback regarding the detection result of a possible leak in the data set. The information processing device according to (14) or (15). (17) a separation unit that separates the data set into a plurality of time series data sets based on a degree of overlap of values ​​of items related to date and time in the data set; The information processing device according to any one of (1) to (16) further comprises: (18) a resampling unit that resamples the data set based on an interval between values ​​of items related to date and time in the data set; The information processing device according to any one of (1) to (17) further comprises: (19) An information processing device, Detecting possible leaks in a dataset based on the characteristics of the dataset used to train a predictive model used in predictive analysis Information processing methods. (20) Detecting possible leaks in a dataset based on the characteristics of the dataset used to train a predictive model used in predictive analysis A program that causes a computer to execute a process. [Explanation of symbols]

[0316] 1 Information processing system, 11 Terminal device, 12 Information processing device, 103 Control unit, 142 Information processing unit, 143 Display control unit, 152 Interpretation unit, 153 Extraction unit, 154 Time series processing unit, 155 Leak detection and correction unit, 156 Learning unit, 157 Evaluation unit, 201 Separation unit, 202 Resampling unit, 203 Change point detection unit, 223 Prediction timing setting unit, 224 Leak detection unit, 225 Leak correction unit

Claims

1. A leak detection unit that detects the possibility of a leak in a dataset based on the characteristics of the dataset used to train a predictive model used in predictive analysis. Preparation, The leak detection unit determines, based on a relationship between a generation date / time or an acquisition date / time of the data included in the data set and a timing at which predictive analysis is performed, that data whose generation date / time or the acquisition date / time is after the timing at which predictive analysis is performed may be leaked. Information processing device.

2. The leak detection unit detects a possibility of a leak of the dataset based on a correlation between an identifying item having discriminability of the dataset and a prediction target of the predictive analysis. The information processing device according to claim 1 .

3. The leak detection unit determines that there is a possibility of a leak in the identification item when there is a correlation between the identification item and the prediction target. The information processing device according to claim 2 .

4. The leak detection unit divides the dataset into a plurality of groups according to a value of at least one item of the dataset, and detects a possibility of data leak of the item based on a result of group K-fold cross-validation. The information processing device according to claim 1 .

5. The leak detection unit detects a possibility of a data leak of the item based on a difference in prediction accuracy between the groups. The information processing device according to claim 4.

6. The leak detection unit divides the dataset into learning data and test data in a plurality of different patterns while maintaining the order of the dataset, and detects a possibility of a leak in the dataset based on a result of verifying the prediction model for each of the patterns. The information processing device according to claim 1 .

7. The leak detection unit detects a possibility of a leak in the sorting order of the data set based on a difference in prediction accuracy between the patterns. The information processing device according to claim 6.

8. The leak detection unit detects a possibility of a leak of the data set based on the item names of the data set. The information processing device according to claim 1 .

9. a display control unit that controls display of the detection result of the possibility of a leak of the data set; The information processing device according to claim 1 further comprising:

10. The display control unit controls display of items or data that may be leaked in the data set and reasons for the possible leaks. The information processing device according to claim 9 further comprising:

11. a leak correction unit that corrects the data set based on a result of detection of a possibility of a leak in the data set; The information processing device according to claim 1 further comprising:

12. The leak correction unit removes potential leak items from the data set. The information processing device according to claim 11.

13. The leak correction unit personalizes a method of correcting the data set based on user feedback regarding the detection result of a possible leak in the data set. The information processing device according to claim 11.

14. a separation unit that separates the data set into a plurality of time series data sets based on a degree of overlap of values ​​of items related to date and time in the data set; The information processing device according to claim 1 further comprising:

15. a resampling unit that resamples the data set based on an interval between values ​​of items related to date and time in the data set; The information processing device according to claim 1 further comprising:

16. An information processing device, Based on the relationship between the occurrence date / time or acquisition date / time of data included in a dataset used to learn a predictive model used in predictive analysis and the timing of performing predictive analysis, data whose occurrence date / time or acquisition date / time is after the timing of performing predictive analysis is determined to have a possibility of leaking. Information processing methods.

17. Based on the relationship between the occurrence date / time or acquisition date / time of data included in a dataset used to learn a predictive model used in predictive analysis and the timing of performing predictive analysis, data whose occurrence date / time or acquisition date / time is after the timing of performing predictive analysis is determined to have a possibility of leaking. A program that causes a computer to execute a process.

Images