Method for creating extended data set for analysis, and computer program for creating extended data set for analysis

Abstract

This invention provides a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis, which enable analysis with higher accuracy even when the number of samples is small. A method for creating an extended data set for analysis according to the present invention comprises: a step for creating a primary data set on the basis of raw data; and a step for creating an extended data set for analysis on the basis of the primary data set. A method for creating an extended data set for analysis according to another aspect of the present invention comprises: a step for creating extended data on the basis of raw data; and a step for creating an extended data set for analysis. A computer program for creating an extended data set for analysis according to another aspect of the present invention causes a computer to execute: a step for creating a primary data set on the basis of raw data; and a step for creating an extended data set for analysis. A computer program for creating an extended data set for analysis according to yet another aspect causes a computer to execute: a step for creating extended data on the basis of raw data; and a creation step for creating an extended data set for analysis.

Description

Method for creating an extended data set for analysis and computer program for creating an extended data set for analysis

[0001] The present invention relates to a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis.

[0002] With the recent trend toward digitalization, data analysis is becoming increasingly important. Data analysis can provide more useful information, improving business efficiency and convenience in our daily lives. For example, in the medical field, accumulating large amounts of data on a particular case and analyzing it can lead to the establishment of new treatment methods and the development of preventive medicine.

[0003] Regarding data analysis technology, for example, Patent Document 1 below discloses a technology for acquiring medical data and generating a learning dataset from the medical data.

[0004] Japanese Patent Application Laid-Open No. 2021-086558

[0005] However, the technology described in Patent Document 1 simply generates a dataset for analysis after selecting the necessary data from the acquired medical data, and has the problem that it is difficult to perform highly accurate analysis when the number of samples of medical data acquired is small to begin with, or when there is an extreme difference in the number of samples.

[0006] In view of the above problems, the present invention aims to provide a method for creating an extended dataset for analysis and a computer program for creating an extended dataset for analysis, which enable more accurate analysis even when the number of samples is small.

[0007] A method for creating an extended dataset for analysis according to one aspect of the present invention that solves the above problem comprises a primary dataset creation step of creating a primary dataset based on raw data, and an extended dataset for analysis step of creating an extended dataset for analysis based on the primary dataset.

[0008] Furthermore, in this respect, although not limited thereto, it is preferable that the step of creating an expanded analysis dataset includes at least one of a stratification step, an imbalance correction step, and a validation step for the primary dataset.

[0009] In addition, in this respect, although not limited thereto, it is preferable that the extended data set creation step deletes personally identifiable data from the primary data set.

[0010] In addition, in this aspect, although not limited thereto, it is preferable that the extended dataset creation step is performed on the primary dataset using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model.

[0011] Furthermore, a method for creating an extended dataset for analysis according to another aspect of the present invention creates an analysis dataset, including an extended data creation step for creating extended data based on raw data, and an extended dataset for analysis step for creating an extended dataset for analysis based on the extended data.

[0012] Furthermore, in this respect, although not limited thereto, it is preferable that the step of creating expanded data for analysis includes at least one of a stratification step, an imbalance correction step, and a validation step for the primary dataset.

[0013] Also, in this respect, although not limited thereto, it is preferable that the augmented data creation step deletes personally identifiable data from the primary data set.

[0014] In addition, in this aspect, although not limited thereto, it is preferable that the augmented data creation step is performed using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model on the primary dataset.

[0015] In addition, a computer program for creating an extended dataset for analysis according to another aspect of the present invention causes a computer to execute a primary dataset creation step of creating a primary dataset based on raw data, and an extended dataset for analysis creation step of creating an extended dataset for analysis based on the primary dataset.

[0016] In addition, a computer program for creating an extended dataset for analysis according to another aspect of the present invention is intended to cause a computer to execute an extended data creation step of creating extended data based on raw data, and an extended dataset for analysis creation step of creating an extended dataset for analysis based on the extended data.

[0017] As described above, the present invention can provide a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis that can perform more accurate analysis even when the number of samples is small.

[0018] FIG. 1 is a diagram showing the processing flow of an analysis dataset creation method according to embodiment 1. FIG. 2 is a diagram showing an image of raw data according to embodiment 1. FIG. 3 is a diagram showing an image of a raw dataset according to embodiment 1. FIG. 4 is a diagram showing an image of an extended analysis dataset according to embodiment 1. FIG. 5 is a diagram showing the processing flow of an analysis dataset creation method according to embodiment 2. FIG. 6 is a diagram showing a primary dataset created in an example. FIG. 7 is a diagram showing an extended analysis dataset created in an example.

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention can be embodied in many different forms and is not limited to the specific examples described in the following embodiments and examples.

[0020] First Embodiment FIG. 1 is a diagram showing a process flow of a method for creating an analysis dataset according to this embodiment (hereinafter referred to as "this method").

[0021] As shown in the figure, this method includes (S1-1) a primary dataset creation step of creating a primary dataset based on raw data, and (S1-2) an analysis extended dataset creation step of creating an analysis extended dataset based on the primary dataset. This method has the advantage that it is possible to create an analysis extended dataset that enables highly accurate analysis even when the number of raw data samples is small.

[0022] This method is executed by an information processing device, i.e., a computer, and is specifically realized by storing a program on a recording medium such as a computer hard disk, and reading and executing this program as needed into a volatile recording medium such as memory. That is, this method is realized by an extended dataset for analysis creation program that causes a computer to execute (S1-1) a primary dataset creation step of creating a primary dataset based on raw data, and (S1-2) an extended dataset for analysis creation step of creating an extended dataset for analysis based on the primary dataset.

[0023] The computer used to execute this program is not limited as long as it has the above functions, but it is preferable that it includes, but is not limited to, components of a typical computer, such as a central processing unit (CPU), a non-volatile recording medium such as a hard disk or flash memory, a volatile recording medium such as memory, a bus connecting these, input devices such as a keyboard or mouse, and a display device such as a monitor.

[0024] The computer may be a laptop or desktop PC, but it may also be a mobile information terminal, specifically a smartphone or tablet terminal, which have become increasingly popular in recent years. However, considering information processing capabilities, it is preferable for the computer to be equipped with a recording medium with sufficient storage capacity and a CPU capable of high-speed processing. A typical mobile information terminal houses and integrates components such as the CPU and display device within a single cover. Furthermore, by placing a sensor on the display device to function as a touch panel, it can also function as an input device, making it very easy to use. Therefore, this aspect can be adopted if this aspect is important. Furthermore, in the case of a mobile information terminal, a program for executing the present method can be recorded and displayed as a so-called app on the mobile information terminal, and the present method can be easily executed by launching the app.

[0025] Here, this method will be explained again. First, this method has (S1-1) a primary data set creation step of creating a primary data set based on raw data.

[0026] Here, "raw data" refers to data before a primary dataset is created, a collection of data acquired from multiple subjects, and raw data that has not been processed for statistical processing or machine learning. In this embodiment, for example, it refers to a collection of data sets containing information on specific items acquired for each of multiple patients. Specifically, raw data preferably contains information on specific items for each of multiple subjects, and is not limited to, for example, data containing information on the subject's name (name data), data containing information on an identification number (identification number data), data containing information on an address (address data), data containing information on sex (sex data), data containing information on the body (for example, weight, height, etc.) (body data (weight data, height data)), data containing information on date of birth, etc. "raw data" includes, but is not limited to, data (date of birth data), data containing information on the name of a disease and information on its occurrence and treatment history (for example, whether or not the person has a particular disease, etc.) (disease data), data containing information on specific components contained in blood (for example, blood glucose, HbA1c, total protein, albumin, AST, ALT, γ-GTP, creatinine, eGFR, uric acid, HDL cholesterol, LDL cholesterol, triglycerides, red blood cells, hemoglobin, white blood cells, platelet count, etc.) (blood data), data containing information on genes (whether or not a specific gene has a mutation, whether or not a gene is expressed, etc.) (gene data). An example of "raw data" is shown in FIG. 2. This diagram shows an example including many sets of data in which gender (SEX), age (AGE), values ​​of specific elements in the blood (blood data, BC1-4), values ​​of expressed genes (gene data, GE1-4), values ​​related to disease (disease data, disease), etc. are recorded in correspondence with an identification number (ID) in the column direction. However, raw data is data containing information obtained as a result of measurement as is, and each of the above data may contain defects, and even clearly abnormal values ​​such as false detections during measurement may be recorded as data as is.

[0027] Furthermore, while the raw data in this example already contains only identification number data, as in the above example, data containing information that can identify an individual (personal information identifiable data), such as name data, address data, and date of birth data, is unnecessary for the analysis process. However, if leaked, this data often causes disadvantage to the individual. Therefore, it is preferable to delete this data in this step, specifically, to delete the personal information identifiable data from the raw data. This personal information identifiable data may be deleted from the raw data, or it may be deleted after the primary dataset is created. However, deleting the personal information identifiable data after performing several processes to create the primary dataset requires additional processing effort, so it is preferable to do so as early as possible.

[0028] In this method, a "primary dataset" refers to a dataset that is created primarily based on raw data and is not expanded. More specifically, it is created based solely on the number of data sets contained in the raw data, and is distinct from the expanded dataset described below, in which the number of sets is increased. Figure 3 shows an example of a primary dataset. This figure shows an example of a dataset created based on the raw data shown in Figure 2 above. Note that the primary dataset shown in this figure is obtained by deleting columns of data that were included in Figure 2 but are deemed unnecessary for the subsequent expansion process, or columns of data that are deemed difficult to use in the subsequent expansion process due to significant value loss. Specifically, the example in Figure 3 is an example in which one blood data item (BC4) from the multiple blood data items present in Figure 2 and one gene data item (GE3) from the multiple gene data items present in Figure 2 have been deleted. In other words, a primary dataset is basically created by deleting and organizing certain data from raw data.

[0029] It is also preferable to perform a data correction process in this step. Here, "correction process" of data refers to a process of complementing missing values ​​in the raw data, or a process of correcting abnormal values ​​that are theoretically unmeasurable to bring the data into a normal range. In this case, the correction process is not limited to, but may involve inputting a representative value (representative value) directly to fill in or convert the data if one is available, or may involve applying dummy variable processing to the raw data and then inputting or converting values ​​that are deemed appropriate as a result.

[0030] It is also preferable to perform a data narrowing process in this step. Here, narrowing refers to the process of deleting unnecessary data, and more specifically, the overall processing of unnecessary measurement data. As described above, raw data contains data related to a large amount of information, but not all of this data is necessary for analysis, and the selection of data may reduce the accuracy of the analysis results. Therefore, narrowing down the data can improve accuracy. This "narrowing process" may simply be a process of deleting item data that is clearly not used by the person performing this method, or it may be possible to narrow down the necessary items by performing statistical processing on this raw data. Note that this data narrowing process may also be performed on the primary dataset. Performing this on the raw dataset has the advantage of reducing the processing burden in subsequent stages, while performing this on the primary dataset has the advantage of potentially improving the accuracy of the analysis results, depending on the conditions.

[0031] The method also includes (S1-2) an extended analysis dataset creation step of creating an extended analysis dataset based on the primary dataset.

[0032] Furthermore, in this method, the term "extended data set for analysis" refers to a data set created based on the primary data set and used for analysis, which includes fictitious data sets (has undergone extension processing). An image of this case is shown in FIG. 4. The example shown in this figure shows an extended data set for analysis created based on the primary data set shown in FIG. 3 above. Specifically, the primary data set in FIG. 3 contains up to 110 data sets (identification numbers 1 to 110), but the extended data set for analysis in FIG. 4 adds fictitious data sets, specifically identification numbers 111 to 150, to the primary data set. In other words, adding fictitious data has the advantage of increasing the number of data sets, enabling more detailed analysis, for example, improving learning accuracy through machine learning processing.

[0033] Although various methods can be used to add this data set, it is preferable to use so-called generative AI rather than random addition. The algorithm is preferably at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model, but is not limited to these.

[0034] Here, "generative adversarial network (GAN)" refers to a program that can learn features from multiple prepared data and generate pseudo-data. Details of GAN are found in, for example, Goodfellow et al.'s publication in 2014 (e.g., Goodfellow et al., "Generative adversarial nets," in Proc. Int. Conf. Neural Inf. Process. Syst., 2014, pp. 2672-2680.), and this can be utilized.

[0035] Here, "Flow-based generative models" refers to a type of generative AI that utilizes variable transformation rules for probability distributions. Details are found, for example, in documents by Ivan Kobyzev et al. (e.g., IEEE Transactions on Pattern Analysis and Machine Intelligence, arXiv:1908.09257v4 [stat.ML] 6 Jun 2020, "Normalizing Flows: An Introduction and Review of Current Methods"), which can be utilized.

[0036] Here, the term "diffusion model" refers to a model that can generate similar data by learning the process of adding noise to data that is the basis of learning and destroying it. For example, details are provided in a document by Jascha Sohl-Dickstein et al. (e.g., arXiv:1503.03585v8 [cs.LG] 18 Nov 2015, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"), and this can be utilized.

[0037] Furthermore, in this step, the number of fictitious data sets to be generated (the number of data sets to be subjected to extension processing) can be adjusted as appropriate depending on the performance of the information processing device used, but if the number of data sets in the primary data set (for example, the number of identification number data sets) is about 30, the number can be 100 times or more, or in some cases about 1,000 times. Note that as long as the performance of the information processing device performing the processing allows, it is possible to increase the number to 10,000 times or more, but if the number is too large, similar data will be created and the effect of improving the accuracy of the analysis will be saturated, so it is important to keep it within an appropriate range.

[0038] In addition, in this method, although not limited thereto, the step of creating an extended data set for analysis preferably includes at least one of a stratification step, an imbalance correction step, and a verification step (S1-2-1) for the primary data set. By performing these steps, it becomes possible to perform the extension process using GAN or the like with higher accuracy.

[0039] Here, the term "stratification step" refers to a step of performing stratification processing on data in a dataset, and the term "stratification processing" refers to a process of grouping data sets in the primary dataset when they have common attributes and comparing them to find features. This stratification processing can be performed using at least one of, but not limited to, correlation analysis, causal analysis, feature engineering, and the like. By performing the stratification step in this manner, the accuracy and reliability of the analytical extended dataset generated by the data extension processing can be improved.

[0040] Furthermore, the "imbalance correction step" here refers to a step of performing imbalance correction processing on the data in the dataset, and the "imbalance correction processing" refers to processing to correct imbalances when there is unbalanced data between data in the primary dataset. For example, if a large gender imbalance occurs even though an equal gender ratio is preferable for analytical processing, processing such as deleting data from the gender group that is heavily biased corresponds to this. This makes it possible to improve the reliability of data processing.

[0041] Furthermore, the term "verification step" refers to a step of performing a verification process on a dataset, and the "verification process" refers to a step of performing a verification process to verify whether the stratification process or imbalance correction process described above is appropriate when the process is performed. This can be confirmed by checking whether the same process has been performed when the stratification process or imbalance correction process is performed again after the process, but is not limited to this.

[0042] As described above, according to this embodiment, it is possible to provide a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis that can generate fictitious data with high accuracy even when the number of samples is small, and perform analysis such as machine learning based on this fictitious data, thereby enabling more accurate analysis. The effects of this will become clear from the examples described below.

[0043] (Embodiment 2) In the above-mentioned embodiment 1, a primary dataset is created from raw data, and an extended dataset for analysis is created based on the primary dataset, but in this embodiment, extended data is created from raw data, and then an extended dataset for analysis is created. A specific description will be given below, but a description of the same configuration and processing as in embodiment 1 will be omitted.

[0044] FIG. 5 is a diagram showing the process flow of the method for creating an extended analysis data set according to this embodiment (hereinafter referred to as "this method").

[0045] As shown in this figure, the method for creating an extended dataset for analysis according to this embodiment (hereinafter referred to as "this method") comprises: (S2-1) an extended data creation step for creating extended data based on raw data; and (S2-2) an extended dataset for analysis creation step for creating an extended dataset for analysis based on the extended data.

[0046] Furthermore, this method can be realized by recording an extended analysis dataset creation program for executing the above steps on a computer recording medium and executing the program, similar to the first embodiment. The explanation is omitted here as it is similar to the first embodiment.

[0047] First, this method includes (S2-1) an extended data creating step of creating extended data based on raw data.

[0048] In this embodiment, the "raw data" is the same as that mentioned in the first embodiment. On the other hand, the "augmented data" is data created based on the raw data, and includes many fictitious data sets in addition to the many data sets originally included in the raw data. By creating the augmented data based on the raw data, it is possible to increase the number of data sets even with a small number of samples, and the same effect as in the first embodiment can be obtained. The augmentation process used in this case is the same as in the first embodiment, and it is also preferable to include a stratification process, etc.

[0049] This method also includes (S2-2) an analysis extended dataset creation step of creating an analysis extended dataset based on the extended data. The analysis extended dataset is the same as in the above-described embodiment 1. Here, the process of creating the analysis extended dataset based on the extended data is the same as in the above-described embodiment 1, but it is possible to employ a process similar to the (S1-1) primary dataset creation step in the above-described embodiment 1 of creating a primary dataset based on raw data.

[0050] As described above, according to this embodiment, similarly to the first embodiment, it is possible to provide a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis that can perform more accurate analysis even when the number of samples is small.

[0051] Here, we actually created an expanded data set for analysis from the raw data and confirmed the high accuracy of the data set. Specifically, we confirmed the effectiveness by using sample data on heart disease that is publicly available on the Internet as raw data.

[0052] The sample data used in this example includes age data (Age) including information on age, sex data (Sex) including information on sex, chest pain type data (ChestPainType) including information on the type of chest pain (TA: typical angina, ATA: atypical angina, NAP: non-anginal pain, ASY: asymptomatic), resting blood pressure data (RestingBP) (mmHg) including information on resting blood pressure, serum cholesterol data (Cholesterol) (mm / dl) including information on serum cholesterol, fasting blood glucose level data (FastingBS) including information on fasting blood glucose level (1: if FastingBS>120 mg / dl, 0: otherwise), and resting electrocardiogram result data (RestingECG) including information on resting electrocardiogram results (Norma The raw data included: 1: normal, ST: abnormal ST-T wave, LVH: tendency to cardiac hypertrophy according to the Estes criteria), maximum heart rate data (MaxHR) including information about maximum heart rate, exercise-induced angina data (ExerciseAngina) including information about exercise-induced angina (Y: yes, N: no), depression tendency data (Oldpeak) including information about depression tendency, cardiac peak slope data during exercise (ST_Slope) including information about the slope of the cardiac peak during exercise (Up: upslope, Flat: flat, Down: downslope), and heart disease data (HeartDisease) including information about the presence or absence of heart disease (1: heart disease, 0: normal). The number of sets of raw data was 511.

[0053] First, this raw data was checked for missing values ​​and obvious abnormal values ​​and corrected to enable analysis, and then, based on this, a narrowing process was performed to narrow down the variables necessary for predicting the presence or absence of heart disease, resulting in a primary data set. This primary data set is shown in Figure 6.

[0054] Next, using the primary dataset created above, we focused on heart disease and expanded the data using generative AI (GAN) so that the sample numbers of healthy people (0 responses) and heart disease people (1 response) were 0:286 and 1:286, respectively, and then adjusted the number of samples to create an extended dataset for analysis. The results are shown in Figure 7. The number of pairs in this extended dataset is 572, with 71 pairs added.

[0055] (Comparative Prediction Accuracy) First, for the primary dataset, prediction was performed using the gradient boosting classification algorithm, with the heart disease data (HeartDisease) as the objective variable and variables other than the heart disease data (HeartDisease) as the explanatory variables. As a result, it was confirmed that the prediction accuracy was 0.7282 and the confusion matrix was as follows: [69,6] [22,6]

[0056] This means that (1) the number of samples predicted as healthy but actually healthy was 69, (2) the number of samples predicted as having heart disease but actually healthy was 6, (3) the number of samples predicted as healthy but actually had heart disease was 22, and (4) the number of samples predicted as having heart disease but actually did have heart disease was 6. This suggests that this predictive model can achieve a certain degree of accuracy if subjects simply answer "healthy" no matter what question they are asked. This means that when using predictive models in the real world, predicting someone as healthy despite actually having heart disease could result in actual harm due to an incorrect judgment. This could also be due to the imbalance in the amount of data between healthy subjects (410 subjects) and heart disease patients (101 subjects), which could lead to bias in the machine learning (i.e., learning too much data from healthy subjects).

[0057] The sample data used this time consisted of 410 healthy people and 101 heart disease patients, resulting in a bias of approximately 4:1. One possible method for correcting this bias would be to reduce the data with a larger number of people (healthy people). However, this would reduce the overall number of samples used to train the predictive model, making it impossible to guarantee sufficient training accuracy.

[0058] (Example Prediction Accuracy) For the expanded data set for analysis created above, prediction was performed using the same method and parameters as above. As a result, the prediction accuracy was 0.8314, which was an improvement. The confusion matrix was as follows: [68, 18] [11, 75]

[0059] This means that (1) the number of samples predicted to be healthy but actually were healthy was 68, (2) the number of samples predicted to have heart disease but actually were healthy was 18, (3) the number of samples predicted to be healthy but actually had heart disease was 11, and (4) the number of samples predicted to have heart disease but actually did have heart disease was 75. In other words, it is inferred that by expanding the data, the imbalance in the number of data points between healthy people and heart disease patients was corrected, resulting in improved learning accuracy.

[0060] As described above, the effects of the present invention were confirmed by this example. Specifically, by performing a generation AI extension process such as GAN on raw data to increase the number of fictitious data sets, it is possible to provide an analysis data set that can improve prediction accuracy beyond that of real data. This particularly has the great possibility that even in the case of sets that cannot be analyzed using real data alone, analysis will be possible by extending the data.

[0061] The present invention has industrial applicability as a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis.

Claims

1. A method for creating an extended dataset for analysis, comprising: a primary dataset creation step of creating a primary dataset based on raw data; and an extended dataset for analysis step of creating an extended dataset for analysis based on the primary dataset.

2. The method for creating an extended dataset for analysis according to claim 1, wherein at least one of the steps of creating the primary dataset and creating the extended dataset for analysis includes at least one of a stratification step, an imbalance correction step, and a validation step.

3. The method for creating an extended data set for analysis according to claim 1, wherein personal information identifiable data is deleted in at least one of the steps of creating the primary data set and creating the extended data set for analysis.

4. The method for creating an extended data set for analysis according to claim 1, wherein the step of creating an extended data set for analysis is performed using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model on the primary data set.

5. A method for creating an extended dataset for analysis, comprising: an extended data creation step of creating extended data based on raw data; and an extended dataset for analysis creation step of creating an extended dataset for analysis based on the extended data.

6. A method for creating an extended data set for analysis according to claim 5, wherein at least one of the extended data creation step and the extended data set for analysis creation step includes at least one of a stratification step, an imbalance correction step, and a verification step.

7. The method for creating an extended data set for analysis according to claim 5, wherein said extended data creating step deletes data that can identify personal information from said raw data.

8. The method for creating an extended data set for analysis according to claim 5, wherein the extended data creation step is performed using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model on the raw data.

9. A computer program for creating an extended dataset for analysis, causing a computer to execute: a primary dataset creation step of creating a primary dataset based on raw data; and an extended dataset for analysis step of creating an extended dataset for analysis based on the primary dataset.

10. A computer program for creating an extended dataset for analysis, causing a computer to execute an extended data creation step of creating extended data based on raw data, and an extended dataset for analysis creation step of creating an extended dataset for analysis based on the extended data.

Images