Analysis expansion dataset creation method and computer program for analysis expansion dataset dataset creation

Abstract

To provide an analysis expansion dataset creation method capable of performing further highly accurate analysis even in the case where the number of samples is small, and a computer program for analysis expansion dataset creation.SOLUTION: According to the present invention, an analysis expansion dataset creation method includes the steps of: creating a primary dataset on the basis of raw data; and creating an analysis expansion dataset on the basis of the primary dataset. A computer program for analysis expansion dataset creation related to another standpoint of the present invention causes a computer to execute the steps of: creating the primary dataset on the basis of the raw data; and creating the analysis expansion dataset.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis. [Background technology]

[0002] With the recent trend toward digitalization, data analysis is becoming increasingly important. Data analysis can provide more useful information, improving business efficiency and convenience in our daily lives. For example, in the medical field, accumulating large amounts of data on a particular case and analyzing it can lead to the establishment of new treatment methods and the development of preventive medicine.

[0003] Regarding data analysis technology, for example, Patent Document 1 below discloses a technology for acquiring medical data and generating a learning dataset from the medical data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-086558 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 above only generates a dataset for analysis after selecting necessary data from the acquired medical data, and has the problem that it is difficult to perform highly accurate analysis when the number of samples of medical data acquired is small to begin with, or when there is an extreme difference in the number of samples.

[0006] In view of the above problems, the present invention aims to provide a method for creating an extended dataset for analysis and a computer program for creating an extended dataset for analysis, which enable more accurate analysis even when the number of samples is small. [Means for solving the problem]

[0007] A method for creating an extended dataset for analysis according to one aspect of the present invention that solves the above problem comprises a primary dataset creation step of creating a primary dataset based on raw data, and an extended dataset for analysis step of creating an extended dataset for analysis based on the primary dataset.

[0008] Furthermore, in this respect, although not limited thereto, it is preferable that the step of creating an expanded data set for analysis includes at least one of a stratification step, an imbalance correction step, and a validation step for the primary data set.

[0009] In addition, in this respect, although not limited thereto, it is preferable that the extended data set creation step deletes personally identifiable data from the primary data set.

[0010] In addition, in this respect, although not limited thereto, it is preferable that the extended dataset creation step is performed on the primary dataset using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model.

[0011] Furthermore, a method for creating an extended dataset for analysis according to another aspect of the present invention creates an analysis dataset, including an extended data creation step for creating extended data based on raw data, and an extended dataset for analysis step for creating an extended dataset for analysis based on the extended data.

[0012] Furthermore, in this respect, although not limited thereto, it is preferable that the step of creating expanded data for analysis includes at least one of a stratification step, an imbalance correction step, and a validation step for the primary dataset.

[0013] Also, in this respect, although not limited thereto, it is preferable that the augmented data creation step deletes personally identifiable data from the primary data set.

[0014] In addition, in this respect, although not limited thereto, it is preferable that the augmented data creation step is performed using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model on the primary dataset.

[0015] In addition, a computer program for creating an extended dataset for analysis according to another aspect of the present invention causes a computer to execute a primary dataset creation step of creating a primary dataset based on raw data, and an extended dataset for analysis creation step of creating an extended dataset for analysis based on the primary dataset.

[0016] In addition, a computer program for creating an extended dataset for analysis according to another aspect of the present invention is intended to cause a computer to execute an extended data creation step of creating extended data based on raw data, and an extended dataset for analysis creation step of creating an extended dataset for analysis based on the extended data. [Effects of the Invention]

[0017] As described above, the present invention can provide a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis that can perform more accurate analysis even when the number of samples is small. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 2 is a diagram showing the processing flow of the analysis dataset creation method according to the first embodiment. [Figure 2] FIG. 2 is an image diagram of raw data according to the first embodiment. [Figure 3] FIG. 2 is an image diagram of a raw data set according to the first embodiment. [Figure 4] FIG. 2 is a conceptual diagram of an extended analysis data set according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing the processing flow of an analysis dataset creation method according to the second embodiment. [Figure 6] FIG. 1 shows a primary data set created in an example. [Figure 7] FIG. 1 shows an expanded data set for analysis created in an example. BEST MODE FOR CARRYING OUT THE INVENTION

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention can be embodied in many different forms and is not limited to the specific examples described in the following embodiments and examples.

[0020] (Embodiment 1) FIG. 1 is a diagram showing the process flow of the analysis dataset creation method according to this embodiment (hereinafter referred to as "this method").

[0021] As shown in the figure, this method comprises (S1-1) a primary dataset creation step of creating a primary dataset based on raw data, and (S1-2) an extended dataset for analysis creation step of creating an extended dataset for analysis based on the primary dataset. This method has the advantage that it is possible to create an extended dataset for analysis that enables highly accurate analysis even when the number of raw data samples is small.

[0022] This method is executed by an information processing device, i.e., a computer, and is specifically realized by storing a program on a recording medium such as a computer hard disk, and reading and executing this program as needed into a volatile recording medium such as memory. That is, this method is realized by an extended dataset for analysis creation program that causes a computer to execute (S1-1) a primary dataset creation step of creating a primary dataset based on raw data, and (S1-2) an extended dataset for analysis creation step of creating an extended dataset for analysis based on the primary dataset.

[0023] The computer used to execute this program is not limited as long as it has the above functions, but it is preferable that it includes, but is not limited to, components of a typical computer, such as a central processing unit (CPU), non-volatile recording media such as a hard disk or flash memory, volatile recording media such as memory, a bus connecting these, input devices such as a keyboard or mouse, and a display device such as a monitor.

[0024] The computer may be a laptop or desktop PC, but it may also be a mobile information terminal (PID), which has become increasingly popular in recent years, specifically a smartphone or tablet. However, considering information processing capabilities, it is preferable for the PID to be equipped with a recording medium with sufficient storage capacity and a CPU capable of high-speed processing. A typical PID integrates components such as the CPU and display device into a single cover. Furthermore, by placing a sensor on the display device to function as a touch panel, it can also function as an input device, making it very easy to use. Therefore, this aspect can be adopted if this aspect is important. Furthermore, in the case of a PID, a program for executing the method can be recorded and displayed on the PID as a so-called app, and the method can be easily executed by launching the app.

[0025] Here, this method will be explained again. First, this method has (S1-1) a primary data set creation step of creating a primary data set based on raw data.

[0026] Here, "raw data" refers to data before a primary data set is created, a collection of data acquired from multiple subjects, and raw data that has not been processed for statistical processing or machine learning. In this embodiment, for example, it refers to a collection of data sets containing information on specific items acquired for each of multiple patients. Specifically, raw data preferably contains information on specific items for each of multiple subjects, and is not limited to, for example, data containing information on the subject's name (name data), data containing information on their identification number (identification number data), data containing information on their address (address data), data containing information on their sex (sex data), data containing information on their body (for example, weight, height, etc.) (body data (weight data, height data)), data containing information on their date of birth, etc. This includes, but is not limited to, data (date of birth data), data (disease data) containing information on the name of a disease and its prevalence and treatment history (e.g., whether or not the patient has a particular disease), data (blood data) containing information on specific components contained in blood (e.g., blood glucose, HbA1c, total protein, albumin, AST, ALT, γ-GTP, creatinine, eGFR, uric acid, HDL cholesterol, LDL cholesterol, triglycerides, red blood cells, hemoglobin, white blood cells, platelet count, etc.), and data (gene data) containing information on genes (e.g., whether or not a specific gene has a mutation, whether or not a gene is expressed, etc.). An example of "raw data" is shown in FIG. 2. This figure shows an example containing many sets of data in which, for each identification number (ID), sex (SEX), age (AGE), values ​​of specific elements in the blood (blood data, BC1-4), values ​​of expressed genes (gene data, GE1-4), values ​​related to a disease (disease data, disease), etc. are recorded in columns. However, raw data is data containing information obtained as a result of measurement as is, and each of the above data may contain missing information. In addition, even if a value is clearly abnormal due to a false detection during measurement, it may be recorded as data as is.

[0027] Furthermore, while the raw data in this example already contains only identification number data, as in the above example, data containing information that can identify an individual (personal information identifiable data), such as name data, address data, and date of birth data, is unnecessary for the analysis process. However, if leaked, this data often causes disadvantage to the individual. Therefore, it is preferable to delete this data in this step, specifically, to delete the personal information identifiable data from the raw data. This personal information identifiable data may be deleted from the raw data, or it may be deleted after the primary dataset is created. However, deleting the personal information identifiable data after performing several processes to create the primary dataset requires additional processing effort, so it is preferable to do so as early as possible.

[0028] In this method, a "primary dataset" refers to a dataset that is created primarily based on raw data and is not expanded. More specifically, it is created based solely on the number of data pairs contained in the raw data, and is distinct from the expanded dataset described below, in which the number of pairs is increased. Figure 3 shows an example of a primary dataset. This figure shows an example of a dataset created based on the raw data shown in Figure 2 above. Note that the primary dataset shown in this figure has been deleted from columns that were included in Figure 2 but are deemed unnecessary for the subsequent expansion process, or columns that have significant missing values ​​and are deemed difficult to use in the subsequent expansion process. Specifically, the example in Figure 3 has been deleted from one of the multiple blood data sets (BC4) and one of the multiple gene data sets (GE3) present in Figure 2. In other words, a primary dataset is basically created by deleting and organizing certain data from raw data.

[0029] It is also preferable to perform a data correction process in this step. Here, "correction process" of data refers to a process of complementing missing values ​​in the raw data, or a process of correcting abnormal values ​​that are theoretically unmeasurable to bring the data into a normal range. In this case, the correction process is not limited to, but may involve inputting a representative value (representative value) directly to fill in or convert the data if one is available, or may involve applying dummy variable processing to the raw data and then inputting or converting values ​​that are deemed appropriate as a result.

[0030] It is also preferable to perform a data narrowing process in this step. Here, narrowing refers to the process of deleting unnecessary data, and more specifically, the overall processing of unnecessary measurement data. As described above, raw data contains data related to a large amount of information, but not all of this data is necessary for analysis, and the selection of data may reduce the accuracy of the analysis results. Therefore, narrowing down the data can improve accuracy. This "narrowing process" may simply be a process of deleting item data that is clearly not used by the person performing this method, or it may be possible to narrow down the necessary items by performing statistical processing on this raw data. Note that this data narrowing process may also be performed on the primary dataset. Performing this on the raw dataset has the advantage of reducing the processing burden in subsequent stages, while performing this on the primary dataset has the advantage of potentially improving the accuracy of the analysis results, depending on the conditions.

[0031] The method also includes (S1-2) an extended analysis dataset creation step of creating an extended analysis dataset based on the primary dataset.

[0032] Furthermore, in this method, the term "extended dataset for analysis" refers to a dataset created based on the primary dataset and used for analysis, which includes fictitious data sets (has undergone extension processing). An image of this case is shown in FIG. 4. The example shown in this figure shows an extended dataset for analysis created based on the primary dataset shown in FIG. 3 above. Specifically, the primary dataset in FIG. 3 contains up to 110 data sets (identification numbers 1 to 110), but the extended dataset for analysis in FIG. 4 adds fictitious data sets, specifically identification numbers 111 to 150, to the primary dataset. In other words, adding fictitious data has the advantage of increasing the number of data sets, enabling more detailed analysis, for example, improving learning accuracy through machine learning processing.

[0033] Although various methods can be used to add this data set, it is preferable to use so-called generative AI rather than random addition. The algorithm is preferably at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model, but is not limited to these.

[0034] Here, "generative adversarial network (GAN)" refers to a program that can learn features from multiple prepared data sets and generate pseudo-data. Details of GANs can be found in, for example, Goodfellow et al.'s 2014 paper (see, for example, "Generative adversarial nets," in Proc. Int. Conf. Neural Inf. Process. Syst., 2014, pp. 2672-2680.), which can be utilized.

[0035] Furthermore, "Flow-based generative models" here refer to a type of generative AI that utilizes the rules for variable transformation of probability distributions. Details are provided, for example, in the literature by Ivan Kobyzev et al. (e.g., IEEE Transactions on Pattern Analysis and Machine Intelligence, arXiv:1908.09257v4 [stat.ML] 6 Jun 2020, "Normalizing Flows: An Introduction and Review of Current Methods"), which can be utilized.

[0036] Here, a "diffusion model" is a model that can generate similar data by learning the process of adding noise to the data that is the basis for learning and destroying it. For example, details are provided in a paper by Jascha Sohl-Dickstein et al. (e.g., arXiv:1503.03585v8 [cs.LG] 18 Nov 2015, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"), and this can be utilized.

[0037] Furthermore, in this step, the number of fictitious data sets to be generated (the number of data sets to be subjected to extension processing) can be adjusted as appropriate depending on the performance of the information processing device used, but if the number of data sets in the primary data set (for example, the number of identification number data sets) is about 30, the number can be 100 times or more, or in some cases about 1,000 times. Note that as long as the performance of the information processing device performing the processing allows, it is possible to increase the number to 10,000 times or more, but if the number is too large, similar data will be created and the effect of improving the accuracy of the analysis will be saturated, so it is important to keep it within an appropriate range.

[0038] In addition, in this method, although not limited thereto, the step of creating an extended data set for analysis preferably includes at least one of a stratification step, an imbalance correction step, and a validation step for the primary data set (S1-2-1). By performing these steps, it becomes possible to perform the extension process using GAN or the like with higher accuracy.

[0039] Here, the term "stratification step" refers to a step of performing stratification processing on data in a dataset, and the term "stratification processing" refers to a process of grouping data sets in the primary dataset when they have common attributes and comparing them to find features. This stratification processing can be performed using at least one of, but not limited to, correlation analysis, causal analysis, feature engineering, and the like. By performing the stratification step in this manner, the accuracy and reliability of the analytical extended dataset generated by the data extension processing can be improved.

[0040] Furthermore, the "imbalance correction step" here refers to a step of performing imbalance correction processing on the data in the dataset, and the "imbalance correction processing" refers to processing to correct imbalances when there is unbalanced data between data in the primary dataset. For example, if a large gender imbalance occurs even though an equal gender ratio is preferable for analytical processing, processing such as deleting data from the gender group that is heavily biased corresponds to this. This makes it possible to improve the reliability of data processing.

[0041] Furthermore, the term "verification step" refers to a step of performing a verification process on a dataset, and the "verification process" refers to a step of performing a verification process to verify whether the stratification process or imbalance correction process described above is appropriate when the process is performed. This can be confirmed by checking whether the same process has been performed when the stratification process or imbalance correction process is performed again after the process, but is not limited to this.

[0042] As described above, according to this embodiment, it is possible to provide a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis that can generate fictitious data with high accuracy even when the number of samples is small, and perform analysis such as machine learning based on this fictitious data, thereby enabling more accurate analysis. The effects of this will become clear from the examples described below.

[0043] (Embodiment 2) In the first embodiment, a primary data set is created from raw data, and an extended data set for analysis is created based on the primary data set, but in this embodiment, extended data is created from raw data, and then an extended data set for analysis is created. A specific description will be given below, but a description of the same configuration and processing as in the first embodiment will be omitted.

[0044] FIG. 5 is a diagram showing the process flow of the method for creating an extended analysis data set according to this embodiment (hereinafter referred to as "this method").

[0045] As shown in this figure, the method for creating an extended dataset for analysis according to this embodiment (hereinafter referred to as "this method") comprises (S2-1) an extended data creation step for creating extended data based on raw data, and (S2-2) an extended dataset for analysis creation step for creating an extended dataset for analysis based on the extended data.

[0046] Furthermore, this method can be realized by recording an extended analysis dataset creation program for executing the above steps on a computer recording medium and executing the program, similar to the first embodiment. The explanation is omitted here as it is similar to the first embodiment.

[0047] First, this method includes (S2-1) an extended data creation step of creating extended data based on raw data.

[0048] In this embodiment, the "raw data" is the same as that mentioned in the first embodiment. On the other hand, the "extended data" is data created based on the raw data, and includes many fictitious data sets in addition to the many data sets originally included in the raw data. By creating extended data based on the raw data, it is possible to increase the number of data sets even with a small number of samples, and the same effect as in the first embodiment can be obtained. The extension process used in this case is the same as in the first embodiment, and it is also preferable to include a stratification process, etc.

[0049] This method also includes (S2-2) an analysis extended dataset creation step of creating an analysis extended dataset based on the extended data. The analysis extended dataset is the same as in the above-described embodiment 1. Here, the process of creating the analysis extended dataset based on the extended data is the same as in the above-described embodiment 1, but it is possible to adopt a process similar to the (S1-1) primary dataset creation step in the above-described embodiment 1 of creating a primary dataset based on raw data.

[0050] As described above, this embodiment, like the first embodiment, can provide a method for creating an extended dataset for analysis and a computer program for creating an extended dataset for analysis that can perform more accurate analysis even when the number of samples is small. [Example]

[0051] Here, we actually created an expanded dataset for analysis from the raw data and confirmed the high accuracy of the dataset. Specifically, we confirmed the effectiveness by using sample data on heart disease that is publicly available on the Internet as raw data.

[0052] The sample data used in this example includes age data (Age) containing information about age, sex data (Sex) containing information about sex, and chest pain type data (ChestPainType) containing information about the type of chest pain (TA: typical angina, ATA: atypical angina, NAP: non-anginal pain, ASY: Asymptomatic), resting blood pressure data (RestingBP) (mmHg) including information on resting blood pressure, serum cholesterol data (Cholesterol) (mm / dl) including information on serum cholesterol, fasting blood glucose level data (FastingBS) including information on fasting blood glucose level (1: if FastingBS>120mg / dl, 0: otherwise), resting electrocardiogram result data (RestingECG) including information on resting electrocardiogram results (Normal: normal, ST: ST-T wave abnormal, LVH: tendency to cardiac hypertrophy according to Estes' criteria), maximum heart rate data (MaxHR) including information on maximum heart rate, exercise-induced angina data (ExerciseAngina) including information on exercise-induced angina (Y: yes, N: no), depression tendency data (Oldpeak) including information on depression tendency, exercise cardiac peak slope data (ST_Slope) including information on the slope of the cardiac peak during exercise (Up: uphill, Flat: flat, Down: The raw data set included 511 sets of heart disease data (HeartDisease) containing information on the presence or absence of heart disease (1: heart disease, 0: normal).

[0053] First, this raw data was checked for missing values ​​and obvious outliers and corrected to make it possible to analyze, and then, based on this, a filtering process was carried out to narrow down the variables necessary to predict the presence or absence of heart disease, resulting in a primary dataset. This primary dataset is shown in Figure 6.

[0054] Next, using the primary dataset created above, we focused on heart disease and expanded it using generative AI (GAN) so that the sample numbers for healthy people (0 responses) and heart disease (1 response) were 0:286 and 1:286, respectively, and then adjusted the number of samples to create an expanded dataset for analysis. The results are shown in Figure 7. The expanded dataset contains 572 pairs, with 71 pairs added.

[0055] (Comparative prediction accuracy) First, for the primary dataset, prediction was performed using the gradient boosting classification algorithm, with the heart disease data (HeartDisease) as the objective variable and variables other than the heart disease data (HeartDisease) as the explanatory variables. As a result, it was confirmed that the prediction accuracy was 0.7282 and the confusion matrix was as shown below. [69,6] [22,6]

[0056] This means that (1) 69 people were predicted as healthy and actually healthy, (2) 6 people were predicted as having heart disease but actually healthy, (3) 22 people were predicted as healthy and actually had heart disease, and (4) 6 people were predicted as having heart disease and actually did have heart disease. This suggests that this predictive model can achieve a certain degree of accuracy if people simply answer "healthy" no matter what question they are asked. This means that in the real world, predicting someone as healthy when they actually have heart disease could result in real harm due to an error in judgment. This could also be due to the imbalance in the number of data points between healthy people (410 people) and heart disease patients (101 people), which could be a cause of bias in the machine learning (training too much data from healthy people).

[0057] The sample data used this time consisted of 410 healthy people and 101 heart disease patients, resulting in a bias of approximately 4:1. One possible way to correct this bias would be to reduce the number of data points with a larger number of people (healthy people). However, this would reduce the overall number of samples used to train the predictive model, making it impossible to guarantee sufficient training accuracy.

[0058] (Example prediction accuracy) In response to this, prediction was performed using the same method and parameters as above on the expanded analysis dataset created above. As a result, the prediction accuracy was 0.8314, which was an improvement. The confusion matrix was as follows. [68,18] [11,75]

[0059] This means that (1) the number of samples predicted to be healthy and actually healthy was 68, (2) the number of samples predicted to have heart disease but actually healthy was 18, (3) the number of samples predicted to be healthy but actually had heart disease was 11, and (4) the number of samples predicted to have heart disease and actually did have heart disease was 75. In other words, it is inferred that by expanding the data, the imbalance in the number of data points between healthy people and heart disease patients was corrected, resulting in improved learning accuracy.

[0060] As described above, the present embodiment has confirmed the effects of the present invention. Specifically, by performing generative AI augmentation processing such as GAN on raw data to increase the number of fictitious data sets, it is possible to provide an analytical dataset that can improve prediction accuracy beyond that of real data. This particularly has the potential to make it possible to analyze sets that cannot be analyzed using real data alone by expanding the dataset. [Industrial Applicability]

[0061] The present invention has industrial applicability as a method for creating an extended data set for analysis and a computer program for creating an extended data set for analysis.

Claims

1. a primary dataset creation step of creating a primary dataset based on the raw data; An extended dataset for analysis creation method comprising an extended dataset for analysis creation step of creating an extended dataset for analysis based on the primary dataset.

2. 2. The method for creating an extended dataset for analysis according to claim 1, wherein at least one of the primary dataset creation step and the extended dataset for analysis creation step includes at least one of a stratification step, an imbalance correction step, and a validation step.

3. 2. The method for creating an extended data set for analysis according to claim 1, wherein data capable of identifying personal information is deleted in at least one of the primary data set creation step and the extended data set for analysis creation step.

4. 2. The method for creating an extended data set for analysis according to claim 1, wherein the step of creating an extended data set for analysis is performed using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model on the primary data set.

5. an augmented data creation step of creating augmented data based on the raw data; An extended data set for analysis creating method comprising an extended data set for analysis creating step of creating an extended data set for analysis based on the extended data.

6. 6. The method for creating an extended data set for analysis according to claim 5, wherein at least one of the extended data creation step and the extended data set for analysis creation step includes at least one of a stratification step, an imbalance correction step, and a verification step.

7. 6. The method for creating an extended data set for analysis according to claim 5, wherein the extended data creating step deletes data that can identify personal information from the raw data.

8. 6. The method for creating an extended data set for analysis according to claim 5, wherein the extended data creation step is performed using at least one of a generative adversarial network (GAN), flow-based generative models, and a diffusion model on the raw data.

9. On the computer, a primary dataset creation step of creating a primary dataset based on the raw data; A computer program for creating an extended data set for analysis, for executing an extended data set for analysis creation step of creating an extended data set for analysis based on the primary data set.

10. On the computer, an augmented data creation step of creating augmented data based on the raw data; A computer program for creating an extended data set for analysis, for executing an extended data set for analysis creation step of creating an extended data set for analysis based on the extended data.

Images