Data correction program, device, and method
By converting categorical variables to one-hot format and adjusting values based on confidence rankings, the method addresses bias in machine learning models, improving accuracy and preventing discriminatory outcomes.
Patent Information
- Application Number
- JP2023555957
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-10-27
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-10-27
AI Technical Summary
Existing methods like DIR are ineffective in addressing bias in machine learning models when dealing with categorical variables, leading to incomplete bias removal and decreased model accuracy.
Convert categorical variables into one-hot representations, rank data based on confidence levels, and adjust values to median levels to correct bias, maintaining consistency in the one-hot format.
Effectively removes bias from machine learning models using categorical variables, enhancing model accuracy and preventing discriminatory predictions.
Smart Images

Figure 0007582506000001 
Figure 0007582506000002 
Figure 0007582506000003
Abstract
Description
[Technical field]
[0001] The disclosed technology relates to a data correction program, a data correction device, and a data correction method. [Background technology]
[0002] The values of certain attributes included in the training data used to train a machine learning model may become biased, causing the judgment results of the machine learning model to be discriminatory. For example, assume a case where a machine learning model that predicts the pass / fail outcome from a person's attributes is trained using training data in which the values of attributes such as a person's gender, age, place of origin, etc. are used as explanatory variables, and the pass / fail outcome of the person in employment, a test, etc. is used as the objective variable. In this case, if the training data is a past history in which being female has been treated unfavorably in terms of pass / fail outcomes, the machine learning model trained using that training data will make discriminatory predictions that make unfavorable judgments against women.
[0003] As mentioned above, in order to prevent machine learning models from making discriminatory predictions, there is a technology called DIR (Disparate Impact Remover) that removes factors that bias discriminatory predictions from training data. DIR reduces the correlation between the attribute to be protected from discriminatory treatment ("gender" in the above example) and the values of other attributes, and modifies the values of other attributes to suppress bias in the data. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, Suresh Venkatasubramanian, "Certifying and Removing Disparate Impact", Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2015, Pages 259-268. Summary of the Invention [Problem to be solved by the invention]
[0005] However, DIR is a method for correcting attributes whose variables are numerical values that have superiority or inferiority, such as large and small values, i.e., so-called numerical features. Therefore, there is a problem in that DIR cannot be applied to correct attributes whose variables are limited numbers or categories, i.e., so-called categorical variables.
[0006] In one aspect, the disclosed technology aims to remove factors that bias machine learning models toward discriminatory predictions from data containing categorical variables. [Means for solving the problem]
[0007] In one embodiment, the disclosed technology generates a second plurality of data by changing a first type of categorical variable included in each of the first plurality of data to a one-hot representation. The disclosed technology also uses a first component of the one-hot representation of the first type of categorical variable included in each of the second plurality of data as a response variable, and uses a portion of the second plurality of data other than the one-hot representation of the first type of categorical variable as an explanatory variable. The disclosed technology inputs the portion of the second plurality of data other than the one-hot representation of the first type of categorical variable to a machine learning model generated based on training data including the response variable and explanatory variables. The disclosed technology ranks each of the second plurality of data based on the estimation result of the machine learning model in that case. The disclosed technology then generates a third plurality of data by correcting bias in each attribute of the first type of categorical variable in the second plurality of data based on the result of the ranking process. Effect of the Invention
[0008] One aspect is that it has the effect of removing factors that bias the machine learning model toward discriminatory predictions from data containing categorical variables. [Brief description of the drawings]
[0009] [Figure 1] FIG. 2 is a functional block diagram of a data correction device. [Diagram 2] FIG. 13 is a diagram illustrating an example of a data set. [Diagram 3] FIG. 13 is a diagram showing an example of a transformed data set. [Figure 4] FIG. 1 is a diagram for explaining problems that arise when applying DIR to a data set that includes categorical variables. [Diagram 5] FIG. 11 is a diagram for explaining calculation of a certainty factor. [Figure 6] FIG. 13 is a diagram for explaining setting of ranks. [Figure 7]FIG. 13 is a diagram showing an example of a combination of a value of a component to be corrected and a certainty factor for each rank, and a median value of the rank. [Figure 8] FIG. 13 is a diagram for explaining correction of inconsistency in one-hot representation. [Figure 9] FIG. 13 is a diagram for explaining correction of inconsistency in one-hot representation. [Figure 10] FIG. 1 is a block diagram showing a schematic configuration of a computer functioning as a data correction device. [Figure 11] 13 is a flowchart illustrating an example of a data correction process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.
[0011] As shown in FIG. 1, a data set including a plurality of data is input to the data correction device 10. Then, the data correction device 10 corrects factors (hereinafter simply referred to as "bias") that are included in the input data set and that bias the discriminatory prediction of the machine learning model, and outputs a corrected data set. The data set is an example of the "first plurality of data" of the disclosed technology, and the corrected data set is an example of the "third plurality of data" of the disclosed technology. FIG. 2 shows an example of a data set. In the example of FIG. 2, each row (each record) is one piece of data, and each piece of data is assigned an "ID" that is identification information of the data. In addition, in the example of FIG. 2, each piece of data has values (variables) for each of attributes such as "gender", "age", "place of origin", and "work type" as features. The value of the attribute "age" is a numerical feature. In addition, the values of the attributes "gender", "place of origin", and "work type" are categorical variables.
[0012] Here, we consider a method of applying DIR to remove bias from a dataset containing categorical variables as shown in Figure 2. As mentioned above, DIR modifies numerical features and cannot be applied to categorical variables. Therefore, the following two simple methods can be considered for applying DIR to a dataset containing categorical variables. The first method is to modify only the numerical features without modifying the categorical variables. The second method is to remove the categorical variable columns from the dataset, leaving only the numerical feature columns, and then apply DIR. However, in the first method, latent bias remains in the categorical variables, so it is not possible to completely remove bias from the dataset. In addition, in the second method, information useful for prediction in a machine learning model is missing from the dataset, which may significantly reduce the accuracy of the machine learning model trained using that dataset.
[0013] In addition, in order to apply DIR to a dataset that contains categorical variables, it is possible to simply convert the categorical variables into numerical features by replacing them with numerical values. However, in this case, even though there is no superiority or inferiority between the categories, the converted numerical values will have superiority or inferiority, such as large and small, which will affect the accuracy of the machine learning model being trained.
[0014] Therefore, it is possible to convert the categorical variable into a one-hot representation and apply DIR. A one-hot representation is a representation in which one of multiple components has a value of 1 and the other components have a value of 0. A one-hot representation converts a single column of categorical variables into multiple columns of numerical features. Figure 3 shows an example of converting the categorical variable "work style" in the example in Figure 2 into a one-hot representation. The categorical variable "work style" containing the categories "full-time" and "part-time" is converted into a one-hot representation with multiple components (columns) of "full-time" and "part-time" as shown in the bold frame in Figure 3. In this example, for each data, one of the components "full-time" and "part-time" in the one-hot representation has a value of 1, and the other has a value of 0. Focusing on each component (column) of the one-hot representation, each component is a numerical feature.
[0015] Furthermore, the DIR algorithm generates rankings for each group based on the values of attributes that are protected from discriminatory treatment, such as gender and nationality (hereinafter referred to as "protected attributes"), based on the values of numerical features that are the values of attributes other than the protected attributes. The DIR algorithm then corrects the values of numerical features other than the protected attributes to the median of the values of numerical features that belong to the same rank. When focusing on a numerical feature that is one component of a one-hot representation, the only possible values are 0 or 1, so when the DIR algorithm described above is applied, there are only two ranks: a rank with a value of 0 and a rank with a value of 1. Therefore, the median of numerical features that belong to a rank with a value of 0 is 0, and the median of numerical features that belong to a rank with a value of 1 is 1, and the values do not change before and after the correction, so bias is not removed.
[0016] A more detailed description will be given with reference to FIG. 4. As shown in FIG. 4, in a data set in which the categorical variable "working style" is converted to a one-hot representation, the protected attribute is "gender" and the numerical feature to be corrected is "full-time", which is one of the components of the one-hot representation of the categorical variable "working style". As shown in A in FIG. 4, the information processing device that executes DIR extracts the value of "full-time" for each piece of data included in a group of data in which the value of the protected attribute "gender" is "male" (hereinafter also referred to as a "male group"). Similarly, the information processing device extracts the value of "full-time" for each piece of data included in a group of data in which the value of the attribute "gender" is "female" (hereinafter also referred to as a "female group"). The information processing device obtains a set of values extracted from each group without duplication. In this case, the information processing device obtains a set of (0, 1) for each group.
[0017] Furthermore, the information processing device sets the size of the ranking to, for example, the smaller of the sizes of the sets acquired from each group, i.e., the number of values included in the set. Here, the size of the sets of all groups is 2, so the size of the ranking is also 2. The information processing device sets a rank for each value included in the acquired set for each group, depending on the size of the value. For example, as shown in B in FIG. 4, for the male group, the information processing device sets rank 1 to the value "1" and rank 2 to the value "0". For the female group, the information processing device sets rank 1 to the value "1" and rank 2 to the value "0". Note that if the size of the set is larger than the size of the ranking, the same rank may be set for two or more values.
[0018] As shown in C of FIG. 4, the information processing device collects data for each rank that has the value set for that rank as the "full-time" value from each group, and calculates the median of the "full-time" values of data in the same rank. Then, as shown in D of FIG. 4, the information processing device corrects the "full-time" value of each data to the median calculated for the rank to which the data belongs. In this case, since all "full-time" values belonging to rank 1 are "1", the median is also "1", and since all "full-time" values belonging to rank 2 are "0", the median is also "0". Therefore, with the above method, the values do not change before and after the correction, and bias is not removed.
[0019] Therefore, in this embodiment, when creating a ranking of data for each group based on the value of the component to be corrected, the ranking is created by further ranking data having the same value as the component to be corrected. The functional configuration of the data correction device 10 according to this embodiment will be described in detail below.
[0020] As shown in FIG. 1, the data correction device 10 functionally includes a conversion unit 12, a ranking unit 14, and a correction unit 16.
[0021] The conversion unit 12 converts the categorical variables to be corrected, which are included in each of the data included in the data set, into a one-hot representation, and generates a converted data set. The categorical variables to be corrected are an example of the "first type of categorical variables" of the disclosed technology, and the converted data set is an example of the "second plurality of data" of the disclosed technology. For example, when the categorical variable to be corrected is "working style" in a data set as shown in FIG. 2, the conversion unit 12 converts the value of the categorical variable of the attribute "working style" into a one-hot representation, and generates a converted data set as shown in FIG. 3. Note that each row (each record) of the converted data set shown in FIG. 3 corresponds to one converted data.
[0022] The ranking unit 14 calculates a proxy value for ranking the values of each component of the One-hot expression of the categorical variable to be corrected in the converted data. The proxy value may be, for example, a confidence level indicating the likelihood of the value of each component of the One-hot expression for the part of the converted data other than the One-hot expression of the categorical variable to be corrected. Specifically, as shown in FIG. 5, the ranking unit 14 sets each component of the One-hot expression of the categorical variable to be corrected included in each of the converted data included in the converted data set (the diagonally shaded columns "Component 1" and "Component 2" in the example of FIG. 5) as the objective variable. In addition, the ranking unit 14 sets the parts of the converted data other than the categorical variable to be corrected ("Attribute 1" to "Attribute 6" in the example of FIG. 5) as the explanatory variables. The ranking unit 14 generates a machine learning model based on training data consisting of the above-mentioned objective variable and explanatory variables. The ranking unit 14 may execute machine learning of the machine learning model using all of the training data. In addition, the ranking unit 14 may perform machine learning of the machine learning model using a portion of the training data (e.g., 4 / 5 of the total), and perform cross-validation to validate the generated machine learning model using the remaining portion of the training data (e.g., 1 / 5 of the total).
[0023] The ranking unit 14 inputs the part of the converted data other than the one-hot representation of the categorical variable to be corrected, i.e., the explanatory variable part of the training data, to the generated machine learning model, and obtains the confidence level for each component of each converted data as the prediction result of the machine learning model. The confidence level of each component is a value between 0.0 and 1.0, and the sum of the confidence levels of each component for each converted data is 1.0.
[0024] The ranking unit 14 ranks each piece of converted data based on the confidence level, which is the prediction result of the machine learning model. Specifically, the ranking unit 14 arranges the converted data by the value of the protected attribute in the order of the component values of the one-hot expression of the categorical variable to be corrected. Furthermore, the ranking unit 14 arranges the converted data having the same value of the component to be corrected in the order of the confidence level, and sets a rank. The protected attribute is an example of the "first attribute" of the disclosed technology.
[0025] A more detailed description will be given with reference to FIG. 6. The ranking unit 14 accepts the designation of the protected attribute. For example, when the designation of the protected attribute accepted is the attribute "gender", the ranking unit 14 extracts the converted data in which the value of the component to be corrected is "1" and the converted data in which the value of the component to be corrected is "0" for each of the male group and the female group. The ranking unit 14 arranges the converted data in which the value of the component to be corrected is "1" in descending order of confidence for each group. Similarly, the ranking unit 14 arranges the converted data in which the value of the component to be corrected is "0" in descending order of confidence for each group. The ranking unit 14 concatenates the converted data in which the value of the component to be corrected is "0" in descending order of confidence for each group, after the converted data in which the value of the component to be corrected is "1" in descending order of confidence for each group. As a result, the converted data in which the value of the component to be corrected is "0" in ascending order of confidence for each group is arranged in descending order of confidence.
[0026] The ranking unit 14 obtains a set of combinations of the values of the components to be corrected and the confidence levels for each group without duplication, and sets the size of the set, i.e., the number of combinations included in the set, whichever is smaller, as the size of the ranking. Then, the ranking unit 14 sets ranks for the converted data arranged in order of the values of the components to be corrected and in order of the confidence levels according to the size of the ranking. For example, for two converted data in which one component of a one-hot representation of a categorical variable has the same value "1", the confidence level for that component is 0.998 for one converted data and 0.940 for the other converted data. In this case, the former converted data is set to a higher rank.
[0027] The correction unit 16 generates a corrected data set by correcting the bias of each attribute of the categorical variable to be corrected in the converted data set based on the result of the ranking process by the ranking unit 14. Specifically, the correction unit 16 corrects the value of the component to be corrected in the converted data set having the same rank set by the ranking unit 14 to the median value of the component to be corrected in the converted data set having the same rank. FIG. 7 shows an example of a combination of the value of the component to be corrected and the confidence level for each rank, and the median value of the rank. Even if the value of the component to be corrected is the same, the rank differs depending on the confidence level, so that the value may differ between the male group and the female group even if the rank is the same (in the example of FIG. 7, the rank is 453). In this case, the value of the component to be corrected in the converted data of one of the groups is changed from the original value. That is, it becomes possible to apply DIR to the categorical variable. When the median value is 0.5, the correction unit 16 may set a predetermined value of 1 or 0 as the median value, that is, the corrected value.
[0028] Furthermore, the correction unit 16 corrects the One-hot representation of the categorical variable to be corrected so as to maintain the consistency of the One-hot representation of the categorical variable to be corrected after changing the component value to the median value. Specifically, as shown in FIG. 8, there may be a plurality of One-hot values, i.e., "1", in the One-hot representation of the categorical variable to be corrected. In this case, the correction unit 16 leaves the value of any one of the plurality of "1"s as "1" based on the confidence of each component, and corrects the values of the other components to "0". For example, the correction unit 16 may leave the value of the component with the highest confidence as "1". In the example of FIG. 8, the values of component 1 and component 2 are "1", but since component 1 has a higher confidence than component 2, the correction unit 16 corrects the value of component 2 from "1" to "0" (thick frame in FIG. 8).
[0029] Also, as shown in FIG. 9, there may be cases where the One-hot value, i.e., "1", does not exist in the One-hot representation of the categorical variable to be corrected. In this case, the correction unit 16 corrects the value of any one of the components to "1" based on the confidence of each component. For example, the correction unit 16 may correct the value of the component with the highest confidence to "1". In the example of FIG. 9, since the values of component 1, component 2, and component 3 are all "0", the correction unit 16 corrects the value of component 1 with the highest confidence from "0" to "1" (bold framed part in FIG. 9). The correction unit 16 also performs the above-mentioned correction to maintain the consistency of the One-hot representation, and then outputs the corrected data set.
[0030] The data modification device 10 may be realized by, for example, a computer 40 shown in Fig. 10. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. The computer 40 also includes an input / output device 44 such as an input unit and a display unit, and an R / W (Read / Write) unit 45 that controls reading and writing of data to a non-temporary storage medium 49. The computer 40 also includes a communication I / F (Interface) 46 that is connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to one another via a bus 47.
[0031] The storage unit 43 may be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 43 as a storage medium stores a data correction program 50 for causing the computer 40 to function as the data correction device 10. The data correction program 50 has a conversion process 52, a ranking process 54, and a correction process 56.
[0032] The CPU 41 reads out the data correction program 50 from the storage unit 43, expands it in the memory 42, and sequentially executes the processes of the data correction program 50. The CPU 41 operates as the conversion unit 12 shown in FIG. 1 by executing the conversion process 52. The CPU 41 also operates as the ranking unit 14 shown in FIG. 1 by executing the ranking process 54. The CPU 41 also operates as the correction unit 16 shown in FIG. 1 by executing the correction process 56. In this way, the computer 40 that has executed the data correction program 50 functions as the data correction device 10. The CPU 41 that executes the program is hardware.
[0033] The functions realized by the data correction program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.
[0034] Next, the operation of the data correction device 10 according to the present embodiment will be described. When a data set is input to the data correction device 10 and an instruction to remove the bias is given, the data correction device 10 executes the data correction process shown in Fig. 11. The data correction process is an example of a data correction method of the disclosed technology.
[0035] In step S10, the conversion unit 12 acquires a data set input to the data correction device 10. In addition, the ranking unit 14 accepts designation of protection attributes from a user. Next, in step S12, the conversion unit 12 converts categorical variables to be corrected that are included in the acquired data set into a one-hot representation, and generates a converted data set. The categorical variables to be corrected may be categorical variables designated by the user, or all categorical variables included in the data set may be sequentially set as categorical variables to be corrected.
[0036] Next, in step S14, the ranking unit 14 generates training data in which each component of the One-hot representation of the categorical variable to be corrected included in the transformed data set is used as the objective variable, and attributes other than the categorical variable to be corrected are used as the explanatory variables. Based on the generated training data, the ranking unit 14 generates a machine learning model that estimates a confidence level indicating the likelihood of the value of each component of the One-hot representation for the part other than the One-hot representation of the categorical variable to be corrected.
[0037] Next, in step S16, the ranking unit 14 inputs the explanatory variable portion of the generated training data to the generated machine learning model. Then, the ranking unit 14 obtains the confidence level for each component of the one-hot expression of the categorical variable to be corrected in each transformed data as the estimation result of the machine learning model.
[0038] Next, in step S18, the ranking unit 14 arranges the transformed data by the values of the components of the one-hot representation of the categorical variable to be corrected, for each value of the protection attribute received in step S10. Furthermore, the ranking unit 14 arranges the transformed data having the same values of the components to be corrected in order of the degree of certainty, and sets a rank.
[0039] Next, in step S20, the correction unit 16 corrects the value of the component to be corrected in the converted data to the median value of the component to be corrected in the set of converted data having the same rank set in step S18. Next, in step S22, for converted data in which there is an inconsistency in that the one-hot representation of the categorical variable to be corrected has multiple "1s" or no "1s", the correction unit 16 corrects the value of any one of the components to be "1" based on the confidence of each component. Then, the correction unit 16 outputs the corrected data set, and the data correction process ends.
[0040] As described above, the data correction device according to the present embodiment generates a converted data set in which the categorical variable to be corrected included in the data set is changed to a one-hot representation. The data correction device generates a machine learning model based on training data in which each component of the one-hot representation of the categorical variable to be corrected included in the converted data set is used as a response variable, and attributes other than the categorical variable to be corrected are used as explanatory variables. The data correction device then ranks each of the converted data based on the prediction result of the machine learning model when a part of the converted data set other than the one-hot representation of the categorical variable to be corrected is input. Furthermore, the data correction device generates and outputs a corrected data set by correcting the bias of the values of each component of the categorical variable to be corrected due to the difference in the value of the protection attribute in the converted data set based on the result of the ranking process. This makes it possible to remove factors that bias the discriminatory prediction of the machine learning model from data including the categorical variable. Furthermore, by generating a machine learning model using the corrected data set generated by the data correction device according to the present embodiment as training data, it is possible to suppress discriminatory prediction by the machine learning model.
[0041] In the above embodiment, the data correction program is stored (installed) in the storage unit in advance, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored in a non-transitory storage medium such as a CD-ROM, a DVD-ROM, or a USB memory. [Explanation of symbols]
[0042] 10 Data correction device 12 Conversion section 14 Ranking Section 16 Correction section 40 Computer 41 CPU 42 Memory 43 Storage section 44 I / O Devices 45 R / W section 46 Communication I / F 47 Bus 49 Storage medium 50 Data Corrector
Claims
1. A second plurality of data is generated by converting a first type of categorical variable included in each of the first plurality of data into a one-hot representation; a first component of a one-hot expression of the first type categorical variable included in each of the second plurality of data is used as a response variable, and a machine learning model is generated based on training data in which a portion of the second plurality of data other than the one-hot expression of the first type categorical variable is used as an explanatory variable, and the portion of the second plurality of data other than the one-hot expression of the first type categorical variable is input to the machine learning model, and each of the second plurality of data is ranked based on an inference result of the machine learning model when the portion of the second plurality of data other than the one-hot expression of the first type categorical variable is input; generating a third plurality of data by correcting bias of each attribute of the first type of categorical variable in the second plurality of data based on a result of the ranking process; A data correction program that causes a computer to execute a process.
2. the ranking process includes arranging the data in order of the value of the first component for each value of a first attribute, and arranging data having the same value of the first component in order of the value indicated by the prediction result to set a rank; The data correction program according to claim 1 .
3. the process of correcting the bias includes correcting a value of the first component of data having the same rank set by the value of the first attribute to a median value of values of the first component in a set of data having the same rank. The data correction program according to claim 2 .
4. The process of correcting the bias includes correcting the one-hot representation of the first type categorical variable after the correction so as to maintain consistency. The data correction program according to any one of claims 1 to 3.
5. The process of correcting the one-hot representation so as to maintain the consistency of the one-hot representation includes, when a plurality of one-hot values exist in the one-hot representation, correcting the one-hot representation so as to leave one of the plurality of one-hot values based on the inference result, and, when a one-hot value does not exist in the one-hot representation, changing a value of any component in the one-hot representation to a one-hot value based on the inference result. The data correction program according to claim 4.
6. The prediction result of the machine learning model is a confidence level of the value of the first component for a portion of the second plurality of data other than the one-hot representation of the first type categorical variable. The data correction program according to any one of claims 1 to 5.
7. performing machine learning of the machine learning model using all of the training data, or performing machine learning of the machine learning model using a portion of the training data and validating the generated machine learning model using the remainder of the training data; 7. The data correction program according to claim 1, wherein the data correction program causes the computer to execute a process.
8. A second plurality of data is generated by converting a first type of categorical variable included in each of the first plurality of data into a one-hot representation; a first component of a one-hot expression of the first type categorical variable included in each of the second plurality of data is used as a response variable, and a machine learning model is generated based on training data in which a portion of the second plurality of data other than the one-hot expression of the first type categorical variable is used as an explanatory variable, and the portion of the second plurality of data other than the one-hot expression of the first type categorical variable is input to the machine learning model, and each of the second plurality of data is ranked based on an inference result of the machine learning model when the portion of the second plurality of data other than the one-hot expression of the first type categorical variable is input; generating a third plurality of data by correcting bias of each attribute of the first type of categorical variable in the second plurality of data based on a result of the ranking process; A data correction device comprising a control unit for executing processing.
9. A second plurality of data is generated by converting a first type of categorical variable included in each of the first plurality of data into a one-hot representation; a first component of a one-hot expression of the first type categorical variable included in each of the second plurality of data is used as a response variable, and a machine learning model is generated based on training data in which a portion of the second plurality of data other than the one-hot expression of the first type categorical variable is used as an explanatory variable, and the portion of the second plurality of data other than the one-hot expression of the first type categorical variable is input to the machine learning model, and each of the second plurality of data is ranked based on an inference result of the machine learning model when the portion of the second plurality of data other than the one-hot expression of the first type categorical variable is input; generating a third plurality of data by correcting bias of each attribute of the first type of categorical variable in the second plurality of data based on a result of the ranking process; A data correction method, the processing of which is carried out by a computer.
Citation Information
Patent Citations
Data complement program, data complement method, and data complement device
JP2020154828A
Information processing system and information processing program
JP2021152838A
Information processing apparatus, method, and program
WO2019234802A1
