Label application device and label application method

Abstract

To enable the generation of training data for a machine learning model that assigns an audit label to accounting specification data.SOLUTION: A label application device 100 comprises: a clustering unit 111 that performs clustering processing by calculating distances based on specified items within accounting specification data that contain amounts different from audit labels, for labeled accounting specification data being accounting specification data with audit labels and unlabeled accounting specification data being accounting specification data without audit labels; and a label assignment unit 112 that assigns the audit labels of the label-containing accounting specification data belonging to the cluster to which the label-less accounting specification data subjected to the clustering processing belongs, to the label-less accounting specification data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a labeling device and a labeling method used for generating teacher data for generating a label prediction model for assigning an audit label.

Background Art

[0002] In accounting audits, auditors assign labels (audit labels) to individual accounting detail data to classify the accounting detail data and pick out fraudulent data, suspicious data, and related data. However, since the number of accounting detail data is enormous, it is desired to streamline this label assignment work. One method for streamlining is to use machine learning technology. To use machine learning technology, it is necessary to prepare a large amount of teacher data, which requires a great deal of effort and cost.

[0003] As an invention for collecting teacher data, there is an information processing system described in Patent Document 1. This information processing system is an information processing system for classifying verification images by a teacher-based image classifier, and includes an image input means for inputting verification images, a similar image extraction means for extracting images similar to the verification images input by the image input means, a teacher data creation means for creating teacher data by assigning labels to the images extracted by the similar image extraction means, and a learning means for learning the teacher-based image classifier using the teacher data created by the teacher data creation means.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By using the invention described in Patent Document 1, it is possible to collect data that can be a teacher data candidate, but the determination of whether to use the collected data as teacher data for learning is manual. Therefore, still, it takes time and cost to prepare teacher data. The present invention has been made in view of such a background, and an object thereof is to provide a labeling device and a labeling method that enable generation of teacher data for a machine learning model that assigns an audit label to accounting detail data.

Means for Solving the Problems

[0006] To solve the above-described problems, a labeling device according to the present invention calculates a distance based on a predetermined item including an amount different from the audit label among the items of the accounting detail data for labeled accounting detail data that is accounting detail data with an audit label and unlabeled accounting detail data that is the accounting detail data without the audit label, and performs a clustering process; and for the unlabeled accounting detail data after the clustering process, a labeling unit that assigns the audit label of the labeled accounting detail data included in the cluster to which the unlabeled accounting detail data belongs to the unlabeled accounting detail data.

Effects of the Invention

[0007] According to the present invention, it is possible to provide a labeling device and a labeling method that enable generation of teacher data for a machine learning model that assigns an audit label to accounting detail data. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Mode for Carrying Out the Invention

[0009] ≪Outline of Labeling Device≫ The labeling device according to the embodiment for carrying out the present invention will be described below. The labeling device assigns an audit label to unassigned accounting detail data based on the audit label (also simply referred to as a label) assigned by an auditor to the accounting detail data. More specifically, the labeling device performs clustering processing on the labeled accounting detail data, which is the accounting detail data with the audit label assigned by the auditor, and the unlabeled accounting detail data, which is the accounting detail data without the audit label assigned by the auditor. Next, the labeling device assigns the audit label of the labeled accounting detail data included in the cluster to which the unlabeled accounting detail data belongs to the unlabeled accounting detail data.

[0010] Subsequently, the labeling device uses the labeled accounting detail data and the unlabeled accounting detail data to which the audit label has been assigned as teacher data to generate a machine learning model (see the label prediction model 141 described later) that assigns (predicts) an audit label to the accounting detail data.

[0011] By using such a labeling device, an audit label can be assigned to the unlabeled accounting detail data. Subsequently, the labor and cost of preparing the teacher data for the machine learning model that assigns an audit label to the accounting detail data can be reduced. Furthermore, by using the machine learning model, an audit label can be assigned to the accounting detail data, and the labeling work can be labor-saving.

[0012] ≪Configuration of Labeling Device≫ Figure 1 is a functional block diagram of the labeling device 100 according to the present embodiment. The labeling device 100 is a computer and includes a control unit 110, a storage unit 120, and an input / output unit 180. User interface devices such as a display, a keyboard, and a mouse are connected to the input / output unit 180. The input / output unit 180 may include a communication device and be capable of data transmission and reception with other devices. Further, a media drive may be connected to the input / output unit 180, and data exchange using a recording medium may be possible.

[0013] <<Labeling device: Storage unit>> The storage unit 120 is configured to include storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 120 stores an accounting detail database 130, a label prediction model database 140, teacher data 150, model history data 160, and a program 128. The program 128 includes a description of a label prediction model generation process (see Figure 3) described later. Note that various storage contents of the storage unit 120 may be read as needed from those stored in an external storage device such as a cloud server.

[0014] <<Storage unit: Accounting detail database>> Figure 2 is a data configuration diagram of the accounting detail database 130 according to the present embodiment. The accounting detail database 130 is, for example, tabular data. Each row (record) of the accounting detail database 130 is one accounting detail data and includes columns (attributes, items) of identification information, date, summary, subject, amount, acquisition date, audit label, and type.

[0015] The identification information (described as "ID" in Figure 2) is the identification information of the accounting detail data. The date indicates the date on which the transaction shown in the accounting detail data occurred. The summary indicates the content of the transaction. The subject is the accounting subject. The amount is the transaction amount. The acquisition date is the date when the accounting detail data is stored in the accounting detail database 130. The audit label is an audit label assigned by the auditor to the accounting detail data. Examples of audit labels include "personal use" and "assets". Not all accounting detail data has an audit label assigned to it.

[0016] The type is the type when classifying the accounting detail data from the perspective of generating and evaluating the label prediction model 141. The types include "with label", "without label", "labeled teacher", and "labeled for evaluation".

[0017] "With label" indicates that the accounting detail data has an audit label assigned by the auditor. The accounting detail data with a label is also referred to as labeled accounting detail data. "Without label" indicates that the accounting detail data does not have an audit label assigned by the auditor. The accounting detail data without a label is also referred to as unlabeled accounting detail data.

[0018] "Labeled teacher" indicates that the labeled accounting detail data is used as the teacher data for the label prediction model 141. The accounting detail data of "labeled teacher" is randomly selected from the accounting detail data of "with label".

[0019] "Labeled for evaluation" indicates that the labeled accounting detail data is used for evaluating the label prediction model 141. The accounting detail data of "labeled for evaluation" is the accounting detail data of "with label" excluding the accounting detail data of "labeled teacher".

[0020] The accounting detail database 130 may include other attributes (items). For example, the accounting detail database 130 may include the department name (department code), counterparty (counterparty code), and identification information of related vouchers related to the transactions indicated by the accounting detail data.

[0021] ≪Memory unit: Label prediction model database≫ Returning to FIG. 1, the description of the storage unit 120 will be continued. In the label prediction model database 140, a label prediction model 141 with a version number is stored. The label prediction model 141 is a machine learning model in which the explanatory variable is accounting detail data and the objective variable is an audit label. The label prediction model 141 is, for example, a neural network or a random forest, but is not limited thereto. By using the label prediction model 141, the audit label of the accounting detail data can be predicted (assigned).

[0022] ≪Storage Unit: Teacher Data≫ The teacher data 150 is the teacher data (learning data, training data) of the label prediction model 141. The explanatory variable of the teacher data 150 is accounting detail data, and the objective variable is an audit label. The type of the accounting detail data in the accounting detail database 130 corresponding to the accounting detail data in the teacher data 150 is "labeled teacher" or "unlabeled".

[0023] ≪Storage Unit: Model History Data≫ In the model history data 160, as the change history of the label prediction model 141, for example, the version number, the update date and time, and the prediction accuracy of the label prediction model 141 evaluated by the evaluation unit 114 described later are stored.

[0024] ≪Label Assignment Device: Control Unit≫ The control unit 110 includes a CPU (Central Processing Unit) and is provided with a clustering unit 111, a label assignment unit 112, a model generation unit 113, and an evaluation unit 114. The control unit 110 may include a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0025] ≪Control Unit: Clustering Unit≫ The clustering unit 111 randomly selects accounting detail data with the type of "with label" and changes the type to "labeled teacher", and changes the rest to "labeled evaluation". Next, the clustering unit 111 performs clustering processing on the accounting detail data with the type of "labeled teacher" and the accounting detail data with the type of "unlabeled" in the accounting detail database 130. The clustering unit 111 calculates the distance between accounting detail data based on all or some of the attributes including the amount among the attributes of the accounting detail data excluding the identification information, acquisition date, audit label, and type, and performs clustering processing.

[0026] Regarding the distance, there is an example of using the Euclidean distance, standard Euclidean distance, or Manhattan distance, where if the values are the same for each attribute, it is 0, and if they are different, it is 1, but it is not limited to this. For example, the Levenshtein distance may be used for the attribute of the abstract, and the difference in amount may be used as the distance for the attribute of the amount. If there is a tree structure (hierarchy) relationship between the subjects for the attribute of the subject, the number of branches connecting the two subjects may be used as the distance. For example, if the parents are the same, the distance between the two subjects may be 2.

[0027] As the clustering method, for example, K-means or Ward's method is used, but it is not limited to this. Regarding the number of clusters, for example, the elbow method is used to determine it, but it is not limited to this. For example, the clustering unit 111 performs clustering processing using K-means while increasing the number of clusters from 1 to a predetermined number. The clustering unit 111 stops increasing the number of clusters at the timing when the improvement in the total sum of squared errors within the clusters becomes less than or equal to a predetermined value, and ends the clustering processing.

[0028] As described above, the labeling device 100 includes a clustering unit 111 that performs clustering processing by calculating the distance based on a predetermined item including an amount different from the audit label among the items of the accounting detail data for the accounting detail data with an audit label (labeled accounting detail data) and the accounting detail data without an audit label (unlabeled accounting detail data).

[0029] <<Control Unit: Label Assignment Unit>> The label assignment unit 112 assigns an audit label of the accounting detail data of the "teacher with label" belonging to the cluster to which the accounting detail data belongs, to the accounting detail data of which the type is "without label". When there are multiple audit labels of the accounting detail data of the "teacher with label" belonging to the cluster, the label assignment unit 112 may assign the audit label of the accounting detail data of the "teacher with label" that is close to (the distance is the minimum) the accounting detail data of "without label", or may assign the audit label of the accounting detail data of the "teacher with label" that is close to the center of the cluster. After the label assignment unit 112 assigns an audit label to the accounting detail data of "without label", it stores the clustered accounting detail data in the teacher data 150. All the accounting detail data stored in the teacher data is assigned an audit label.

[0030] As described above, the label assignment device 100 includes a label assignment unit 112 that assigns an audit label of the labeled accounting detail data included in the cluster to which the unlabeled accounting detail data (after the clustering process executed by the clustering unit 111) belongs, to the unlabeled accounting detail data. The label assignment unit 112 assigns the label of the accounting detail data of the "teacher with label" with the minimum distance to the unlabeled accounting detail data, among the accounting detail data of the "teacher with label" belonging to the cluster to which the unlabeled accounting detail data belongs. The label assignment unit 112 assigns the label of the accounting detail data of the "teacher with label" with the minimum distance to the center of the cluster, among the accounting detail data of the "teacher with label" belonging to the cluster to which the unlabeled accounting detail data belongs, to the unlabeled accounting detail data.

[0031] <<Control Unit: Model Generation Unit and Evaluation Unit>> The model generation unit 113 generates a label prediction model 141 using the accounting detail data assigned with the audit label stored in the teacher data 150. The evaluation unit 114 evaluates the accuracy of the label prediction model 141 using accounting statement data of the type "labeled evaluation" in the accounting statement database 130. The evaluation unit 114 evaluates the accuracy using, for example, AUC (Area Under the Curve), but other measures may also be used.

[0032] As described above, the labeling device 100 includes a model generation unit 113 that generates a label prediction model 141, which is a machine learning model that uses labeled accounting detail data and labeled, unlabeled accounting detail data as training data 150, unlabeled accounting detail data as an explanatory variable, and audit labels as a target variable. The labeling device 100 also includes an evaluation unit 114 that evaluates the prediction accuracy of the label prediction model 141 using labeled accounting statement data (type: "labeled evaluation").

[0033] <Prediction model generation process> 3 is a flowchart of the prediction model generation process according to this embodiment. At the start of the prediction model generation process, accounting detail data is stored in the accounting detail database 130, and the type has already been assigned either "labeled" or "unlabeled."

[0034] Step S11 is training data generation processing. Details of the training data generation processing will be described later with reference to Fig. 4. As a result of the training data generation processing, training data 150 for generating the label prediction model 141 is generated. In step S12 , the model generation unit 113 generates the label prediction model 141 using the training data 150 and stores it in the label prediction model database 140 .

[0035] In step S13, the evaluation unit 114 evaluates the prediction accuracy of the label prediction model 141 using the accounting detail data in the accounting detail database 130 that is classified as "labeled evaluation." In step S14, the evaluation unit 114 resets the types in the accounting details database 130. More specifically, the evaluation unit 114 changes the types of "labeled teacher" and "labeled evaluation" back to "labeled".

[0036] In step S15, if the accuracy calculated in step S13 is higher than that of the existing latest label prediction model 141 (step S15 → YES), the evaluation unit 114 proceeds with the process to step S16. If the accuracy has not improved (step S15 → NO), the evaluation unit 114 returns the process to step S11. The accuracy of the existing latest label prediction model 141 is stored in the model history data 160.

[0037] In step S16, the model generation unit 113 replaces the label prediction model 141. More specifically, the model generation unit 113 assigns a new version number to the label prediction model 141 generated in step S12, and stores the version number, the current date and time, and the prediction accuracy evaluated in step S13 in the model history data 160.

[0038] ≪Teacher data generation process≫ FIG. 4 is a flowchart of the teacher data generation process (refer to step S11 shown in FIG. 3) according to the present embodiment. In step S21, the clustering unit 111 randomly selects a predetermined number or ratio of accounting detail data with the type of "labeled" in the accounting detail database 130, and sets their types to "labeled teacher", and sets the types of the remaining data to "labeled evaluation".

[0039] In step S22, the clustering unit 111 performs clustering processing on the accounting detail data with the type of "labeled teacher" and the accounting detail data with the type of "unlabeled". In step S23, the labeling unit 112 assigns an audit label to the accounting detail data with the type of "unlabeled".

[0040] In step S24, the labeling unit 112 stores accounting detail data with the type "labeled teacher" and accounting detail data with the type "unlabeled" in the teacher data 150. The accounting detail data in the teacher data 150 is either of the type "labeled teacher" and originally has an audit label assigned by an auditor, or of the type "unlabeled" and has an audit label assigned in step S23.

[0041] ≪Features of the Labeling Device≫ In the teacher data generation process (see Fig. 4), the labeling device 100 assigns an audit label to the unlabeled accounting detail data based on the labeled accounting detail data with an audit label assigned by the auditor. Subsequently, the labeling device 100 uses the labeled accounting detail data and the unlabeled accounting detail data with an audit label assigned as the teacher data 150 to generate the label prediction model 141.

[0042] By using such a labeling device 100, an audit label can be assigned to the unlabeled accounting detail data. Subsequently, the labor and cost required to prepare the teacher data 150 for assigning an audit label to the accounting detail data can be reduced. Furthermore, by using the label prediction model 141, an audit label can be assigned to the accounting detail data, and the labeling operation can be streamlined.

[0043] Each time the clustering unit 111 executes the teacher data generation process (see Fig. 4), it randomly selects the labeled accounting detail data that will become the teacher data 150 (see step S21). Therefore, the teacher data 150 is different each time. For this reason, the label prediction models 141 generated in each of the repetitive processes of steps S11 to S15 in Fig. 3 are different, and it can be expected that a label prediction model 141 with higher accuracy than the label prediction model 141 generated in the previous prediction model generation process (see Fig. 3) will be generated. In particular, when the prediction model generation process is executed at the timing when the labeled accounting detail data is added to the accounting detail database 130, it can be expected that a label prediction model 141 with high accuracy will be generated.

[0044] As described above, when the prediction accuracy of the (label prediction model 141) is less than that of the previously generated label prediction model (refer to NO in step S15 of FIG. 3), the clustering unit 111 performs clustering processing on accounting detail data with a label that is different from the processing target of the previous clustering processing, and accounting detail data without a label. In addition, the labeling unit 112 assigns, to the accounting detail data without a label, the audit label of the accounting detail data with a label included in the cluster to which the accounting detail data without a label belongs. The model generation unit 113 generates a label prediction model 141 using the accounting detail data with a label and the accounting detail data without a label to which a label has been assigned as teacher data.

[0045] ≪Modification Example: Selection of Teacher Data≫ The clustering unit 111 divides the accounting detail data with a "label" into teacher use and evaluation use (refer to step S21 described in FIG. 4). Part or all of the accounting detail data with a label for teacher use and evaluation use may overlap. All of the accounting detail data without a "label" is clustered (refer to step S22) to become teacher data 150. Only a part of the accounting detail data without a "label" may be clustered and an audit label may be assigned to make it teacher data 150.

[0046] ≪Modification Example: Re-clustering Processing≫ When there are multiple audit labels for the accounting detail data with a label belonging to one cluster, the clustering unit 111 may re-cluster the cluster.

[0047] As described above, when there are multiple audit labels for the accounting detail data with a label included in a cluster, the clustering unit 111 performs clustering processing on the accounting detail data belonging to the cluster.

[0048] ≪Modification Example: Label Assignment≫ When there is no labeled accounting detail data in the cluster to which the unlabeled accounting detail data belongs, the labeling unit 112 may assign the audit label of the most recent labeled accounting detail data to the unlabeled accounting detail data. Alternatively, when there is no labeled accounting detail data in the cluster to which the unlabeled accounting detail data belongs, the labeling unit 112 may not include the unlabeled accounting detail data in the teacher data 150.

[0049] <<Modification Example: Label Prediction Device>> By using the label prediction model 141, a label prediction device that assigns an audit label to accounting detail data can be configured. More specifically, the label prediction unit provided in the label prediction device uses the label prediction model 141 generated by the label assignment device 100 to assign an audit label to the accounting detail data.

[0050] As described above, the label prediction device includes a label prediction unit that uses the label prediction model 141 generated by the label assignment device 100 to assign an audit label to unlabeled accounting detail data, which is accounting detail data without an audit label.

[0051] <<Other Modification Examples>> Although some embodiments and modification examples of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various changes such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and their modifications are included in the scope and gist of the invention described in this specification and the like, and are also included in the invention described in the claims and its equivalent scope.

[0052] <<Hardware Configuration>> The labeling apparatus 100 according to the embodiment described above is realized by a computer 900 having a configuration as shown in Fig. 5, for example. Fig. 5 is a hardware configuration diagram showing an example of the computer 900 that realizes the functions of the labeling apparatus 100 according to the embodiment. The computer 900 includes a CPU 901, a ROM 902, a RAM 903, an SSD 904, an input / output interface 905 (referred to as an input / output I / F (Interface) in Fig. 5), a communication interface 906 (referred to as a communication I / F in Fig. 5), and a media interface 907 (referred to as a media I / F in Fig. 5). The computer 900 may include an HDD (Hard Disc Drive) instead of the SSD 904, or may include an HDD in addition to the SSD 904.

[0053] The CPU 901 operates based on a program stored in the ROM 902 or the SSD 904, and performs control by the control unit 110 in Fig. 1. The ROM 902 stores a boot program executed by the CPU 901 when the computer 900 starts up, programs related to the hardware of the computer 900, and the like. The CPU 901 controls an input device 910 such as a mouse or keyboard, and an output device 911 such as a display or printer, via an input / output interface 905. The CPU 901 acquires data from the input device 910 via the input / output interface 905, and outputs generated data to the output device 911.

[0054] The SSD 904 stores programs executed by the CPU 901 and data used by the programs. The communication interface 906 receives data from other devices (not shown) via a communication network and outputs the data to the CPU 901, and also transmits data generated by the CPU 901 to other devices via the communication network. The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 is an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical disk), a magnetic recording medium, a conductor memory tape medium, or a semiconductor memory.

[0055] For example, when the computer 900 functions as the labeling device 100 according to the above-described embodiment, the CPU 901 of the computer 900 realizes the functions of the labeling device 100 by executing the program 128 (see FIG. 1) loaded onto the RAM 903. The CPU 901 reads and executes the program from the recording medium 912. In addition, the CPU 901 may read a program from another device via a communication network, or may install and execute the program 128 from the recording medium 912 to the SSD 904.

Explanation of Signs

[0056] 100 Labeling device 111 Clustering unit 112 Labeling unit 113 Model generation unit 114 Evaluation unit 130 Accounting detail database 140 Label prediction model database 141 Label prediction model 150 Teacher data 160 Model history data

Claims

1. A clustering unit that performs clustering processing by calculating a distance based on a predetermined item including an amount different from the audit label among the items of the accounting detail data for the labeled accounting detail data having an audit label and the unlabeled accounting detail data that is the accounting detail data without the audit label; A label assigning unit that assigns an audit label of the labeled accounting detail data included in the cluster to which the unlabeled accounting detail data belongs to the unlabeled accounting detail data, for the unlabeled accounting detail data after the clustering processing. A label assigning device.

2. The label assigning unit: Among the labeled accounting detail data belonging to the cluster to which the unlabeled accounting detail data belongs, assigns the label of the labeled accounting detail data with the minimum distance to the unlabeled accounting detail data to the unlabeled accounting detail data. The label assigning device according to claim 1.

3. The label assigning unit: Among the labeled accounting detail data belonging to the cluster to which the unlabeled accounting detail data belongs, assigns the label of the labeled accounting detail data with the minimum distance to the center of the cluster to the unlabeled accounting detail data. The label assigning device according to claim 1.

4. The clustering unit: When there are a plurality of audit labels for the labeled accounting detail data included in the cluster, performs clustering processing on the accounting detail data belonging to the cluster. The label assigning device according to claim 1.

5. Further includes a model generation unit that generates a label prediction model, which is a machine learning model using the labeled accounting detail data and the unlabeled accounting detail data with labels as teacher data, the unlabeled accounting detail data as explanatory variables, and the audit label as the target variable. The label assigning device according to claim 1.

6. Further includes an evaluation unit that evaluates the prediction accuracy of the label prediction model using the labeled accounting detail data, and When the prediction accuracy is less than the prediction accuracy of the label prediction model generated last time, The clustering unit performs clustering processing on labeled accounting detail data different from the processing target of the previous clustering processing and the unlabeled accounting detail data. The label - attaching unit attaches, to the accounting detail data without a label, an audit label of the accounting detail data with a label included in the cluster to which the accounting detail data without a label belongs. The model - generating unit generates the label prediction model using the accounting detail data with a label and the accounting detail data without a label to which a label has been attached as teacher data. The label - attaching device according to claim 5.

7. A label prediction unit that uses the label prediction model generated by the label - attaching device according to claim 5 to attach the audit label to the accounting detail data without a label, which is accounting detail data without an attached audit label. Label prediction device.

8. The label - attaching device performs a clustering process by calculating a distance based on a predetermined item including an amount different from the audit label among the items of the accounting detail data for the accounting detail data with an audit label (labeled accounting detail data) and the accounting detail data without the audit label (unlabeled accounting detail data); and for the unlabeled accounting detail data after the clustering process, attaches the audit label of the labeled accounting detail data included in the cluster to which the unlabeled accounting detail data belongs to the unlabeled accounting detail data. Label - attaching method.

Images