Learning device, learning method, and learning program

Abstract

To provide a training device, a training method, and a training program with which it is possible to further improve the classification accuracy of a classification model.SOLUTION: Provided is a training device that retrains a classification model that refers to an antecedents extraction table and takes the antecedents extracted from a document, as input, and outputs the category to which the document belongs, as a classification result. The training device includes at least one processor, the processor presenting an acceptance screen that accepts the input of a correction indication to correct errors in the classification result and a correction ground based on information included in the document, updating the antecedents extraction table on the basis of the correction ground when the correction indication and the correction ground have been accepted, and retraining the classification model using the modified correct classification result and the antecedents extraction table after the updating.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, and a learning program. [Background technology]

[0002] Conventionally, as a machine learning model, a classification model that automatically classifies input words, sentences, etc. into categories to which they belong and outputs the classification results has been known. For example, Patent Document 1 describes a classification model that classifies word categories. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-12457 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, one method of retraining when the classification result of a classification model is incorrect is a method in which a user corrects the incorrect classification result and retrains the classification model using the correct classification result after the correction. For example, in the technology described in Patent Document 1, the classification model is retrained using the word classification corrected by the user. However, the technology described in Patent Document 1 may not be sufficient to improve the accuracy of the retrained classification model, and there is room for improvement.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide a learning device, a learning method, and a learning program that can further improve the classification accuracy of a classification model. [Means for solving the problem]

[0006] In order to achieve the above object, a learning device of a first aspect of the present disclosure is a learning device that re-trains a classification model that inputs features extracted from a document by referring to a feature extraction table and outputs the category to which the document belongs as a classification result, and includes at least one processor. The processor presents a reception screen that accepts input of a correction instruction to correct an error in the classification result and a basis for the correction based on information contained in the document. When the correction instruction and the basis for the correction are accepted, the processor updates the feature extraction table based on the basis of the correction basis, and re-trains the classification model using the corrected, correct classification result and the updated feature extraction table.

[0007] The learning device of the second aspect is the learning device of the first aspect, wherein the items of the feature extraction table include at least words to be extracted as features, or words to be extracted as features and weights assigned to each word.

[0008] A third aspect of the learning device is the learning device of the second aspect, wherein the processor updates at least one of the entries in the feature extraction table based on the correction grounds.

[0009] A fourth aspect of the learning device is the learning device of the third aspect, wherein the update is the addition of a word included in the correction basis.

[0010] A fifth aspect of the learning device is the learning device of the third aspect, The feature extraction table includes at least the words to be extracted as features and the weights assigned to each word. The update is an update of the weights assigned to the words included in the revision basis.

[0011] In a sixth aspect of the learning device, in the learning device of the first aspect, the processor displays selectable candidates for the basis for revision extracted from the document on a reception screen, and accepts input of the basis for revision by allowing the user to select a candidate.

[0012] A seventh aspect of the learning device is the learning device of the sixth aspect, wherein the processor extracts at least words from the document as candidates for correction basis.

[0013] In the eighth aspect of the learning device, in the seventh aspect of the learning device, the processor extracts sentences in addition to words as candidates for correction reasons, and displays the extracted words and sentences mixed together on the reception screen.

[0014] A ninth aspect of the learning device is a learning device according to the first aspect, in which the reception screen has an input field in which the user can enter the reason for the correction in free text, and the processor displays the information contained in the document in an identifiable manner from the input free text.

[0015] A tenth aspect of the learning device is the learning device of the ninth aspect, wherein the processor displays information contained in the document in a distinguishable manner by highlighting the information.

[0016] The learning device of an eleventh aspect is the learning device of the first aspect, wherein when the semantic content of the document represents a situation of a machine failure, the categories include the cause of the failure.

[0017] A twelfth aspect of the learning device is the learning device of the first aspect, wherein the reception screen includes a display area for displaying the document and the classification result.

[0018] In the learning device of the thirteenth aspect, in the learning device of the first aspect, the feature extraction table includes images related to the document, and the images are input as features to the classification model in addition to the features extracted from the document.

[0019] A learning device of a fourteenth aspect is the learning device of the first aspect, wherein the document contains symbols, and the items of the feature extraction table contain at least symbols to be extracted as features.

[0020] Furthermore, in order to achieve the above-mentioned object, a learning device of a fifteenth aspect of the present disclosure includes at least one processor, which inputs features extracted from documents by referring to a feature extraction table, displays documents previously input to a classification model that outputs the category to which the document belongs as a classification result, displays the classification results output from the classification model according to the documents previously input, accepts correction instructions to correct errors in the displayed classification results, accepts correction grounds that serve as the basis for the correction instructions, and re-learns using the accepted correction instructions and correction grounds.

[0021] Furthermore, in order to achieve the above object, a training method of a sixteenth aspect of the present disclosure is a training method in which a processor included in a training device that retrains a classification model that inputs features extracted from a document by referencing a feature extraction table and outputs the category to which the document belongs as a classification result presents a reception screen that accepts input of a correction instruction to correct an error in the classification result and a reason for the correction based on information included in the document, and when the correction instruction and the reason for the correction are accepted, updates the feature extraction table based on the reason for the correction, and executes processing to retrain the classification model using the corrected, correct classification result and the updated feature extraction table.

[0022] In order to achieve the above object, a learning method according to a seventeenth aspect of the present disclosure includes: A learning method executed by a processor included in a learning device, The learning method includes the steps of: displaying documents previously input into a classification model that inputs features extracted from documents by referring to a feature extraction table and outputs the category to which the document belongs as a classification result; displaying the classification results output from the classification model according to the documents previously input; accepting correction instructions to correct errors in the displayed classification results; accepting correction grounds that serve as the basis for the correction instructions; and relearning using the accepted correction instructions and correction grounds.

[0023] Furthermore, in order to achieve the above object, the learning program of an eighteenth aspect of the present disclosure is configured to cause a processor of a learning device that re-trains a classification model that receives as input features extracted from a document by referencing a feature extraction table and outputs the category to which the document belongs as a classification result, to present a reception screen that receives input of a correction instruction to correct an error in the classification result and a reason for the correction based on information contained in the document, and when the correction instruction and the reason for the correction are received, to update the feature extraction table based on the reason for the correction, and to execute a process of re-training the classification model using the corrected, correct classification result and the updated feature extraction table. [Effects of the Invention]

[0024] According to the present disclosure, the classification accuracy of a classification model can be further improved. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 2 is a diagram illustrating classification of categories to which documents belong using a classification model according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a learning device. [Figure 3] FIG. 10 is a diagram illustrating an example of a data management DB. [Figure 4] FIG. 10 is a diagram illustrating an example of a feature extraction table. [Figure 5] FIG. 2 is a block diagram showing an example of the functional configuration of a learning device in a learning phase. [Figure 6] 10 is a flowchart illustrating an example of a learning process. [Figure 7] FIG. 2 is a block diagram illustrating an example of the functional configuration of a learning device in an operation phase. [Figure 8] 10 is a flowchart illustrating an example of a classification process. [Figure 9] FIG. 2 is a block diagram showing an example of the functional configuration of a learning device in a re-learning phase. [Figure 10] FIG. 4 is a diagram showing an example of a display state of a reception screen according to the first embodiment. [Figure 11] 5 is a flowchart illustrating an example of a relearning process according to the first embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a display state of a reception screen on which a correct classification result and a reason for correction have been input. [Figure 13] 4 is a diagram for explaining extraction of correction basis words by the UI presenting unit of the first embodiment. FIG. [Figure 14] FIG. 4 is a diagram showing an example of a display state of a reception screen according to the first embodiment. [Figure 15] FIG. 10 is a diagram for explaining updating of a feature extraction table. [Figure 16] FIG. 10 is a diagram illustrating an example of a display state of a reception screen according to the second embodiment. [Figure 17] 10 is a flowchart illustrating an example of a relearning process according to the second embodiment. [Figure 18] FIG. 10 is a diagram illustrating an example of a display state of a reception screen according to the second embodiment. [Figure 19] 10 is a diagram for explaining extraction of correction basis words by a UI presenting unit of the second embodiment. FIG. [Figure 20] FIG. 10 is a diagram illustrating an example of a display state of a reception screen according to the second embodiment. [Figure 21] FIG. 10 is a diagram showing another example of the display state of the reception screen in the second embodiment. [Figure 22] FIG. 11 is a diagram illustrating an example of a display state of a reception screen according to the third embodiment. [Figure 23] 10 is a flowchart illustrating an example of a relearning process according to the third embodiment. [Figure 24] FIG. 11 is a diagram illustrating an example of a display state of a reception screen according to the third embodiment. [Figure 25] 13 is a diagram for explaining extraction of correction basis words by a UI presenting unit according to the third embodiment. FIG. [Figure 26] FIG. 11 is a diagram showing another example of the display state of the reception screen in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the technology of the present disclosure is not limited to the embodiments.

[0027] [First embodiment] First, the classification model learned by the learning device of this embodiment will be described. Fig. 1 shows a diagram for explaining prediction of the classification of a category to which a document 50 belongs by a classification model 37 used in this embodiment (hereinafter simply referred to as "classification of document 50").

[0028] The classification model 37 receives as input features 52 extracted from a document 50 by the feature extraction unit 40 with reference to a feature extraction table 34 (described in detail below) and vectorized according to weighting, and outputs the category to which the document 50 belongs as a classification result 54. In this embodiment, the document 50 is information whose components are at least one of a word and a sentence. For example, the document 50 may contain only one word, or may contain multiple sentences. The term "feature" refers to an inherent property, and in machine learning, it refers to information that serves as a clue for classifying data, i.e., a feature. In this embodiment, feature 52 represents the characteristics of document 50, and an example is the words contained in document 50.

[0029] Fig. 2 is a configuration diagram showing an example of the hardware configuration of a learning device 10 that learns and re-learns the classification model 37. As shown in Fig. 2, the learning device 10 of this embodiment includes a CPU (Central Processing Unit) 20, a memory 21 as a temporary storage area, a non-volatile storage unit 22, a display 23 such as a liquid crystal display, an input unit 24, and a network I / F (Interface) 26. The CPU 20, memory 21, storage unit 22, display 23, input unit 24, and network I / F 26 are connected via a bus 29 such as a system bus and a control bus so that various information can be exchanged between them.

[0030] The input unit 24 is for receiving user operations and is, for example, a touch panel, buttons, a keyboard, a mouse, etc. It is also possible to adopt a touch panel display that integrates the display 23 and the input unit 24. The network I / F 26 performs wired or wireless communication with an external device (not shown).

[0031] The storage unit 22 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage unit 22 stores a learning program 30A, a classification program 30B, and a relearning program 30C. When training the classification model 37, the CPU 20 reads the learning program 30A from the storage unit 22, loads it into the memory 21, and executes the loaded learning program 30A. When using the classification model 37 to classify documents 50, the CPU 20 reads the classification program 30B from the storage unit 22, loads it into the memory 21, and executes the loaded classification program 30B. When relearning the classification model 37, the CPU 20 reads the relearning program 30C from the storage unit 22, loads it into the memory 21, and executes the loaded relearning program 30C. The CPU 20 is an example of a processor of the present disclosure, and the relearning program 30C is an example of a learning program of the present disclosure.

[0032] The storage unit 22 also stores a data management DB 32. The data management DB 32 is used to manage documents 50. The data management DB 32 stores a plurality of learning data sets each combining a document 50 and a correct classification result 54. The data management DB 32 also stores a combination of a document 50 and a classification result 54 obtained by actually classifying the document 50 using the classification model 37. FIG. 3 shows an example of the data management DB 32 according to this embodiment. As shown in FIG. 3, the document 50 and the classification result 54 are managed in association with an "ID (Identification)" for management purposes. As an example, the semantic content of the document 50 according to this embodiment represents the status of a machine failure, and the category includes a cause classification for classifying the cause of the machine failure. As shown in FIG. 3, the document 50 includes fields for "Title," "Description," "Response Result," "Model," and "OS (Operating System)." The "Title" is a title assigned to the document 50. The "Description" is an explanation of the details of the failure that occurred. The "Response Result" is the result of the maintenance worker's response to the failure. "Model name" is the model name of the device in which the failure occurred. "OS" is the name of the OS that was in use when the failure occurred. As shown in Figure 3, the classification result 54 also includes a "cause classification." The "cause classification" is a category according to the cause of the failure.

[0033] The storage unit 22 also stores a feature extraction table 34. The feature extraction table 34 stores extraction rules for extracting features from a document 50. The feature extraction table 34 is a table referenced when the feature extraction unit 40 extracts features 52 from a document 50. The holding unit 40 of this embodiment is a table that records rules for extracting features 52 from a document 50. FIG. 4 shows an example of the feature extraction table 34 of this embodiment. As shown in FIG. 4, the items in the feature extraction table 34 include the "field" of the document 50 from which the feature 52 is extracted, the "type" of that field, a specific "feature," and the "weight" of that feature. In the example shown in FIG. 4, the "fields" shown are "description," "response result," and "model name." Of these, "description" and "response result" are of the "text" type. When the type is "text," for example, morphemes obtained by morphologically analyzing text using a known morphological analyzer such as JUMAN may be used. An n-gram (a sequence of n morphemes) may be used as a "feature." Furthermore, the "weight" may be weighted using a known scale such as TF-IDF (Term Frequency - Inverse Document Frequency). Meanwhile, the type of "model name" is a "nominal scale." When the type is a "nominal scale," the presence or absence of a possible value is used as a feature. For example, in the case of "Desket" listed as a "model name" in the data management DB 32 shown in FIG. 3, a value indicating whether or not it is a Desket is stored as a feature in the feature extraction table 34. Note that a "nominal" is a "surface name," and a "scale" is a standard for evaluating something. In other words, a "nominal scale" is a scale that does not have any meaning in terms of numerical magnitude or relative magnitude.

[0034] The storage unit 22 also stores a classification model DB (DataBase) 36. The classification model DB 36 is a database in which the above-described classification model 37 is stored.

[0035] Next, the operation of the learning device 10 of this embodiment will be described.

[0036] (Learning phase) First, the learning phase in which the classification model 37 of this embodiment is trained will be described. FIG. 5 shows an example of the functional configuration of the learning device 10 in the learning phase. When training the classification model 37, the CPU 20 of the learning device 10 executes the learning program 30A, causing the CPU 20 to function as a feature extraction unit 40 and a model training unit 42. In this embodiment, generating the classification model 37 by training a machine learning model using training data is referred to as "training the classification model 37." In this embodiment, "training" the classification model 37 refers to training the classification model 37 using training data that combines documents 50 and correct classification results 54.

[0037] The feature extraction unit 40 acquires, as training data, combinations of documents 50 and correct classification results 54 from the data management DB 32. In the present embodiment, as an example, a predetermined ID is associated with the training data. For example, if the IDs associated with the training data are "00001" to "09999," the feature extraction unit 40 acquires, as training data, combinations of documents 50 and correct classification results 54 associated with IDs in the range of "00001" to "09999" from the data management DB 32. For example, in the example of the data management DB 32 shown in FIG. 3, the feature extraction unit 40 acquires, as training data, combinations of documents 50 and correct classification results 54 with an ID of "01234." The feature extraction unit 40 extracts elements specified in the feature extraction table 34 from the acquired documents 50 as features 52. The feature extraction unit 40 vectorizes the extracted features 52 using weights assigned to the features 52. The feature extraction unit 40 outputs the vectorized features 52 and the correct classification result 54 acquired from the data management DB 32 to the model learning unit 42.

[0038] The model training unit 42 trains the classification model 37 based on the features 52 of the documents 50, which are training data input from the feature extraction unit 40, and the correct classification result 54. The model training unit 42 of this embodiment receives the vectorized features 52 as input and trains the classification model 37 so that the output is the correct classification result 54. Note that any machine learning method may be used as long as it can perform classification, such as a well-known support vector machine or random forest. The model training unit 42 stores the trained classification model 37 in the classification model DB 36.

[0039] When the learning device 10 learns the classification model 37, the CPU 20 executes the learning program 30A, thereby performing the learning process shown in Fig. 6. The learning process is performed, for example, at a predetermined timing for performing learning or when a user issues an instruction to start the learning process via the input unit 24.

[0040] 6, the feature extraction unit 40, as described above, acquires from the data management DB 32 a combination of a document 50 and a correct classification result 54, which constitutes learning data. In the next step, S102, the feature extraction unit 40, as described above, references the feature extraction table 34, extracts features 52 from the acquired document 50, and vectorizes them according to the assigned weights. The feature extraction unit 40 outputs the vectorized features 52 and the acquired correct classification result 54 to the model training unit 42.

[0041] In the next step S104, the model learning unit 42 learns the classification model 37 based on the vectorized features 52 and the correct classification result, as described above, and stores the learned classification model 37 in the classification model DB 36.

[0042] In the next step S106, the feature extraction unit 40 determines whether or not to end the learning process. If a predetermined learning process end condition is met, the feature extraction unit 40 ends the learning process. As an example, the feature extraction unit 40 of this embodiment employs the predetermined learning process end condition as being when the classification model 37 has been trained using a predetermined amount of training data, specifically, when the processes of steps S100 to S104 have been performed a predetermined number of times. Until the predetermined learning process end condition is met, the determination of step S106 is negative, and the process returns to step S100, and the processes of steps S100 to S104 are repeated. On the other hand, if the predetermined learning process end condition is met, the determination of step S106 is positive. If the determination of step S106 is positive, the learning process shown in FIG. 6 ends.

[0043] In this way, in the learning phase, the learning device 10 executes the learning process shown in FIG. 6, and the learned classification model 37 is stored in the classification model DB 36.

[0044] (Operational phase) Next, we will explain the operation phase in which documents 50 are classified using the classification model 37 of this embodiment. Figure 7 shows an example of the functional configuration of the learning device 10 in the operation phase. When the classification model 37 is operated, the CPU 20 of the learning device 10 executes the classification program 30B, causing the CPU 20 to function as a feature extraction unit 40 and a classification prediction unit 44.

[0045] The feature extraction unit 40 acquires the document 50 to be actually classified from the data management DB 32. In this embodiment, as an example, the document 50 to be classified using the classification model 37 is also stored in the data management DB 32. Of the multiple documents 50 stored in the data management DB 32, the document 50 to be classified using the classification model 37 (hereinafter referred to as the "document 50 to be classified") has a blank field in the corresponding classification result 54. Therefore, as an example, the feature extraction unit 40 in this embodiment acquires, from the multiple documents 50 stored in the data management DB 32, a document 50 for which the blank field in the classification result 54 is located. The feature extraction unit 40 references the feature extraction table 34 and extracts elements specified in the feature extraction table 34 from the acquired document 50 as features 52. The feature extraction unit 40 vectorizes the extracted features 52 using weights assigned to the features 52. The feature extraction unit 40 outputs the vectorized features 52 to the category prediction unit 44.

[0046] The classification prediction unit 44 inputs the vectorized features 52 received from the feature extraction unit 40 into the classification model 37 stored in the classification model DB 36, and obtains the classification result 54 output from the classification model 37. As described above, if the semantic content of the document 50 describes the state of a machine failure, the classification model 37 outputs the cause of the machine failure as the classification result 54. The classification prediction unit 44 stores the obtained classification result 54 in association with the document 50 to be classified that is stored in the data management DB 32. As described above, the column for the classification result 54 corresponding to the document 50 to be classified that is stored in the data management DB 32 is blank, and therefore the classification result 54 output from the classification model 37 is recorded in that blank column.

[0047] When the learning device 10 classifies documents 50 using the classification model 37, the CPU 20 executes the classification program 30B, thereby performing the classification process shown in Fig. 8. The classification process is performed, for example, when a user issues an instruction to start the process via the input unit 24, or when a predetermined number or more of documents 50 to be classified have been accumulated in the data management DB 32.

[0048] 8, the feature extraction unit 40, as described above, acquires one document 50 to be classified from the data management DB 32. In the next step S202, as described above, the feature extraction unit 40 references the feature extraction table 34, extracts features 52 from the acquired document 50 to be classified, and vectorizes them according to the assigned weights. The feature extraction unit 40 outputs the vectorized features 52 to the category prediction unit 44.

[0049] In the next step S204, the classification prediction unit 44, as described above, acquires the classification result 54 output from the classification model 37. In the next step S206, the classification prediction unit 44, as described above, stores the acquired classification result 54 in the data management DB 32 in association with the document to be classified.

[0050] In the next step S208, the feature extraction unit 40 determines whether to end the classification process. The feature extraction unit 40 ends the classification process when a predetermined classification process end condition is met. As an example, the feature extraction unit 40 of this embodiment uses, as the classification process end condition, a condition where there are no documents 50 stored in the data management DB 32 for which the classification result 54 field is blank, in other words, where all documents 50 stored in the data management DB 32 are associated with and stored as classification results 54. Until the predetermined classification process end condition is met, the determination in step S208 is negative, and the process returns to step S200, where the next document 50 is obtained and the processes in steps S204 and S206 are repeated. On the other hand, if the predetermined classification process end condition is met, the determination in step S208 is positive. If the determination in step S208 is positive, the classification process shown in FIG. 8 ends. 8, the classification process is described as being performed by sequentially retrieving the documents 50 to be classified one by one from the data management DB 32 and sequentially retrieving the classification results 54, but this is not limiting. For example, in step S200 of the classification process, all documents 50 to be classified that are stored in the data management DB 32 may be retrieved. In this case, the process of step S208 in the classification process described above is not required, and the classification process may be terminated upon completion of step S206.

[0051] In this manner, in the operational phase, the learning device 10 executes the classification process shown in FIG. 8 to classify the document 50 using the classification model 37, and the classification result 54 is associated with the document 50 to be classified and stored in the data management DB 32.

[0052] (Relearning phase) Next, the re-learning phase, in which the classification model 37 of this embodiment re-learns the document 50, will be described. As described above, the classification result 54 obtained by classifying the document 50 using the classification model 37 in the operation phase may be incorrect. When the classification result 54 is incorrect in this way, the learning device 10 of this embodiment accepts the correct classification result 54 and the basis for correction to the correct classification result 54 input by the user, and updates the feature extraction table 34 based on the accepted basis for correction. Furthermore, the learning device 10 refers to the updated feature extraction table 34 and re-learns the classification model 37 using the correct classification result 54.

[0053] 9 shows an example of the functional configuration of the learning device 10 in the relearning phase. When relearning the classification model 37, the CPU 20 of the learning device 10 executes the relearning program 30C, causing the CPU 20 to function as a feature extraction unit 40, a model learning unit 42, a UI (User Interface) presentation unit 46, and a feature extraction table update unit 48.

[0054] The UI presentation unit 46 presents, as a UI, a reception screen 60 for receiving a correction instruction, which is a correct classification result for correcting an error in the classification result 54, and the reason for the correction. Note that the reception screen 60 presented by the UI presentation unit 46 of this embodiment also receives a correction instruction as to whether or not to correct the classification result 54 output from the classification model 37.

[0055] FIG. 10 illustrates an example of a reception screen 60 presented by the UI presentation unit 46. FIG. 10 illustrates an example of the initial state of the reception screen 60 presented when correcting the classification result 54 of a document 50 with the ID "12478" stored in the data management DB 32 illustrated in FIG. 3. The reception screen 60 illustrated in FIG. 10 has an area in which the document 50 to be classified is displayed. The document 50 with the ID "12478" includes a title field of "Device Freeze," a description field of "The user reported that Windows (registered trademark) has started to freeze frequently," a response result field of "HDD replacement performed," a model name field of "Desktop," and an OS field of "Windows 10 (registered trademark)" (see FIG. 3). Therefore, the reception screen 60 illustrated in FIG. 10 displays the document 50 including these fields.

[0056] 10, the reception screen 60 includes a classification result field 64, a corrected classification result field 66, a correction basis field 68, and a correction basis word field 70. The classification result field 64 displays the classification result 54 obtained by acquiring the document 50 from the classification model 37 and stored in the data management DB 32. In the example shown in FIG. 10, the cause category "S / W (Software) defect", which is the classification result 54 associated with the document 50 having the ID "12478", is displayed in the classification result field 64 on the reception screen 60. In the initial state, the corrected classification result field 66, the correction basis field 68, and the correction basis word field 70 on the reception screen 60 are blank.

[0057] 10, the reception screen 60 further includes a correction instruction button 62. The correction instruction button 62 includes an execution instruction button 62A for instructing that the classification result 54 displayed in the classification result field 64 be corrected, and a non-execution instruction button 62B for instructing not to correct the classification result 54 displayed in the classification result field 64.

[0058] The UI presentation unit 46 accepts the correct classification result input by the user in the corrected classification result field 66 on the reception screen 60. The UI presentation unit 46 rewrites the classification result 54 associated with the document 50 that is the target of correction, among the documents 50 stored in the data management DB 32, with the correct classification result accepted on the reception screen 60.

[0059] The UI presentation unit 46 also accepts a correction basis input by the user in a correction basis field 68 of the reception screen 60. The UI presentation unit 46 also accepts a word serving as a basis for correction input by the user in a correction basis word field 70 of the reception screen 60. The UI presentation unit 46 outputs the accepted word serving as a basis for correction to the feature extraction table update unit 48.

[0060] The feature extraction table update unit 48 updates the feature extraction table 34 based on the correction basis received by the UI presentation unit 46. The feature extraction table update unit 48 updates the feature extraction table 34 by adding words as features 52, adding word combinations as features 52, and updating the weights assigned to the features 52.

[0061] The feature extraction unit 40 acquires re-learning data from the data management DB 32. The re-learning data is a combination of a document 50 whose classification result 54 has been corrected using the reception screen 60 presented by the UI presentation unit 46 and the classification result 54 associated with that document 50. In this case, the classification result 54 included in the re-learning data is a correct classification result due to the user's correction. The feature extraction unit 40 references the feature extraction table 34 updated by the feature extraction table update unit 48 to extract features 52 from the acquired document 50. The feature extraction unit 40 vectorizes the extracted features 52 using weights assigned to the extracted features 52. The feature extraction unit 40 outputs the vectorized features 52 and the correct classification result 54 acquired from the data management DB 32 to the model training unit 42.

[0062] The model training unit 42 retrains the classification model 37 based on the features 52 of the documents 50, which are the retraining data input from the feature extraction unit 40, and the correct classification results 54. The model training unit 42 of this embodiment inputs the vectorized features 52 and retrains the classification model 37 so that the output is the correct classification result 54. The retraining method can be the same as the method used by the model training unit 42 to train the classification model 37 in the learning phase described above. The model training unit 42 stores the retrained classification model 37 in the classification model DB 36.

[0063] When the learning device 10 learns the classification model 37, the CPU 20 executes the re-learning program 30C, thereby performing the re-learning process shown in Fig. 11. The re-learning process is performed, for example, when a user issues an instruction to start the process via the input unit 24. The re-learning process shown in Fig. 11 is performed for each document 50.

[0064] 11, the UI presentation unit 46 acquires one document 50 and the classification result 54 associated with the document 50 from the data management DB 32. Specifically, one combination of a document 50 that has actually been classified by the classification model 37 and has not been used as re-learning data, from among the multiple documents 50 stored in the data management DB 32, and the classification result 54 associated with the document 50 is acquired. Here, as in the above, an example will be described in which the UI presentation unit 46 acquires a combination of a document 50 that has an ID of "12478" and is stored in the data management DB 32 shown in FIG. 3.

[0065] In the next step S302, the UI presenting unit 46 displays the reception screen 60 on the display 23. In this step, the reception screen 60 displayed on the display 23 is in the display state shown in FIG.

[0066] The user refers to the document 50 displayed on the reception screen 60 and determines whether or not to correct the classification result displayed in the classification result field 64. If no correction is to be made, the user uses the input unit 24 to select the no-execution instruction button 62B of the correction instruction button 62. If the UI presentation unit 46 receives that the no-execution instruction button 62B has been selected, the determination in the next step S304 is affirmative, and the re-learning process shown in FIG. 11 is terminated.

[0067] On the other hand, if the UI presentation unit 46 has not received the designation of the non-execution instruction button 62B, the determination at step S304 is negative, and the process proceeds to step S306. To correct the classification result 54 displayed in the classification result field 64, the user inputs a correct classification result into the corrected classification result field 66 using the input unit 24. In one embodiment, the user selects a correct classification result from a pull-down menu 66P in the corrected classification result field 66. Specifically, when the user operates the pull-down menu 66P using the input unit 24, candidate classification results are displayed. The user selects a correct classification result from the displayed candidate classification results using the input unit 24. The user also inputs a field that serves as the basis for correction into the correction basis field 68 using the input unit 24. In one embodiment, the user selects a field that serves as the basis for correction from the pull-down menu 68P in the correction basis field 68. Specifically, when the user operates the pull-down menu 68P using the input unit 24, candidate fields of the document 50 that serve as the basis for correction are displayed. The user uses the input unit 24 to select a field to be used as the basis for correction from among the displayed fields that are candidates for the basis for correction.

[0068] 12 shows, as an example, a state in which "H / W (Hardware) failure" is entered as the correct classification result in the post-correction classification result column 66, "response result" is entered as the field for the correction basis in the first correction basis column 68, and "OS" is entered as the field for the correction basis in the second correction basis column 68. Note that, a case will be described in which the reception screen 60 of this embodiment is configured to be able to accept two fields for the correction basis by displaying two correction basis columns 68, but the number of fields accepted by the reception screen 60 as the correction basis is not limited to the number exemplified in this embodiment.

[0069] Therefore, in step S306, the UI presentation unit 46 determines whether or not the correct classification result entered in the corrected classification result field 66 and the basis for correction entered in the basis for correction field 68 have been accepted. If the correct classification result entered in the corrected classification result field 66 and the basis for correction entered in the basis for correction field 68 have not been accepted, the determination in step S306 is negative, and the process returns to step S304. On the other hand, if the correct classification result entered in the corrected classification result field 66 and the basis for correction entered in the basis for correction field 68 have been accepted, the determination in step S306 is positive, and the process proceeds to step S308. In the example shown in FIG. 12 , the UI presentation unit 46 accepts "H / W failure" as the correct classification result, and accepts "response result" and "OS" as the fields for the basis for correction.

[0070] In the next step S308, the UI presentation unit 46 extracts candidates for correction basis words from the sentences or words of the correction basis and displays them in the correction basis word field 70. Note that the processing of step S308 and step S310, which will be described later, is executed when the type of the field of the received correction basis is text, and is not executed when the type of the field of the received correction basis is not text. Therefore, in the example shown in Fig. 12, the UI presentation unit 46 executes the processing of this step only for the "correspondence result" of which type is text, among the fields of the received correction basis, to extract candidates for correction basis words.

[0071] As shown in FIG. 13, the UI presentation unit 46 performs morphological analysis on the sentence "Replacement of HDD carried out," which is text in the "Response result" field accepted as the basis for correction, and generates a morpheme list in which morphemes and parts of speech are paired. In the example shown in FIG. 13, the UI presentation unit 46 generates a morpheme list including "HDD: noun," "no: particle," "exchange: noun," "o: particle," and "implementation: noun." Also, as shown in FIG. 13, the UI presentation unit 46 extracts, from the generated morpheme list, morphemes excluding particles, general words, and the like, as candidates for correction basis words. In the example shown in FIG. 13, "HDD," "replacement," and "implementation" are extracted as candidates for correction basis words.

[0072] As shown in FIG. 14 , the UI presentation unit 46 displays the extracted correction basis word candidates with checkboxes in the correction basis word field 70, thereby displaying the correction basis word candidates extracted from the document 50 in a selectable manner. The user selects the correction basis words that serve as the basis for correction by checking the checkboxes in the correction basis word field 70 using the input unit 24. The example shown in FIG. 14 illustrates a case in which the user has selected "HDD" and "replace." The number of correction basis words to be selected is not particularly limited, and may be one or more, or may be all of the correction basis words displayed in the correction basis word field 70. The UI presentation unit 46 accepts, as correction basis words, the correction basis word candidates with checked checkboxes among the correction basis word candidates displayed in the correction basis word field 70.

[0073] Therefore, in the next step S310, the UI presentation unit 46 determines whether or not the selection of a basis word for correction has been accepted. Until the UI presentation unit 46 accepts the selection of a basis word for correction, the determination in step S310 is negative. On the other hand, if the UI presentation unit 46 accepts the selection of a basis word for correction, the determination in step S310 is positive, and the process proceeds to step S312.

[0074] After inputting the correct classification result and selecting the basis for correction and the word for correction, the user instructs the implementation of correction of the classification result 54 by specifying the implementation instruction button 62A using the input unit 24. When the implementation instruction button 62A is specified, the UI presentation unit 46 accepts the implementation of correction of the classification result 54.

[0075] Until the UI presentation unit 46 accepts a request to make corrections, the determination in step S312 is negative, the process returns to step S306, and the processing in steps S306 to S310 is repeated. On the other hand, if the UI presentation unit 46 accepts a request to make corrections, the determination in step S312 is positive, and the process proceeds to step S314.

[0076] In step S314, the UI presentation unit 46 corrects the classification result 54 stored in the feature extraction table 34 to the received correct classification result. In the above example, the classification result 54 associated with the document 50 with the ID "12478" is corrected from the incorrect "S / W defect" to the correct classification result "H / W failure." Through this process, the combination of the document 50 with the ID "12478" and the classification result 54 becomes data for re-learning.

[0077] In the next step S316, the feature extraction table update unit 48 updates the feature extraction table 34 based on the correction basis accepted by the UI presentation unit 46. FIG. 15 shows an example of the feature extraction table 34 updated by the feature extraction table update unit 48. For a field for which a correction basis word has been selected, the feature extraction table update unit 48 updates the feature extraction table 34 by adding each correction basis word and each combination of correction basis words to the feature item, with the type set to "additional feature" for that field. As described above, when "HDD" and "replacement" are accepted as correction basis words, as shown in FIG. 15, the feature extraction table update unit 48 adds each of "HDD," "replacement," and "HDD replacement (a combination of "HDD" and "replacement")" to the feature item, with the field set to "correspondence result" and the type set to "additional feature."

[0078] Furthermore, when a feature is added to the feature extraction table 34, the UI presentation unit 46 assigns a weight of "1" to the feature. If the feature to be added already exists in the feature extraction table 34, the feature extraction table update unit 48 updates only the weight value by adding "1" to the weight value assigned to the feature, instead of adding the feature as an "added feature."

[0079] Furthermore, when the UI presentation unit 46 receives a field of type nominal scale as the basis for correction, the feature extraction table update unit 48 adds that field to the feature extraction table 34 as type "nominal scale." As described above, when "OS" is received as the basis for correction, the feature extraction table update unit 48 adds the "OS" field of type "nominal scale" to the feature extraction table 34 and assigns a weight value of "1." Note that when the field to be added already exists in the feature extraction table 34, the feature extraction table update unit 48 updates only the weight value by adding "1" to the weight value assigned to that field, instead of adding the field.

[0080] The method for updating the feature extraction table 34 is not limited to the method described in this embodiment. When the type of accumulation basis is text, any method may be used as long as the correction basis word selected as the correction basis is newly extracted from the document 50 as a feature 52, or, if it is an existing feature 52, the importance of that feature 52 is increased. For example, the weight value assigned or added does not have to be "1." When the type of correction basis is nominal, any method may be used as long as the existence of the field selected as the correction basis is added as a feature 52, or, if it is an existing field, the importance of that field is increased.

[0081] In the next step S318, the feature extraction unit 40 obtains from the data management DB 32 a combination of the document 50 serving as re-learning data and the correct classification result 54, and extracts, from the obtained document 50, elements specified in the updated feature extraction table 34 as features 52. The feature extraction unit 40 vectorizes the extracted feature 52 using the weighting assigned to the feature 52. The feature extraction unit 40 outputs the vectorized feature 52 and the correct classification result 54 obtained from the data management DB 32 to the model training unit 42.

[0082] In the next step S320, the model training unit 42 retrains the classification model 37 based on the vectorized features 52 and the correct classification results, as described above, and stores the retrained classification model 37 in the classification model DB 36. When step S320 is completed, the retraining process shown in FIG.

[0083] The re-learning process is not limited to the form shown in Fig. 11. For example, although the re-learning process shown in Fig. 11 is performed for each document 50, the re-learning process is not limited to this form. For example, the processes of steps S300 to S316 and the processes of steps S318 and S320 may be performed in separate flows, and each process may be performed sequentially for a predetermined number of documents 50.

[0084] The relearning phase in the learning device 10 is not limited to this embodiment, and may be, for example, the second and third embodiments described below, or a combination of these embodiments may be used as appropriate. In the second and third embodiments described below, the reception screen 60 presented by the UI presentation unit 46 is different from the reception screen 60 presented by the UI presentation unit 46 in the first embodiment, and therefore the relearning process is different.

[0085] [Second embodiment] The relearning phase in the second embodiment will be described. Specifically, the reception screen 60 presented by the UI presentation unit 46 of this embodiment and the relearning process will be described with reference to the drawings. FIG. 16 shows an example of the reception screen 60 presented by the UI presentation unit 46 of this embodiment. Similarly to the first embodiment, FIG. 16 shows an example of the reception screen 60 in its initial state, which is presented when correcting the classification result 54 of the document 50 to which "12478" is assigned as an ID and stored in the data management DB 32 shown in FIG. 3 . The reception screen 60 shown in FIG. 16 includes a correction basis field 72 instead of the correction basis field 68 and the correction basis word field 70 included in the reception screen 60 of the first embodiment (see FIG. 10 ). In the initial state, the corrected classification result field 66 and the correction basis field 72 of the reception screen 60 are blank. The correction basis field 72 is an input field in which the user can enter the reason for the correction in free text. As in the first embodiment, the user inputs the correct classification result in the corrected classification result field 66, and inputs the reason for correcting the classification result in free text in the correction reason field 72. The UI presentation unit 46 accepts the correction reason input as free text in the correction reason field 72.

[0086] 17 shows a flowchart of an example of the relearning process executed by the learning device 10 of this embodiment. Note that steps S330, S334, S342, S344, S348, and S350 of the relearning process shown in Fig. 17 are similar to steps S300, S304, S312, S314, S318, and S320 of the relearning process of the first embodiment (see Fig. 11), and therefore detailed description thereof will be omitted.

[0087] 17, the UI presentation unit 46 acquires one document 50 and the classification result 54 associated with the document 50 from the data management DB 32. Here, as in the above, an example will be described in which the UI presentation unit 46 acquires a combination of the document 50 and the classification result 54 to which "12478" is assigned as an ID and which are stored in the data management DB 32 shown in FIG.

[0088] In the next step S332, the UI presenting unit 46 displays the reception screen 60 on the display 23. The reception screen 60 displayed on the display 23 by this step is in the display state shown in FIG.

[0089] If the UI presenting unit 46 receives that the non-execution instruction button 62B has been designated by the user, the determination in the next step S334 is affirmative, and the re-learning process shown in FIG. 17 is terminated.

[0090] On the other hand, if the UI presentation unit 46 has not received the designation of the non-execution instruction button 62B, the determination in step S334 is negative, and the process proceeds to step S336. The classification result 54 displayed in the classification result field 64 is corrected in the same manner as in the first embodiment. Meanwhile, the user uses the input unit 24 to input the reason for the correction in free text into the correction reason field 72. As an example, FIG. 18 shows a state in which "H / W failure" is input as the correct classification result in the corrected classification result field 66, and the free text "because the HDD is being replaced" is input into the correction reason field 72.

[0091] Therefore, in step S336, the UI presentation unit 46 determines whether or not the correct classification result entered in the corrected classification result field 66 and the basis for correction entered in the correction basis field 72 have been accepted. If the correct classification result entered in the corrected classification result field 66 and the basis for correction entered in the correction basis field 72 have not been accepted, the determination in step S336 is negative, and the process returns to step S334. On the other hand, if the correct classification result entered in the corrected classification result field 66 and the basis for correction entered in the correction basis field 72 have been accepted, the determination in step S336 is positive, and the process proceeds to step S338.

[0092] In the next step S338, the UI presentation unit 46 extracts candidates for correction basis words from the document 50 acquired in step S330. As shown in Fig. 19, the UI presentation unit 46 performs morphological analysis on the sentence "It seems that Windows has started to freeze frequently. The HDD was replaced," which is text included in the document 50 to be corrected for correcting the classification result, and generates a morpheme list that pairs morphemes with parts of speech.

[0093] In the example shown in Fig. 19, the UI presentation unit 46 generates a morpheme list including "frequent: noun," "ni: particle," "Windows: noun," "o: particle," and "implementation: noun," etc. Also, as shown in Fig. 19, the UI presentation unit 46 extracts, from the generated morpheme list, morphemes excluding particles, general words, etc., as candidates for correction basis words. In the example shown in Fig. 19, "frequent," "Windows," "freeze," "HDD," "exchange," and "implementation" are extracted as candidates for correction basis words.

[0094] In the next step S340, the UI presentation unit 46 highlights candidate correction words included in the correction basis entered in the correction basis field 72. FIG. 20 shows an example of the reception screen 60 in a state in which candidate correction words included in the correction basis entered in the correction basis field 72 are highlighted. As shown in FIG. 19, the UI presentation unit 46 compares the extracted candidate correction basis words with the free text that is the correction basis entered in the correction basis field 72, and highlights candidate correction basis words included in the free text that is the correction basis. As a result, in the example shown in FIG. 20, "HDD" and "exchange" are highlighted in the free text that is the correction basis entered in the correction basis field 72. Note that, in the present embodiment, a form in which candidate correction basis words are highlighted from the free text that is the correction basis entered in the correction basis field 72 has been described. However, the present invention is not limited to this form as long as information included in the document 50 can be identifiably displayed from the free text. For example, instead of highlighting, the information contained in document 50 may be displayed in a identifiable manner from the free text by emphasizing it compared to other words, such as by changing at least one of the font style and size of the characters, or by blinking the characters. Note that "identifiable" here means that it is identifiable as information to be used during relearning.

[0095] Furthermore, if a correction basis word included in the free text of the correction basis entered in the correction basis field 72 is already registered as a feature 52 in the feature extraction table 34, the correction basis word may be displayed in a different color from unregistered correction basis words so that it can be distinguished from them. In this case, the color may be made darker depending on the magnitude of the assigned weight, making it possible to distinguish between large and small weight values. In this way, by displaying the correction basis words included in the free text of the correction basis entered in the correction basis field 72, it is possible to confirm the degree to which the highlighted word will be emphasized in retraining the classification model 37.

[0096] By referring to the highlighting applied to the free text of the reason for correction entered in the correction reason field 72, the user can recognize words that will be added as features 52 to the feature extraction table 34 or whose weight value will be updated. Therefore, for example, if the free text of the reason for correction entered in the correction reason field 72 is not highlighted, the feature extraction table 34 will not be updated, and the user will be able to change the entered free text, etc.

[0097] After inputting the correct classification result and the basis for the correction in free text, the user uses the input unit 24 to select the implementation instruction button 62A to instruct the implementation of correction of the classification result 54. When the implementation instruction button 62A is selected, the UI presentation unit 46 accepts the implementation of correction of the classification result 54.

[0098] The determination at step S342 is negative until the UI presentation unit 46 accepts a request to make corrections, and the process returns to step S336 and repeats the processing at steps S336 to S340. On the other hand, if the UI presentation unit 46 accepts a request to make corrections, the determination at step S342 is positive, and the process proceeds to step S344.

[0099] In step S344, the UI presentation unit 46 corrects the classification result 54 stored in the feature extraction table 34 to the accepted correct classification result, and sets it as data for re-learning.

[0100] In the next step S346, the feature extraction table update unit 48 updates the feature extraction table 34 based on the highlighted correction basis word. Note that the updating method of the feature extraction table update unit 48 is similar to the method used by the feature extraction table update unit 48 in the first embodiment, in which the type is "added feature," a feature is added to the feature item, and the weight value assigned to that feature is updated, and therefore a description thereof will be omitted.

[0101] In the next step S348, the feature extraction unit 40 acquires re-learning data from the data management DB 32, extracts elements specified in the updated feature extraction table 34 from the acquired documents 50 as features 52, and vectorizes the features 52 using the assigned weighting.

[0102] In the next step S350, the model training unit 42 retrains the classification model 37 based on the vectorized features 52 and the correct classification results, as described above, and stores the retrained classification model 37 in the classification model DB 36. When step S320 is completed, the retraining process shown in FIG.

[0103] 17 may be repeatedly performed at predetermined intervals while the reception screen 60 is displayed. That is, the free text may be highlighted as needed while the user is freely entering the reason for the correction in the correction reason field 72. Alternatively, instead of the reception screen 60 described above, a reception screen 60 may be configured as shown in FIG. 21, in which a confirmation button 73 is provided in the correction reason field 72, and when the confirmation button 73 is pressed by the user, the free text of the reason for the correction entered in the correction reason field 72 is highlighted.

[0104] [Third embodiment] The relearning phase in the third embodiment will be described. Specifically, the reception screen 60 presented by the UI presentation unit 46 of this embodiment and the relearning process will be described with reference to the drawings. FIG. 22 shows an example of the reception screen 60 presented by the UI presentation unit 46 of this embodiment. Similarly to the first embodiment, FIG. 22 shows an example of the reception screen 60 in its initial state, which is presented when correcting the classification result 54 of the document 50 to which "12478" is assigned as an ID and stored in the data management DB 32 shown in FIG. 3 . The reception screen 60 shown in FIG. 22 includes a correction basis field 74 instead of the correction basis field 68 and the correction basis word field 70 included in the reception screen 60 of the first embodiment (see FIG. 10 ). In the initial state, the post-correction classification result field 66 of the reception screen 60 is blank. Meanwhile, the correction basis field 74 displays candidate correction basis sentences extracted from the document 50 with checkboxes, allowing the user to select a correction basis sentence. As in the first embodiment, the user inputs the correct classification result in the post-correction classification result field 66, and selects a correction basis sentence that will be the basis for correcting the classification result from the correction basis sentence candidates displayed in the correction basis field 72. The UI presentation unit 46 accepts the correction basis input as free text in the correction basis field 72.

[0105] 23 shows a flowchart of an example of the relearning process executed by the learning device 10 of this embodiment. Note that steps S360, S364, S372, S374, S378, and S380 of the relearning process shown in Fig. 23 are similar to steps S300, S304, S312, S314, S318, and S320 of the relearning process of the first embodiment (see Fig. 11), and therefore detailed description thereof will be omitted.

[0106] 23, the UI presentation unit 46 acquires one document 50 and the classification result 54 associated with the document 50 from the data management DB 32. Here, as in the above, an example will be described in which the UI presentation unit 46 acquires a combination of the document 50 and the classification result 54 to which "12478" is assigned as an ID and which are stored in the data management DB 32 shown in FIG.

[0107] In the next step S361, the UI presentation unit 46 extracts correction basis sentence candidates from the document 50 acquired in step S360. As an example, for a field in the document 50 whose type is text, the UI presentation unit 46 of this embodiment divides the text into single sentences and sets each of the divided single sentences as a correction basis sentence candidate. Note that in this disclosure, a "sentence" also includes a single-word sentence containing only one word. In the example shown in FIG. 22, "Device froze" is extracted as a correction basis sentence candidate from the "Title" field of the document 50. Furthermore, "Apparently, Windows has started to freeze frequently." is extracted as a correction basis sentence candidate from the "Description" field of the document 50. Furthermore, "The HDD was replaced." is extracted as a correction basis sentence candidate from the "Response result" field of the document 50.

[0108] Furthermore, for a field in the document 50 whose type is a nominal scale, the UI presentation unit 46 sets the description of that field contained in the document 50 as a candidate basis sentence for correction. In the example shown in FIG. 22, "Desket" is extracted as a candidate basis sentence for correction from the field "Model Name" in the document 50. Furthermore, "Windows 10" is extracted as a candidate basis sentence for correction from the field "OS" in the document 50.

[0109] In the next step S362, the UI presenting unit 46 displays the reception screen 60 on the display 23. The reception screen 60 displayed on the display 23 by this step is in the display state shown in FIG.

[0110] If the UI presenting unit 46 receives that the non-execution instruction button 62B has been designated by the user, the determination in the next step S364 is affirmative, and the re-learning process shown in FIG. 23 is terminated.

[0111] On the other hand, if the UI presentation unit 46 has not received the designation of the non-execution instruction button 62B, the determination at step S364 is negative, and the process proceeds to step S366. The classification result 54 displayed in the classification result field 64 is corrected in the same manner as in the first embodiment. The selection of a correction basis sentence is performed by the user using the input unit 24 to check a checkbox corresponding to a correction basis sentence candidate to be selected as the correction basis from among the correction basis sentence candidates displayed in the correction basis field 74. As an example, FIG. 24 shows a state in which "H / W failure" is entered as the correct classification result in the corrected classification result field 66, and "because the HDD is being replaced" is selected as the correction basis sentence from among the correction basis sentence candidates displayed in the correction basis field 74. The number of correction basis sentences selected by the user is not particularly limited and may be one or more. Alternatively, all of the correction basis sentence candidates displayed in the correction basis field 74 may be selected.

[0112] Therefore, in step S366, the UI presentation unit 46 determines whether or not the selection of the correct classification result and the correction basis sentence entered in the corrected classification result field 66 has been accepted. If the selection of the correct classification result and the correction basis sentence entered in the corrected classification result field 66 has not been accepted, the determination in step S366 is negative, and the process returns to step S364. On the other hand, if the selection of the correct classification result and the correction basis sentence entered in the corrected classification result field 66 has been accepted, the determination in step S366 is positive, and the process proceeds to step S372.

[0113] After inputting the correct classification result and selecting the correction basis sentence, the user instructs the correction of the classification result 54 by specifying the implementation instruction button 62A using the input unit 24. When the implementation instruction button 62A is specified, the UI presentation unit 46 accepts the execution of the correction of the classification result 54.

[0114] The determination at step S372 remains negative until the UI presentation unit 46 accepts a request to make corrections, and the process returns to step S366. On the other hand, if the UI presentation unit 46 accepts a request to make corrections, the determination at step S372 remains positive, and the process proceeds to step S374.

[0115] In step S374, the UI presentation unit 46 corrects the classification result 54 stored in the feature extraction table 34 to the accepted correct classification result, and sets it as data for re-learning.

[0116] In the next step S375, the UI presentation unit 46 extracts correction basis words from the document 50 based on the received correction basis sentence. If the type of the field corresponding to the received correction basis sentence is text, the UI presentation unit 46 performs morphological analysis on the correction basis sentence to generate a morpheme list in which morphemes and parts of speech are paired. In the example shown in Figure 25, the correction basis sentence is "HDD replacement performed," the corresponding field is "response result," and its type is text, so morphological analysis is performed on the sentence "HDD replacement performed," to generate a morpheme list in which morphemes and parts of speech are paired.

[0117] In the example shown in Fig. 25, the UI presentation unit 46 generates a morpheme list including "HDD: noun," "no: particle," "exchange: noun," "o: particle," and "execution: noun." Also, as shown in Fig. 25, the UI presentation unit 46 extracts, from the generated morpheme list, words excluding particles, general words, etc., as correction basis words. In the example shown in Fig. 25, "HDD," "exchange," and "execution" are extracted as correction basis words.

[0118] In the next step S376, the feature extraction table update unit 48 updates the feature extraction table 34 based on the extracted correction basis words. Note that the updating method of the feature extraction table update unit 48 is similar to the method used by the feature extraction table update unit 48 in the first embodiment, in which the type is set to "added feature," a feature is added to the feature item, and the weight value assigned to that feature is updated, so a description thereof will be omitted.

[0119] If a field whose type is nominal scale is selected from the candidate revision basis sentences displayed in the revision basis column 74, the processing of step S375 may not be performed, and the feature extraction table 34 may be updated by adding the presence or absence of possible values ​​as a feature, as described above in the first embodiment.

[0120] In the next step S378, the feature extraction unit 40 acquires re-learning data from the data management DB 32, extracts elements specified in the updated feature extraction table 34 from the acquired documents 50 as features 52, and vectorizes the features 52 using the assigned weighting.

[0121] In the next step S380, the model training unit 42 retrains the classification model 37 based on the vectorized features 52 and the correct classification results, as described above, and stores the retrained classification model 37 in the classification model DB 36. When step S380 is completed, the retraining process shown in FIG. 23 ends.

[0122] Instead of the reception screen 60 described above, a reception screen 60 shown in FIG. 26 may include a correction basis field 68 with a pull-down menu 68P. In this case, similar to the correction basis field 68 included in the reception screen 60 of the first embodiment, the UI presentation unit 46 accepts a field selected by the user as the correction basis. Then, text sentences corresponding to the accepted field are displayed in the correction basis field 74 so that they can be selected as correction basis sentences. In the example shown in FIG. 26, the UI presentation unit 46 accepts, as correction basis sentences, the sentence "Windows frequently freezes" corresponding to the response result field and the sentence "The HDD was replaced" corresponding to the description field. In addition, "OS," a nominal scale type, is accepted as the correction basis.

[0123] As described above, the learning device 10 of the above embodiment is a learning device that retrains a classification model 37 that receives as input features 52 extracted from a document 50 by referencing a feature extraction table 34 and outputs a category to which the document 50 belongs as a classification result 54. The learning device 10 includes a CPU 20, which presents a reception screen 60 that accepts input of a correct classification result as a correction instruction for correcting an error in the classification result 54, and a correction basis word as a correction basis based on information included in the document 50. When the correct classification result and the correction basis are received, the CPU 20 updates the feature extraction table 34 based on the correction basis, and retrains the classification model 37 using the corrected, correct classification result and the updated feature extraction table 34.

[0124] In this way, the learning device 10 of this embodiment updates the feature extraction table 34 based on the correction reason entered by the user, and re-learns using the updated feature extraction table 34, thereby further improving the classification accuracy of the classification model 37.

[0125] In the above embodiments, the features 52 are described as words. However, features 52 may be other than words, or may include other features in addition to words. For example, if a document 50 contains a character string that does not have a general meaning, such as an error code, the character string may be treated as a feature 52. If a document 50 contains a symbol, the symbol may be treated as a feature 52. For example, if the feature extraction table 34 contains an image related to the document 50, the image may be treated as a feature 52, and the image treated as a feature 52 may be input to the classification model 37 in addition to the features 52 extracted from the document 50.

[0126] In addition, in each of the above embodiments, the semantic content of document 50 represents the state of a machine failure, and the category of the classification result includes the cause of the failure, but the semantic content of document 50 and the category of the classification result are not limited to this. For example, document 50 may represent the medical record and condition of a patient, and the category of the classification result may be the name of a disease. Furthermore, although the reception screen 60 in each of the above embodiments has been described as including the document 50 and the classification result field 64, it is not essential that these fields be included. For example, the document 50, the classification result field 64, and the correction instruction button 62 may be displayed on one screen, and when the user designates the instruction button 62A and the UI presentation unit 46 receives an instruction to perform correction, the reception screen 60 may transition to one that does not include the document 50 and the classification result field 64. Furthermore, the reception screen 60 may be displayed on one display 23, or may be displayed on multiple displays 23. That is, a multi-display may be adopted as the display 23. Furthermore, in each of the above embodiments, the learning device 10 performs learning, operation, and re-learning of the classification model 37. However, each of these may be performed by a separate device.

[0127] In the above embodiment, the following various processors may be used as the hardware configuration of processing units that perform various processes, such as the feature extraction unit 40, the model learning unit 42, the classification prediction unit 44, the UI presentation unit 46, and the feature extraction table update unit 48. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits, such as programmable logic devices (PLDs) whose circuit configuration can be changed after manufacture, such as field programmable gate arrays (FPGAs), and application specific integrated circuits (ASICs), which are processors with a circuit configuration designed specifically for performing specific processes.

[0128] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0129] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0130] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0131] In addition, in the above-described embodiments, the learning program 30A, the classification program 30B, and the relearning program 30C are each pre-stored (installed) in the storage unit 22, but this is not limiting. The learning program 30A, the classification program 30B, and the relearning program 30C may each be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. Furthermore, the learning program 30A, the classification program 30B, and the relearning program 30C may each be downloaded from an external device via a network. In other words, the programs (program products) described in the present embodiments may be provided on a recording medium or distributed from an external computer.

[0132] The following additional notes are provided regarding the above-described embodiment.

[0133] (Appendix 1) A learning device that re-trains a classification model that receives input of features extracted from a document by referring to a feature extraction table and outputs a category to which the document belongs as a classification result, at least one processor; The processor: presenting a reception screen for receiving input of a correction instruction for correcting an error in the classification result and a correction basis based on information included in the document; When the instruction for correction and the grounds for correction are accepted, The feature extraction table is updated based on the correction basis, and the classification model is retrained using the corrected classification result and the updated feature extraction table. Learning device.

[0134] (Appendix 2) The items of the feature extraction table include at least words to be extracted as features, or words to be extracted as features and weights assigned to each of the words. 2. The learning device of claim 1.

[0135] (Appendix 3) The processor: updating at least one of the items in the feature extraction table based on the correction grounds; 3. The learning device according to claim 2.

[0136] (Appendix 4) The update is the addition of a word included in the revision basis. 4. The learning device according to claim 1, wherein the learning device is a

[0137] (Appendix 5) The update is an update of the weights assigned to words included in the revision basis. 5. A learning device according to any one of claims 1 to 4.

[0138] (Appendix 6) The processor: displaying, on the reception screen, candidates for the revision grounds extracted from the document in a selectable manner; The input of the reason for the correction is accepted by allowing the candidate to be selected. 6. A learning device according to any one of claims 1 to 5.

[0139] (Appendix 7) The processor: extracting at least words from the document as candidates for the revision basis; 7. The learning device according to claim 6.

[0140] (Appendix 8) The processor: extracting sentences in addition to the words as candidates for the correction reasons; The extracted words and the sentences are displayed together on the reception screen. 8. The learning device of claim 7.

[0141] (Appendix 9) the reception screen has an input field in which a user can input the reason for the correction in free text, The processor: The information contained in the document is displayed in a identifiable manner from the input free text. 6. A learning device according to any one of claims 1 to 5.

[0142] (Appendix 10) The processor: The information contained in the document is highlighted to make it identifiable. 10. The learning device according to claim 9.

[0143] (Appendix 11) When the semantic content of the document represents a situation of a machine failure, the category includes a cause of the failure. 11. The learning device according to any one of claims 1 to 10.

[0144] (Appendix 12) The reception screen includes a display area for displaying the document and the classification result. 12. A learning device according to any one of claims 1 to 11.

[0145] (Appendix 13) The feature extraction table includes images related to the document; The image is input as a feature to the classification model in addition to the features extracted from the document. 13. A learning device according to any one of claims 1 to 12.

[0146] (Appendix 14) The document contains symbols, and the feature extraction table contains at least symbols to be extracted as features. 14. The learning device according to any one of claims 1 to 13.

[0147] (Appendix 15) at least one processor; The processor: Displaying previously input documents in a classification model that inputs features extracted from documents by referring to a feature extraction table and outputs the category to which the document belongs as a classification result; Displaying the classification results output from the classification model according to the previously input documents; Accepts correction instructions to correct errors in the displayed classification results, Accepting a correction basis that is the basis for the correction instruction; Re-learning is performed using the accepted correction instructions and correction grounds. Learning device.

[0148] (Appendix 16) a processor included in a learning device that retrains a classification model that receives features extracted from a document by referring to a feature extraction table and outputs a category to which the document belongs as a classification result; presenting a reception screen for receiving input of a correction instruction for correcting an error in the classification result and a correction basis based on information included in the document; When the instruction for correction and the grounds for correction are accepted, The feature extraction table is updated based on the correction basis, and the classification model is retrained using the corrected classification result and the updated feature extraction table. How to learn to perform a process.

[0149] (Appendix 17) a step of displaying documents previously input to a classification model that inputs features extracted from documents by referring to a feature extraction table and outputs a category to which the document belongs as a classification result; displaying the classification results output from the classification model according to the previously input documents; receiving a correction instruction to correct an error in the displayed classification result; receiving a correction basis that is a basis for the correction instruction; a step of relearning using the received correction instruction and correction basis; Learning methods including.

[0150] (Appendix 18) A processor included in a learning device that retrains a classification model that receives features extracted from a document by referring to a feature extraction table and outputs a category to which the document belongs as a classification result, presenting a reception screen for receiving input of a correction instruction for correcting an error in the classification result and a correction basis based on information included in the document; When the instruction for correction and the grounds for correction are accepted, The feature extraction table is updated based on the correction basis, and the classification model is retrained using the corrected classification result and the updated feature extraction table. A learning program to execute the process. [Explanation of symbols]

[0151] 10 Learning Device 20 CPU 21 Memory 22 Memory section 23 Display 24 Input section 26 Network I / F 29 Bus 30A Learning Program, 30B Classification Program, 30C Refresher Program 32 Data Management DB 34 Feature Extraction Table 36 Classification Model DB 37 Classification Models 40 Feature Extraction Unit 42 Model Learning Section 44 Classification and Prediction Unit 46 UI presentation part 48 Feature extraction table update unit 50 documents 52 Background 54 Classification results 60 Reception screen 62 Correction execution instruction button, 62A Execution instruction button, 62B Non-execution instruction button 64 Classification result column 66 Corrected classification result column, 66P pull-down menu 68 Reason for amendment, 68P pull-down menu 70 Correction basis word column 72, 74 Amendment Basis Column 73 Confirm button

Claims

1. A learning device that re-trains a classification model that receives input of features extracted from a document by referring to a feature extraction table and outputs a category to which the document belongs as a classification result, at least one processor; The processor: presenting a reception screen for receiving input of a correction instruction for correcting an error in the classification result and a correction basis based on information included in the document; When the instruction for correction and the grounds for correction are accepted, The feature extraction table is updated based on the correction basis, and the classification model is retrained using the corrected classification result and the updated feature extraction table. Learning device.

2. The items of the feature extraction table include at least words to be extracted as features, or words to be extracted as features and weights assigned to each of the words. The learning device according to claim 1 .

3. The processor: updating at least one of the items in the feature extraction table based on the correction grounds; The learning device according to claim 2 .

4. The update is the addition of a word included in the revision basis. The learning device according to claim 3 .

5. The items of the feature extraction table include at least words to be extracted as features and weights assigned to each of the words, The update is an update of the weights assigned to words included in the revision basis. The learning device according to claim 3 .

6. The processor: displaying, on the reception screen, candidates for the revision grounds extracted from the document in a selectable manner; The input of the reason for the correction is accepted by allowing the candidate to be selected. The learning device according to claim 1 .

7. The processor: extracting at least words from the document as candidates for the revision basis; The learning device according to claim 6.

8. The processor: extracting sentences in addition to the words as candidates for the correction reasons; The extracted words and the sentences are displayed together on the reception screen. The learning device according to claim 7 .

9. the reception screen has an input field in which a user can input the reason for the correction in free text, The processor: The information contained in the document is displayed in a identifiable manner from the input free text. The learning device according to claim 1 .

10. The processor: The information contained in the document is highlighted to make it identifiable. The learning device according to claim 9 .

11. When the semantic content of the document represents a situation of a machine failure, the category includes a cause of the failure. The learning device according to claim 1 .

12. The reception screen includes a display area for displaying the document and the classification result. The learning device according to claim 1 .

13. The feature extraction table includes images related to the document; The image is input as a feature to the classification model in addition to the features extracted from the document. The learning device according to claim 1 .

14. The document contains symbols, and the feature extraction table contains at least symbols to be extracted as features. The learning device according to claim 1 .

15. at least one processor; The processor: Displaying previously input documents in a classification model that inputs features extracted from documents by referring to a feature extraction table and outputs the category to which the document belongs as a classification result; Displaying the classification results output from the classification model according to the previously input documents; Accepts correction instructions to correct errors in the displayed classification results, Accepting a correction basis that is the basis for the correction instruction; Re-learning is performed using the accepted correction instructions and correction grounds. Learning device.

16. a processor included in a learning device that retrains a classification model that receives features extracted from a document by referring to a feature extraction table and outputs a category to which the document belongs as a classification result; presenting a reception screen for receiving input of a correction instruction for correcting an error in the classification result and a correction basis based on information included in the document; When the instruction for correction and the grounds for correction are accepted, The feature extraction table is updated based on the correction basis, and the classification model is retrained using the corrected classification result and the updated feature extraction table. How to learn to perform a process.

17. A learning method executed by a processor included in a learning device, comprising: a step of displaying documents previously input to a classification model that inputs features extracted from documents by referring to a feature extraction table and outputs a category to which the document belongs as a classification result; displaying the classification results output from the classification model according to the previously input documents; receiving a correction instruction to correct an error in the displayed classification result; receiving a correction basis that is a basis for the correction instruction; a step of relearning using the received correction instruction and correction basis; Learning methods including.

18. A processor included in a learning device that retrains a classification model that receives features extracted from a document by referring to a feature extraction table and outputs a category to which the document belongs as a classification result, presenting a reception screen for receiving input of a correction instruction for correcting an error in the classification result and a correction basis based on information included in the document; When the instruction for correction and the grounds for correction are accepted, The feature extraction table is updated based on the correction basis, and the classification model is retrained using the corrected classification result and the updated feature extraction table. A learning program to execute the process.

Images