Information processing system, information processing method, and program

Abstract

To support work for confirming whether or not scoring by a AI model is appropriate.SOLUTION: A AI model predictor that predicts a scoring result of a target answer by using a AI model that has learned a correspondence relationship between a scoring standard of a question and a learning answer, a prediction evidence extractor that calculates a prediction evidence indicating an evidence of prediction by the AI model, a prediction evidence visualizer that generates an evidence image in which the prediction evidence is associated with wording described in the target answer, and an outputter that outputs image information of the evidence image generated by the prediction evidence visualizer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] There is a technology for automatic scoring that allows a computer to score written questions such as those used in academic tests, qualification exams, etc. For example, Patent Document 1 discloses a technology that aims to reduce the number of prepared learning patterns and improve the accuracy of the judgment in a system in which an AI judges the similarity of the input results of a respondent by natural language classification. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-86037 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technology did not take into consideration the task of a human grader checking whether the scoring results of the AI ​​model were valid. Because grading academic ability tests and qualification exams requires thoroughgoing care, a human grader must check the validity of the scores of the AI ​​model, which is time-consuming.

[0005] The present invention has been made in consideration of the above circumstances, and aims to provide an information processing system, an information processing method, and a program that can assist in the task of confirming whether scoring by an AI model is appropriate. [Means for solving the problem]

[0006] The information processing system of the present invention comprises an AI model prediction unit that predicts the scoring result of a target answer using an AI model that has learned the correspondence between the scoring criteria for written questions and learning answers; a prediction basis extraction unit that calculates prediction basis that indicates the basis for the prediction by the AI ​​model; a prediction basis visualization unit that generates a basis image in which the prediction basis is associated with the wording described in the target answer; and an output unit that outputs image information of the basis image generated by the prediction basis visualization unit.

[0007] The information processing method of the present invention is an information processing method performed by a computer, in which an AI model prediction unit predicts the scoring result of a target answer using an AI model that has learned the correspondence between the scoring criteria for written questions and learning answers, a prediction basis extraction unit calculates a prediction basis that indicates the basis for the prediction by the AI ​​model, a prediction basis visualization unit generates a basis image that matches the prediction basis with the wording written in the target answer, and an output unit outputs image information of the basis image generated by the prediction basis visualization unit.

[0008] The program of the present invention causes a computer to predict the scoring result of a target answer using an AI model that has learned the correspondence between the scoring criteria for essay questions and the learning answers, The program calculates prediction basis indicating the basis for the prediction by the AI ​​model, generates a basis image that matches the prediction basis with the wording written in the target answer, and outputs image information of the generated basis image. [Effects of the Invention]

[0009] According to the present invention, it is possible to assist in the task of confirming whether the scoring by the AI ​​model is appropriate. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing system 1 according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an example of an essay question in the embodiment. [Figure 3]FIG. 2 is a diagram showing an example of information stored in a scoring criteria storage unit 12 in the embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an image generated by a prediction basis visualization unit 22 in the embodiment. [Figure 5] FIG. 2 is a diagram showing an example of an image generated by a prediction basis visualization unit 22 in the embodiment. [Figure 6] FIG. 2 is a diagram showing an example of an image generated by a prediction basis visualization unit 22 in the embodiment. [Figure 7] FIG. 2 is a diagram showing an example of an image generated by a prediction basis visualization unit 22 in the embodiment. [Figure 8] 2 is a sequence diagram showing the flow of processing performed by the information processing system 1 of the embodiment. FIG. [Figure 9] 2 is a sequence diagram showing the flow of processing performed by the information processing system 1 of the embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] (About Information Processing System 1)

[0013] The information processing system 1 is a system that supports the grading of essay questions. Essay questions are questions that require the answer to be written. The information processing system 1 uses an AI model to predict the grading result of the answer, and presents an image (basis image G, described later) that allows a grader (human) to confirm the validity of the grading result by the AI ​​model. This can support the task of the grader (human) confirming whether the grading by the AI ​​model is valid.

[0014] Fig. 1 is a block diagram showing an example of the configuration of an information processing system 1 according to an embodiment. As shown in Fig. 1, the information processing system 1 includes, for example, a scoring device 10 and a scoring basis visualization device 20. The scoring device 10 predicts the scoring result of an answer using an AI model. The scoring basis visualization device 20 presents an image (basis image G, described later) that allows a grader (human) to confirm the validity of the prediction result by the AI ​​model.

[0015] In the information processing system 1, the scoring device 10 and the scoring basis visualization device 20 are communicatively connected via a communication network or the like. This diagram shows an example in which the scoring device 10 and the scoring basis visualization device 20 are configured as different devices in the information processing system 1, but the functions of the scoring device 10 and the scoring basis visualization device 20 may be realized by one device, or by three or more devices.

[0016] 2 is a diagram showing an example of an essay question in the embodiment, in which actual answers by test takers are shown for the example question and the correct answer. In this diagram, the example question asks, "Explain why the shape of the moon appears to change from day to day." The correct answer is, "Because the relative positions of the moon and sun as seen from Earth change." As shown in Figure 2, in written questions, answers are rarely identical to the sample correct answers, and a variety of sentences are given as answers. If the answer is "Because the relative positions of the moon and sun as seen from the earth change," the marking result will be "correct." If the answer is "Because the relative positions of the earth, moon, and sun change," the marking result will be "correct." On the other hand, if the answer is "Because the shape of the moon changes," the marking result will be "incorrect." If the answer is "Because different shaped moons appear alternately," the marking result will be "incorrect." In the information processing system 1, the AI ​​model scores the answers written in various sentences and predicts whether the answer is "correct" or "incorrect." In this embodiment, to make it easier to confirm the validity of the prediction (scoring result) by the AI ​​model, the basis for the prediction by the AI ​​model is shown in the basis image G (see FIGS. 4 to 7). For example, by using a technology called XAI (Extensible AI), it is possible to derive the basis for predictions made by an AI model. By showing the basis for predictions in the basis image G, it becomes easier to confirm whether the scoring made by the AI ​​model is appropriate.

[0017] (Regarding the scoring device 10) The scoring device 10 is a computer that predicts the scoring results of answers using an AI model. The scoring device 10 is realized by a cloud, a server device, a PC (personal computer), or the like. The scoring device 10 includes, for example, an answer data storage unit 11, a scoring criterion storage unit 12, a learning answer data storage unit 13, a prediction result storage unit 14, an AI model prediction unit 15, an AI model storage unit 16, and an AI model learning unit 17.

[0018] The marking criteria storage unit 12 stores information about marking criteria, including, for example, written questions, marking criteria, and sample answers.

[0019] FIG. 3 is a diagram illustrating an example of information stored in the scoring criteria storage unit 12 of the information processing system 1 according to the embodiment. As shown in FIG. 3, the marking criteria storage unit 12 stores information corresponding to each item, such as a question ID, a question, a marking criteria, and a correct answer example. The question ID field stores identification information for identifying a question. The question field stores information indicating the specific content of the question identified by the question ID. The item of the marking criteria stores information indicating the criteria for marking the question identified by the question ID. For example, the marking criteria created by the question creator P1 along with the question is stored here. The item of correct answer example stores information indicating a sentence that shows an example of the correct answer to the question identified by the question ID. In this diagram, the question "Answer why the shape of the moon appears to change from day to day" has three scoring criteria: (Scoring Criterion 1) mentioning the change in position, (Scoring Criterion 2) including the words sun, moon, and earth in the answer, and (Scoring Criterion 3) providing a sentence explaining the reason.

[0020] The learning answer data storage unit 13 stores information about learning answer data. The learning answer data is an answer (written text) to a written question to be graded, and is information that associates an answer used for learning with the graded result. As the learning answer data, for example, answers to written questions that ask the same content as the question created by the question creator P1 among written questions that have been given in the past, and the results of the grades of those answers, can be used. Alternatively, if the question creator P1 has created a sample answer and its graded result, this can be used as the learning answer data.

[0021] The AI ​​model learning unit 17 trains the AI ​​model. The AI ​​model learning unit 17 trains the AI ​​model to learn the correspondence between the scoring criteria stored in the scoring criteria storage unit 12 and the learning answer data stored in the learning answer data storage unit 13. In this way, the AI ​​model is trained to be able to predict whether an answer input to the AI ​​model is correct or incorrect based on the scoring criteria. When the AI ​​model learning unit 17 completes learning of the AI ​​model, it stores information for constructing the AI ​​model for which learning has been completed in the AI ​​model storage unit 16.

[0022] The AI ​​model storage unit 16 stores information for constructing an AI model that has been trained by the AI ​​model training unit 17.

[0023] The answer data storage unit 11 stores information about answer data. The answer data is information that indicates answers (text) to written questions that are the subject of grading.

[0024] The AI ​​model prediction unit 15 predicts the grading result of the answer using the AI ​​model. The AI ​​model prediction unit 15 reads information for constructing the AI ​​model from the AI ​​model storage unit 16, and constructs the AI ​​model using the read information. The AI ​​model prediction unit 15 also reads and acquires answer data from the answer data storage unit 11. The AI ​​model prediction unit 15 inputs the answer data to the AI ​​model, and acquires the grading result predicted by the AI ​​model in response to the input of the answer data. The AI ​​model prediction unit 15 sets the grading result predicted by the AI ​​model as the grading result of the answer. The AI ​​model prediction unit 15 stores the grading result by the AI ​​model in the prediction result storage unit 14.

[0025] The prediction result storage unit 14 stores the prediction result by the AI ​​model prediction unit 15 (the prediction result by the AI ​​model).

[0026] The memory units (including answer data memory unit 11, scoring criteria memory unit 12, learning answer data memory unit 13, prediction result memory unit 14, and AI model memory unit 16) provided in scoring device 10 are configured with storage media such as HDD (Hard Disk Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access Read / Write Memory), ROM (Read Only Memory), or a combination of these. The memory units provided in scoring device 10 store programs for executing various processes in scoring device 10 and temporary data used when performing various processes.

[0027] In addition, the functional units (including the AI ​​model prediction unit 15 and the AI ​​model learning unit 17) of the scoring device 10 are realized by having a CPU (Central Processing Unit) and / or GPU (Graphics Processing Unit) that the scoring device 10 has as hardware execute a program.

[0028] (About the scoring basis visualization device 20) The scoring basis visualization device 20 is a computer that generates a basis image G. The scoring basis visualization device 20 is realized by a cloud, a server device, a PC (personal computer), etc. The scoring basis visualization device 20 includes, for example, a prediction basis extraction unit 21, a prediction basis visualization unit 22, and an image output unit 23.

[0029] The prediction basis extraction unit 21 calculates the basis for the prediction made by the AI ​​model. The basis here is information indicating which wording in the sentence written in the input (answer) was focused on to predict the scoring result (whether the answer is correct or incorrect). The prediction basis extraction unit 21 calculates the prediction basis using XAI (Explainable AI), that is, explainable AI. The prediction basis extraction unit 21 can use any conventional XAI method. For example, attention can be used as XAI. Attention here is a mechanism that assigns a score indicating the strength of the relationship between words as a value indicating which part of the input data the AI ​​model should focus on. By visualizing this attention, it is possible to calculate the prediction basis indicating "which part of the input data the AI ​​model focused on to make the prediction." The prediction basis extraction unit 21 outputs information indicating the calculated prediction basis to the prediction basis visualization unit 22.

[0030] The prediction basis visualization unit 22 generates an image (basis image G) that visualizes the prediction basis. The prediction basis visualization unit 22 acquires information indicating the prediction basis from the prediction basis extraction unit 21. The information indicating the prediction basis is information in which a score ranging from 0 (zero) to 1 is assigned to each word in the sentence written as input data (answer). Here, a word assigned a score closer to 0 (zero) indicates that it has received less attention in the prediction, and a word assigned a score closer to 1 indicates that it has received more attention in the prediction.

[0031] For example, the prediction basis visualization unit 22 calculates the strength of the relationship between words in the sentences described in the input data (answers) that have been assigned a score equal to or greater than a threshold value and the scoring criteria. Any method such as morphological analysis or natural language processing can be used to calculate the strength of the relationship. For example, when a word that has been assigned a score equal to or greater than a threshold value or a word similar to that word is included in the sentence indicating the scoring criteria, the prediction basis visualization unit 22 calculates an index value indicating the strength of the relationship so that the strength of the relationship is large.

[0032] Based on the calculated index value, the prediction basis visualization unit 22 identifies words in the sentences described in the input data (answer) that have been assigned a score equal to or greater than a threshold value and scoring criteria that have a high relevance. For example, when the index value indicating the strength of relevance is equal to or greater than a threshold value, the prediction basis visualization unit 22 determines that the word is highly relevant to the scoring criteria. The prediction basis visualization unit 22 generates an image that displays words in the sentences described in the input data (answer) that have been assigned a score equal to or greater than a threshold value and scoring criteria that have a high relevance in association with each other, as a basis image G (see FIGS. 4 to 7).

[0033] Alternatively, the prediction basis visualizing unit 22 may generate, as the basis image G, an image in which, among the sentences described in the input data (answer), words that have been assigned a score equal to or greater than a threshold are highlighted, for example. Here, the prediction basis visualizing unit 22 may set a plurality of thresholds and generate, as the basis image G, an image in which words that have been assigned a score equal to or greater than each threshold are highlighted in a plurality of stages. For example, the prediction basis visualizing unit 22 generates, as the basis image G, an image in which words that have been assigned a score of 0.8 or greater are highlighted by using a red background color, and words that have been assigned a score of 0.6 or greater (and less than 0.8) are highlighted by using a pink background color.

[0034] The image output unit 23 outputs image information of the basis image G. For example, the image output unit 23 outputs the image information of the basis image G generated by the prediction basis visualization unit 22 to a display (not shown) visually recognized by the grader P2, thereby displaying the basis image G on the display. By visually recognizing the basis image G displayed on the display, the grader P2 can easily determine whether the scoring prediction by the AI ​​model is valid.

[0035] The storage unit included in the scoring basis visualization device 20 is configured by a storage medium such as an HDD, flash memory, EEPROM, RAM, or ROM, or a combination of these. The storage unit included in the scoring basis visualization device 20 stores programs for executing various processes in the scoring basis visualization device 20 and temporary data used when performing the various processes.

[0036] In addition, the functional units (including the prediction basis extraction unit 21, the prediction basis visualization unit 22, and the image output unit 23) of the scoring basis visualization device 20 are realized by having the CPU and / or GPU that the scoring basis visualization device 20 has as hardware execute a program.

[0037] Here, examples of the basis image G generated by the prediction basis visualizing unit 22 will be described with reference to Fig. 4 to Fig. 7. Fig. 4 to Fig. 7 are diagrams showing examples of images (basis image G) generated by the prediction basis visualizing unit 22 in the embodiment. 4 to 7, the information corresponding to the implementation period, grade, subject, question number, etc. is shown at the top of the basis image G as information for identifying the question. Here, it is shown that it is a science question for first-year junior high school students. The question here may be any question, but for the sake of explanation, it is assumed to be the question shown in FIG. 3, "Answer why the shape of the moon appears to change depending on the day." The left side of Figures 4 to 7 shows an "answer ID list." The answer ID list shows a list of identification information (answer IDs) of answers predicted by the AI ​​model. When an operation to select one of the answer IDs in this answer ID list is performed, a basis image G of the answer corresponding to the selected answer ID is displayed on the right side of the "answer ID list."

[0038] Figure 4 shows an example of evidence image G when the scoring result predicted by the AI ​​model (AI scoring result) is "correct." The answer shown is an example of the sentence "Because the positions of the sun, earth, and moon change." In this figure, evidence image G highlights the background of the part of the answer where the phrase "sun, earth, moon" is written in a first color (e.g., yellow) as the basis for prediction, and also highlights the beginning of the statement in the scoring criteria that "(scoring criteria 2) the words sun, moon, and earth are included in the answer" in the same first color (e.g., yellow). This shows that the phrase "sun, earth, moon" that was noticed in the AI ​​model's prediction is related to (scoring criteria 2). Furthermore, in evidence image G in this figure, the background of the part of the answer where the phrase "position changes" is written is highlighted in a second color (e.g., red) as the basis for prediction, and the beginning of the statement "(Scoring Criteria 1) mentions position changes" in the scoring criteria is highlighted in the same second color (e.g., red). This shows that the phrase "position changes," which was noted in the AI ​​model's prediction, is related to (Scoring Criteria 1). Furthermore, in the evidence image G in this figure, the background of the part of the answer where the word "kara" is written at the end is highlighted in a third color (e.g., green) as the basis for prediction, and the beginning of the statement "(Scoring Criteria 3) is written in a sentence that explains the reason" in the scoring criteria is highlighted in the same third color (e.g., green). This shows that the word "kara" that was noticed in the AI ​​model's prediction is related to (Scoring Criteria 3). By visually checking the evidence image G, grader P2 can easily determine that the "correct answer" predicted by the AI ​​model (AI scoring result) is valid simply by confirming that the answer contains wording corresponding to each of the scoring criteria (scoring criteria 1) to (scoring criteria 3) and that the correspondence between the scoring criteria and the wording is valid.

[0039] Figure 5 shows an example of evidence image G when the scoring result predicted by the AI ​​model (AI scoring result) is "incorrect." The answer shown is an example in which the sentence "Because the shape of the moon changes" is written. In this figure, evidence image G highlights the background of the part of the answer where the word "moon" is written in a first color (e.g., yellow) as the basis for prediction, and also highlights the beginning of the statement in the scoring criteria that "(Scoring Criteria 2) The answer contains the words sun, moon, and earth" in the same first color (e.g., yellow). This shows that the word "moon," which was noted in the AI ​​model's prediction, is related to (Scoring Criteria 2). Furthermore, in the evidence image G in this figure, the background of the part of the answer where the word "kara" is written at the end is highlighted in a third color (e.g., green) as the basis for prediction, and the beginning of the statement "(Scoring Criteria 3) is written in a sentence that explains the reason" in the scoring criteria is highlighted in the same third color (e.g., green). This shows that the word "kara" that was noticed in the AI ​​model's prediction is related to (Scoring Criteria 3). In addition, in the evidence image G in this figure, the background is not highlighted in the second color (e.g., red) wherever any wording is written in the answer, which indicates that the wording related to (Scoring Criteria 1) is not written in the answer. By visually examining the evidence image G, grader P2 can confirm that the answer does not contain the wording corresponding to (Scoring Criterion 1) and that the answer is missing the wording corresponding to (Scoring Criterion 2), and can easily determine that the "incorrect answer" predicted by the AI ​​model (AI scoring result) is appropriate.

[0040] 4 and 5, the prediction basis visualizing unit 22 generates, as the basis image G, an image in which the wording and scoring criteria described in the answer (the target answer that is the prediction target of the AI ​​model) are displayed, the specific wording described in the answer and the scoring criteria corresponding to the specific wording are displayed in a first display mode, and the wording described in the answer that is different from the specific wording and the scoring criteria that do not correspond to the specific wording are displayed in a second display mode that is different from the first display mode. In this way, the prediction basis visualizing unit 22 can generate, as the basis image G, an image in which the specific wording described in the answer and the scoring criteria related to the specific wording are displayed in an emphasized manner. By visually checking the evidence image G, the grader P2 can easily determine whether the answer contains wording corresponding to each of the scoring criteria, from (Scoring Criteria 1) to (Scoring Criteria 3). Therefore, the grader P2 can easily determine whether the scoring result predicted by the AI ​​model (AI scoring result) is valid, simply by checking whether the answer contains wording corresponding to the scoring criteria and whether the correspondence between the scoring criteria and the wording is valid.

[0041] Figure 6 shows an example of evidence image G when the scoring result predicted by the AI ​​model (AI scoring result) is "correct." As an answer, an example is shown in which the sentence "Because the positions of the sun, earth, and moon change," as in Figure 4, is written. In the evidence image G in this figure, when an operation is performed to select one of (Grading Criteria 1) to (Grading Criteria 3), the wording related to the selected grading criterion (wording written in the answer) is highlighted. In this figure, a cursor icon is displayed where (Scoring Standard 2) is displayed, indicating that (Scoring Standard 2) has been selected. In this case, the words "Sun, Earth, Moon" in the answer are highlighted in the evidence image G. Note that, although the above description has been given with reference to an example in which a related wording is highlighted in response to the selection of a scoring criterion, the present invention is not limited to this. The prediction basis visualizing unit 22 may generate, as the basis image G, an image in which, in response to the selection of a wording described in an answer, the scoring criterion related to the selected wording is highlighted. That is, the prediction basis visualizing unit 22 generates, as the basis image G, an image in which, in response to the selection of either a specific wording described in the answer or a scoring criterion corresponding to the specific wording, the other is highlighted.

[0042] Figure 7 shows an example of evidence image G when the scoring result predicted by the AI ​​model (AI scoring result) is "incorrect." As an answer, the example shows the sentence "Because the shape of the moon changes," as in Figure 5. In the evidence image G in this figure, as in Figure 6, when an operation is performed to select one of (Grading Criteria 1) to (Grading Criteria 3), the wording related to the selected grading criterion (wording written in the answer) is highlighted. In this figure, a cursor icon is displayed where (Scoring Standard 1) is displayed, indicating that (Scoring Standard 1) has been selected. In this case, in the evidence image G, no part of the answer where any wording is written is highlighted. This indicates that the answer does not contain any wording related to (Scoring Standard 1).

[0043] Here, the flow of processing performed by the information processing system 1 will be described with reference to Fig. 8 and Fig. 9. Fig. 8 and Fig. 9 are sequence diagrams for explaining the flow of processing performed by the information processing system 1 according to the embodiment.

[0044] Figure 8 mainly shows the flow of a series of processes for predicting scoring results using an AI model.

[0045] The question creator P1 registers the marking criteria (step S10). For example, the marking person P2 operates the marking device 10 to store the marking criteria in the marking criteria storage unit 12. The AI ​​model learning unit 17 of the scoring device 10 trains the AI ​​model. The AI ​​model learning unit 17 acquires the scoring criteria (step S11). The AI ​​model learning unit 17 acquires the scoring criteria by referring to the scoring criteria storage unit 12 and reading out the scoring criteria. The AI ​​model learning unit 17 also acquires learning answer data (sentences written as answers used for learning and their scoring results) (step S12). The AI ​​model learning unit 17 acquires learning answer data by referring to the learning answer data storage unit 13 and reading out the learning answer data. The AI ​​model learning unit 17 trains the AI ​​model to learn the correspondence between the acquired scoring criteria, the learning answers, and the scoring results of the learning answers (step S13). When learning into the AI ​​model is completed, the AI ​​model learning unit 17 stores information for constructing the AI ​​model (the AI ​​model for which learning has been completed) in the AI ​​model storage unit 16 (step S14).

[0046] Meanwhile, the AI ​​model prediction unit 15 of the scoring device 10 acquires the scoring criteria (step S15). The AI ​​model prediction unit 15 acquires the scoring criteria by referring to the scoring criteria storage unit 12 and reading out the scoring criteria. The AI ​​model prediction unit 15 also acquires answer data. The AI ​​model prediction unit 15 acquires answer data (sentences written as answers) by referring to the answer data storage unit 11 and reading out the answer data. The AI ​​model prediction unit 15 also acquires (information for constructing) an AI model (step S17). The AI ​​model prediction unit 15 acquires information for constructing an AI model by referring to the AI ​​model storage unit 16. The AI ​​model prediction unit 15 predicts the scoring result of the answer using the AI ​​model (step S18). The AI ​​model prediction unit 15 constructs an AI model using the information acquired in step S17, and inputs the scoring criteria and answer data (sentences written as answers) to the constructed AI model. The AI ​​model predicts the scoring result of the input answer in response to the input scoring criteria and answer data, and outputs the predicted result. The AI ​​model prediction unit 15 stores the predicted result output from the AI ​​model in 14 as the predicted result by the AI ​​model (step S19).

[0047] FIG. 9 mainly shows the flow of a series of processes from generating the basis image G to displaying it.

[0048] The prediction basis extraction unit 21 of the scoring basis visualization device 20 acquires the prediction result by the AI ​​model (step S20). The prediction basis extraction unit 21 acquires the prediction result by the AI ​​model by referring to the prediction result storage unit 14 and reading out the prediction result by the AI ​​model. The prediction basis extraction unit 21 also acquires (information for constructing) the AI ​​model (step S21). The prediction basis extraction unit 21 acquires information for constructing the AI ​​model by referring to the AI ​​model storage unit 16. The prediction basis extraction unit 21 calculates the prediction basis (basis for prediction by the AI ​​model) (step S22). The prediction basis extraction unit 21 calculates, as the prediction basis, for example, the attention level assigned as Attention to words included in the sentence indicated as the answer during the prediction process, and a value (score) indicating the strength of the association between words in the sentence described in the answer. The prediction basis visualization unit 22 of the scoring basis visualization device 20 acquires the prediction basis calculated by the prediction basis extraction unit 21 (step S23). The prediction basis visualization unit 22 uses the acquired prediction basis to generate an image (basis image G) that visualizes the prediction basis (step S24). The prediction basis visualization unit 22 generates, as the basis image G, an image that associates, for example, a word that has attracted attention in the prediction process with a scoring criterion that is highly relevant to that word. The image output unit 23 of the scoring basis visualization device 20 presents the basis image G to the grader P2. The image output unit 23 acquires image information of the basis image G generated by the prediction basis visualization unit 22 (step S25). The image output unit 23 outputs the acquired image information to a display visually recognized by the grader P2 (step S26). The display displays the basis image G. The grader P2 performs final grading of the answer based on the basis image G (step S27). The grader P2 checks the prediction basis shown in the basis image G, for example, the relationship between the focused wording in the prediction of the AI ​​model and the scoring criteria, and if the scoring prediction by the AI ​​model is appropriate, the scoring prediction is used as the final scoring result. On the other hand, if the grader P2 determines that the scoring prediction by the AI ​​model is not appropriate, the grader P2 modifies the scoring prediction by the AI ​​model and uses the modified score as the final scoring result.

[0049] As described above, the information processing system 1 of the embodiment includes an AI model prediction unit 15, a prediction basis extraction unit 21, a prediction basis visualization unit 22, and an image output unit 23 (output unit). The AI ​​model prediction unit 15 predicts the scoring result of a target answer using an AI model that has learned the correspondence between the scoring criteria for written questions and learning answers. The prediction basis extraction unit 21 calculates prediction basis indicating the basis for the prediction by the AI ​​model. The prediction basis visualization unit 22 generates a basis image G in which the prediction basis is associated with the wording written in the target answer. The image output unit 23 outputs image information of the basis image G generated by the prediction basis visualization unit 22. As a result, the information processing system 1 of the embodiment can indicate, in the basis image G, what wording in the answer to the written question was used as the basis for the AI ​​model's determination that the answer is correct (or incorrect), allowing the grader P2 to easily determine whether the scoring by the AI ​​model is appropriate. This can support the task of confirming the appropriateness of the scoring results by AI.

[0050] Furthermore, in the information processing system 1 of the embodiment, the prediction basis extraction unit 21 calculates, as the prediction basis, the degree to which the AI ​​model paid attention to wording described in the target answer in its prediction (score). The prediction basis visualization unit 22 generates, as a basis image G, an image showing, among the wording described in the target answer, wording that the AI ​​model paid attention to in its prediction in association with the scoring criteria. As a result, in the information processing system 1 of the embodiment, it is possible to show, in association with the scoring criteria, wording that the AI ​​model paid attention to in the process of predicting the score, allowing the grader P2 to easily determine whether the score by the AI ​​model is appropriate.

[0051] Furthermore, in the information processing system 1 of the embodiment, the prediction basis visualization unit 22 generates, as the basis image G, an image in which the wording and scoring criteria described in the target answer are displayed, the specific wording described in the target answer and the scoring criteria corresponding to the specific wording are displayed in a first display mode, and the wording described in the target answer that is different from the specific wording and the scoring criteria that do not correspond to the specific wording are displayed in a display mode different from the first display mode. As a result, in the information processing system 1 of the embodiment, the specific wording described in the answer and the scoring criteria corresponding to the specific wording can be highlighted in the basis image G, making it possible for the grader P2 to easily determine whether the scoring by the AI ​​model is appropriate.

[0052] Furthermore, in the information processing system 1 of the embodiment, the prediction basis visualization unit 22 generates an image in which the wording described in the target answer and the scoring criteria are displayed, and in response to the selection of either a specific wording described in the target answer or the scoring criteria corresponding to the specific wording, the other is highlighted, as the basis image G. This allows the information processing system 1 of the embodiment to achieve the same effects as those described above.

[0053] Furthermore, in the information processing system 1 of the embodiment, the prediction basis extraction unit 21 calculates the degree to which the AI ​​model paid attention to wording written in the target answer in its prediction as the prediction basis. The prediction basis visualization unit 22 generates, as a basis image G, an image that highlights the wording written in the target answer that the AI ​​model paid attention to in its prediction. As a result, the information processing system 1 of the embodiment can highlight the wording that the AI ​​model paid attention to in the process of predicting the score, thereby achieving the same effect as the effect described above.

[0054] All or part of the information processing system 1 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may be designed to implement some of the functions described above, or may be capable of implementing the functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0055] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0056] 1. Information processing system 10...Scoring device 11...Answer data storage section 12...Scoring criteria memory section 13...Study answer data storage unit 14...Prediction result memory unit 15…AI model prediction section 16...AI model memory section 17…AI model learning section 20...Scoring basis visualization device 21...Prediction basis extraction unit 22...Prediction Basis Visualization Department 23...Image output unit

Claims

1. an AI model prediction unit that predicts the scoring result of a target answer using an AI model that has learned the correspondence between the scoring criteria for the essay questions and the learning answers; a prediction basis extraction unit that calculates a prediction basis indicating a basis for the prediction by the AI ​​model; a prediction basis visualization unit that generates a basis image in which the prediction basis is associated with a statement written in the target answer; an output unit that outputs image information of the basis image generated by the prediction basis visualization unit; An information processing system comprising:

2. The prediction basis extraction unit calculates the degree to which the AI ​​model paid attention to the wording described in the target answer in its prediction as the prediction basis, The prediction basis visualization unit generates, as the basis image, an image showing a word that the AI ​​model focused on in the prediction among the wordings described in the target answer in association with a scoring criterion. The information processing system according to claim 1 .

3. The prediction basis visualization unit generates, as the basis image, an image in which the wording described in the target answer and the grading criteria are displayed, specific wording described in the target answer and the grading criteria corresponding to the specific wording are displayed in a first display mode, and wording described in the target answer that is different from the specific wording and the grading criteria that does not correspond to the specific wording are displayed in a second display mode that is different from the first display mode. The information processing system according to claim 1 .

4. The prediction basis visualization unit generates, as the basis image, an image in which the wording described in the target answer and the grading criteria are displayed, and in response to selection of either a specific wording described in the target answer or a grading criterion corresponding to the specific wording, the other is highlighted. The information processing system according to claim 1 .

5. The prediction basis extraction unit calculates the degree to which the AI ​​model paid attention to the wording described in the target answer in its prediction as the prediction basis, The prediction basis visualization unit generates, as the basis image, an image that highlights the wording that the AI ​​model focused on in the prediction among the wordings described in the target answer. The information processing system according to claim 1 .

6. An information processing method performed by a computer, comprising: The AI ​​model prediction unit predicts the scoring result of the target answer using an AI model that has learned the correspondence between the scoring criteria for the written questions and the learning answers, A prediction basis extraction unit calculates a prediction basis indicating the basis for the prediction by the AI ​​model, a prediction basis visualization unit that generates a basis image in which the prediction basis is associated with a wording described in the target answer; an output unit that outputs image information of the basis image generated by the prediction basis visualization unit; Information processing methods.

7. On the computer, Using an AI model that has learned the correspondence between the scoring criteria for essay questions and the answers for study, the scoring results for the target answers are predicted. Calculating prediction grounds indicating the grounds for prediction by the AI ​​model; generating a basis image in which the prediction basis is associated with the wording described in the target answer; outputting image information of the generated basis image; program.

Images