Evaluation system, evaluation method, and control program

Abstract

To provide an evaluation system, an evaluation method and a control program capable of quantitatively evaluating the effect of learning.SOLUTION: The evaluation system evaluates the effect of learning. At each time point before and after the learning, questions related to the content of the learning are asked to the subject of the learning. The evaluation system includes an acquisition unit, an evaluation unit, and an output unit. The acquisition unit acquires natural language data indicating an answer to a question. The evaluation unit evaluates the answer to the question by performing natural language processing on the natural language data. The output unit outputs an evaluation result of an answer to a question asked at each time point.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an evaluation system, an evaluation method, and a control program. [Background technology]

[0002] Japanese Patent Publication No. 7058016 (Patent Document 1) discloses a computing device that implements an implicit evaluation framework, in which students are evaluated based on their behavior during a presentation (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7058016 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an evaluation system, an evaluation method, and a control program that can quantitatively evaluate the effectiveness of learning. [Means for solving the problem]

[0005] An evaluation system according to one aspect of the present invention evaluates the effectiveness of learning. Questions related to the content of the learning are asked to the learner at each time point before and after the learning. The evaluation system includes an acquisition unit, an evaluation unit, and an output unit. The acquisition unit acquires natural language data indicating answers to the questions. The evaluation unit evaluates the answers to the questions by performing natural language processing on the natural language data. The output unit outputs evaluation results of the answers to the questions asked at each time point.

[0006] In this evaluation system, answers to questions posed before and after learning are evaluated using natural language processing, and the evaluation results are output. Therefore, this evaluation system makes it possible to compare the evaluation results of the learner before and after learning, making it possible to quantitatively evaluate the effectiveness of learning.

[0007] In the evaluation system, the output unit may output the evaluation results of the answers to the questions asked at each time point in association with information indicating the time point at which the questions were asked.

[0008] In this evaluation system, the evaluation results of answers to questions asked at each point in time are output in association with information indicating the point in time at which the questions were asked. Therefore, this evaluation system allows the user to relatively easily recognize the changes in the evaluation results of the learner before and after learning.

[0009] In the above evaluation system, questions may be asked to the subject of learning before, during and after learning.

[0010] In this evaluation system, questions related to the content of the learning are asked to the learner before, during, and after the learning. Therefore, this evaluation system makes it possible to compare the evaluation results of the learner before, during, and after the learning, allowing for a more detailed evaluation of the learning effect.

[0011] In the above evaluation system, the output unit may output matrix data showing a matrix in which evaluation results of answers to questions asked at each time point are associated with each subject.

[0012] This evaluation system outputs matrix data showing a matrix in which the evaluation results of answers to questions posed at each time point are associated with each subject. Therefore, this evaluation system makes it relatively easy to analyze a group consisting of multiple subjects by processing the matrix data.

[0013] In the evaluation system, the output unit may perform statistical processing on the matrix data and further output the results of the statistical processing.

[0014] In this evaluation system, statistical processing is performed on the matrix data and the results of the statistical processing are output. Therefore, this evaluation system allows the user to relatively easily recognize the analysis results of a group consisting of multiple subjects.

[0015] In the above evaluation system, the natural language processing may include sentence classification analysis, in which the natural language data is classified into one of multiple classes to evaluate the answer, and the multiple classes may be treated as ordinal scale data.

[0016] This evaluation system performs a sentence classification analysis to evaluate the answers to questions before and after learning. Therefore, this evaluation system makes it possible to compare the evaluation results of the learner before and after learning, allowing for a quantitative evaluation of the effectiveness of learning.

[0017] The evaluation system may further include a memory unit that stores sample data of answers to questions, and the natural language processing may include sentence similarity analysis, in which the similarity between the natural language data and the sample data stored in the memory unit is calculated, and the lower the calculated similarity, the higher the evaluation of the answer.

[0018] This evaluation system performs a sentence similarity analysis to evaluate answers to questions before and after learning. Therefore, this evaluation system makes it possible to compare the evaluation results of the learner before and after learning, allowing for a quantitative evaluation of the effectiveness of learning.

[0019] In the above evaluation system, the memory unit may store question data indicating a question, and the evaluation system may further include a control unit that instructs the generative AI model to generate sample data of an answer to a question asked of the learning subject when the question asked of the learning subject is substantially different from the question indicated by the question data, and in the sentence similarity analysis, when the question asked of the learning subject is substantially different from the question indicated by the question data, the similarity between the natural language data and the sample data generated by the generative AI model may be calculated, and the lower the calculated similarity, the higher the evaluation of the answer may be.

[0020] In this evaluation system, when the questions posed to the learner are substantially different from the questions indicated by the question data, the generative AI model generates sample data of answers to the questions posed to the learner. Therefore, according to this evaluation system, sample data is generated according to questions actually posed to the learner, so that even if the questions posed to the learner are substantially different from the questions indicated by the question data, an appropriate evaluation can be made for each answer.

[0021] In the above evaluation system, the output unit may perform dimensionality reduction on the matrix data using the questions as explanatory variables, and output the result of the dimensionality reduction.

[0022] According to this evaluation system, dimensionality reduction is performed on the matrix data using the questions as explanatory variables, and high-dimensional matrix data is converted into low-dimensional matrix data, making it relatively easy to evaluate the position of each subject within a group.

[0023] In the above evaluation system, the output unit may perform dimensionality reduction on the matrix data using the subject as an explanatory variable, and output the result of the dimensionality reduction.

[0024] According to this evaluation system, dimensionality reduction is performed on the matrix data using the subjects as explanatory variables, and high-dimensional matrix data is converted into low-dimensional matrix data, making it relatively easy to evaluate the effectiveness of each question.

[0025] In the evaluation system, the output unit may perform a clustering process on the result of the dimensionality reduction, and further output the result of the clustering process.

[0026] In this evaluation system, clustering processing is performed on the results of dimensionality reduction, and the results of the clustering processing are output. With this evaluation system, by referring to the results of the clustering processing, the user can relatively easily recognize the effect of learning.

[0027] According to another aspect of the present invention, an evaluation method evaluates the effectiveness of learning. Questions related to the content of the learning are asked to a learning subject at each time point before and after the learning. The evaluation method includes acquiring natural language data indicating answers to the questions, evaluating the answers to the questions by performing natural language processing on the natural language data, and outputting evaluation results of the answers to the questions asked at each time point.

[0028] According to this evaluation method, the evaluation results of the learner before and after learning can be compared, so that the effect of learning can be quantitatively evaluated.

[0029] A control program according to another aspect of the present invention causes a computer to execute a process for evaluating the effectiveness of learning. Questions related to the content of the learning are asked to a learner at each time point before and after the learning. The control program causes the computer to execute a process for acquiring natural language data indicating answers to the questions, a process for evaluating the answers to the questions by performing natural language processing on the natural language data, and a process for outputting the evaluation results of the answers to the questions posed at each time point.

[0030] According to this control program, the evaluation results of the learner before and after learning can be compared, so that the effect of learning can be quantitatively evaluated. [Effects of the Invention]

[0031] According to the present invention, it is possible to provide an evaluation system, an evaluation method, and a control program that are capable of quantitatively evaluating the effect of learning. [Brief explanation of the drawings]

[0032] [Figure 1] FIG. 1 is a diagram for explaining an outline of a procedure for evaluating a learning effect in an evaluation system. [Figure 2] FIG. 1 is a diagram schematically illustrating a configuration of an evaluation system. [Figure 3] FIG. 2 is a block diagram illustrating a hardware configuration of an evaluation server. [Figure 4] FIG. 10 is a diagram for explaining an example of a sentence classification analysis using a sentence classification analysis model. [Figure 5] FIG. 10 is a diagram for explaining an example of text similarity analysis using a text similarity analysis model. [Figure 6] FIG. 10 is a diagram for explaining data managed by an evaluation result DB. [Figure 7] FIG. 2 is a block diagram illustrating a hardware configuration of a user terminal. [Figure 8] 10 is a flowchart showing a procedure for evaluating answers of a learner. [Figure 9] 10 is a flowchart showing the procedure of a text data conversion process. [Figure 10] FIG. 2 is a diagram schematically illustrating a first example of a portion of matrix data. [Figure 11] FIG. 10 is a diagram schematically illustrating a second example of a portion of matrix data. [Figure 12] FIG. 10 is a diagram schematically illustrating a third example of a portion of matrix data. [Figure 13] FIG. 10 is a diagram schematically illustrating a first example of the results of principal component analysis. [Figure 14]FIG. 10 is a diagram schematically illustrating a second example of the results of principal component analysis. DETAILED DESCRIPTION OF THE INVENTION

[0033] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described in detail below with reference to the drawings. Note that the same or corresponding parts in the drawings are designated by the same reference numerals, and their description will not be repeated. Furthermore, for ease of understanding, each drawing is drawn schematically with objects appropriately omitted or exaggerated.

[0034] [1. Overview] Many waste disposal facilities have environmental awareness functions such as awareness facilities, for example, for the purpose of raising public awareness of environmental issues. The largest users of these environmental awareness functions are elementary school students. The environmental awareness functions are used during social studies field trips at elementary schools. Conventionally, the learning effect of using the environmental awareness functions has not necessarily been quantitatively evaluated. In the evaluation system 10 (described below) according to this embodiment, the learning effect of using the environmental awareness functions is quantitatively evaluated. The quantitative evaluation results are used, for example, to improve the environmental awareness functions.

[0035] FIG. 1 is a diagram for explaining an outline of the procedure for evaluating the learning effect in the evaluation system 10. Referring to FIG. 1, in a typical elementary school social studies field trip, (1) preparatory learning, (2) facility tour, and (3) wrap-up learning are conducted in this order. Each of the preparatory learning and wrap-up learning is conducted, for example, in an elementary school classroom through a textbook. The facility tour is conducted, for example, by visiting the educational equipment at a waste treatment facility.

[0036] In the evaluation system 10, one or more questions related to the learning content are asked to the learner (hereinafter also referred to as the "learner") at each of the following points: pre-learning, facility tour, and wrap-up learning. The questions asked at each point are the same. The learner's answers at each point are then collected, and a score for each answer is calculated. In the evaluation system 10, for example, the progress of the learner's answer scores throughout the pre-learning, facility tour, and wrap-up learning is presented to the user. Examples of users of the evaluation system 10 include developers, managers, and operators of waste disposal facilities. For example, by checking the difference between the score of the answer during the facility tour and the score of the answer during the pre-learning (score S2 - score S1), the user can quantitatively recognize the instantaneous learning effect of the facility tour. Furthermore, for example, by checking the difference between the score of the answer during the wrap-up learning and the score of the answer during the pre-learning (score S3 - score S1), the user can quantitatively recognize the continuous learning effect of the facility tour. The evaluation system 10 will be described in detail below.

[0037] [2. Configuration] <2-1. Evaluation system configuration> Fig. 2 is a diagram schematically illustrating the configuration of the evaluation system 10. As shown in Fig. 2, the evaluation system 10 includes an evaluation server 100, a user terminal 200, and a response collection terminal 300. In the evaluation system 10, the evaluation server 100, the user terminal 200, and the response collection terminal 300 communicate with each other via the Internet N1.

[0038] The answer collection terminal 300 is configured to collect data (hereinafter also referred to as "answer data") indicating answers of the learner to questions asked at each point in time of the pre-learning, facility tour, and comprehensive learning. The answer data is data indicating natural language (hereinafter also referred to as "natural language data"). The answer collection terminal 300 is configured, for example, by a PC (Personal Computer), a tablet, or a smartphone.

[0039] The answer collection terminal 300 collects, for example, voice data, text data, and image data indicating answers from the learner. The answer collection terminal 300 may collect voice data by generating voice data based on voice captured through a microphone, or may collect voice data from a voice recorder external to the answer collection terminal 300. The answer collection terminal 300 may also collect text data from the learner's device (e.g., a PC, tablet, or smartphone). The answer collection terminal 300 may also collect image data by, for example, photographing an answer sheet with a camera.

[0040] The answer data collected by the answer collection terminal 300 is transmitted to the assessment server 100 via the Internet N1. If the answer data is voice data or image data, the answer data is converted into text data in the assessment server 100. In the assessment server 100, natural language processing is performed on each answer data (text data) to derive a score for each answer data. In the present embodiment, each answer data is evaluated by deriving a score for each answer data, but the method for evaluating each answer data is not limited to this. Each answer data may be evaluated, for example, by a scale rating that assigns AE or 1-5 to each answer data. Data indicating the score for each answer data (hereinafter also referred to as "score data") is transmitted to the user terminal 200 via the Internet N1. In the user terminal 200, the score for each answer is presented to the user. For example, by comparing the scores at each point in time, the user can quantitatively evaluate the learning effect through the environmental awareness function of the waste disposal facility.

[0041] <2-2. Evaluation Server Configuration> Fig. 3 is a block diagram showing a schematic hardware configuration of the assessment server 100. The assessment server 100 is realized by, for example, a general-purpose computer. As shown in Fig. 3, the assessment server 100 includes a control unit 110, a communication I / F (interface) 130, and a storage unit 120. Each component is electrically connected via a bus.

[0042] The control unit 110 includes a central processing unit (CPU) 112, a random access memory (RAM) 114, a read only memory (ROM) 116, and the like, and is configured to control each component in accordance with information processing.

[0043] The communication I / F 130 is configured to communicate with the user terminal 200 and the response collection terminal 300 (FIG. 1) via the Internet N1. The communication I / F 130 is configured, for example, with a wired LAN (Local Area Network) module or a wireless LAN module.

[0044] The storage unit 120 is configured with an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 120 stores, for example, a control program 122, a sentence classification analysis model 124, a sentence similarity analysis model 126, and a DB (database) 128 for evaluation results, etc. The control program 122 is executed by the CPU 112, thereby realizing various functions of the evaluation server 100.

[0045] The text classification analysis model 124 is configured to perform text classification analysis on natural language data. For example, in response to input text data indicating a learner's response, the text classification analysis model 124 classifies the text data into one of multiple classification classes. The multiple classification classes are treated as ordinal scale data. For example, the text classification analysis model 124 classifies the text data into one of 11 classification classes ranging from dislike (0 points) to like (10 points). The text classification analysis model 124 classifies the text data, deriving a score for the text data. The text classification analysis model 124 is generated, for example, through machine learning. Various well-known machine learning methods, such as neural networks, deep learning, decision tree learning, association rule learning, and Bayesian networks, can be applied as the text classification analysis model 124. Furthermore, for example, a trained large-scale language model, or a language model fine-tuned based on the learner's attributes and learning content, can be applied as the text classification analysis model 124.

[0046] FIG. 4 is a diagram illustrating an example of text classification analysis by the text classification analysis model 124. Referring to FIG. 4, in this example, the subject is asked the following question before, during, and after a tour of a waste treatment facility: "What is your impression of waste incineration facilities?" Before the tour, the subject answers, "They're smelly and dirty. I don't want one near my house." During the tour, the subject answers, "They're necessary, but I don't want one near my house." After the tour, the subject answers, "I'd like to help improve local waste disposal." The text classification analysis model 124 classifies the responses before, during, and after the tour into classification classes of 3, 7, and 8 points, respectively.

[0047] Referring again to FIG. 3 , the sentence similarity analysis model 126 is configured to perform sentence similarity analysis on natural language data. A plurality of sample data (not shown) is stored in the storage unit 120. The plurality of sample data are samples of answer data. In response to input of text data indicating answers from a learner, the sentence similarity analysis model 126 analyzes the similarity between the text data and each of the plurality of sample data, and derives a score for the text data based on the analysis results. The sentence similarity analysis model 126 is generated, for example, through machine learning. Various well-known methods such as neural networks, deep learning, decision tree learning, association rule learning, and Bayesian networks can be applied as machine learning. Furthermore, for example, a trained large-scale language model can be applied as the sentence similarity analysis model 126, and a language model that has been fine-tuned based on the attributes of the learner and the learning content can be applied.

[0048] When a learner who has visited a waste disposal facility is asked what he or she would like to do in the future, the learner may reflexively answer only morally correct things such as "reduce waste," "recycle," or "not litter." In such cases, it is highly likely that the true educational goal of helping the learner understand how to be involved in waste disposal as a citizen has not been achieved. In this embodiment, each of the multiple sample data indicates an answer that remains morally correct. In the text similarity analysis model 126, the similarity between the text data indicating the learner's answer and each of the multiple sample data is calculated, and the lower the similarity, the higher the score of the answer is derived.

[0049] FIG. 5 is a diagram illustrating an example of text similarity analysis by the text similarity analysis model 126. Referring to FIG. 5, in this example, contrasting texts SE1, SE2, and SE3 are provided as sample data. For example, contrasting text SE1 is a text saying, "I want to reduce waste." Contrasting text SE2 is a text saying, "I will recycle." Contrasting text SE3 is a text saying, "I will try my best to separate waste." Answer A1 of the learner is a text saying, "I wanted to reduce waste." Answer A2 of the learner is a text saying, "I wanted to separate waste." Answer A3 of the learner is a text saying, "I wanted to separate waste so that the energy generated by burning the waste could be used efficiently."

[0050] Answer A1 has a high similarity to contrasting sentence SE1 and a low similarity to contrasting sentences SE2 and SE3. Answer A2 has a low similarity to contrasting sentences SE1 and SE2 and a high similarity to contrasting sentence SE3. Answer A3 has a low similarity to each of contrasting sentences SE1, SE2, and SE3. The text similarity analysis model 126 assigns 1 point, 1 point, and 3 points to answers A1, A2, and A3, respectively. In other words, the text similarity analysis model 126 assigns the highest score to answer A3, which has a low similarity to each of contrasting sentences SE1, SE2, and SE3.

[0051] 3 again, the evaluation result DB 128 manages answer data of learners and data indicating the evaluation results (scores) of the answer data (hereinafter also referred to as "evaluation result data"). The details will be described later, but the evaluation result data is, for example, matrix data indicating a matrix in which the scores of answers to questions asked at each point in time are associated with each learner.

[0052] Fig. 6 is a diagram for explaining data managed by the evaluation result etc. DB 128. Referring to Fig. 6, the evaluation result etc. DB 128 manages response data and evaluation result data. In the evaluation result etc. DB 128, the response data and evaluation result data are managed for each facility to be toured, by group that toured the facility. In this example, the response data and evaluation result data of each of the groups AG1, AG2, and AG3 that toured facility F1 are managed so as to be distinguishable from one another, and the response data and evaluation result data of each of the groups AG4, AG5, and AG6 that toured facility F2 are managed so as to be distinguishable from one another.

[0053] <2-3. User terminal configuration> Fig. 7 is a block diagram schematically showing the hardware configuration of user terminal 200. User terminal 200 is realized by, for example, a PC, a tablet, or a smartphone. As shown in Fig. 7, user terminal 200 includes a control unit 210, a communication I / F 230, an operation unit 240, a display 250, and a storage unit 220. In user terminal 200, each component is electrically connected via a bus.

[0054] The control unit 210 includes a CPU, RAM, ROM, etc., and is configured to control each component in accordance with information processing. The communication I / F 230 is configured to communicate with the evaluation server 100 via the Internet N1. The communication I / F 230 is configured, for example, by a wired LAN module or a wireless LAN module.

[0055] The operation unit 240 is configured to receive input from a user. The operation unit 240 is configured, for example, to include some or all of a touch panel, a keyboard, a mouse, and a microphone. The display 250 is configured to display an image. The display 250 is configured, for example, to include a monitor such as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor. The display 250 displays, for example, an image showing evaluation result data.

[0056] The storage unit 220 is, for example, an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 220 stores, for example, a control program 222. When the control program 222 is executed by the CPU of the control unit 210, various functions of the user terminal 200 are realized.

[0057] [3. Operation] 8 is a flowchart showing the procedure for evaluating the answers of the learner. The process shown in this flowchart is executed by the control unit 110 of the evaluation server 100.

[0058] 8, control unit 110 determines whether or not answer data to questions at each time point before, during, and after the tour of the waste treatment facility has been received (step S100). If it is determined that answer data to questions at each time point has not been received (NO in step S100), control unit 110 executes the process of step S100 again.

[0059] On the other hand, when it is determined that answer data to the question at each time point has been received (YES in step S100), control unit 110 converts the answer data into text data according to the data format of the answer data (step S110). Specifically, when the answer data is voice data or image data, control unit 110 converts the answer data into text data.

[0060] 9 is a flowchart showing the procedure of the text data conversion process. The process shown in this flowchart is executed by the control unit 110 of the evaluation server 100 in step S110 of FIG.

[0061] 9, control unit 110 determines whether the received answer data is voice data (step S200). If it is determined that the received answer data is voice data (YES in step S200), control unit 110 converts the voice data into text data by performing a transcription process on the voice data (step S210). For the transcription process, various known techniques such as natural language processing are used.

[0062] On the other hand, if it is determined that the received answer data is not voice data (NO in step S200), control unit 110 determines whether the received answer data is image data (step S220).If it is determined that the received answer data is image data (YES in step S220), control unit 110 converts the image data into text data by executing OCR (Optical Character Recognition / Reader) processing (step S230).

[0063] If the received answer data is voice data or image data, the control unit 110 controls the memory unit 120 to store the converted text data as answer data, and if the received answer data is text data, the control unit 110 controls the memory unit 120 to store the received text data as answer data (step S240).

[0064] Control unit 110 determines whether processing has been completed for all of the received answer data (step S250). If it is determined that processing has not been completed for at least some of the received answer data (NO in step S250), control unit 110 executes the processing of step S200 again for the answer data for which processing has not been completed. On the other hand, if it is determined that processing has been completed for all of the received answer data (YES in step S250), the processing shown in this flowchart ends.

[0065] 8 again, when the text data conversion process is completed, the control unit 110 executes a score derivation process for each piece of answer data (step S120). Specifically, the control unit 110 executes a process of deriving a score for each piece of answer data by inputting each piece of answer data into the sentence classification analysis model 124, and a process of deriving a score for each piece of answer data by inputting each piece of answer data into the sentence similarity analysis model 126. Through the score derivation process, a score for each answer of each learner is derived.

[0066] The control unit 110 generates matrix data (evaluation result data) showing a matrix in which the scores of answers to questions asked at each time point are associated with each learning subject (step S130). The control unit 110 generates, for example, multiple types of matrix data.

[0067] FIG. 10 is a diagram schematically illustrating a first example of a portion of matrix data. Referring to FIG. 10, each of the symbols P1-P5 indicates a learner ID. The suffixes "X," "Y," and "Z" in the symbols E1X-E4X, E1Y-E3Y, and E1Z-E2Z correspond to time flags. The time flags X, Y, and Z correspond to before, during, and after the tour, respectively. The symbols E1X-E4X, E1Y-E3Y, and E1Z-E2Z, "E1," "E2," "E3," and "E4," indicate evaluation methods. The evaluation methods E1-E4 differ from one another in terms of analysis methods (sentence classification analysis or sentence similarity analysis) or question content. In matrix MA1, the scores of each learner are managed for each evaluation method.

[0068] Fig. 11 is a diagram schematically illustrating a second example of a portion of the matrix data. Referring to Fig. 11, in matrix MA2, instead of the time point flags in matrix MA1, group average scores are managed for each assessment method. For example, by referring to matrix MA2, the user can recognize the average scores for each assessment method in the group made up of learning subjects P1-P5.

[0069] Fig. 12 is a diagram schematically illustrating a third example of a portion of the matrix data. Referring to Fig. 12, a matrix MA3 is a transposed matrix of the matrix MA1 (Fig. 10). Control unit 110 generates, for example, matrices MA1, MA2, and MA3.

[0070] 8, once the various matrix data have been generated, the control unit 110 performs a principal component analysis process on each of the matrix data generated in step S130 (step S140). The control unit 110, for example, converts high-dimensional matrix data into two-dimensional matrix data through the principal component analysis process. The control unit 110 may, for example, perform the principal component analysis process on the entire matrix data, or may perform the principal component analysis process on a portion of the matrix data.

[0071] FIG. 13 is a diagram schematically illustrating a first example of the results of principal component analysis. Referring to FIG. 13, the results show the results of principal component analysis on data included in matrix MA1 shown in FIG. 10 and assigned with "X" as the time point flag. In this principal component analysis, the evaluation method, such as questions, is used as an explanatory variable. The horizontal axis represents the first principal component, and the vertical axis represents the second principal component. For example, by referring to the results of this principal component analysis, the user can recognize the position of each learner in the group prior to the tour.

[0072] FIG. 14 is a diagram illustrating a second example of the results of principal component analysis. Referring to FIG. 14, this result shows the results of principal component analysis on the data assigned "E1" and "E2" as the evaluation methods among the data included in matrix MA3 shown in FIG. 12. In this principal component analysis, the learner is used as the explanatory variable. The horizontal axis represents the first principal component, and the vertical axis represents the second principal component. By referring to the results of this principal component analysis, for example, the user can recognize the progress of the scores for evaluation methods E1 and E2 before, during, and after the tour. Furthermore, for example, an evaluation method that results in almost no change in scores before, during, and after the tour is an inappropriate evaluation method from the perspective of evaluating the learning effect of a facility tour. By referring to the results of this principal component analysis, for example, the user can identify inappropriate evaluation methods.

[0073] 8, when the principal component analysis process is completed, the control unit 110 executes a clustering process on the results of the principal component analysis process as necessary (step S150). The control unit 110 performs the desired clustering by using a clustering method such as the K-means method. For example, in FIG. 13, the results of the principal component analysis are clustered into a "high score group" and a "low score group."

[0074] When the clustering process is completed, the control unit 110 controls the storage unit 120 to store the matrix data (evaluation result data) generated in step S130, the result data of the principal component analysis process, the result data of the clustering process, etc. (hereinafter also referred to as "evaluation result data, etc."), and also controls the communication I / F 130 to transmit the evaluation result data, etc. to the user terminal 200 (step S160). When the evaluation result data, etc. is received, the control unit 210 of the user terminal 200 controls the display 250 to display a screen showing, for example, some or all of the data included in the evaluation result data, etc. By referring to the screen, the user can recognize the learning effect of visiting the waste treatment facility.

[0075] [4. Features] As described above, in evaluation system 10 according to the present embodiment, scores for answers to questions before and after learning are derived through natural language processing, and the derived scores are output. Therefore, evaluation system 10 makes it possible to compare the scores of a learner before and after learning, for example, and therefore makes it possible to quantitatively evaluate the effectiveness of learning.

[0076] Furthermore, in the evaluation system 10 according to the present embodiment, the score of the answer to the question asked at each time point is output in association with information indicating the time point at which the question was asked (time point flag). Therefore, with the evaluation system 10, for example, the user can relatively easily recognize the change in the student's score before and after the study.

[0077] Furthermore, in the evaluation system 10 according to the present embodiment, questions related to the content of the learning are asked to the learner before, during, and after the learning. Therefore, the evaluation system 10 makes it possible to compare the scores of the learner before, during, and after the learning, for example, and therefore makes it possible to evaluate the effectiveness of the learning in more detail.

[0078] Furthermore, evaluation system 10 according to the present embodiment outputs matrix data showing a matrix in which the scores of answers to questions asked at each time point are associated with each learner. Therefore, evaluation system 10 allows, for example, analysis of a group made up of multiple learners to be carried out relatively easily by processing the matrix data.

[0079] Furthermore, in the evaluation system 10 according to the present embodiment, statistical processing (e.g., calculation of an average score) is performed on the matrix data, and the results of the statistical processing are output. Therefore, with the evaluation system 10, for example, a user can relatively easily recognize the analysis results of a group made up of multiple learners.

[0080] Furthermore, in evaluation system 10 according to the present embodiment, clustering processing is performed on the results of the principal component analysis, and the results of the clustering processing are output. With evaluation system 10, by referring to the results of the clustering processing, for example, the user can relatively easily recognize the effect of learning.

[0081] 5. Other Embodiments The concept of the above embodiment is not limited to the embodiment described above. Hereinafter, examples of other embodiments to which the concept of the above embodiment can be applied will be described.

[0082] <5-1> In the above embodiment, the evaluation system 10 is used to evaluate the learning effect through a tour of a waste treatment facility. However, the subject evaluated by the evaluation system 10 is not limited to the learning effect through a tour of a waste treatment facility. The evaluation system 10 may also be used, for example, to evaluate the learning effect through tours of various factories, the learning effect through in-house safety training, or the learning effect through instruction at a cram school.

[0083] <5-2> Furthermore, in the above embodiment, the evaluation result data etc. of a specific group that visited a specific waste disposal facility was transmitted from the assessment server 100 to the user terminal 200. However, the data transmitted from the assessment server 100 to the user terminal 200 is not limited to this. In addition to the above data, evaluation result data etc. of other groups that visited other waste disposal facilities may also be transmitted from the assessment server 100 to the user terminal 200. For example, by comparing the evaluation result data etc. of a specific group that visited a specific waste disposal facility with the evaluation result data etc. of other groups that visited other waste disposal facilities, problems with the environmental awareness function of each waste disposal facility may become clear.

[0084] <5-3> In the evaluation system 10 according to the above embodiment, the evaluation server 100 may be realized by a plurality of servers.

[0085] <5-4> In the above embodiment, principal component analysis is used to reduce the dimension of the matrix data. However, the method used to reduce the dimension of the matrix data is not limited to this. For example, principal coordinate analysis, factor analysis, or T-sne may be used instead of principal component analysis.

[0086] <5-5> Furthermore, in the above embodiment, the sample data of the contrasting sentences used in the sentence similarity analysis is always sample data that is pre-stored in the storage unit 120. However, the sample data of the contrasting sentences used in the sentence similarity analysis does not necessarily have to be sample data that is pre-stored in the storage unit 120.

[0087] For example, it is assumed that the sample data pre-stored in the storage unit 120 is sample data (a list of undesirable answers) for the question "Why do we dispose of garbage?" In this case, for example, question data indicating the question "Why do we dispose of garbage?" is pre-stored in the storage unit 120. An example of the sample data in this case is an answer list (hereinafter also referred to as "answer list A") including the following information:

[0088] ·I don't know Because it smells Because it's dirty ·somehow Because it's better that way Because that's what's decided

[0089] In such cases, in actual learning situations, it is conceivable that a question slightly different from "Why do we dispose of garbage?" would be asked. For example, it is conceivable that a question such as "Why do we throw away garbage?" would be asked. Sample data for the question "Why do we throw away garbage?" would not necessarily match Answer List A. An example of sample data for the question "Why do we throw away garbage?" would be an answer list containing the following information (hereinafter also referred to as "Answer List B").

[0090] ·I don't know Because it's in the way -It takes up space Because it's a house rule Because adults say so · Because I don't need it

[0091] For example, if a question different from the question indicated by the question data pre-stored in the storage unit 120 is asked in the actual learning situation, the text similarity analysis should use sample data different from the sample data pre-stored in the storage unit 120. In the above example, the text similarity analysis should calculate the similarity between the natural language data indicating the answer to the question asked in the actual learning situation and each answer included in the answer list B.

[0092] For example, when a question different from the question indicated by the question data pre-stored in the storage unit 120 is asked in the actual learning situation, the control unit 110 of the assessment server 100 may instruct the generative AI model to generate sample data for the question asked in the actual learning situation, based on the question data and sample data stored in the storage unit 120. The control unit 110 may, for example, control the communication I / F 130 to send a prompt indicating such an instruction to the generative AI model.

[0093] For example, the control unit 110 transmits to the generation AI model a prompt including a question such as, "Possible answers to the question 'Why do we dispose of garbage?' include 'I don't know,' 'Because it smells,' 'Because it's dirty,' 'Just because,' 'Because it's better,' and 'Because that's the way it is.' What are some possible answers to the question 'Why do we throw out garbage?'?" This prompt is generated by storing a prompt template in the storage unit 120, such as, "Possible answers to the question 'Why do we dispose of garbage?' include 'I don't know,' 'Because it smells,' 'Because it's dirty,' 'Just because,' 'Because that's better,' and 'Because that's the way it is.' What are some possible answers to the question '#Actual Question'?" The prompt is generated by filling in the '#Actual Question' portion of the template with an actual question. The prompt is generated, for example, by the control unit 110.

[0094] The generative AI model is generated through learning using information periodically acquired from various websites via the Internet. The generative AI model is configured to accept input of natural language and generate various answers corresponding to the natural language based on common sense. Examples of generative AI models include ChatGPT and Marvin.

[0095] In this case, the control unit 110 may calculate the similarity between natural language data indicating answers to questions asked in an actual learning situation and the sample data generated by the generative AI model. The lower the calculated similarity, the higher the score derived for the answer. According to this evaluation system, sample data is generated according to questions actually asked to the learning subject, so that even if the questions asked to the learning subject are substantially different from the questions indicated by the question data pre-stored in the storage unit 120, an appropriate score can be derived for each answer.

[0096] In addition, the control unit 110 may perform fine tuning by first having the generative AI model learn the differences between the questions indicated by the question data stored in the memory unit 120 and the questions asked in the actual learning environment, and then instruct the generative AI model to generate sample data for the questions asked in the actual learning environment based on the question data and sample data stored in the memory unit 120.

[0097] The above describes exemplary embodiments of the present invention. That is, the detailed description and the accompanying drawings are disclosed for the purpose of illustrative explanation. Therefore, some of the components described in the detailed description and the accompanying drawings may be non-essential components for solving the problems. Therefore, just because these non-essential components are described in the detailed description and the accompanying drawings, it should not be immediately recognized that these non-essential components are essential.

[0098] Furthermore, the above-described embodiments are merely illustrative of the present invention in all respects. Various improvements and modifications to the above-described embodiments are possible within the scope of the present invention. For example, at least a portion of the configuration of any of the embodiments may be combined with at least a portion of the configuration of any of the other embodiments. In other words, when implementing the present invention, specific configurations can be appropriately adopted depending on the embodiment. [Explanation of symbols]

[0099] 10 Evaluation system, 100 Evaluation server, 110, 210 Control unit, 112 CPU, 114 RAM, 116 ROM, 120, 220 Memory unit, 122, 222 Control program, 124 Text classification analysis model, 126 Text similarity analysis model, 128 Evaluation result DB, etc., 130, 230 Communication I / F, 200 User terminal, 240 Operation unit, 250 Display, 300 Response collection terminal.

Claims

1. An evaluation system for evaluating the effectiveness of learning, questions related to the content of the learning are asked to the subject of the learning at each time point before and after the learning; an acquisition unit that acquires natural language data indicating an answer to the question; an evaluation unit that evaluates an answer to the question by performing natural language processing on the natural language data; an output unit that outputs evaluation results of the answers to the questions made at each of the points in time.

2. The evaluation system according to claim 1 , wherein the output unit outputs the evaluation results of the answers to the questions asked at each time point in association with information indicating the time point at which the questions were asked.

3. 3. The evaluation system according to claim 1, wherein the questions are asked to the subject of the learning before, during, and after the learning.

4. The evaluation system according to claim 1 , wherein the output unit outputs matrix data indicating a matrix in which evaluation results of answers to the questions given at each of the time points are associated with each of the subjects.

5. The evaluation system according to claim 4 , wherein the output unit performs statistical processing on the matrix data and further outputs a result of the statistical processing.

6. The natural language processing includes text classification analysis; In the sentence classification analysis, the answer is evaluated by classifying the natural language data into one of a plurality of classes; 3. The evaluation system according to claim 1, wherein the plurality of classes are treated as ordinal scale data.

7. a storage unit for storing sample data of answers to the questions; The natural language processing includes a sentence similarity analysis, 3. The evaluation system according to claim 1, wherein the sentence similarity analysis calculates a similarity between the natural language data and the sample data stored in the storage unit, and the lower the calculated similarity, the higher the evaluation of the answer.

8. the storage unit stores question data indicating the question; A control unit is further provided that, when a question posed to the learning subject is substantially different from a question indicated by the question data, instructs the generation AI model to generate sample data of an answer to the question posed to the learning subject; 8. The evaluation system of claim 7, wherein, in the text similarity analysis, if a question posed to the learning subject is substantially different from a question indicated by the question data, a similarity between the natural language data and sample data generated by the generative AI model is calculated, and the lower the calculated similarity, the higher the evaluation of the answer.

9. 5. The evaluation system according to claim 4, wherein the output unit performs dimensionality reduction on the matrix data using the questions as explanatory variables, and outputs the result of the dimensionality reduction.

10. The evaluation system according to claim 4 , wherein the output unit performs dimensionality reduction on the matrix data using the subject as an explanatory variable, and outputs the result of the dimensionality reduction.

11. 11. The evaluation system according to claim 9, wherein the output unit performs a clustering process on the result of the dimensionality reduction and further outputs the result of the clustering process.

12. An evaluation method for evaluating the effectiveness of learning, comprising: questions related to the content of the learning are asked to the subject of the learning at each time point before and after the learning; obtaining natural language data indicating an answer to the question; evaluating an answer to the question by performing natural language processing on the natural language data; and outputting evaluation results of the answers to the questions made at each of the points in time.

13. A control program for causing a computer to execute a process for evaluating the effectiveness of learning, questions related to the content of the learning are asked to the subject of the learning at each time point before and after the learning; obtaining natural language data indicating an answer to the question; evaluating an answer to the question by performing natural language processing on the natural language data; and outputting evaluation results of the answers to the questions made at each point in time.

Images