Educational support systems, educational support methods, and programs

Abstract

To provide an automatic scoring program, server, automatic scoring method, and automatic scoring system, which enable easy checking of automatic scoring results.SOLUTION: An automatic scoring program disclosed herein causes a processor to perform steps of receiving input of information including answers to questions, scoring the answers according to one or more evaluation items preset for the questions, and displaying scoring results in different ways according to evaluation items matching the answers.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to Educational support systems, educational support methods, and programs .

Background Art

[0002] In recent years, online classes have been introduced. In such online classes, tests and the like are also conducted online. Systems that can perform automatic scoring in such online tests and the like have also been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to improve the accuracy of automatic scoring, it is desirable that the final scoring result be confirmed by a person. stomach.

[0005] The present invention An educational support system, educational support method, and program that allows setting an answer sheet suitable for grading or answering. aims to provide

Means for Solving the Problems

[0006] The education support device according to the present invention includes setting means for setting an electronic answer sheet on which a slip for a respondent to write an answer to a question is arranged. When the type of answer type for each of a plurality of questions is selected by the questioner, the setting means sets the answer sheet so that the arrangement of the slips corresponding to the unselected answer types on the answer sheet by the respondent is prohibited. Furthermore, the types of the aforementioned answers include mathematical formulas, graphs, or tables. It is characterized by this.

[0007] The educational support method according to the present invention is an educational support method implemented by an educational support system, which sets up an electronic answer sheet on which sticky notes are placed for the respondent to write their answers to the questions. Includes a setting process, and the setting process The answer sheet is configured such that, when the question setter selects a type of answer for each of the multiple questions, the answerer is prohibited from placing sticky notes on the answer sheet corresponding to the answer types that were not selected. Furthermore, the types of the aforementioned answers include mathematical formulas, graphs, or tables. It is characterized by the following:

[0008] The program according to the present invention causes a computer to function as a setting means for setting up an electronic answer sheet on which sticky notes are placed for respondents to write their answers to questions. The setting means sets up the answer sheet such that, when the type of answer to a question is selected by the question setter for each of a plurality of questions, the respondent is prohibited from placing sticky notes on the answer sheet corresponding to the type of answer that has not been selected. Furthermore, the types of the aforementioned answers include mathematical formulas, graphs, or tables. It is characterized by the following: [Effects of the Invention]

[0010] According to the present invention, An educational support system, educational support method, and program that allows setting an answer sheet suitable for grading or answering. We can provide this. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows an example of the system configuration according to one embodiment of the present invention. [Figure 2] Figure 2 shows an example of a screen display by an application running on the web browser of terminal 20a. [Figure 3] Figure 3 is a sequence diagram showing an example of the system's operation when creating answer sheets for exams, etc. [Figure 4A] Figure 4A shows an example of a dialog box that appears on paper. [Figure 4B] Figure 4B shows an example of a dialog box that appears on paper. [Figure 4C]FIG. 4C is a diagram showing an example of a dialog displayed on paper. [Figure 5] FIG. 5 is a sequence diagram showing an example of the operation of the system when answering questions. [Figure 6] FIG. 6 is a diagram showing an example of the display screen of terminal <* [Figure 7] FIG. 7 is a sequence diagram showing an example of the operation during scoring. [Figure 8] FIG. 8 is a flowchart showing an example of the server processing during automatic scoring. [Figure 9] FIG. 9 is a diagram showing an example of the display screen of terminal 20a when the paper with the automatic scoring result is displayed. [Figure 10] FIG. 10 is a diagram showing an example of the display screen of terminal 20a when the paper with the automatic scoring result in English is displayed.

MODE FOR CARRYING OUT THE INVENTION

[0012] An embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a system 1 according to an embodiment of the present invention. The system 1 includes a server 10 and terminals 20a and 20b. The server 10 and the terminals 20a and 20b are communicably connected via a network 30. The network 30 is, for example, the Internet. The number of terminals is not limited to two. There may be at least one terminal 20a and at least one terminal 20b.

[0013] The server 10 has a processor 11, a ROM 12, a RAM 13, a storage 14, and a communication device 15. Each of these is connected to each other via a system bus 19.

[0014] The processor 11 may be an integrated circuit including a CPU (Central Processing Unit) or the like. The ROM 12 records information used for the operation of the processor 11 or the like. The RAM 13 is the main memory device for the operation of the processor 11 or the like. The storage 14 stores various programs such as a server control program used by the processor 11, arithmetic programs for executing various operations, and parameters. The server control program includes an automatic scoring program. The processor 11 controls the operation of the server 10 according to the programs stored in the storage 14. As the processor 11, a processor other than the CPU, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a GPU (Graphic Processing unit), or the like may be used. The communication device 15 includes a circuit for communicating with an external communication network such as the network 30.

[0015] The terminals 20a and 20b may be electronic devices such as personal computers (PCs), tablet terminals, smartphones, or the like. Also, the terminals 20a and 20b may be function calculators equipped with a communication function. The terminal 20a is a terminal operated by a grader for an exam such as a teacher. The terminal 20b is a terminal operated by an examinee for an exam such as a student. Hereinafter, assuming that the terminal 20a and the terminal 20b have the same configuration, the configuration of the terminal 20a will be described. Also, hereinafter, the grader may be distinguished as the first user and the examinee may be distinguished as the second user as necessary.

[0016] The terminal 20a has a CPU 21, a ROM 22, a RAM 23, a storage 24, an input device 25, a display device 26, and a communication device 27. Each of these is connected to each other via a system bus 29. Note that the terminal 20a and the terminal 20b do not necessarily have the same configuration.

[0017] The CPU 21 is a processor that controls various operations of the terminal 20a. The ROM 22 stores the startup program for the terminal 20, etc. The RAM 23 is the main memory for the CPU 21 and other components. The storage 24 stores various programs, parameters, etc., such as the terminal control program used by the CPU 21. The CPU 21 controls the operation of the terminal 20a by executing various programs in response to input signals from the input device 25 and received signals from the communication device 27. Various programs may be downloaded to the storage 24 from a web server (not shown) via the network 30 and the communication device 27. The communication device 27 includes a circuit for communicating with an external communication network such as the network 30.

[0018] The input device 25 includes a keyboard, mouse, touch panel, etc. In response to user operations via the input device 25, signals indicating the content of those user operations are input to the CPU 21 via the system bus 29.

[0019] The display device 26 is a liquid crystal display, an organic EL display, or the like. The display device 26 may be integrated with the terminal 20a or may be provided separately from the terminal 20a. Various images are displayed on the display device 26.

[0020] In one example, the user specifies the address of server 10 in a web browser running on terminal 20a. At this time, a display screen for the web application stored on server 10 is displayed on the web browser on terminal 20a. A request is issued to server 10 in response to operations performed via the input device 25 on this display screen. These operations include, for example, operations related to grading an exam taken by a user on terminal 20b. Server 10 executes processing in response to the request and returns the processing result as a response to terminal 20a. Terminal 20a displays information in accordance with the user's operations according to the response from server 10. In this way, in system 1, the functionality of a web application for exams, etc., is realized by a program running on the web browser of terminal 20a and a calculation program on server 10. Similarly, in system 1, the functionality of a web application for exams, etc., is realized by a program running on the web browser of terminal 20b and a calculation program on server 10.

[0021] Such web applications could be used, for example, in mathematics classes in schools where ICT (Information and Communication Technology) is becoming increasingly prevalent.

[0022] Figure 2 shows an example of a screen display by an application running on the web browser of terminal 20a.

[0023] The screen 26a displayed on the display device 26 of terminal 20a includes an upper area 100a and a lower area 100b. The upper area 100a is displayed at the top of screen 26a. The upper area 100a is a longer and narrower area than the lower area 100b. The upper area 100a displays a new paper creation icon 100c. The upper area 100a also displays an answer sheet creation button 100d. The lower area 100b is an area located below the upper area 100a on screen 26a. In the following description, the lower area 100b will also be referred to as paper 100. Various "sticky notes" 101 may be displayed on paper 100. Sticky notes 101 are display areas for displaying various information related to the web application. For example, sticky note 101 includes formula sticky notes for creating mathematical formulas, graph sticky notes for creating graphs, table sticky notes for creating tables, shape sticky notes for creating shapes, and comment sticky notes for creating comments. Sticky note 101 may be a floating object. A floating object is an object (display) that is displayed on the screen and whose display position can be changed in response to user operations.

[0024] In this embodiment, System 1 can create various sticky notes starting from a blank sheet of paper 100. Note that the same sheet of paper 100 as in Terminal 20a may also be displayed on Terminal 20b. However, the answer sheet creation button 100d does not need to be displayed on the sheet of paper 100 on Terminal 20b.

[0025] The following describes the sequence of processes in System 1. Figure 3 is a sequence diagram showing an example of the operation of System 1 when creating answer sheets for exams, etc. Here, it is assumed that prior to the process in Figure 3, the questions and scoring criteria for the questions have been created by a first user, such as a teacher. The scoring criteria for the questions are stored in Server 10 as a scoring criteria information file. The scoring criteria information file is a file that contains the scoring criteria information as text data in formats such as JSON or XML. The following description will explain an example of creating answer sheets for mathematics.

[0026] The scoring criteria information includes, for example, "question name," "answer type," "model answer," "scoring criteria," and "comments." "Question name," "answer type," "model answer," "scoring criteria," and "comments" can be stored for each question. For example, if there are three questions, three sets of "question name," "answer type," "model answer," "scoring criteria," and "comments" can be stored for each question.

[0027] The "Question Name" is information that represents the name of a question. The "Question Name" is unique to each question within the same exam. The "Question Name" may be entered by the first user as text, or it may be selected by the first user from several options.

[0028] The "Answer Type" setting indicates the type of sticky note that can be used to answer the corresponding question. For example, if the "Answer Type" for a mathematics question is "Mathematical Formula," then only sticky notes containing mathematical formulas can be used to answer that question.

[0029] The "model answer" is information about a model answer as one of the evaluation criteria assumed by the first user, such as a teacher. When the answer of the second user, such as a student, matches the model answer described in the "model answer," that answer is judged to be correct.

[0030] The "scoring criteria" is information about the evaluation items that serve as the basis for automatic scoring. Specifically, the "scoring criteria" is information about evaluation items such as how many points to give as a baseline score when a second user's answer matches the model answer, how many points to deduct from the baseline score when the answer does not match the model answer but is mathematically equivalent, and how many points to add when the answer does not match the model answer and is not mathematically equivalent but should be given points. These criteria can be freely described by the first user, such as a teacher. "Mathematically equivalent" means that the answer is equivalent to the model answer, although only the format of expression differs, and it matches one of the evaluation items other than the model answer. For example, if the model answer is in the decimal form "0.25" and the second user's answer is in the fraction form "1 / 4", it will be judged as "mathematically equivalent". The "scoring criteria" may be described by combining evaluation formulas and variables to assess whether an answer, predefined in the web application, is mathematically equivalent to a model answer. Exclusion conditions may also be described as part of the "scoring criteria." Answers that meet the exclusion conditions may be judged as not matching the model answer and not being mathematically equivalent. The format of the information described in the "scoring criteria" is not limited to any particular format, as long as it can be used for automatic scoring on server 10.

[0031] "Comments" are text information added by teachers or other staff. Entering "Comments" is optional.

[0032] The process shown in Figure 3 begins when terminal 20a requests server 10 to launch the web application. Note that login processing, such as entering an ID and password, may occur when requesting the web application to launch. Upon receiving the web application launch request, server 10 sends the web application program, including the initial screen data, to terminal 20a. In response, terminal 20 displays the initial screen in its web browser. On the initial screen, paper 100 has not yet been created, and only a new paper creation icon 100c and an answer sheet creation button 100d are displayed in the upper area 100a. The process shown in Figure 3 is executed through the cooperation of terminal 20a's CPU 21 and server 10's processor 11.

[0033] In step S1, the first user, who is responsible for creating answer sheets for teachers and other personnel, operates the input device 25 of terminal 20a to select the new sheet creation icon 100c. In step S2, terminal 20a sends a request to server 10 to create a new sheet 100. In step S3, server 10 creates the new sheet 100 and sends it to terminal 20a.

[0034] In step S4, the first user adds a sticky note as needed. Adding a sticky note may involve, for example, selecting a sticky note addition icon (not shown) displayed on the new paper 100, or selecting an item to add a sticky note from a menu displayed by right-clicking the mouse. After adding the sticky note, the first user may, as needed, write supplementary comments on the sticky note. In step S5, terminal 20a sends the sticky note information entered by the first user to server 10. Server 10 associates the sticky note information with paper 100. The sticky note information includes, for example, the location information of the sticky note on paper 100 and information such as the text written on the sticky note. The process in steps S4-S5 may be repeated until the first user selects the answer sheet creation button 100d in step S6.

[0035] In step S6, after the first user has finished writing supplementary comments on the sticky note, they operate the input device 25 and select the answer sheet creation button 100d. In step S7, terminal 20a sends an answer sheet creation request to server 10. In step S8, server 10 generates a dialog for creating the answer sheet and sends the generated dialog to terminal 20a. Terminal 20a displays the received dialog on the paper 100.

[0036] Figures 4A, 4B, and 4C are examples of dialogs displayed on paper 100. First, a dialog box is sent from server 10 to terminal 20a to prompt the user to select the scoring criteria information file shown in Figure 4A. The dialog box 201 for the scoring criteria information file includes a file name display field 201a, a select button 201b, a cancel button 201c, and a next button 201d. The file name display field 201a is a display field that shows the file name of the currently selected scoring criteria information file. The select button 201b is a button that accepts the operation to display the scoring criteria information file selection screen. When the select button 201b is selected, a list of scoring criteria information files stored in server 10 is displayed. When the first user selects the desired scoring criteria information file, the file name of the selected scoring criteria information file is displayed in the file name display field 201a. The cancel button 201c is a button that accepts the operation to cancel the selection of the scoring criteria information file. If the Cancel button 201c is selected, the selection of the scoring criteria information file is canceled, and the display of dialog 201 is also terminated. The Next button 201d is a button that accepts the operation to confirm the selection of the scoring criteria information file.

[0037] In step S9, the first user sets the scoring criteria. Specifically, the first user selects a scoring criteria information file in dialog 201. After selecting the scoring criteria information file, the first user selects the Next button 201d. In step S10, terminal 20a sends the information of the selected scoring criteria information file, for example, the file name of the scoring criteria information file, to server 10. In step S11, server 10 adds the scoring criteria information to paper 100 based on the information in the scoring criteria information file. Specifically, server 10 associates paper 100 with the scoring criteria information file selected by the first user. After that, server 10 sends a dialog for the next setting to terminal 20a. Terminal 20a displays the received dialog on paper 100.

[0038] A dialog box for setting the answer deadline, as shown in Figure 4B, is sent from server 10 to terminal 20a. The dialog box 202 for setting the answer deadline includes an answer deadline display field 202a, a cancel button 202b, and a next button 202c. The answer deadline display field 202a is a display field that shows the currently set answer deadline. The first user can set the answer deadline, including the units of hours, minutes, and seconds, by selecting the answer deadline display field 202a. The answer deadline may be set, for example, by entering text or selecting from a calendar. The cancel button 202b is a button that accepts the operation to cancel the setting of the answer deadline. When the cancel button 202b is selected, the setting of the answer deadline is canceled, and the display of dialog box 202 is also terminated. In this case, the selection of the scoring criteria information file may also be canceled. The next button 202c is a button that accepts the operation to confirm the setting of the answer deadline.

[0039] In step S12, the first user sets the deadline for answering in dialog 202. After setting the deadline, the first user selects the Next button 202c. In step S13, terminal 20a sends the set deadline information to server 10. In step S14, server 10 adds the deadline information to paper 100. Specifically, server 10 associates the deadline information with paper 100. Server 10 then generates a unique URL for each piece of paper 100. Server 10 then sends a dialog containing the generated URL to terminal 20a. Terminal 20a displays the received dialog on paper 100.

[0040] A dialog box is sent from server 10 to terminal 20a to notify it that the creation of the answer sheet shown in Figure 4C has been completed. The dialog box 203 that notifies the completion of the answer sheet includes a URL display field 203a, a copy button 203b, and a close button 203c. The URL display field 203a is the URL for accessing the answer sheet. The copy button 203b is a button for copying the string representing the URL displayed in the URL display field 203a. The string copied by the copy button 203b can be pasted, for example, into an email. This email notifies a second user, such as a student, of the URL. The close button 203c is a button that accepts the operation to close dialog box 203.

[0041] In step S15, the first user checks the URL, copies it if necessary, and sends an email to the corresponding second user's terminal 20b. The email may be sent by the server 10. After these processes, the first user selects the close button 203c. In step S16, terminal 20a sends a notification to the server 10 that the process of creating the answer sheet is complete. The server 10 then recognizes that the process is finished. The display of the paper 100 on terminal 20a may also end upon completion of the process.

[0042] Figure 5 is a sequence diagram showing an example of the operation of System 1 when answering a question. The process in Figure 5 starts when the web browser is launched by terminal 20b. The process in Figure 5 is mainly executed by the cooperation between the CPU 21 of terminal 20b and the processor 11 of server 10.

[0043] In step S101, the second user, such as a student, operates terminal 20b and enters a URL previously specified by the first user, such as a teacher, via email, into a web browser. In step S102, terminal 20b accesses server 10 according to the entered URL. Login processing, such as entering an ID and password, may be performed when accessing the server.

[0044] Server 10 determines whether the status of the paper at the corresponding URL is a status error. For example, Server 10 determines whether the deadline for answering set in the paper at the corresponding URL has passed. Server 10 also determines whether the submission status of the second user for the paper at the corresponding URL is "submitted". As will be explained later, when the second user answers the question and "submits" the answer, Server 10 sets the submission status of that second user from "not submitted" to "submitted". Server 10 determines the submission status of the second user who accessed the page. If the deadline for answering has passed or the second user's submission status is "submitted", Server 10 determines that there is a status error. If a status error is determined, in step S103, Server 10 notifies terminal 20b of the error and then terminates processing. This error includes, for example, a message indicating that an answer cannot be submitted for the paper at the corresponding URL. On the other hand, if the deadline for responses has not passed and the submission status of the second user is "not submitted," the server 10 determines that there is no status error. If it is determined that there is no status error, in step S104, the server 10 sends the paper with the corresponding URL to the terminal 20b. The terminal 20b displays the received paper in its web browser.

[0045] Figure 6 shows an example of the display screen of terminal 20b when the answer sheet 300 is displayed. Similar to Figure 2, the screen displayed on the display device 26 of terminal 20b includes an upper area 300a and a lower area 300b. The upper area 300a corresponds to the upper area 100a, and the lower area 300b corresponds to the lower area 100b. When the answer sheet 300 is displayed, the answer submission button 300c is displayed in the upper area 300a.

[0046] On answer sheet 300, a second user may add sticky notes for each question. The second user can create sticky notes for answers when answering. Figure 6 shows three sticky notes 301, 302, and 303. In other words, Figure 6 shows that the second user has created three sticky notes. Sticky notes 301, 302, and 303 can be added by the second user by selecting from the aforementioned formula sticky notes, graph sticky notes, table sticky notes, and shape sticky notes.

[0047] Users can input various elements corresponding to the added sticky notes, such as creating mathematical formulas, graphs, tables, and shapes. Users can also have the server 10 perform calculations on the formulas. The calculation results of the formulas are displayed, for example, in the lower right corner of the sticky note as the answer to the question. Each of the sticky notes 301, 302, and 303 has check boxes 301a, 301b, and 301c, respectively. For example, when sticky note 303 is selected, a check mark appears in the corresponding check box 303a. In addition to the check mark, selected sticky notes and unselected sticky notes may be distinguished by other indicators. Each sticky note may also be assigned an answer label. When a second user selects from the dropdown list provided for the answer, a list of available question names for the corresponding sticky note is displayed. By the second user selecting the desired question name from the list, the answer label is assigned to the sticky note. Figure 6 shows an example where sticky note 301 is assigned the answer label 301b, and sticky note 302 is assigned the answer label 302b. Here, in the list of question names, question names of answer types that do not match the sticky notes created by the second user are not displayed. For example, for formula sticky notes, only question names where the answer type is set to formula are displayed in the list. Therefore, sticky note 303 cannot be assigned a question name where the answer type is set to formula.

[0048] In step S105, the second user adds sticky notes to paper 300 to answer the questions. In step S106, terminal 20b sends information about the added sticky notes to server 10. Server 10 associates the sticky note information with paper 300 for each second user. The sticky note information includes, for example, identification information, location information, and answer information written on each sticky note on paper 300. In step S107, the second user assigns an answer label to the sticky note. In step S108, terminal 20b sends information about the answer label to server 10. Server 10 associates the answer label information with the sticky note information for each second user. The answer label information includes, for example, the name of the question represented by the answer label. The process in steps S105-S108 may be repeated until the second user selects the answer submission button 300c in step S109.

[0049] The second user creates a sticky note, writes the answer on it, and then assigns an answer label to the sticky note. Then, at step S109, when the second user wishes to submit the answer, they select the answer submission button 300c. At step S110, terminal 20b sends a notification to server 10 that the answer has been submitted.

[0050] Server 10 determines whether there are any unanswered questions by aggregating the information of the answer labels associated with the paper 300 indicating that an answer has been submitted. If it is determined that there are unanswered questions, in step S111, Server 10 sends a warning dialog to Terminal 20b. The warning dialog includes, for example, a message asking whether it is alright to continue the submission process even though there are unsubmitted answers. The warning dialog may also have buttons to accept the operation to continue submitting the answers and buttons to accept the operation to cancel submitting the answers. In this case, Server 10 may determine whether to continue or cancel the answer submission process depending on the button selected by the operation of the second user's input device 25. If it is determined that the answer submission process should be continued, the process proceeds to step S112. If it is determined that the answer submission process should be canceled, the process returns to step S105. The process also proceeds to step S112 if it is determined that there are no unanswered questions.

[0051] In step S112, the server 10 changes the submission status of paper 300 for the second user, who is the user of terminal 20b that requested the submission of the answer, to "Submitted". The server 10 then generates a unique URL for each second user associated with the submitted paper 300. The server 10 then sends a dialog containing the generated URL to terminal 20b. Terminal 20b displays the received dialog on paper 300. The dialog displayed on terminal 20b in step S112 may be the same as the dialog shown in Figure 4C, differing only in that the displayed message indicates that the submission of the answer has been completed. The second user can check their submitted paper 300 using the URL sent in step S112. However, the second user is restricted from editing their submitted paper 300.

[0052] In step S113, the server 10 copies the URL and sends an email to the terminal 20a of the first user. The first user can then use the URL sent in step S113 to check the paper 300 answered by the second user, for example, for grading purposes.

[0053] Figure 7 is a sequence diagram showing an example of the scoring process. The process in Figure 7 begins when the web browser is launched by terminal 20a. The process in Figure 7 is mainly executed through the cooperation between the CPU 21 of terminal 20a and the processor 11 of server 10.

[0054] In step S201, the first user, such as a teacher, operates terminal 20a and enters the URL previously notified by server 10 via email or other means into a web browser. In step S202, terminal 20a accesses server 10 according to the entered URL. Login processing, such as entering an ID and password, may be performed when accessing the server.

[0055] In step S202, the server 10 retrieves the paper 300 from the entered URL. The server 10 then extracts the answer from the second user from the sticky note information associated with paper 300. Subsequently, the server 10 performs automatic scoring by comparing the answer from the second user with the scoring criteria information associated with paper 300. After the automatic scoring is complete, the server 10 sends the paper containing the automatic scoring results to terminal 20a. Terminal 20a displays the received paper in a web browser.

[0056] Figure 8 is a flowchart showing an example of the processing performed by the server 10 during automatic scoring. In step S301, the server 10 selects one answer according to the order of the question labels. If there are any missing question labels, the server 10 assigns a score of 0 to those question labels.

[0057] In step S302, the server 10 obtains the question name of the question label and the reference score for the corresponding question name from the scoring criteria information, and assigns the reference score to the second user's score for the corresponding question label. As mentioned above, the reference score is the score given when the second user's answer is correct.

[0058] In step S303, the server 10 determines whether the answer provided by the selected second user matches the model answer for the corresponding question. If it is determined in step S303 that the answer provided by the second user matches the model answer, the process proceeds to step S304. If it is determined in step S303 that the answer provided by the second user does not match the model answer, the process proceeds to step S305.

[0059] In step S304, the server 10 sets the background color of the sticky note corresponding to the question label to "green". The process then proceeds to step S310. The background color is the background color of the sticky note included in the paper displayed on terminal 20a as a result of automatic scoring. The background color is changed according to the result of automatic scoring based on the scoring criteria information. In other words, the background color is set to "green" when the answer by the second user matches the model answer. The score for the corresponding question label for the second user is the standard score.

[0060] In step S305, the server 10 determines, based on the scoring criteria information, whether the answer by the second user is mathematically equivalent to the model answer. Here, in step S305, if the answer by the second user matches any of the exclusion conditions described in the scoring criteria information, it may be determined that it is not mathematically equivalent to the model answer. If, in step S305, it is determined that the answer by the second user is mathematically equivalent to the model answer, the process proceeds to step S306. If, in step S305, it is determined that the answer by the second user is not mathematically equivalent to the model answer, the process proceeds to step S308.

[0061] In step S306, the server 10 deducts points from the score of the corresponding question label from the baseline score according to the evaluation criteria of the scoring standards.

[0062] In step S307, server 10 sets the background color of the sticky note corresponding to the question label to "yellow". Then the process moves to step S310. In other words, the background color is set to "yellow" when the answer by the second user does not match the model answer but is mathematically equivalent.

[0063] In step S308, server 10 sets the score for the corresponding question label to 0. However, in step S308, server 10 may add points to the score from 0 depending on the evaluation item of the scoring criteria.

[0064] In step S309, server 10 sets the background color of the sticky note corresponding to the question label to "red". Then the process moves to step S310. In other words, the background color is set to "red" when the answer by the second user is incorrect.

[0065] In step S310, the server 10 determines whether or not to terminate automatic scoring. For example, if scoring is completed for all question labels, it is determined to terminate automatic scoring. If it is determined in step S310 not to terminate automatic scoring, the process returns to step S301. If it is determined in step S310 to terminate automatic scoring, the process proceeds to step S311.

[0066] In step S311, the server 10 calculates the total score of the points assigned to each question label.

[0067] In step S312, the server 10 creates a paper with the automatic scoring results. After that, the server 10 terminates the process shown in Figure 8.

[0068] Figure 9 shows an example of the display screen of terminal 20a when the automatic scoring result paper 400 is displayed. The screen displayed on the display device 26 of terminal 20a includes an upper area 400a and a lower area 400b. The upper area 400a corresponds to the upper area 100a, and the lower area 400b corresponds to the lower area 100b. When the automatic scoring result paper 400 is displayed, the scoring complete button 400c is displayed in the upper area 400a.

[0069] Paper 400, which displays the automatically graded results, shows the sticky notes added by the user to paper 300, as well as the corresponding sticky notes. In Figure 9, three sticky notes, 401, 402, and 403, are displayed. The content displayed on sticky notes 401, 402, and 403 is the same as the sticky notes added to paper 300. However, unlike the sticky notes for answers, sticky notes 401, 402, and 403 cannot be edited.

[0070] Here, sticky notes 401, 402, and 403 have background colors set. These background colors are set according to the results of automatic scoring. As mentioned above, when the answer matches the model answer, the background color of the sticky note is set to "green", when the answer does not match the model answer but is mathematically equivalent to the model answer, the background color is set to "yellow", and when the answer does not match the model answer and is not mathematically equivalent, the background color of the sticky note is set to "red". Figure 9 shows an example where the background color of sticky note 401 is "yellow", the background color of sticky note 402 is "green", and the background color of sticky note 403 is "red".

[0071] Furthermore, the scoring criteria sticky note 404 is displayed on paper 400, which contains the results of the automatic scoring. The scoring criteria sticky note 404 is a list of evaluation items for the scoring criteria for each question, which is associated with the corresponding answer sheet 300.

[0072] The scoring criteria sticky note 404 includes a model answer 404a, evaluation items 404b, and a total score 404c. The model answer 404a is the "model answer" recorded in the scoring criteria information for the corresponding question. The evaluation items 404b are text representing each evaluation item of the "scoring criteria" recorded in the scoring criteria information for the corresponding question. The total score 404c is the sum of the scores for each question calculated as a result of automatic scoring. The numerical value displayed in the total score 404c can be modified by the first user.

[0073] In the scoring criteria sticky note 404, the section for model answer 404a is colored according to the results of the automatic scoring. For example, if the answer matches the model answer, the section for model answer 404a is colored "green". If the answer does not match the model answer but is mathematically equivalent to the model answer, the section for model answer 404a is colored "yellow". For example, if it matches an evaluation item other than the model answer, it will be judged to be mathematically equivalent. If the answer does not match the model answer and is not mathematically equivalent, the section for model answer 404a is colored "red". In other words, the section for model answer 404a is colored the same as the background color of sticky notes 401, 402, and 403. Note that the color scheme shown here is just an example and may be changed as appropriate.

[0074] Now, let's return to the explanation of Figure 7. In step S204, after the paper 400 shown in Figure 9 is displayed on terminal 20a, the first user adds sticky notes as needed. After adding sticky notes, the first user writes feedback comments, etc., on the sticky notes as needed. In step S205, terminal 20a sends the sticky note information entered by the first user to server 10. Server 10 associates the sticky note information with paper 400. The sticky note information includes, for example, information such as the location of the sticky note on paper 400 and the text written on the sticky note. Also, in step S206, if the result of the automatic scoring is not what the first user wanted, they correct the total score 404c. In step S207, terminal 20a sends the total score information corrected by the first user to server 10. Server 10 corrects the total score information of the corresponding user associated with paper 300. The process in steps S204-S205 may be repeated until the first user selects the scoring completion button 400c in step S206.

[0075] The first user selects the scoring completion button 400c at step S208, when the scoring result is what they wanted. At step S209, terminal 20a sends a notification to server 10 that scoring is complete. Server 10 then recognizes that the process is finished. The display of paper 400 on terminal 20a may also be terminated upon completion of the process. Furthermore, paper 400 may be configured to be uneditable once scoring is complete.

[0076] As described above, according to this embodiment, answers submitted by a second user, such as a student, are automatically scored according to scoring criteria information that includes predetermined evaluation items. As a result of the automatic scoring, answers are displayed separately depending on which evaluation items the answer matches, such as answers that match the model answer, answers that do not match the model answer but are substantially equivalent to the model answer, and answers that do not match the model answer and are not substantially equivalent. This allows the first user, such as a teacher, to intuitively understand what scoring criteria were used for the automatic scoring of each question. In particular, for questions colored "yellow," scoring is not based on the absolute criterion of whether or not the answer matches the model answer, but rather on evaluation items created in advance based on the first user's scoring approach. Therefore, the first user, such as a teacher, only needs to focus on checking the questions colored "yellow." In this way, the burden and errors of visually checking the results of automatic scoring by the first user are reduced. As a result, the quality of automatic scoring is improved.

[0077] [Differentiation] The following describes modifications of the embodiment. The embodiment describes an example of automatic scoring of answers to mathematics questions. However, the method of the embodiment can be applied to the automatic scoring of answers to various questions other than mathematics. Figure 10 shows an example of the display screen of terminal 20a when paper 500 of the automatic scoring results for English is displayed. The screen displayed on the display device 26 of terminal 20a also includes an upper area 500a and a lower area 500b. The upper area 500a corresponds to the upper area 100a, and the lower area 500b corresponds to the lower area 100b. Also, when paper 500 of the automatic scoring results is displayed, a scoring complete button 500c is displayed in the upper area 500a. These are the same as those for paper 400.

[0078] Paper 500, which displays the automatically graded results, shows the sticky notes added by the user to answer sheet 300, along with their corresponding sticky notes. In Figure 10, one sticky note, 501, is displayed. This sticky note corresponds to sticky notes 401, 402, and 403 on paper 400. For example, Figure 10 is a question where the user must fill in the parentheses with the semantically correct word in the correct form.

[0079] The background color of sticky note 501 is set according to the result of automatic scoring. For example, if the answer matches the model answer, the background color will be "green"; if the answer is semantically the same as the model answer but has a problem with its expression, the background color will be "yellow"; and if the answer does not semantically match the model answer, the background color will be "red". Problems with expression include answers that contain errors that are expected to have the same meaning as the model answer, such as "a word that should be written in lowercase contains an uppercase letter," "the verb form is different," or "a word contains a spelling mistake," as shown in Figure 10.

[0080] In the modified version, the evaluation items that match the answer are colored. For example, in the example in Figure 10, the model answer is the word "assess" or "measure," but an example is shown where the answer contains a spelling mistake, such as "asessed," and is written in the past tense. In this case, the evaluation items "-3 points for incorrect verb form" and "-1 point for spelling mistake" are colored yellow. This coloring allows the first user to more intuitively understand what scoring criteria were used. This coloring of each evaluation item may also be done in the example in Figure 9 mentioned above. Furthermore, if multiple evaluation items are colored as in Figure 10, different colors may be used for these multiple evaluation items.

[0081] In this embodiment, the server 10 performs automatic scoring according to scoring criteria created by the first user and colors the answers according to the results of the automatic scoring. Alternatively, the server 10 may perform automatic scoring using artificial intelligence that has learned the scoring criteria created by the first user. In this case as well, the server 10 displays and distinguishes the answers according to the results of the automatic scoring.

[0082] Furthermore, in this embodiment, the background color of the sticky notes is color-coded to distinguish between answers that match the model answer, answers that do not match the model answer but are substantially equivalent, and answers that do not match the model answer and are not substantially equivalent. However, it is sufficient that the answers that match the model answer, answers that do not match the model answer but are substantially equivalent, and answers that do not match the model answer and are not substantially equivalent are displayed in a way that allows the first user to distinguish them visually. For example, the display may be done by changing the shape of the sticky note frame, changing the color of the sticky note frame, or changing the thickness of the frame.

[0083] Furthermore, in this embodiment, the background color of the sticky notes is differentiated into three categories: answers that match the model answer, answers that do not match the model answer but are substantially equivalent, and answers that do not match the model answer and are not substantially equivalent. Alternatively, the background color of the sticky notes may be colored only for answers that do not match the model answer but are substantially equivalent.

[0084] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate as possible, and in that case, the combined effects can be obtained. Moreover, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriate combinations of the multiple constituent elements disclosed. For example, if some constituent elements are deleted from all the constituent elements shown in an embodiment, and the problem described in the section on the problem the invention aims to solve is solved and the effects described in the section on the effects of the invention are obtained, then the configuration with these constituent elements deleted can be extracted as an invention.

[0085] The invention described in the original claims of this application is listed below. [1] Accepting input of information including answers to questions, The answers to the aforementioned questions are scored according to one or more pre-set evaluation criteria, The scoring results will be displayed and distinguished according to the evaluation items that match the aforementioned answers, An automated scoring program that instructs the processor to execute. [2] The evaluation items include model answers to the above answers, Distinguishing in the above display includes distinguishing between cases where the answer matches the model answer, cases where the answer does not match the model answer but is equivalent, and cases where the answer neither matches nor is equivalent to the model answer. The automated scoring program described in [1]. [3] The evaluation items include model answers to the above answers, Distinguishing in the above-mentioned display includes distinguishing only when the answer does not match the model answer but is equivalent. The automated scoring program described in [1]. [4] Distinguishing in the display includes distinguishing in the display of the area where the answer is displayed. The automatic scoring program described in any one of [1]-[3]. [5] Distinguishing in the display includes distinguishing in the display the parts of the evaluation items that match the answer, The automatic scoring program described in any one of [1]-[4]. [6] Distinguishing in the display includes color-coding the scoring results according to the matching evaluation items, The automatic scoring program described in any one of [1]-[5]. [7] A server that can communicate with a terminal, We accept input of information including answers to the questions. The answers to the aforementioned questions are scored according to one or more pre-set evaluation criteria. The scoring results will be displayed and distinguished according to the evaluation items that match the aforementioned answers. The results of the scoring are transmitted to the terminal. A server equipped with a processor configured in such a way. [8] An automated scoring method performed by an automated scoring system, The system accepts input of information, including answers to questions, The answers to the aforementioned questions are scored according to one or more pre-set evaluation criteria, The scoring results will be displayed and distinguished according to the evaluation items that match the aforementioned answers, An automated scoring method comprising the following features. [9] Accept input of information including answers to questions, The answers to the aforementioned questions are scored according to one or more pre-set evaluation criteria. The scoring results will be displayed and distinguished according to the evaluation items that match the aforementioned answers. The results of the aforementioned scoring are transmitted. A server comprising a first processor configured as follows, Upon receiving the scoring results transmitted from the server, The received scoring results are displayed on the display device. A terminal comprising a second processor configured as follows, An automated scoring system. [Explanation of symbols]

[0086] 1 System, 10 Server, 11 Processor, 12 ROM, 13 RAM, 14 Storage, 15 Communication device, 19 System bus, 20a, 20b Terminal, 21 CPU, 22 ROM, 23 RAM, 24 Storage, 25 Input device, 26 Display device, 27 Communication device, 29 System bus, 30 Network.

Claims

1. The system includes a setting means for setting up an electronic answer sheet on which sticky notes are placed for respondents to write their answers to the questions. The setting means configures the answer sheet so that, when the question setter selects a type of answer for each of the multiple questions, the answerer is prohibited from placing sticky notes on the answer sheet corresponding to the type of answer that was not selected. The aforementioned answer types include mathematical formulas, graphs, or tables. An educational support system characterized by the following features.

2. The setting means sets the answer sheet so that when the type of answer to each of the plurality of questions is selected by the question setter, the answer sheet is set to allow the answerer to place sticky notes corresponding to the selected type of answer on the answer sheet. The educational support system according to feature 1.

3. An educational support method implemented by an educational support system, This includes a setup process for configuring an electronic answer sheet on which sticky notes are placed for respondents to write their answers to the questions. The setting process configures the answer sheet so that, when the question setter selects a type of answer for each of the multiple questions, the answerer is prohibited from placing sticky notes on the answer sheet corresponding to the type of answer that was not selected. The aforementioned answer types include mathematical formulas, graphs, or tables. An educational support method characterized by the following features.

4. A computer, It functions as a setting means for configuring an electronic answer sheet on which sticky notes are placed for respondents to write their answers to the questions. The setting means configures the answer sheet so that, when the question setter selects a type of answer for each of the multiple questions, the answerer is prohibited from placing sticky notes on the answer sheet corresponding to the type of answer that was not selected. The aforementioned answer types include mathematical formulas, graphs, or tables. A program characterized by the following features.

Images