system

Abstract

We provide the system. [Solution] The data receiving device includes means for receiving written answers, The preprocessing device includes means for preprocessing the received answer data by tokenization or normalization, The analysis device includes means for analyzing pre-processed answer data using natural language processing technology and calculating a scoring score, The recording device includes means for storing the analyzed score, The output device includes means for providing scoring results and feedback to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional test system, descriptive questions are often scored manually, which is a great burden on teachers. Also, manual scoring leaves room for fluctuations and subjectivity in scoring criteria due to human factors, and there is a possibility of lack of fairness. Furthermore, since scoring takes time, students cannot receive prompt feedback, and the efficiency of learning is impaired.

Means for Solving the Problems

[0005] This invention solves the above problems by providing a system in which a data receiving device receives written answers, a preprocessing device performs preprocessing such as tokenization, and an analysis device analyzes the answers using natural language processing technology to calculate a scoring score. Furthermore, a recording device stores the analysis results, and an output device quickly provides the scoring results and feedback to the user, thereby realizing a fair and efficient scoring process.

[0006] A "data receiving device" is a device that receives data transmitted from an external source and takes it in a format that can be processed internally.

[0007] A "preprocessing device" is a device that performs preprocessing, such as normalization and tokenization, on received data in order to convert it into a format suitable for analysis.

[0008] An "analysis device" is a device that uses natural language processing technology based on pre-processed data to analyze and evaluate input answers.

[0009] A "scoring score" is a numerical evaluation calculated as a result of an analysis device evaluating the answer.

[0010] A "recording device" is a device that stores analyzed scores and other data so that they can be referenced later as needed.

[0011] An "output device" is a device that displays or communicates information in order to provide analysis results or feedback to the user.

[0012] "Natural language processing technology" refers to technologies that use computer processing to understand, interpret, and generate natural language that humans use on a daily basis.

[0013] "Feedback" refers to information such as comments and suggestions for improvement regarding the answer provided to the user based on evaluation by the analysis device. [Brief explanation of the drawing]

[0014] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. <000S083> [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The automated scoring system for written response questions in this invention primarily operates through the collaborative efforts of multiple devices installed on a server and a user's terminal. When a user inputs and submits their answer to a written response question using their terminal, a data receiving device on the server receives it. Subsequently, a preprocessing device performs data cleansing, such as normalization and tokenization, to prepare the answer data for natural language processing.

[0036] Next, the analysis device uses natural language processing technology to analyze the pre-processed answer data. The analysis calculates a score based on criteria such as the accuracy of the answer, logical structure, and comprehensiveness of related information. For example, if a student answers the history question, "Explain the causes of World War II," with the answer, "The effects of World War I and the economic crisis were the causes," the analysis device evaluates the accuracy and depth of this explanation.

[0037] After analysis, the recording device saves the calculated scores to a database for later review by the user. This data is used for performance evaluation and other learning analysis tools. The output device provides the user's terminal with the scoring results and feedback for their answers. This allows the user to quickly understand the evaluation of their answers and obtain specific guidance on areas for improvement.

[0038] As a concrete example, when a user takes an online test and enters their answers, the entire process is executed automatically, and the scoring results are displayed on the device within minutes. For example, feedback such as "Your score is 75 points. Your understanding of the questions is good, but your explanation of the economic background is insufficient" is provided.

[0039] In summary, the present invention reduces the workload of teachers and improves the efficiency and fairness of grading through the automatic scoring of essay-type questions.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The user uses their device to enter their answers to written questions and then presses the submit button to send the answers to the server.

[0043] Step 2:

[0044] The server's data receiving device receives the transmitted answer data, verifies that the data is in a receivable format, and records it in the log.

[0045] Step 3:

[0046] The server's preprocessor normalizes the received answer data, converting the text to lowercase and removing unnecessary spaces and special characters. It also performs spell checking and tokenizes the text into words.

[0047] Step 4:

[0048] The server's analysis device receives the pre-processed data and begins analysis using natural language processing technology. Here, it evaluates the relevance of the answers, the comprehensiveness of the perspectives, and the logical structure, and calculates a scoring score.

[0049] Step 5:

[0050] The server's recording device stores the scores calculated by the analysis device in a database, managing them so that each user's scoring results can be referenced later.

[0051] Step 6:

[0052] The server's output device generates analysis results and feedback, which are then provided to the user's terminal. This allows the user to quickly check information such as their score and areas for improvement regarding their answers.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] Grading essay-type questions is typically time-consuming and labor-intensive, and prone to inconsistencies in evaluation among graders. Furthermore, evaluating a large volume of responses quickly and fairly is difficult, hindering efficient feedback provision in educational settings.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes means for an information receiving device to receive answers to written questions transmitted from a user terminal, means for an information processing device to preprocess the received answers by normalization and tokenization and convert them into a format suitable for natural language processing, and means for an analysis device to analyze the preprocessed answers using a generative AI model and calculate a score based on the accuracy, logical structure, and information comprehensiveness of the answers. This enables the rapid, fair, and automatic scoring of answers to written questions and the efficient provision of feedback.

[0058] An "information receiving device" is a device used to receive answers to written questions transmitted from a user terminal.

[0059] An "information processing device" is a device that preprocesses received responses through normalization and tokenization, and converts them into a format suitable for natural language processing technology.

[0060] An "analysis device" is a device that uses a generative AI model to analyze the content of pre-processed answers and calculates a score based on the accuracy, logical structure, and comprehensiveness of the answers.

[0061] An "information recording device" is a device used to store analyzed scores as data.

[0062] An "information output device" is a device that provides users with scoring results and feedback, and makes that information available through an online interface.

[0063] A "generative AI model" is a model based on AI technology used to perform natural language analysis on written answers.

[0064] This invention relates to a system for automatically scoring answers to written response questions, and the server includes an information receiving device, an information processing device, an analysis device, an information recording device, and an information output device. Users use a terminal (including PCs and smartphones) to input and transmit answers to written response questions via an online interface. The server's information receiving device receives these answers. The main technologies and procedures used are described below in detail.

[0065] The information processing device performs normalization and tokenization of the received answer data. This is done using natural language processing libraries (e.g., NLTK and spaCy) to preprocess the answers, converting them into a data format that is easy for the generative AI model to analyze.

[0066] The analysis device uses a generative AI model to analyze pre-processed answers. The generative AI model utilizes transformer technology (e.g., BERT and GPT) to evaluate the accuracy, logical structure, and information coverage of the answers, and calculates a specific score. Because this model has already learned evaluation criteria for input answers, it can evaluate user answers quickly and fairly.

[0067] The data recording device securely stores the analyzed scores in a database. The database manages data for each user via a database management system such as MySQL® or PostgreSQL, allowing for later reference.

[0068] The information output device provides users with scoring results and feedback. Users can check the results on their terminals via an online interface, and the device further enhances their motivation to learn by highlighting areas for improvement.

[0069] For example, if a user answers a history essay question with "It was caused by the effects of World War I and the economic crisis," the analysis device will analyze this and generate feedback such as, "Your answer is worth 75 points. Please elaborate on the economic background." An example of a prompt message could be a specific question such as, "Please evaluate the following answer: It was caused by the effects of World War I and the economic crisis." In this way, the present invention enables efficient and fair automatic scoring of essay questions.

[0070] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0071] Step 1:

[0072] The user enters their answers to the written questions using a terminal and clicks the submit button. This sends the answers to the server as text data.

[0073] Step 2:

[0074] The server's information receiving device receives the answer sent by the user. The input answer data is first temporarily stored and then passed on to subsequent processing. The input here is the user's answer, and the output is the receipt and initial storage of that answer data.

[0075] Step 3:

[0076] The server's information processing unit normalizes and tokenizes the received answers. Specifically, it uses a natural language processing library to remove unnecessary characters, spell check, and split the text into individual words. The input is the user's answer data, and the output is pre-processed, parseable data.

[0077] Step 4:

[0078] The server's analysis device analyzes pre-processed data using a generative AI model. A transformer model is utilized to calculate and analyze the answer. The input is pre-processed data, and the output is a score based on the accuracy, logical structure, and comprehensiveness of the information in the answer.

[0079] Step 5:

[0080] The server's data recording device saves the analysis results to a database. The score is associated with a user ID and is securely managed. The input is the analyzed score, and the output is the record stored in the database.

[0081] Step 6:

[0082] The server's information output device generates scoring results and feedback for the user and displays them on the terminal. Through the online interface, the user can check how their answers were evaluated, receive feedback, and use it for future learning. The input is the saved score and feedback content, and the output is the result display on the user's terminal.

[0083] (Application Example 1)

[0084] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0085] In today's educational environment, grading essay-type questions is time-consuming and laborious, placing a burden on educators. Furthermore, ensuring fairness in grading while providing timely feedback is a challenging task. There is also a need for a system that allows test-takers to quickly and specifically identify areas for improvement.

[0086] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0087] In this invention, the server includes a data acquisition device that acquires descriptive answers, an initial processing device that performs initial processing of the acquired answer information by information segmentation and formatting, and an analysis device that analyzes the initially processed answer information using natural language processing technology and calculates an evaluation score. This enables automated scoring and rapid feedback provision.

[0088] A "data acquisition device" is a means of obtaining written responses from users.

[0089] An "initial processing device" is a means of performing initial processing on acquired response information by performing information segmentation and formatting.

[0090] An "analysis device" is a means of analyzing initially processed response information using natural language processing technology and calculating an evaluation score.

[0091] A "storage device" is a means of storing the analyzed evaluation scores so that they can be referenced later.

[0092] An "output device" is a means of providing evaluation results and areas for improvement to the user.

[0093] A "communication terminal" is a device that allows users to check evaluation results and areas for improvement.

[0094] This invention is a system for automatically grading written response questions. Specifically, a server and a user's terminal work together. The user first inputs the answer to the written response question from their terminal. The terminal then transmits this answer data to the server's data acquisition device.

[0095] On the server, the received data is processed by the initial processing unit. The initial processing unit partitions and formats the data to convert it into a format suitable for natural language processing. This process involves tokenization and normalization of the information.

[0096] Next, the analysis device analyzes this pre-processed data using natural language processing techniques. The analysis device calculates evaluation scores and assesses the accuracy, logical structure, and relevance of the answers. For natural language processing, software such as spaCy or BERT is used.

[0097] The results obtained from the analysis are recorded by a storage device. This allows the user to later review the evaluation of their answers. The output device provides the evaluation results and individual feedback to the user's terminal.

[0098] As a concrete example, suppose a high school student uses an online learning app to answer a history essay question. When the user enters their answer, it is instantly sent to the server, and the entire process is executed automatically. A few minutes later, the user receives feedback such as, "Your score is 75 points. You should delve a little deeper into the details of the cause."

[0099] Furthermore, an example of a prompt sentence generated using an AI model is: "Evaluate your answer to the following history question: 'The causes of World War II were the effects of World War I and the economic crisis.'" In this way, the invention can be effectively implemented.

[0100] This system significantly streamlines the evaluation process for educators and enables prompt and fair feedback to test takers.

[0101] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0102] Step 1:

[0103] When a user enters their answer to a written question into their terminal and presses the submit button, the answer data is sent to the server's data acquisition device. The input is text data provided by the user, and the output is data in a format that can be parsed on the server side.

[0104] Step 2:

[0105] The initial processing unit on the server performs initial processing on the acquired answer data, such as tokenization and normalization. The input is text data sent by the user, which is then divided into tokens and formatted to remove unnecessary shapes and characters. The output is data in a format suitable for natural language processing.

[0106] Step 3:

[0107] The analysis system on the server analyzes the pre-processed answer data using natural language processing techniques. The input is the pre-processed data, and at this stage, spaCy and BERT are used to analyze the logical structure and relationships between the content, and to calculate the evaluation score. The output is the scoring result and a feedback message.

[0108] Step 4:

[0109] The server's storage device stores the analysis results and scores in a database. The input is the score data provided by the analysis device, and the output is the data stored for future reference.

[0110] Step 5:

[0111] The server's output device sends the scoring results and feedback to the user's terminal. The output is feedback information provided to the user, consisting of a score and advice text displayed on the terminal.

[0112] Step 6:

[0113] Users receive rapid feedback through their devices, allowing them to identify areas for further learning and improvement. At this point, users can understand which parts of their answers need improvement based on the information provided.

[0114] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0115] The automated scoring system for written questions of the present invention begins with the user inputting a written answer. When the user inputs and submits the answer using a terminal, the answer is received by a data receiving device installed on the server. The received data is normalized and tokenized by a preprocessing device and prepared in a format suitable for natural language processing.

[0116] After preprocessing, the server's analysis system uses natural language processing technology to analyze the answers in detail and calculate a scoring score. The analysis system scores the answers based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. This score is stored in a recording device and managed in a database.

[0117] Furthermore, this invention incorporates an emotion engine. The emotion engine automatically recognizes the user's emotions from their responses during the analysis process. This emotion information is used to generate feedback by the output device. For example, if the system recognizes from the responses that the user is feeling anxious, an encouraging message is automatically generated.

[0118] This feedback is provided to the user's terminal via an output device. The system allows users to quickly check their score and receive emotion-based advice for their answers. For example, if the answer to the history question, "Explain the impact of the Meiji Restoration," is "Japan advanced political reforms and achieved modernization. However, challenges remained in agricultural development," the analysis device will assign a score of 85 points, and the emotion engine will generate emotion-based feedback such as, "You seem a little nervous. Please answer with confidence."

[0119] Thus, this system not only automatically grades written questions but also provides feedback that takes into account the user's emotions, thereby realizing a more personalized learning experience.

[0120] The following describes the processing flow.

[0121] Step 1:

[0122] The user uses their device to enter their answers to written questions and clicks the submit button to send the answer data to the server.

[0123] Step 2:

[0124] The server's data receiving device receives the transmitted answer data and records a log of the data reception.

[0125] Step 3:

[0126] The server's preprocessor normalizes, tokenizes, and spell-checks the received answer data, preparing it for natural language processing.

[0127] Step 4:

[0128] The server's analysis device analyzes the pre-processed data and calculates a scoring score based on the relevance of the answers, the comprehensiveness of the perspectives, and the logical structure.

[0129] Step 5:

[0130] The server's emotion engine recognizes emotions from the user's text included in the answer data and adds that information to the analysis results.

[0131] Step 6:

[0132] The server's recording device stores the calculated score and emotion recognition results in a database, and this data is then organized and managed.

[0133] Step 7:

[0134] The server's output device provides the user's terminal with scoring results and sentiment-based feedback. For example, it might generate a message such as, "Your score is 85 points. You seem a little nervous. Please continue answering with confidence."

[0135] (Example 2)

[0136] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0137] In automated scoring of essay-type questions, there is a need to enhance the objectivity of evaluation and improve learner motivation by providing feedback based on the respondent's emotions. However, conventional automated scoring systems have had difficulty accurately analyzing the content of written responses and generating individual emotional feedback.

[0138] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0139] In this invention, the server includes a data receiving device for receiving written answers, a preprocessing device for tokenizing and normalizing the received answer data, an analysis device for analyzing the preprocessed answer data using natural language processing techniques and calculating an evaluation score, an emotion recognition device for extracting emotions from the answers, and an output device for providing the terminal with the scoring results and emotion-based feedback. This enables highly accurate analysis of written content and the provision of emotion-based, personalized feedback.

[0140] A "data receiving device" is a component that has the function of receiving written answers sent by a user from their terminal.

[0141] A "preprocessor" is a component that tokenizes and normalizes received answer data, preparing it for analysis in a suitable format.

[0142] An "analysis device" is a component that has the function of analyzing pre-processed answer data using natural language processing technology, evaluating its content, and calculating a score.

[0143] The "evaluation score" is a numerical value calculated by the analysis device based on the relevance, comprehensiveness, and logical structure of the answer.

[0144] A "recording device" is a component used to record and manage the scores obtained through analysis.

[0145] An "emotion recognition device" is a component that analyzes and identifies emotions extracted from the user's responses.

[0146] An "output device" is a component that provides the user with feedback generated based on the analysis results and emotions.

[0147] A "remote interface" is a set of protocols and software that allows users to communicate with a server online and check results and feedback.

[0148] In an embodiment of the present invention, the server first uses specific hardware and software to provide automatic scoring of written questions and sentiment-based feedback. This system consists of the following main elements:

[0149] The server side is equipped with a data receiving device, preprocessing device, analysis device, emotion recognition device, and output device. The data receiving device is responsible for receiving written answers that users submit using their terminals. Users using terminals input and submit answers to history questions, such as "Explain the impact of the Meiji Restoration."

[0150] The preprocessor uses natural language processing libraries (e.g., NLTK or SpaCy) to tokenize and normalize the received response data. This preprocessing step prepares the data for analysis.

[0151] The analysis device uses a generative AI model (e.g., GPT, BERT) employing natural language processing technology to analyze the content of the answers and calculate an evaluation score based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. For example, the model assigns a score of 85 points to the specific answer, "Japan advanced political reforms and achieved modernization. However, challenges remained in the development of agriculture."

[0152] The emotion recognition device utilizes an emotion analysis library (e.g., TextBlob or VADER) to extract emotions from the user's responses during the analysis process. Based on the word choices in the responses, it determines whether the user is experiencing a specific emotion, such as anxiety or relief.

[0153] The output device generates feedback derived from the analysis results and sentiment analysis, and provides it to the user's terminal. This allows the user to see their score for their answer along with sentiment-based feedback, such as "Approach this with confidence."

[0154] As described above, this system performs specific analysis and generates feedback using a generation AI model and example prompts. A possible example of a prompt is, "Analyze the student's answer and generate emotion-based feedback." By notifying the user of the feedback through methods such as pop-up notifications, the system allows them to receive the information immediately.

[0155] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0156] Step 1:

[0157] The user uses a terminal to input and submit written answers. The information entered is data in text format, with the user providing the answers. The terminal sends this answer data to the server via the internet.

[0158] Step 2:

[0159] The server's data receiving device receives the answer data sent from the user's terminal. The input is the data sent by the user, and the output is the answer data stored in the receiving device. At this stage, the answer data is stored on the server and passed on to the next processing step.

[0160] Step 3:

[0161] The server's preprocessing unit tokenizes and normalizes the received response data. The input is the received response data, and the output is data in a format suitable for the natural language processing model. This preprocessing removes extraneous whitespace and special characters, and tokenizes sentences into word units.

[0162] Step 4:

[0163] The server's analysis device performs analysis using a generative AI model with pre-processed data. The input is tokenized answer data, and the output is the evaluation score of the answers. The generative AI model is used to evaluate the relevance, logic, and comprehensiveness of the answers and assign a score. For example, a score of 85 points may be calculated.

[0164] Step 5:

[0165] The server's emotion recognition device extracts emotions from the data obtained during the analysis process. The input is pre-processed answer data, and the output is information indicating the user's emotions. The user's feelings are identified by the frequency of positive or negative expressions in the answers.

[0166] Step 6:

[0167] The server's output device generates feedback based on the scoring score and sentiment information, and sends it to the terminal. The input is the analysis result and sentiment information, and the output is the feedback message presented to the user. Specifically, an encouraging message such as "Your answer is correct. Please keep your confidence." is generated and displayed on the user's screen.

[0168] (Application Example 2)

[0169] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0170] Conventional automated scoring systems for written answers could evaluate the content of the answers, but they could not provide feedback based on the user's emotional state. Therefore, it was difficult to provide a more personalized learning experience for individual users.

[0171] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0172] In this invention, the server includes means for a data receiving device to receive descriptive answers, means for a preprocessing device to preprocess the received answer data by tokenization and normalization, and means for an analysis device to analyze the preprocessed answer data using natural language processing technology and calculate a scoring score. This makes it possible to evaluate the user's answers and, further, to recognize the user's emotions from the answers using an emotion recognition device, to provide emotion-based feedback.

[0173] A "data receiving device" is a device that has the function of receiving written answers sent by a user.

[0174] A "preprocessing device" is a device that has the function of preparing received answer data into a format suitable for natural language processing through tokenization and normalization.

[0175] An "analysis device" is a device that uses natural language processing technology to analyze pre-processed answer data in detail and evaluate the answers.

[0176] A "scoring score" is a numerical value that indicates the evaluation of an answer calculated by an analysis device.

[0177] A "recording device" is a device used to store analyzed scoring scores and manage them in a database.

[0178] An "emotion recognition device" is a device that automatically recognizes emotions from a user's answers and processes that information.

[0179] "Feedback" refers to information used to provide users with scoring results and advice tailored to their emotions regarding their answers.

[0180] An "output device" is a device that presents and allows users to review scoring results and feedback.

[0181] The system realizing this invention operates primarily with a server, a terminal, and a user. The server consists of a data receiving device, a preprocessing device, an analysis device, a recording device, an emotion recognition device, and an output device. The server receives written answers sent from the user via the data receiving device. The received answer data is tokenized and normalized by the preprocessing device and prepared in a format suitable for natural language processing.

[0182] This organized data is then analyzed in detail using an analysis device, and the answers are evaluated using natural language processing technology to calculate a scoring score. Software such as Google® Cloud Natural Language API is used in this process. In addition, an emotion recognition device is used to automatically recognize the user's emotions from the answers using Microsoft® Azure® Text Analytics.

[0183] The analysis results and sentiment information are integrated by a recording device and managed in a database. The output device provides the scoring results to the user's terminal and generates and displays sentiment-based feedback. A text generation API is used for this generation. For example, if a middle school student answers the question, "Explain the economic development of the Showa era," with "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged," this answer is analyzed and scored appropriately. If the elements targeted for analysis are evaluated as sufficient to meet the scoring criteria, a message such as, "Your answer is 90 points. Very well done! You seem a little unsure, but the content is solid," is generated and displayed as feedback.

[0184] Examples of input prompts for a generative AI model:

[0185] User's answer: "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged."

[0186] Grading criteria: "Economic growth, industry, economic power, environmental issues"

[0187] It generates analysis and emotional feedback to provide responses to the user.

[0188] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0189] Step 1:

[0190] The user enters and submits written answers using their own device. The user's answer data is generated as input. The user's device then sends this data to a data receiving device.

[0191] Step 2:

[0192] The server's data receiving device receives the user's answer data. It handles the answer data sent from the terminal as input. After this, the answer data is transferred to the preprocessing device.

[0193] Step 3:

[0194] The server's preprocessing unit preprocesses the received answer data through tokenization and normalization. The input is the answer data, and the output is data in a format suitable for natural language processing techniques. Specifically, this involves removing extraneous spaces and splitting words.

[0195] Step 4:

[0196] The server's analysis system uses natural language processing technology to analyze pre-processed answer data and calculates a scoring score. It uses pre-processed data as input and generates a scoring score as output. The Google Cloud Natural Language API is used for the analysis to evaluate the relevance and comprehensiveness of the answers.

[0197] Step 5:

[0198] The server's emotion recognition system recognizes the user's emotions based on the analyzed data and generates that information as output. Specifically, Microsoft Azure Text Analytics is used in this process to analyze the emotions contained in the responses.

[0199] Step 6:

[0200] The server's recording device stores the scoring scores generated by the analysis device and the results of emotion recognition in a database. It uses scores and emotion data as input and manages the data accordingly.

[0201] Step 7:

[0202] The server's output device generates feedback for the user based on saved scoring results and sentiment feedback. The input is the saved results, and the output is the feedback message. A generative AI model and text generation API are used to create specific, sentiment-based advice.

[0203] Step 8:

[0204] The user's device receives feedback messages sent from the server and displays them to the user. It takes messages from the server as input and outputs them in a format that the user can confirm.

[0205] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0206] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0207] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0208] [Second Embodiment]

[0209] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0210] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0211] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0212] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0213] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0215] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0216] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0217] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0218] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0219] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0220] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0221] The automated scoring system for written response questions in this invention primarily operates through the collaborative efforts of multiple devices installed on a server and a user's terminal. When a user inputs and submits their answer to a written response question using their terminal, a data receiving device on the server receives it. Subsequently, a preprocessing device performs data cleansing, such as normalization and tokenization, to prepare the answer data for natural language processing.

[0222] Next, the analysis device uses natural language processing technology to analyze the pre-processed answer data. The analysis calculates a score based on criteria such as the accuracy of the answer, logical structure, and comprehensiveness of related information. For example, if a student answers the history question, "Explain the causes of World War II," with the answer, "The effects of World War I and the economic crisis were the causes," the analysis device evaluates the accuracy and depth of this explanation.

[0223] After analysis, the recording device saves the calculated scores to a database for later review by the user. This data is used for performance evaluation and other learning analysis tools. The output device provides the user's terminal with the scoring results and feedback for their answers. This allows the user to quickly understand the evaluation of their answers and obtain specific guidance on areas for improvement.

[0224] As a concrete example, when a user takes an online test and enters their answers, the entire process is executed automatically, and the scoring results are displayed on the device within minutes. For example, feedback such as "Your score is 75 points. Your understanding of the questions is good, but your explanation of the economic background is insufficient" is provided.

[0225] In summary, the present invention reduces the workload of teachers and improves the efficiency and fairness of grading through the automatic scoring of essay-type questions.

[0226] The following describes the processing flow.

[0227] Step 1:

[0228] The user uses their device to enter their answers to written questions and then presses the submit button to send the answers to the server.

[0229] Step 2:

[0230] The server's data receiving device receives the transmitted answer data, verifies that the data is in a receivable format, and records it in the log.

[0231] Step 3:

[0232] The server's preprocessor normalizes the received answer data, converting the text to lowercase and removing unnecessary spaces and special characters. It also performs spell checking and tokenizes the text into words.

[0233] Step 4:

[0234] The server's analysis device receives the pre-processed data and begins analysis using natural language processing technology. Here, it evaluates the relevance of the answers, the comprehensiveness of the perspectives, and the logical structure, and calculates a scoring score.

[0235] Step 5:

[0236] The server's recording device stores the scores calculated by the analysis device in a database, managing them so that each user's scoring results can be referenced later.

[0237] Step 6:

[0238] The server's output device generates analysis results and feedback, which are then provided to the user's terminal. This allows the user to quickly check information such as their score and areas for improvement regarding their answers.

[0239] (Example 1)

[0240] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0241] Grading essay-type questions is typically time-consuming and labor-intensive, and prone to inconsistencies in evaluation among graders. Furthermore, evaluating a large volume of responses quickly and fairly is difficult, hindering efficient feedback provision in educational settings.

[0242] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0243] In this invention, the server includes means for an information receiving device to receive answers to written questions transmitted from a user terminal, means for an information processing device to preprocess the received answers by normalization and tokenization and convert them into a format suitable for natural language processing, and means for an analysis device to analyze the preprocessed answers using a generative AI model and calculate a score based on the accuracy, logical structure, and information comprehensiveness of the answers. This enables the rapid, fair, and automatic scoring of answers to written questions and the efficient provision of feedback.

[0244] An "information receiving device" is a device used to receive answers to written questions transmitted from a user terminal.

[0245] An "information processing device" is a device that preprocesses received responses through normalization and tokenization, and converts them into a format suitable for natural language processing technology.

[0246] An "analysis device" is a device that uses a generative AI model to analyze the content of pre-processed answers and calculates a score based on the accuracy, logical structure, and comprehensiveness of the answers.

[0247] An "information recording device" is a device used to store analyzed scores as data.

[0248] An "information output device" is a device that provides users with scoring results and feedback, and makes that information available through an online interface.

[0249] A "generative AI model" is a model based on AI technology used to perform natural language analysis on written answers.

[0250] This invention relates to a system for automatically scoring answers to written response questions, and the server includes an information receiving device, an information processing device, an analysis device, an information recording device, and an information output device. Users use a terminal (including PCs and smartphones) to input and transmit answers to written response questions via an online interface. The server's information receiving device receives these answers. The main technologies and procedures used are described below in detail.

[0251] The information processing device performs normalization and tokenization of the received answer data. This is done using natural language processing libraries (e.g., NLTK and spaCy) to preprocess the answers, converting them into a data format that is easy for the generative AI model to analyze.

[0252] The analysis device uses a generative AI model to analyze pre-processed answers. The generative AI model utilizes transformer technology (e.g., BERT and GPT) to evaluate the accuracy, logical structure, and information coverage of the answers, and calculates a specific score. Because this model has already learned evaluation criteria for input answers, it can evaluate user answers quickly and fairly.

[0253] The information recording device securely stores the analyzed scores in a database. The database manages data for each user via a database management system such as MySQL or PostgreSQL, allowing for later reference.

[0254] The information output device provides users with scoring results and feedback. Users can check the results on their terminals via an online interface, and the device further enhances their motivation to learn by highlighting areas for improvement.

[0255] For example, if a user answers a history essay question with "It was caused by the effects of World War I and the economic crisis," the analysis device will analyze this and generate feedback such as, "Your answer is worth 75 points. Please elaborate on the economic background." An example of a prompt message could be a specific question such as, "Please evaluate the following answer: It was caused by the effects of World War I and the economic crisis." In this way, the present invention enables efficient and fair automatic scoring of essay questions.

[0256] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0257] Step 1:

[0258] The user enters their answers to the written questions using a terminal and clicks the submit button. This sends the answers to the server as text data.

[0259] Step 2:

[0260] The server's information receiving device receives the answer sent by the user. The input answer data is first temporarily stored and then passed on to subsequent processing. The input here is the user's answer, and the output is the receipt and initial storage of that answer data.

[0261] Step 3:

[0262] The server's information processing unit normalizes and tokenizes the received answers. Specifically, it uses a natural language processing library to remove unnecessary characters, spell check, and split the text into individual words. The input is the user's answer data, and the output is pre-processed, parseable data.

[0263] Step 4:

[0264] The server's analysis device analyzes pre-processed data using a generative AI model. A transformer model is utilized to calculate and analyze the answer. The input is pre-processed data, and the output is a score based on the accuracy, logical structure, and comprehensiveness of the information in the answer.

[0265] Step 5:

[0266] The server's data recording device saves the analysis results to a database. The score is associated with a user ID and is securely managed. The input is the analyzed score, and the output is the record stored in the database.

[0267] Step 6:

[0268] The server's information output device generates scoring results and feedback for the user and displays them on the terminal. Through the online interface, the user can check how their answers were evaluated, receive feedback, and use it for future learning. The input is the saved score and feedback content, and the output is the result display on the user's terminal.

[0269] (Application Example 1)

[0270] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0271] In today's educational environment, grading essay-type questions is time-consuming and laborious, placing a burden on educators. Furthermore, ensuring fairness in grading while providing timely feedback is a challenging task. There is also a need for a system that allows test-takers to quickly and specifically identify areas for improvement.

[0272] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0273] In this invention, the server includes a data acquisition device that acquires descriptive answers, an initial processing device that performs initial processing of the acquired answer information by information segmentation and formatting, and an analysis device that analyzes the initially processed answer information using natural language processing technology and calculates an evaluation score. This enables automated scoring and rapid feedback provision.

[0274] A "data acquisition device" is a means of obtaining written responses from users.

[0275] An "initial processing device" is a means of performing initial processing on acquired response information by performing information segmentation and formatting.

[0276] An "analysis device" is a means of analyzing initially processed response information using natural language processing technology and calculating an evaluation score.

[0277] A "storage device" is a means of storing the analyzed evaluation scores so that they can be referenced later.

[0278] An "output device" is a means of providing evaluation results and areas for improvement to the user.

[0279] A "communication terminal" is a device that allows users to check evaluation results and areas for improvement.

[0280] This invention is a system for automatically grading written response questions. Specifically, a server and a user's terminal work together. The user first inputs the answer to the written response question from their terminal. The terminal then transmits this answer data to the server's data acquisition device.

[0281] On the server, the received data is processed by the initial processing unit. The initial processing unit partitions and formats the data to convert it into a format suitable for natural language processing. This process involves tokenization and normalization of the information.

[0282] Next, the analysis device analyzes these initially processed data using natural language processing technology. The analysis device calculates an evaluation score and evaluates the accuracy, logical structure, and relevance of the answer. At this time, for natural language processing, software such as spaCy or BERT is used.

[0283] The results obtained from the analysis are recorded by the storage device. Thereby, the user can later confirm the evaluation of their own answer. The output device provides the evaluation result and individual feedback to the user's terminal.

[0284] As a specific example, suppose a high school student user answers a descriptive history question using an online learning app. When the user inputs the answer, it is immediately sent to the server, and the entire process is automatically executed. After a few minutes, the user receives feedback such as "Your score is 75 points. It would be good to delve a little deeper into the details of the reasons."

[0285] Also, as an example of a prompt sentence using a generative AI model, there is "Please evaluate the answer to the following descriptive history question: 'The causes of World War II are the impact of World War I and the economic crisis.'" In this way, the invention can be effectively implemented.

[0286] With this system, the evaluation work of educators is greatly streamlined, and prompt and fair feedback to examinees becomes possible.

[0287] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0288] Step 1:

[0289] When the user inputs the answer to the descriptive question on the terminal and presses the send button, the answer data is sent to the data acquisition device of the server. The input is text data by the user, and the output is data in a format that can be analyzed on the server side.

[0290] Step 2:

[0291] The initial processing unit on the server performs initial processing on the acquired answer data, such as tokenization and normalization. The input is text data sent by the user, which is then divided into tokens and formatted to remove unnecessary shapes and characters. The output is data in a format suitable for natural language processing.

[0292] Step 3:

[0293] The analysis system on the server analyzes the pre-processed answer data using natural language processing techniques. The input is the pre-processed data, and at this stage, spaCy and BERT are used to analyze the logical structure and relationships between the content, and to calculate the evaluation score. The output is the scoring result and a feedback message.

[0294] Step 4:

[0295] The server's storage device stores the analysis results and scores in a database. The input is the score data provided by the analysis device, and the output is the data stored for future reference.

[0296] Step 5:

[0297] The server's output device sends the scoring results and feedback to the user's terminal. The output is feedback information provided to the user, consisting of a score and advice text displayed on the terminal.

[0298] Step 6:

[0299] Users receive rapid feedback through their devices, allowing them to identify areas for further learning and improvement. At this point, users can understand which parts of their answers need improvement based on the information provided.

[0300] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0301] The automated scoring system for written questions of the present invention begins with the user inputting a written answer. When the user inputs and submits the answer using a terminal, the answer is received by a data receiving device installed on the server. The received data is normalized and tokenized by a preprocessing device and prepared in a format suitable for natural language processing.

[0302] After preprocessing, the server's analysis system uses natural language processing technology to analyze the answers in detail and calculate a scoring score. The analysis system scores the answers based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. This score is stored in a recording device and managed in a database.

[0303] Furthermore, this invention incorporates an emotion engine. The emotion engine automatically recognizes the user's emotions from their responses during the analysis process. This emotion information is used to generate feedback by the output device. For example, if the system recognizes from the responses that the user is feeling anxious, an encouraging message is automatically generated.

[0304] This feedback is provided to the user's terminal via an output device. The system allows users to quickly check their score and receive emotion-based advice for their answers. For example, if the answer to the history question, "Explain the impact of the Meiji Restoration," is "Japan advanced political reforms and achieved modernization. However, challenges remained in agricultural development," the analysis device will assign a score of 85 points, and the emotion engine will generate emotion-based feedback such as, "You seem a little nervous. Please answer with confidence."

[0305] In this way, this system not only automatically grades descriptive questions, but also provides feedback considering the user's emotions, thus realizing a more personalized learning experience.

[0306] The following describes the processing flow.

[0307] Step 1:

[0308] The user uses the terminal to input an answer to a descriptive question and clicks the send button to send the answer data to the server.

[0309] Step 2:

[0310] The data receiving device of the server receives the transmitted answer data and records the data reception log.

[0311] Step 3:

[0312] The preprocessing device of the server normalizes, tokenizes, and spell-checks the received answer data, and arranges it in a form suitable for natural language processing.

[0313] Step 4:

[0314] The analysis device of the server analyzes the preprocessed data and calculates a scoring score based on the relevance of the answer content, comprehensiveness of viewpoints, and logical structure.

[0315] Step 5:

[0316] The emotion engine of the server recognizes the emotion from the user's text included in the answer data and adds the information to the analysis result.

[0317] Step 6:

[0318] The recording device of the server saves the calculated score and emotion recognition result in the database, and organizes and manages this data.

[0319] Step 7:

[0320] The server's output device provides the user's terminal with scoring results and sentiment-based feedback. For example, it might generate a message such as, "Your score is 85 points. You seem a little nervous. Please continue answering with confidence."

[0321] (Example 2)

[0322] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0323] In automated scoring of essay-type questions, there is a need to enhance the objectivity of evaluation and improve learner motivation by providing feedback based on the respondent's emotions. However, conventional automated scoring systems have had difficulty accurately analyzing the content of written responses and generating individual emotional feedback.

[0324] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0325] In this invention, the server includes a data receiving device for receiving written answers, a preprocessing device for tokenizing and normalizing the received answer data, an analysis device for analyzing the preprocessed answer data using natural language processing techniques and calculating an evaluation score, an emotion recognition device for extracting emotions from the answers, and an output device for providing the terminal with the scoring results and emotion-based feedback. This enables highly accurate analysis of written content and the provision of emotion-based, personalized feedback.

[0326] A "data receiving device" is a component that has the function of receiving written answers sent by a user from their terminal.

[0327] A "preprocessor" is a component that tokenizes and normalizes received answer data, preparing it for analysis in a suitable format.

[0328] An "analysis device" is a component that has the function of analyzing pre-processed answer data using natural language processing technology, evaluating its content, and calculating a score.

[0329] The "evaluation score" is a numerical value calculated by the analysis device based on the relevance, comprehensiveness, and logical structure of the answer.

[0330] A "recording device" is a component used to record and manage the scores obtained through analysis.

[0331] An "emotion recognition device" is a component that analyzes and identifies emotions extracted from the user's responses.

[0332] An "output device" is a component that provides the user with feedback generated based on the analysis results and emotions.

[0333] A "remote interface" is a set of protocols and software that allows users to communicate with a server online and check results and feedback.

[0334] In an embodiment of the present invention, the server first uses specific hardware and software to provide automatic scoring of written questions and sentiment-based feedback. This system consists of the following main elements:

[0335] The server side is equipped with a data receiving device, preprocessing device, analysis device, emotion recognition device, and output device. The data receiving device is responsible for receiving written answers that users submit using their terminals. Users using terminals input and submit answers to history questions, such as "Explain the impact of the Meiji Restoration."

[0336] The preprocessor uses natural language processing libraries (e.g., NLTK or SpaCy) to tokenize and normalize the received response data. This preprocessing step prepares the data for analysis.

[0337] The analysis device uses a generative AI model (e.g., GPT, BERT) employing natural language processing technology to analyze the content of the answers and calculate an evaluation score based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. For example, the model assigns a score of 85 points to the specific answer, "Japan advanced political reforms and achieved modernization. However, challenges remained in the development of agriculture."

[0338] The emotion recognition device utilizes an emotion analysis library (e.g., TextBlob or VADER) to extract emotions from the user's responses during the analysis process. Based on the word choices in the responses, it determines whether the user is experiencing a specific emotion, such as anxiety or relief.

[0339] The output device generates feedback derived from the analysis results and sentiment analysis, and provides it to the user's terminal. This allows the user to see their score for their answer along with sentiment-based feedback, such as "Approach this with confidence."

[0340] As described above, this system performs specific analysis and generates feedback using a generation AI model and example prompts. A possible example of a prompt is, "Analyze the student's answer and generate emotion-based feedback." By notifying the user of the feedback through methods such as pop-up notifications, the system allows them to receive the information immediately.

[0341] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0342] Step 1:

[0343] The user uses a terminal to input and submit written answers. The information entered is data in text format, with the user providing the answers. The terminal sends this answer data to the server via the internet.

[0344] Step 2:

[0345] The server's data receiving device receives the answer data sent from the user's terminal. The input is the data sent by the user, and the output is the answer data stored in the receiving device. At this stage, the answer data is stored on the server and passed on to the next processing step.

[0346] Step 3:

[0347] The server's preprocessing unit tokenizes and normalizes the received response data. The input is the received response data, and the output is data in a format suitable for the natural language processing model. This preprocessing removes extraneous whitespace and special characters, and tokenizes sentences into word units.

[0348] Step 4:

[0349] The server's analysis device performs analysis using a generative AI model with pre-processed data. The input is tokenized answer data, and the output is the evaluation score of the answers. The generative AI model is used to evaluate the relevance, logic, and comprehensiveness of the answers and assign a score. For example, a score of 85 points may be calculated.

[0350] Step 5:

[0351] The server's emotion recognition device extracts emotions from the data obtained during the analysis process. The input is pre-processed answer data, and the output is information indicating the user's emotions. The user's feelings are identified by the frequency of positive or negative expressions in the answers.

[0352] Step 6:

[0353] The server's output device generates feedback based on the scoring score and sentiment information, and sends it to the terminal. The input is the analysis result and sentiment information, and the output is the feedback message presented to the user. Specifically, an encouraging message such as "Your answer is correct. Please keep your confidence." is generated and displayed on the user's screen.

[0354] (Application Example 2)

[0355] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0356] Conventional automated scoring systems for written answers could evaluate the content of the answers, but they could not provide feedback based on the user's emotional state. Therefore, it was difficult to provide a more personalized learning experience for individual users.

[0357] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0358] In this invention, the server includes means for a data receiving device to receive descriptive answers, means for a preprocessing device to preprocess the received answer data by tokenization and normalization, and means for an analysis device to analyze the preprocessed answer data using natural language processing technology and calculate a scoring score. This makes it possible to evaluate the user's answers and, further, to recognize the user's emotions from the answers using an emotion recognition device, to provide emotion-based feedback.

[0359] A "data receiving device" is a device that has the function of receiving written answers sent by a user.

[0360] A "preprocessing device" is a device that has the function of preparing received answer data into a format suitable for natural language processing through tokenization and normalization.

[0361] An "analysis device" is a device that uses natural language processing technology to analyze pre-processed answer data in detail and evaluate the answers.

[0362] A "scoring score" is a numerical value that indicates the evaluation of an answer calculated by an analysis device.

[0363] A "recording device" is a device used to store analyzed scoring scores and manage them in a database.

[0364] An "emotion recognition device" is a device that automatically recognizes emotions from a user's answers and processes that information.

[0365] "Feedback" refers to information used to provide users with scoring results and advice tailored to their emotions regarding their answers.

[0366] An "output device" is a device that presents and allows users to review scoring results and feedback.

[0367] The system realizing this invention operates primarily with a server, a terminal, and a user. The server consists of a data receiving device, a preprocessing device, an analysis device, a recording device, an emotion recognition device, and an output device. The server receives written answers sent from the user via the data receiving device. The received answer data is tokenized and normalized by the preprocessing device and prepared in a format suitable for natural language processing.

[0368] This organized data is then analyzed in detail using an analysis device, and the answers are evaluated using natural language processing technology to calculate a scoring score. Software such as the Google Cloud Natural Language API is used in this process. In addition, an emotion recognition device is used, and Microsoft Azure Text Analytics is used to automatically recognize the user's emotions from the answers.

[0369] The analysis results and sentiment information are integrated by a recording device and managed in a database. The output device provides the scoring results to the user's terminal and generates and displays sentiment-based feedback. A text generation API is used for this generation. For example, if a middle school student answers the question, "Explain the economic development of the Showa era," with "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged," this answer is analyzed and scored appropriately. If the elements targeted for analysis are evaluated as sufficient to meet the scoring criteria, a message such as, "Your answer is 90 points. Very well done! You seem a little unsure, but the content is solid," is generated and displayed as feedback.

[0370] Examples of input prompts for a generative AI model:

[0371] User's answer: "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged."

[0372] Grading criteria: "Economic growth, industry, economic power, environmental issues"

[0373] It generates analysis and emotional feedback to provide responses to the user.

[0374] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0375] Step 1:

[0376] The user enters and submits written answers using their own device. The user's answer data is generated as input. The user's device then sends this data to a data receiving device.

[0377] Step 2:

[0378] The server's data receiving device receives the user's answer data. It handles the answer data sent from the terminal as input. After this, the answer data is transferred to the preprocessing device.

[0379] Step 3:

[0380] The server's preprocessing unit preprocesses the received answer data through tokenization and normalization. The input is the answer data, and the output is data in a format suitable for natural language processing techniques. Specifically, this involves removing extraneous spaces and splitting words.

[0381] Step 4:

[0382] The server's analysis system uses natural language processing technology to analyze pre-processed answer data and calculates a scoring score. It uses pre-processed data as input and generates a scoring score as output. The Google Cloud Natural Language API is used for the analysis to evaluate the relevance and comprehensiveness of the answers.

[0383] Step 5:

[0384] The server's emotion recognition system recognizes the user's emotions based on the analyzed data and generates that information as output. Specifically, Microsoft Azure Text Analytics is used in this process to analyze the emotions contained in the responses.

[0385] Step 6:

[0386] The server's recording device stores the scoring scores generated by the analysis device and the results of emotion recognition in a database. It uses scores and emotion data as input and manages the data accordingly.

[0387] Step 7:

[0388] The server's output device generates feedback for the user based on saved scoring results and sentiment feedback. The input is the saved results, and the output is the feedback message. A generative AI model and text generation API are used to create specific, sentiment-based advice.

[0389] Step 8:

[0390] The user's device receives feedback messages sent from the server and displays them to the user. It takes messages from the server as input and outputs them in a format that the user can confirm.

[0391] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0392] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0393] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0394] [Third Embodiment]

[0395] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0396] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0397] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0398] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0399] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0400] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0401] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0402] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0403] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0404] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0405] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0406] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0407] The automated scoring system for written response questions in this invention primarily operates through the collaborative efforts of multiple devices installed on a server and a user's terminal. When a user inputs and submits their answer to a written response question using their terminal, a data receiving device on the server receives it. Subsequently, a preprocessing device performs data cleansing, such as normalization and tokenization, to prepare the answer data for natural language processing.

[0408] Next, the analysis device uses natural language processing technology to analyze the pre-processed answer data. The analysis calculates a score based on criteria such as the accuracy of the answer, logical structure, and comprehensiveness of related information. For example, if a student answers the history question, "Explain the causes of World War II," with the answer, "The effects of World War I and the economic crisis were the causes," the analysis device evaluates the accuracy and depth of this explanation.

[0409] After analysis, the recording device saves the calculated scores to a database for later review by the user. This data is used for performance evaluation and other learning analysis tools. The output device provides the user's terminal with the scoring results and feedback for their answers. This allows the user to quickly understand the evaluation of their answers and obtain specific guidance on areas for improvement.

[0410] As a concrete example, when a user takes an online test and enters their answers, the entire process is executed automatically, and the scoring results are displayed on the device within minutes. For example, feedback such as "Your score is 75 points. Your understanding of the questions is good, but your explanation of the economic background is insufficient" is provided.

[0411] In summary, the present invention reduces the workload of teachers and improves the efficiency and fairness of grading through the automatic scoring of essay-type questions.

[0412] The following describes the processing flow.

[0413] Step 1:

[0414] The user uses their device to enter their answers to written questions and then presses the submit button to send the answers to the server.

[0415] Step 2:

[0416] The server's data receiving device receives the transmitted answer data, verifies that the data is in a receivable format, and records it in the log.

[0417] Step 3:

[0418] The server's preprocessor normalizes the received answer data, converting the text to lowercase and removing unnecessary spaces and special characters. It also performs spell checking and tokenizes the text into words.

[0419] Step 4:

[0420] The server's analysis device receives the pre-processed data and begins analysis using natural language processing technology. Here, it evaluates the relevance of the answers, the comprehensiveness of the perspectives, and the logical structure, and calculates a scoring score.

[0421] Step 5:

[0422] The server's recording device stores the scores calculated by the analysis device in a database, managing them so that each user's scoring results can be referenced later.

[0423] Step 6:

[0424] The server's output device generates analysis results and feedback, which are then provided to the user's terminal. This allows the user to quickly check information such as their score and areas for improvement regarding their answers.

[0425] (Example 1)

[0426] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0427] Grading essay-type questions is typically time-consuming and labor-intensive, and prone to inconsistencies in evaluation among graders. Furthermore, evaluating a large volume of responses quickly and fairly is difficult, hindering efficient feedback provision in educational settings.

[0428] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0429] In this invention, the server includes means for an information receiving device to receive answers to written questions transmitted from a user terminal, means for an information processing device to preprocess the received answers by normalization and tokenization and convert them into a format suitable for natural language processing, and means for an analysis device to analyze the preprocessed answers using a generative AI model and calculate a score based on the accuracy, logical structure, and information comprehensiveness of the answers. This enables the rapid, fair, and automatic scoring of answers to written questions and the efficient provision of feedback.

[0430] An "information receiving device" is a device used to receive answers to written questions transmitted from a user terminal.

[0431] An "information processing device" is a device that preprocesses received responses through normalization and tokenization, and converts them into a format suitable for natural language processing technology.

[0432] An "analysis device" is a device that uses a generative AI model to analyze the content of pre-processed answers and calculates a score based on the accuracy, logical structure, and comprehensiveness of the answers.

[0433] An "information recording device" is a device used to store analyzed scores as data.

[0434] An "information output device" is a device that provides users with scoring results and feedback, and makes that information available through an online interface.

[0435] A "generative AI model" is a model based on AI technology used to perform natural language analysis on written answers.

[0436] This invention relates to a system for automatically scoring answers to written response questions, and the server includes an information receiving device, an information processing device, an analysis device, an information recording device, and an information output device. Users use a terminal (including PCs and smartphones) to input and transmit answers to written response questions via an online interface. The server's information receiving device receives these answers. The main technologies and procedures used are described below in detail.

[0437] The information processing device performs normalization and tokenization of the received answer data. This is done using natural language processing libraries (e.g., NLTK and spaCy) to preprocess the answers, converting them into a data format that is easy for the generative AI model to analyze.

[0438] The analysis device uses a generative AI model to analyze pre-processed answers. The generative AI model utilizes transformer technology (e.g., BERT and GPT) to evaluate the accuracy, logical structure, and information coverage of the answers, and calculates a specific score. Because this model has already learned evaluation criteria for input answers, it can evaluate user answers quickly and fairly.

[0439] The information recording device securely stores the analyzed scores in a database. The database manages data for each user via a database management system such as MySQL or PostgreSQL, allowing for later reference.

[0440] The information output device provides users with scoring results and feedback. Users can check the results on their terminals via an online interface, and the device further enhances their motivation to learn by highlighting areas for improvement.

[0441] For example, if a user answers a history essay question with "It was caused by the effects of World War I and the economic crisis," the analysis device will analyze this and generate feedback such as, "Your answer is worth 75 points. Please elaborate on the economic background." An example of a prompt message could be a specific question such as, "Please evaluate the following answer: It was caused by the effects of World War I and the economic crisis." In this way, the present invention enables efficient and fair automatic scoring of essay questions.

[0442] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0443] Step 1:

[0444] The user enters their answers to the written questions using a terminal and clicks the submit button. This sends the answers to the server as text data.

[0445] Step 2:

[0446] The server's information receiving device receives the answer sent by the user. The input answer data is first temporarily stored and then passed on to subsequent processing. The input here is the user's answer, and the output is the receipt and initial storage of that answer data.

[0447] Step 3:

[0448] The server's information processing unit normalizes and tokenizes the received answers. Specifically, it uses a natural language processing library to remove unnecessary characters, spell check, and split the text into individual words. The input is the user's answer data, and the output is pre-processed, parseable data.

[0449] Step 4:

[0450] The server's analysis device analyzes pre-processed data using a generative AI model. A transformer model is utilized to calculate and analyze the answer. The input is pre-processed data, and the output is a score based on the accuracy, logical structure, and comprehensiveness of the information in the answer.

[0451] Step 5:

[0452] The server's data recording device saves the analysis results to a database. The score is associated with a user ID and is securely managed. The input is the analyzed score, and the output is the record stored in the database.

[0453] Step 6:

[0454] The server's information output device generates scoring results and feedback for the user and displays them on the terminal. Through the online interface, the user can check how their answers were evaluated, receive feedback, and use it for future learning. The input is the saved score and feedback content, and the output is the result display on the user's terminal.

[0455] (Application Example 1)

[0456] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0457] In today's educational environment, grading essay-type questions is time-consuming and laborious, placing a burden on educators. Furthermore, ensuring fairness in grading while providing timely feedback is a challenging task. There is also a need for a system that allows test-takers to quickly and specifically identify areas for improvement.

[0458] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0459] In this invention, the server includes a data acquisition device that acquires descriptive answers, an initial processing device that performs initial processing of the acquired answer information by information segmentation and formatting, and an analysis device that analyzes the initially processed answer information using natural language processing technology and calculates an evaluation score. This enables automated scoring and rapid feedback provision.

[0460] A "data acquisition device" is a means of obtaining written responses from users.

[0461] An "initial processing device" is a means of performing initial processing on acquired response information by performing information segmentation and formatting.

[0462] An "analysis device" is a means of analyzing initially processed response information using natural language processing technology and calculating an evaluation score.

[0463] A "storage device" is a means of storing the analyzed evaluation scores so that they can be referenced later.

[0464] An "output device" is a means of providing evaluation results and areas for improvement to the user.

[0465] A "communication terminal" is a device that allows users to check evaluation results and areas for improvement.

[0466] This invention is a system for automatically grading written response questions. Specifically, a server and a user's terminal work together. The user first inputs the answer to the written response question from their terminal. The terminal then transmits this answer data to the server's data acquisition device.

[0467] On the server, the received data is processed by the initial processing unit. The initial processing unit partitions and formats the data to convert it into a format suitable for natural language processing. This process involves tokenization and normalization of the information.

[0468] Next, the analysis device analyzes this pre-processed data using natural language processing techniques. The analysis device calculates evaluation scores and assesses the accuracy, logical structure, and relevance of the answers. For natural language processing, software such as spaCy or BERT is used.

[0469] The results obtained from the analysis are recorded by a storage device. This allows the user to later review the evaluation of their answers. The output device provides the evaluation results and individual feedback to the user's terminal.

[0470] As a concrete example, suppose a high school student uses an online learning app to answer a history essay question. When the user enters their answer, it is instantly sent to the server, and the entire process is executed automatically. A few minutes later, the user receives feedback such as, "Your score is 75 points. You should delve a little deeper into the details of the cause."

[0471] Furthermore, an example of a prompt sentence generated using an AI model is: "Evaluate your answer to the following history question: 'The causes of World War II were the effects of World War I and the economic crisis.'" In this way, the invention can be effectively implemented.

[0472] This system significantly streamlines the evaluation process for educators and enables prompt and fair feedback to test takers.

[0473] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0474] Step 1:

[0475] When a user enters their answer to a written question into their terminal and presses the submit button, the answer data is sent to the server's data acquisition device. The input is text data provided by the user, and the output is data in a format that can be parsed on the server side.

[0476] Step 2:

[0477] The initial processing unit on the server performs initial processing on the acquired answer data, such as tokenization and normalization. The input is text data sent by the user, which is then divided into tokens and formatted to remove unnecessary shapes and characters. The output is data in a format suitable for natural language processing.

[0478] Step 3:

[0479] The analysis system on the server analyzes the pre-processed answer data using natural language processing techniques. The input is the pre-processed data, and at this stage, spaCy and BERT are used to analyze the logical structure and relationships between the content, and to calculate the evaluation score. The output is the scoring result and a feedback message.

[0480] Step 4:

[0481] The server's storage device stores the analysis results and scores in a database. The input is the score data provided by the analysis device, and the output is the data stored for future reference.

[0482] Step 5:

[0483] The server's output device sends the scoring results and feedback to the user's terminal. The output is feedback information provided to the user, consisting of a score and advice text displayed on the terminal.

[0484] Step 6:

[0485] Users receive rapid feedback through their devices, allowing them to identify areas for further learning and improvement. At this point, users can understand which parts of their answers need improvement based on the information provided.

[0486] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0487] The automated scoring system for written questions of the present invention begins with the user inputting a written answer. When the user inputs and submits the answer using a terminal, the answer is received by a data receiving device installed on the server. The received data is normalized and tokenized by a preprocessing device and prepared in a format suitable for natural language processing.

[0488] After preprocessing, the server's analysis system uses natural language processing technology to analyze the answers in detail and calculate a scoring score. The analysis system scores the answers based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. This score is stored in a recording device and managed in a database.

[0489] Furthermore, this invention incorporates an emotion engine. The emotion engine automatically recognizes the user's emotions from their responses during the analysis process. This emotion information is used to generate feedback by the output device. For example, if the system recognizes from the responses that the user is feeling anxious, an encouraging message is automatically generated.

[0490] This feedback is provided to the user's terminal via an output device. The system allows users to quickly check their score and receive emotion-based advice for their answers. For example, if the answer to the history question, "Explain the impact of the Meiji Restoration," is "Japan advanced political reforms and achieved modernization. However, challenges remained in agricultural development," the analysis device will assign a score of 85 points, and the emotion engine will generate emotion-based feedback such as, "You seem a little nervous. Please answer with confidence."

[0491] Thus, this system not only automatically grades written questions but also provides feedback that takes into account the user's emotions, thereby realizing a more personalized learning experience.

[0492] The following describes the processing flow.

[0493] Step 1:

[0494] The user uses their device to enter their answers to written questions and clicks the submit button to send the answer data to the server.

[0495] Step 2:

[0496] The server's data receiving device receives the transmitted answer data and records a log of the data reception.

[0497] Step 3:

[0498] The server's preprocessor normalizes, tokenizes, and spell-checks the received answer data, preparing it for natural language processing.

[0499] Step 4:

[0500] The server's analysis device analyzes the pre-processed data and calculates a scoring score based on the relevance of the answers, the comprehensiveness of the perspectives, and the logical structure.

[0501] Step 5:

[0502] The server's emotion engine recognizes emotions from the user's text included in the answer data and adds that information to the analysis results.

[0503] Step 6:

[0504] The server's recording device stores the calculated score and emotion recognition results in a database, and this data is then organized and managed.

[0505] Step 7:

[0506] The server's output device provides the user's terminal with scoring results and sentiment-based feedback. For example, it might generate a message such as, "Your score is 85 points. You seem a little nervous. Please continue answering with confidence."

[0507] (Example 2)

[0508] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0509] In automated scoring of essay-type questions, there is a need to enhance the objectivity of evaluation and improve learner motivation by providing feedback based on the respondent's emotions. However, conventional automated scoring systems have had difficulty accurately analyzing the content of written responses and generating individual emotional feedback.

[0510] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0511] In this invention, the server includes a data receiving device for receiving written answers, a preprocessing device for tokenizing and normalizing the received answer data, an analysis device for analyzing the preprocessed answer data using natural language processing techniques and calculating an evaluation score, an emotion recognition device for extracting emotions from the answers, and an output device for providing the terminal with the scoring results and emotion-based feedback. This enables highly accurate analysis of written content and the provision of emotion-based, personalized feedback.

[0512] A "data receiving device" is a component that has the function of receiving written answers sent by a user from their terminal.

[0513] A "preprocessor" is a component that tokenizes and normalizes received answer data, preparing it for analysis in a suitable format.

[0514] An "analysis device" is a component that has the function of analyzing pre-processed answer data using natural language processing technology, evaluating its content, and calculating a score.

[0515] The "evaluation score" is a numerical value calculated by the analysis device based on the relevance, comprehensiveness, and logical structure of the answer.

[0516] A "recording device" is a component used to record and manage the scores obtained through analysis.

[0517] An "emotion recognition device" is a component that analyzes and identifies emotions extracted from the user's responses.

[0518] An "output device" is a component that provides the user with feedback generated based on the analysis results and emotions.

[0519] A "remote interface" is a set of protocols and software that allows users to communicate with a server online and check results and feedback.

[0520] In an embodiment of the present invention, the server first uses specific hardware and software to provide automatic scoring of written questions and sentiment-based feedback. This system consists of the following main elements:

[0521] The server side is equipped with a data receiving device, preprocessing device, analysis device, emotion recognition device, and output device. The data receiving device is responsible for receiving written answers that users submit using their terminals. Users using terminals input and submit answers to history questions, such as "Explain the impact of the Meiji Restoration."

[0522] The preprocessor uses natural language processing libraries (e.g., NLTK or SpaCy) to tokenize and normalize the received response data. This preprocessing step prepares the data for analysis.

[0523] The analysis device uses a generative AI model (e.g., GPT, BERT) employing natural language processing technology to analyze the content of the answers and calculate an evaluation score based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. For example, the model assigns a score of 85 points to the specific answer, "Japan advanced political reforms and achieved modernization. However, challenges remained in the development of agriculture."

[0524] The emotion recognition device utilizes an emotion analysis library (e.g., TextBlob or VADER) to extract emotions from the user's responses during the analysis process. Based on the word choices in the responses, it determines whether the user is experiencing a specific emotion, such as anxiety or relief.

[0525] The output device generates feedback derived from the analysis results and sentiment analysis, and provides it to the user's terminal. This allows the user to see their score for their answer along with sentiment-based feedback, such as "Approach this with confidence."

[0526] As described above, this system performs specific analysis and generates feedback using a generation AI model and example prompts. A possible example of a prompt is, "Analyze the student's answer and generate emotion-based feedback." By notifying the user of the feedback through methods such as pop-up notifications, the system allows them to receive the information immediately.

[0527] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0528] Step 1:

[0529] The user uses a terminal to input and submit written answers. The information entered is data in text format, with the user providing the answers. The terminal sends this answer data to the server via the internet.

[0530] Step 2:

[0531] The server's data receiving device receives the answer data sent from the user's terminal. The input is the data sent by the user, and the output is the answer data stored in the receiving device. At this stage, the answer data is stored on the server and passed on to the next processing step.

[0532] Step 3:

[0533] The server's preprocessing unit tokenizes and normalizes the received response data. The input is the received response data, and the output is data in a format suitable for the natural language processing model. This preprocessing removes extraneous whitespace and special characters, and tokenizes sentences into word units.

[0534] Step 4:

[0535] The server's analysis device performs analysis using a generative AI model with pre-processed data. The input is tokenized answer data, and the output is the evaluation score of the answers. The generative AI model is used to evaluate the relevance, logic, and comprehensiveness of the answers and assign a score. For example, a score of 85 points may be calculated.

[0536] Step 5:

[0537] The server's emotion recognition device extracts emotions from the data obtained during the analysis process. The input is pre-processed answer data, and the output is information indicating the user's emotions. The user's feelings are identified by the frequency of positive or negative expressions in the answers.

[0538] Step 6:

[0539] The server's output device generates feedback based on the scoring score and sentiment information, and sends it to the terminal. The input is the analysis result and sentiment information, and the output is the feedback message presented to the user. Specifically, an encouraging message such as "Your answer is correct. Please keep your confidence." is generated and displayed on the user's screen.

[0540] (Application Example 2)

[0541] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0542] Conventional automated scoring systems for written answers could evaluate the content of the answers, but they could not provide feedback based on the user's emotional state. Therefore, it was difficult to provide a more personalized learning experience for individual users.

[0543] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0544] In this invention, the server includes means for a data receiving device to receive descriptive answers, means for a preprocessing device to preprocess the received answer data by tokenization and normalization, and means for an analysis device to analyze the preprocessed answer data using natural language processing technology and calculate a scoring score. This makes it possible to evaluate the user's answers and, further, to recognize the user's emotions from the answers using an emotion recognition device, to provide emotion-based feedback.

[0545] A "data receiving device" is a device that has the function of receiving written answers sent by a user.

[0546] A "preprocessing device" is a device that has the function of preparing received answer data into a format suitable for natural language processing through tokenization and normalization.

[0547] An "analysis device" is a device that uses natural language processing technology to analyze pre-processed answer data in detail and evaluate the answers.

[0548] A "scoring score" is a numerical value that indicates the evaluation of an answer calculated by an analysis device.

[0549] A "recording device" is a device used to store analyzed scoring scores and manage them in a database.

[0550] An "emotion recognition device" is a device that automatically recognizes emotions from a user's answers and processes that information.

[0551] "Feedback" refers to information used to provide users with scoring results and advice tailored to their emotions regarding their answers.

[0552] An "output device" is a device that presents and allows users to review scoring results and feedback.

[0553] The system realizing this invention operates primarily with a server, a terminal, and a user. The server consists of a data receiving device, a preprocessing device, an analysis device, a recording device, an emotion recognition device, and an output device. The server receives written answers sent from the user via the data receiving device. The received answer data is tokenized and normalized by the preprocessing device and prepared in a format suitable for natural language processing.

[0554] This organized data is then analyzed in detail using an analysis device, and the answers are evaluated using natural language processing technology to calculate a scoring score. Software such as the Google Cloud Natural Language API is used in this process. In addition, an emotion recognition device is used, and Microsoft Azure Text Analytics is used to automatically recognize the user's emotions from the answers.

[0555] The analysis results and sentiment information are integrated by a recording device and managed in a database. The output device provides the scoring results to the user's terminal and generates and displays sentiment-based feedback. A text generation API is used for this generation. For example, if a middle school student answers the question, "Explain the economic development of the Showa era," with "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged," this answer is analyzed and scored appropriately. If the elements targeted for analysis are evaluated as sufficient to meet the scoring criteria, a message such as, "Your answer is 90 points. Very well done! You seem a little unsure, but the content is solid," is generated and displayed as feedback.

[0556] Examples of input prompts for a generative AI model:

[0557] User's answer: "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged."

[0558] Grading criteria: "Economic growth, industry, economic power, environmental issues"

[0559] It generates analysis and emotional feedback to provide responses to the user.

[0560] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0561] Step 1:

[0562] The user enters and submits written answers using their own device. The user's answer data is generated as input. The user's device then sends this data to a data receiving device.

[0563] Step 2:

[0564] The server's data receiving device receives the user's answer data. It handles the answer data sent from the terminal as input. After this, the answer data is transferred to the preprocessing device.

[0565] Step 3:

[0566] The server's preprocessing unit preprocesses the received answer data through tokenization and normalization. The input is the answer data, and the output is data in a format suitable for natural language processing techniques. Specifically, this involves removing extraneous spaces and splitting words.

[0567] Step 4:

[0568] The server's analysis system uses natural language processing technology to analyze pre-processed answer data and calculates a scoring score. It uses pre-processed data as input and generates a scoring score as output. The Google Cloud Natural Language API is used for the analysis to evaluate the relevance and comprehensiveness of the answers.

[0569] Step 5:

[0570] The server's emotion recognition system recognizes the user's emotions based on the analyzed data and generates that information as output. Specifically, Microsoft Azure Text Analytics is used in this process to analyze the emotions contained in the responses.

[0571] Step 6:

[0572] The server's recording device stores the scoring scores generated by the analysis device and the results of emotion recognition in a database. It uses scores and emotion data as input and manages the data accordingly.

[0573] Step 7:

[0574] The server's output device generates feedback for the user based on saved scoring results and sentiment feedback. The input is the saved results, and the output is the feedback message. A generative AI model and text generation API are used to create specific, sentiment-based advice.

[0575] Step 8:

[0576] The user's device receives feedback messages sent from the server and displays them to the user. It takes messages from the server as input and outputs them in a format that the user can confirm.

[0577] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0578] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0579] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0580] [Fourth Embodiment]

[0581] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0582] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0583] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0584] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0585] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0586] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0587] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0588] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0589] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0590] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0591] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0592] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0593] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0594] The automated scoring system for written response questions in this invention primarily operates through the collaborative efforts of multiple devices installed on a server and a user's terminal. When a user inputs and submits their answer to a written response question using their terminal, a data receiving device on the server receives it. Subsequently, a preprocessing device performs data cleansing, such as normalization and tokenization, to prepare the answer data for natural language processing.

[0595] Next, the analysis device uses natural language processing technology to analyze the pre-processed answer data. The analysis calculates a score based on criteria such as the accuracy of the answer, logical structure, and comprehensiveness of related information. For example, if a student answers the history question, "Explain the causes of World War II," with the answer, "The effects of World War I and the economic crisis were the causes," the analysis device evaluates the accuracy and depth of this explanation.

[0596] After analysis, the recording device saves the calculated scores to a database for later review by the user. This data is used for performance evaluation and other learning analysis tools. The output device provides the user's terminal with the scoring results and feedback for their answers. This allows the user to quickly understand the evaluation of their answers and obtain specific guidance on areas for improvement.

[0597] As a concrete example, when a user takes an online test and enters their answers, the entire process is executed automatically, and the scoring results are displayed on the device within minutes. For example, feedback such as "Your score is 75 points. Your understanding of the questions is good, but your explanation of the economic background is insufficient" is provided.

[0598] In summary, the present invention reduces the workload of teachers and improves the efficiency and fairness of grading through the automatic scoring of essay-type questions.

[0599] The following describes the processing flow.

[0600] Step 1:

[0601] The user uses their device to enter their answers to written questions and then presses the submit button to send the answers to the server.

[0602] Step 2:

[0603] The server's data receiving device receives the transmitted answer data, verifies that the data is in a receivable format, and records it in the log.

[0604] Step 3:

[0605] The server's preprocessor normalizes the received answer data, converting the text to lowercase and removing unnecessary spaces and special characters. It also performs spell checking and tokenizes the text into words.

[0606] Step 4:

[0607] The server's analysis device receives the pre-processed data and begins analysis using natural language processing technology. Here, it evaluates the relevance of the answers, the comprehensiveness of the perspectives, and the logical structure, and calculates a scoring score.

[0608] Step 5:

[0609] The server's recording device stores the scores calculated by the analysis device in a database, managing them so that each user's scoring results can be referenced later.

[0610] Step 6:

[0611] The server's output device generates analysis results and feedback, which are then provided to the user's terminal. This allows the user to quickly check information such as their score and areas for improvement regarding their answers.

[0612] (Example 1)

[0613] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0614] Grading essay-type questions is typically time-consuming and labor-intensive, and prone to inconsistencies in evaluation among graders. Furthermore, evaluating a large volume of responses quickly and fairly is difficult, hindering efficient feedback provision in educational settings.

[0615] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0616] In this invention, the server includes means for an information receiving device to receive answers to written questions transmitted from a user terminal, means for an information processing device to preprocess the received answers by normalization and tokenization and convert them into a format suitable for natural language processing, and means for an analysis device to analyze the preprocessed answers using a generative AI model and calculate a score based on the accuracy, logical structure, and information comprehensiveness of the answers. This enables the rapid, fair, and automatic scoring of answers to written questions and the efficient provision of feedback.

[0617] An "information receiving device" is a device used to receive answers to written questions transmitted from a user terminal.

[0618] An "information processing device" is a device that preprocesses received responses through normalization and tokenization, and converts them into a format suitable for natural language processing technology.

[0619] An "analysis device" is a device that uses a generative AI model to analyze the content of pre-processed answers and calculates a score based on the accuracy, logical structure, and comprehensiveness of the answers.

[0620] An "information recording device" is a device used to store analyzed scores as data.

[0621] An "information output device" is a device that provides users with scoring results and feedback, and makes that information available through an online interface.

[0622] A "generative AI model" is a model based on AI technology used to perform natural language analysis on written answers.

[0623] This invention relates to a system for automatically scoring answers to written response questions, and the server includes an information receiving device, an information processing device, an analysis device, an information recording device, and an information output device. Users use a terminal (including PCs and smartphones) to input and transmit answers to written response questions via an online interface. The server's information receiving device receives these answers. The main technologies and procedures used are described below in detail.

[0624] The information processing device performs normalization and tokenization of the received answer data. This is done using natural language processing libraries (e.g., NLTK and spaCy) to preprocess the answers, converting them into a data format that is easy for the generative AI model to analyze.

[0625] The analysis device uses a generative AI model to analyze pre-processed answers. The generative AI model utilizes transformer technology (e.g., BERT and GPT) to evaluate the accuracy, logical structure, and information coverage of the answers, and calculates a specific score. Because this model has already learned evaluation criteria for input answers, it can evaluate user answers quickly and fairly.

[0626] The information recording device securely stores the analyzed scores in a database. The database manages data for each user via a database management system such as MySQL or PostgreSQL, allowing for later reference.

[0627] The information output device provides users with scoring results and feedback. Users can check the results on their terminals via an online interface, and the device further enhances their motivation to learn by highlighting areas for improvement.

[0628] For example, if a user answers a history essay question with "It was caused by the effects of World War I and the economic crisis," the analysis device will analyze this and generate feedback such as, "Your answer is worth 75 points. Please elaborate on the economic background." An example of a prompt message could be a specific question such as, "Please evaluate the following answer: It was caused by the effects of World War I and the economic crisis." In this way, the present invention enables efficient and fair automatic scoring of essay questions.

[0629] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0630] Step 1:

[0631] The user enters their answers to the written questions using a terminal and clicks the submit button. This sends the answers to the server as text data.

[0632] Step 2:

[0633] The server's information receiving device receives the answer sent by the user. The input answer data is first temporarily stored and then passed on to subsequent processing. The input here is the user's answer, and the output is the receipt and initial storage of that answer data.

[0634] Step 3:

[0635] The server's information processing unit normalizes and tokenizes the received answers. Specifically, it uses a natural language processing library to remove unnecessary characters, spell check, and split the text into individual words. The input is the user's answer data, and the output is pre-processed, parseable data.

[0636] Step 4:

[0637] The server's analysis device analyzes pre-processed data using a generative AI model. A transformer model is utilized to calculate and analyze the answer. The input is pre-processed data, and the output is a score based on the accuracy, logical structure, and comprehensiveness of the information in the answer.

[0638] Step 5:

[0639] The server's data recording device saves the analysis results to a database. The score is associated with a user ID and is securely managed. The input is the analyzed score, and the output is the record stored in the database.

[0640] Step 6:

[0641] The server's information output device generates scoring results and feedback for the user and displays them on the terminal. Through the online interface, the user can check how their answers were evaluated, receive feedback, and use it for future learning. The input is the saved score and feedback content, and the output is the result display on the user's terminal.

[0642] (Application Example 1)

[0643] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0644] In today's educational environment, grading essay-type questions is time-consuming and laborious, placing a burden on educators. Furthermore, ensuring fairness in grading while providing timely feedback is a challenging task. There is also a need for a system that allows test-takers to quickly and specifically identify areas for improvement.

[0645] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0646] In this invention, the server includes a data acquisition device that acquires descriptive answers, an initial processing device that performs initial processing of the acquired answer information by information segmentation and formatting, and an analysis device that analyzes the initially processed answer information using natural language processing technology and calculates an evaluation score. This enables automated scoring and rapid feedback provision.

[0647] A "data acquisition device" is a means of obtaining written responses from users.

[0648] An "initial processing device" is a means of performing initial processing on acquired response information by performing information segmentation and formatting.

[0649] An "analysis device" is a means of analyzing initially processed response information using natural language processing technology and calculating an evaluation score.

[0650] A "storage device" is a means of storing the analyzed evaluation scores so that they can be referenced later.

[0651] An "output device" is a means of providing evaluation results and areas for improvement to the user.

[0652] A "communication terminal" is a device that allows users to check evaluation results and areas for improvement.

[0653] This invention is a system for automatically grading written response questions. Specifically, a server and a user's terminal work together. The user first inputs the answer to the written response question from their terminal. The terminal then transmits this answer data to the server's data acquisition device.

[0654] On the server, the received data is processed by the initial processing unit. The initial processing unit partitions and formats the data to convert it into a format suitable for natural language processing. This process involves tokenization and normalization of the information.

[0655] Next, the analysis device analyzes this pre-processed data using natural language processing techniques. The analysis device calculates evaluation scores and assesses the accuracy, logical structure, and relevance of the answers. For natural language processing, software such as spaCy or BERT is used.

[0656] The results obtained from the analysis are recorded by a storage device. This allows the user to later review the evaluation of their answers. The output device provides the evaluation results and individual feedback to the user's terminal.

[0657] As a concrete example, suppose a high school student uses an online learning app to answer a history essay question. When the user enters their answer, it is instantly sent to the server, and the entire process is executed automatically. A few minutes later, the user receives feedback such as, "Your score is 75 points. You should delve a little deeper into the details of the cause."

[0658] Furthermore, an example of a prompt sentence generated using an AI model is: "Evaluate your answer to the following history question: 'The causes of World War II were the effects of World War I and the economic crisis.'" In this way, the invention can be effectively implemented.

[0659] This system significantly streamlines the evaluation process for educators and enables prompt and fair feedback to test takers.

[0660] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0661] Step 1:

[0662] When a user enters their answer to a written question into their terminal and presses the submit button, the answer data is sent to the server's data acquisition device. The input is text data provided by the user, and the output is data in a format that can be parsed on the server side.

[0663] Step 2:

[0664] The initial processing unit on the server performs initial processing on the acquired answer data, such as tokenization and normalization. The input is text data sent by the user, which is then divided into tokens and formatted to remove unnecessary shapes and characters. The output is data in a format suitable for natural language processing.

[0665] Step 3:

[0666] The analysis system on the server analyzes the pre-processed answer data using natural language processing techniques. The input is the pre-processed data, and at this stage, spaCy and BERT are used to analyze the logical structure and relationships between the content, and to calculate the evaluation score. The output is the scoring result and a feedback message.

[0667] Step 4:

[0668] The server's storage device stores the analysis results and scores in a database. The input is the score data provided by the analysis device, and the output is the data stored for future reference.

[0669] Step 5:

[0670] The server's output device sends the scoring results and feedback to the user's terminal. The output is feedback information provided to the user, consisting of a score and advice text displayed on the terminal.

[0671] Step 6:

[0672] Users receive rapid feedback through their devices, allowing them to identify areas for further learning and improvement. At this point, users can understand which parts of their answers need improvement based on the information provided.

[0673] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0674] The automated scoring system for written questions of the present invention begins with the user inputting a written answer. When the user inputs and submits the answer using a terminal, the answer is received by a data receiving device installed on the server. The received data is normalized and tokenized by a preprocessing device and prepared in a format suitable for natural language processing.

[0675] After preprocessing, the server's analysis system uses natural language processing technology to analyze the answers in detail and calculate a scoring score. The analysis system scores the answers based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. This score is stored in a recording device and managed in a database.

[0676] Furthermore, this invention incorporates an emotion engine. The emotion engine automatically recognizes the user's emotions from their responses during the analysis process. This emotion information is used to generate feedback by the output device. For example, if the system recognizes from the responses that the user is feeling anxious, an encouraging message is automatically generated.

[0677] This feedback is provided to the user's terminal via an output device. The system allows users to quickly check their score and receive emotion-based advice for their answers. For example, if the answer to the history question, "Explain the impact of the Meiji Restoration," is "Japan advanced political reforms and achieved modernization. However, challenges remained in agricultural development," the analysis device will assign a score of 85 points, and the emotion engine will generate emotion-based feedback such as, "You seem a little nervous. Please answer with confidence."

[0678] Thus, this system not only automatically grades written questions but also provides feedback that takes into account the user's emotions, thereby realizing a more personalized learning experience.

[0679] The following describes the processing flow.

[0680] Step 1:

[0681] The user uses their device to enter their answers to written questions and clicks the submit button to send the answer data to the server.

[0682] Step 2:

[0683] The server's data receiving device receives the transmitted answer data and records a log of the data reception.

[0684] Step 3:

[0685] The server's preprocessor normalizes, tokenizes, and spell-checks the received answer data, preparing it for natural language processing.

[0686] Step 4:

[0687] The server's analysis device analyzes the pre-processed data and calculates a scoring score based on the relevance of the answers, the comprehensiveness of the perspectives, and the logical structure.

[0688] Step 5:

[0689] The server's emotion engine recognizes emotions from the user's text included in the answer data and adds that information to the analysis results.

[0690] Step 6:

[0691] The server's recording device stores the calculated score and emotion recognition results in a database, and this data is then organized and managed.

[0692] Step 7:

[0693] The server's output device provides the user's terminal with scoring results and sentiment-based feedback. For example, it might generate a message such as, "Your score is 85 points. You seem a little nervous. Please continue answering with confidence."

[0694] (Example 2)

[0695] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0696] In automated scoring of essay-type questions, there is a need to enhance the objectivity of evaluation and improve learner motivation by providing feedback based on the respondent's emotions. However, conventional automated scoring systems have had difficulty accurately analyzing the content of written responses and generating individual emotional feedback.

[0697] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0698] In this invention, the server includes a data receiving device for receiving written answers, a preprocessing device for tokenizing and normalizing the received answer data, an analysis device for analyzing the preprocessed answer data using natural language processing techniques and calculating an evaluation score, an emotion recognition device for extracting emotions from the answers, and an output device for providing the terminal with the scoring results and emotion-based feedback. This enables highly accurate analysis of written content and the provision of emotion-based, personalized feedback.

[0699] A "data receiving device" is a component that has the function of receiving written answers sent by a user from their terminal.

[0700] A "preprocessor" is a component that tokenizes and normalizes received answer data, preparing it for analysis in a suitable format.

[0701] An "analysis device" is a component that has the function of analyzing pre-processed answer data using natural language processing technology, evaluating its content, and calculating a score.

[0702] The "evaluation score" is a numerical value calculated by the analysis device based on the relevance, comprehensiveness, and logical structure of the answer.

[0703] A "recording device" is a component used to record and manage the scores obtained through analysis.

[0704] An "emotion recognition device" is a component that analyzes and identifies emotions extracted from the user's responses.

[0705] An "output device" is a component that provides the user with feedback generated based on the analysis results and emotions.

[0706] A "remote interface" is a set of protocols and software that allows users to communicate with a server online and check results and feedback.

[0707] In an embodiment of the present invention, the server first uses specific hardware and software to provide automatic scoring of written questions and sentiment-based feedback. This system consists of the following main elements:

[0708] The server side is equipped with a data receiving device, preprocessing device, analysis device, emotion recognition device, and output device. The data receiving device is responsible for receiving written answers that users submit using their terminals. Users using terminals input and submit answers to history questions, such as "Explain the impact of the Meiji Restoration."

[0709] The preprocessor uses natural language processing libraries (e.g., NLTK or SpaCy) to tokenize and normalize the received response data. This preprocessing step prepares the data for analysis.

[0710] The analysis device uses a generative AI model (e.g., GPT, BERT) employing natural language processing technology to analyze the content of the answers and calculate an evaluation score based on the relevance of the content, the comprehensiveness of the perspectives, and the logical structure. For example, the model assigns a score of 85 points to the specific answer, "Japan advanced political reforms and achieved modernization. However, challenges remained in the development of agriculture."

[0711] The emotion recognition device utilizes an emotion analysis library (e.g., TextBlob or VADER) to extract emotions from the user's responses during the analysis process. Based on the word choices in the responses, it determines whether the user is experiencing a specific emotion, such as anxiety or relief.

[0712] The output device generates feedback derived from the analysis results and sentiment analysis, and provides it to the user's terminal. This allows the user to see their score for their answer along with sentiment-based feedback, such as "Approach this with confidence."

[0713] As described above, this system performs specific analysis and generates feedback using a generation AI model and example prompts. A possible example of a prompt is, "Analyze the student's answer and generate emotion-based feedback." By notifying the user of the feedback through methods such as pop-up notifications, the system allows them to receive the information immediately.

[0714] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0715] Step 1:

[0716] The user uses a terminal to input and submit written answers. The information entered is data in text format, with the user providing the answers. The terminal sends this answer data to the server via the internet.

[0717] Step 2:

[0718] The server's data receiving device receives the answer data sent from the user's terminal. The input is the data sent by the user, and the output is the answer data stored in the receiving device. At this stage, the answer data is stored on the server and passed on to the next processing step.

[0719] Step 3:

[0720] The server's preprocessing unit tokenizes and normalizes the received response data. The input is the received response data, and the output is data in a format suitable for the natural language processing model. This preprocessing removes extraneous whitespace and special characters, and tokenizes sentences into word units.

[0721] Step 4:

[0722] The server's analysis device performs analysis using a generative AI model with pre-processed data. The input is tokenized answer data, and the output is the evaluation score of the answers. The generative AI model is used to evaluate the relevance, logic, and comprehensiveness of the answers and assign a score. For example, a score of 85 points may be calculated.

[0723] Step 5:

[0724] The server's emotion recognition device extracts emotions from the data obtained during the analysis process. The input is pre-processed answer data, and the output is information indicating the user's emotions. The user's feelings are identified by the frequency of positive or negative expressions in the answers.

[0725] Step 6:

[0726] The server's output device generates feedback based on the scoring score and sentiment information, and sends it to the terminal. The input is the analysis result and sentiment information, and the output is the feedback message presented to the user. Specifically, an encouraging message such as "Your answer is correct. Please keep your confidence." is generated and displayed on the user's screen.

[0727] (Application Example 2)

[0728] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0729] Conventional automated scoring systems for written answers could evaluate the content of the answers, but they could not provide feedback based on the user's emotional state. Therefore, it was difficult to provide a more personalized learning experience for individual users.

[0730] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0731] In this invention, the server includes means for a data receiving device to receive descriptive answers, means for a preprocessing device to preprocess the received answer data by tokenization and normalization, and means for an analysis device to analyze the preprocessed answer data using natural language processing technology and calculate a scoring score. This makes it possible to evaluate the user's answers and, further, to recognize the user's emotions from the answers using an emotion recognition device, to provide emotion-based feedback.

[0732] A "data receiving device" is a device that has the function of receiving written answers sent by a user.

[0733] A "preprocessing device" is a device that has the function of preparing received answer data into a format suitable for natural language processing through tokenization and normalization.

[0734] An "analysis device" is a device that uses natural language processing technology to analyze pre-processed answer data in detail and evaluate the answers.

[0735] A "scoring score" is a numerical value that indicates the evaluation of an answer calculated by an analysis device.

[0736] A "recording device" is a device used to store analyzed scoring scores and manage them in a database.

[0737] An "emotion recognition device" is a device that automatically recognizes emotions from a user's answers and processes that information.

[0738] "Feedback" refers to information used to provide users with scoring results and advice tailored to their emotions regarding their answers.

[0739] An "output device" is a device that presents and allows users to review scoring results and feedback.

[0740] The system realizing this invention operates primarily with a server, a terminal, and a user. The server consists of a data receiving device, a preprocessing device, an analysis device, a recording device, an emotion recognition device, and an output device. The server receives written answers sent from the user via the data receiving device. The received answer data is tokenized and normalized by the preprocessing device and prepared in a format suitable for natural language processing.

[0741] This organized data is then analyzed in detail using an analysis device, and the answers are evaluated using natural language processing technology to calculate a scoring score. Software such as the Google Cloud Natural Language API is used in this process. In addition, an emotion recognition device is used, and Microsoft Azure Text Analytics is used to automatically recognize the user's emotions from the answers.

[0742] The analysis results and sentiment information are integrated by a recording device and managed in a database. The output device provides the scoring results to the user's terminal and generates and displays sentiment-based feedback. A text generation API is used for this generation. For example, if a middle school student answers the question, "Explain the economic development of the Showa era," with "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged," this answer is analyzed and scored appropriately. If the elements targeted for analysis are evaluated as sufficient to meet the scoring criteria, a message such as, "Your answer is 90 points. Very well done! You seem a little unsure, but the content is solid," is generated and displayed as feedback.

[0743] Examples of input prompts for a generative AI model:

[0744] User's answer: "During the Showa era, industry developed rapidly, and Japan grew into the world's second-largest economy. However, environmental problems also emerged."

[0745] Grading criteria: "Economic growth, industry, economic power, environmental issues"

[0746] It generates analysis and emotional feedback to provide responses to the user.

[0747] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0748] Step 1:

[0749] The user enters and submits written answers using their own device. The user's answer data is generated as input. The user's device then sends this data to a data receiving device.

[0750] Step 2:

[0751] The server's data receiving device receives the user's answer data. It handles the answer data sent from the terminal as input. After this, the answer data is transferred to the preprocessing device.

[0752] Step 3:

[0753] The server's preprocessing unit preprocesses the received answer data through tokenization and normalization. The input is the answer data, and the output is data in a format suitable for natural language processing techniques. Specifically, this involves removing extraneous spaces and splitting words.

[0754] Step 4:

[0755] The server's analysis system uses natural language processing technology to analyze pre-processed answer data and calculates a scoring score. It uses pre-processed data as input and generates a scoring score as output. The Google Cloud Natural Language API is used for the analysis to evaluate the relevance and comprehensiveness of the answers.

[0756] Step 5:

[0757] The server's emotion recognition system recognizes the user's emotions based on the analyzed data and generates that information as output. Specifically, Microsoft Azure Text Analytics is used in this process to analyze the emotions contained in the responses.

[0758] Step 6:

[0759] The server's recording device stores the scoring scores generated by the analysis device and the results of emotion recognition in a database. It uses scores and emotion data as input and manages the data accordingly.

[0760] Step 7:

[0761] The server's output device generates feedback for the user based on saved scoring results and sentiment feedback. The input is the saved results, and the output is the feedback message. A generative AI model and text generation API are used to create specific, sentiment-based advice.

[0762] Step 8:

[0763] The user's device receives feedback messages sent from the server and displays them to the user. It takes messages from the server as input and outputs them in a format that the user can confirm.

[0764] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0765] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0766] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0767] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0768] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0769] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0770] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0771] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0772] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0773] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0774] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0775] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0776] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0777] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0778] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0779] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0780] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0781] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0782] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0783] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0784] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0785] The following is further disclosed regarding the embodiments described above.

[0786] (Claim 1)

[0787] The data receiving device includes means for receiving written answers,

[0788] The preprocessing device includes means for preprocessing the received answer data by tokenization or normalization,

[0789] The analysis device includes means for analyzing pre-processed answer data using natural language processing technology and calculating a scoring score,

[0790] The recording device includes means for storing the analyzed score,

[0791] The output device includes means for providing scoring results and feedback to the user,

[0792] A system that includes this.

[0793] (Claim 2)

[0794] The system according to claim 1, wherein the analysis device comprises means for evaluating the relevance, comprehensiveness of perspectives, and logical structure of the content of the answer.

[0795] (Claim 3)

[0796] The system according to claim 1, wherein the output device is provided with means for providing information so that the user can view the scoring results and feedback through an online interface.

[0797] "Example 1"

[0798] (Claim 1)

[0799] The information receiving device includes means for receiving answers to written questions transmitted from a user terminal,

[0800] The information processing device includes means for preprocessing the received response by normalization and tokenization and converting it into a format suitable for natural language processing,

[0801] The analysis device includes means for analyzing pre-processed answers using a generative AI model and calculating a score based on the accuracy, logical structure, and information coverage of the answers,

[0802] The information recording device includes means for storing the score obtained through analysis,

[0803] The information output device includes means for providing the user with feedback including scoring results and areas for improvement,

[0804] A system that includes this.

[0805] (Claim 2)

[0806] The system according to claim 1, wherein the analysis device comprises means for analyzing the accuracy, logical structure, and information comprehensiveness of the answer in detail based on multiple evaluation criteria.

[0807] (Claim 3)

[0808] The system according to claim 1, wherein the information output device is provided with means for providing information so that a user can quickly obtain scoring results and feedback through an online interface.

[0809] "Application Example 1"

[0810] (Claim 1)

[0811] The data acquisition device includes means for acquiring descriptive answers,

[0812] The initial processing device includes means for initial processing of acquired response information through information partitioning and formatting,

[0813] The analysis device includes means for analyzing the initially processed response information using natural language processing technology and calculating an evaluation score,

[0814] The storage device includes means for storing the analyzed scores,

[0815] The output device provides a means for providing evaluation results and areas for improvement to the user,

[0816] A means of displaying the results on a communication terminal,

[0817] A system that includes this.

[0818] (Claim 2)

[0819] The system according to claim 1, wherein the analysis device comprises means for evaluating the relevance, comprehensiveness of perspectives, and logical structure of the information in the responses.

[0820] (Claim 3)

[0821] The system according to claim 1, wherein the output device provides means for providing information so that a user can check the evaluation results and areas for improvement through an online interface.

[0822] "Example 2 of combining an emotion engine"

[0823] (Claim 1)

[0824] The data receiving device includes means for receiving written answers,

[0825] The preprocessor includes means for tokenizing and normalizing the received answer data,

[0826] The analysis device includes means for analyzing pre-processed answer data using natural language processing technology and calculating an evaluation score,

[0827] The recording device is a means for recording the analyzed score,

[0828] The emotion recognition device provides means for extracting emotions from the answer,

[0829] The output device provides means for providing the terminal with scoring results and emotion-based feedback,

[0830] A system that includes this.

[0831] (Claim 2)

[0832] The system according to claim 1, further comprising means for evaluating the relevance of the content of the answers, the comprehensiveness of the perspectives, and the logical structure of the answers.

[0833] (Claim 3)

[0834] The system according to claim 1, further comprising means for providing information so that the output device can view evaluation results and emotional feedback via a remote interface.

[0835] "Application example 2 when combining with an emotional engine"

[0836] (Claim 1)

[0837] The data receiving device includes means for receiving written answers,

[0838] The preprocessing device includes means for preprocessing the received answer data by tokenization or normalization,

[0839] The analysis device includes means for analyzing pre-processed answer data using natural language processing technology and calculating a scoring score,

[0840] The recording device includes means for storing the analyzed score,

[0841] An emotion recognition device recognizes emotions from the answer and uses this information to generate feedback.

[0842] The output device includes means for providing scoring results and feedback to the user,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, wherein the analysis device comprises means for evaluating the relevance, comprehensiveness of perspectives, and logical structure of the content of the answer, and also comprises means for recognizing the user's emotions using an emotion recognition device.

[0846] (Claim 3)

[0847] The system according to claim 1, wherein the output device provides information to enable the user to view scoring results and feedback through an online interface, and provides means for generating emotion-based advice. [Explanation of symbols]

[0848] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. The data receiving device includes means for receiving written answers, The preprocessing device includes means for preprocessing the received answer data by tokenization or normalization, The analysis device includes means for analyzing pre-processed answer data using natural language processing technology and calculating a scoring score, The recording device includes means for storing the analyzed score, The output device includes means for providing scoring results and feedback to the user, A system that includes this.

2. The system according to claim 1, wherein the analysis device comprises means for evaluating the relevance, comprehensiveness of perspectives, and logical structure of the content of the answer.

3. The system according to claim 1, wherein the output device is provided with means for providing information so that the user can check the scoring results and feedback through an online interface.

Images