System

Abstract

A system is provided.SOLUTION: A system comprising: means for registering a correct answer by a teacher; means for photographing an answer sheet of a student to generate image data; means for transmitting the generated image data to a server; means for recognizing a character and a figure from the received image data; means for comparing the recognized character and figure data with the correct answer; means for detecting an error based on a comparison result and calculating a partial score; means for transmitting a calculation result to a terminal of the teacher; and means for displaying the calculation result on the terminal of the teacher.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In addition to their busy teaching schedules, today's school teachers face a heavy burden from administrative tasks such as grading tests. Grading proof questions in particular is often time-consuming and labor-intensive due to their complexity, with teachers routinely working overtime in excess of 80 hours per month. This excessive workload can lead to health problems for teachers and a decline in the quality of education. Therefore, there is a need for a system that supports efficient and accurate test grading. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means: a system including means for a teacher to register correct answers, means for photographing students' answer sheets to generate image data, means for transmitting the generated image data to a server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answers, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to a teacher's terminal, and means for displaying the calculation results on the teacher's terminal. Furthermore, by including means for logically verifying answers to proof problems based on the recognized character and figure data, and means for highlighting errors in the scoring results displayed on the teacher's terminal, efficient and accurate test scoring is achieved.

[0006] A "teacher" is a person whose occupation is to teach knowledge and skills to students in an educational institution.

[0007] An "answer" is a written response to a test question or assignment.

[0008] A "student" is a learner who acquires knowledge and skills from a teacher at an educational institution.

[0009] An "answer sheet" is a paper or digital format in which a student records answers to an exam or assignment.

[0010] "Image data" is digital information of an image captured using a camera or scanner.

[0011] A "server" is a computer system that provides services to other computers on a network.

[0012] "Optical Character Recognition (OCR)" is a technology that identifies characters from image data and extracts them as digital text.

[0013] "Graphic recognition" is a technology that identifies figures and symbols from image data.

[0014] A "database" is a computer system that can efficiently store, manage, and retrieve data.

[0015] "Partial credit" is credit awarded for a student's answer being partially correct, even if it is not completely correct.

[0016] "Score Results" is information that includes the calculated score based on the student's answers and details of any errors.

[0017] A "terminal" is a device, such as a computer or smartphone, that a user uses to input or display information.

[0018] An "error" is an inaccurate part compared to the correct answer.

[0019] "Logical verification" is a method of checking the logical progression and conclusion based on the consistency and accuracy of the answer.

[0020] "Highlighting" is the use of visual techniques such as changing color or font to make certain information stand out. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The present invention is a system that aims to improve the efficiency of test grading by school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades students' answers by having the teacher register the correct answers and then photographing the students' answers with a camera and sending them to a server. Below, we will explain how to implement this system.

[0043] User interface and correct answer registration

[0044] User (Teacher):

[0045] The teacher launches the app and enters their account information on the login screen. After logging in, the main screen appears, displaying a "Register correct answers" button. Pressing this button activates the camera, takes a picture of the test answer sheet, and generates image data. The generated image data is sent to the server.

[0046] Device:

[0047] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is then processed for character recognition (OCR) and graphic recognition.

[0048] server:

[0049] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database.

[0050] Reading and grading student responses

[0051] User (Teacher):

[0052] The system switches to "Test Marking" mode and begins taking photos of students' answer sheets one by one, generating image data for each answer sheet.

[0053] Device:

[0054] The terminal sends the image data of the generated student answer sheet to the server.

[0055] server:

[0056] The server performs character recognition (OCR) and graphic recognition on the received image data and analyzes the student's answers as digital data. The analyzed data is compared with the correct answer data already saved, and errors and partial correct answers are detected. In particular, for proof questions, the accuracy of the logical steps and conclusions is checked. Partial points are calculated based on errors and partial correct answers, and the final score is generated.

[0057] Sending and viewing graded results

[0058] server:

[0059] The server sends the generated grading results to the teacher's terminal.

[0060] Device:

[0061] The device then displays the received score results on the UI, including the score for each question, any errors made, and details of the partial score.

[0062] User (Teacher):

[0063] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[0064] Specific examples

[0065] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[0066] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where errors and partial marks are highlighted. The teacher then provides feedback to the student based on this.

[0067] This system will streamline teachers' marking work, saving them a huge amount of time and effort.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] User (teacher): Start the app, enter your account information, and log in. After logging in, the main screen will appear, and you can press the "Register correct answer" button.

[0071] Step 2:

[0072] Device: When the correct answer registration button is pressed, the camera is activated and the teacher takes a picture of the answer sheet. The captured image data is generated.

[0073] Step 3:

[0074] Terminal: Sends the generated image data to the server.

[0075] Step 4:

[0076] Server: Initially processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols.

[0077] Step 5:

[0078] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[0079] Step 6:

[0080] User (teacher): Selects "Test marking" mode and takes photos of students' answer sheets one after another.

[0081] Step 7:

[0082] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[0083] Step 8:

[0084] Server: Performs character recognition (OCR) and graphic recognition on the received image data. Extracts student answers and graphics and analyzes them as digital data.

[0085] Step 9:

[0086] Server: Compares the student's answers with the registered correct answers. For each problem, matches the text and figures. For proof problems, checks the accuracy of the logical steps and conclusions.

[0087] Step 10:

[0088] Server: Determines errors, partial correct answers, and complete correct answers in detail. Highlights errors and calculates partial points.

[0089] Step 11:

[0090] Server: Generates and compiles the scoring results, including the score for each question, the total score, and details of errors.

[0091] Step 12:

[0092] Server: Sends the scoring results to the device.

[0093] Step 13:

[0094] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[0095] Step 14:

[0096] User (Teacher): Checks the marks and provides feedback to students, highlighting errors and visually checking partial marks details.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] Grading tests in the past required a great deal of time and effort from teachers, especially for complex problems like mathematical proofs. This increased the burden on teachers and made it difficult to grade quickly and accurately. Additionally, calculating partial marks and identifying errors had to be done manually, further reducing work efficiency.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes a means for the teacher to register correct answer data, a means for photographing the student's answer sheet to generate image data, and a means for transmitting the generated image data to an information processing device. This reduces the burden on the teacher and enables quick and accurate grading. Furthermore, by including a means for recognizing characters and figures from the received image data and comparing the recognized character and figure data with correct answer data, and a means for detecting errors based on the comparison results and calculating partial points, more advanced grading can be achieved. Furthermore, by including a means for highlighting details of errors and partial points, the teacher can provide efficient and visual feedback.

[0102] "Means for teachers to register correct answer data" refers to a device or application that allows teachers to input or record correct answers to test questions.

[0103] "Means for photographing students' answer sheets and generating image data" refers to means for photographing students' answers with a device such as a camera or scanner and generating image data thereof.

[0104] The "means for transmitting the generated image data to the information processing device" refers to a communication method or protocol for transmitting image data from a terminal to a central information processing device (such as a server).

[0105] "Means for recognizing characters and figures from received image data" refers to technology that extracts characters and figures from image data using OCR (optical character recognition) technology or a figure recognition algorithm.

[0106] "Means for comparing recognized character and graphic data with correct answer data" refers to an algorithm for comparing extracted character and graphic data with pre-registered correct answer data and evaluating the recognition results.

[0107] "Means for detecting errors based on the comparison results and calculating partial points" refers to a process for analyzing the comparison results, detecting errors or partial correct answers, and calculating points based on the results.

[0108] "Means for transmitting the calculation results to the teacher's terminal" refers to a method for communicating the calculation results from the server to the terminal used by the teacher.

[0109] "Means for displaying calculation results on the teacher's terminal" refers to an interface for visually displaying calculation results on the terminal screen.

[0110] "Quality check and communication means for image data by the terminal" refers to a method for verifying that the captured image data is accurate and complete, and then communicating the data.

[0111] "Means for analyzing data in an information processing device" refers to a method by which a central computer such as a server analyzes received data and extracts and processes the necessary information.

[0112] "Means for highlighting details of errors and partial marks" refers to visual devices that highlight the parts of the marking results that require careful checking.

[0113] The present invention is a system that allows school teachers to automatically grade complex problems, such as mathematical proofs. This system automatically grades students' answers by having teachers register the correct answer data and taking photos of the students' answers with a camera and sending them to a server. Specific embodiments for implementing this system are described below.

[0114] User interface and correct answer registration

[0115] User (Teacher):

[0116] The teacher launches the application and logs in by entering their account information on the login screen. After logging in, the main screen appears, and a "Register correct answers" button appears. The teacher presses this button to take a photo of the answer sheet with the camera and generate image data.

[0117] Device:

[0118] The device compresses the generated image data of the answer sheet and checks the image quality before sending it to the server. Possible devices include smartphones and tablets. Wi-Fi and mobile data communications are used as communication methods. Libraries such as OpenCV and Tesseract OCR can be used to process the image data.

[0119] server:

[0120] The server analyzes the received image data and uses an OCR engine to extract text from the image. It also uses a shape recognition algorithm to detect figures and mathematical formulas. This extracted data is structured and stored in a database. MySQL or MongoDB can be used as the database management system.

[0121] Read and submit student responses

[0122] User (Teacher):

[0123] The teacher switches to "test marking" mode and takes pictures of the students' answer sheets one by one with the camera, which generates image data of the students' answer sheets.

[0124] Device:

[0125] The device sends the image data of the generated student answer sheet to the server. Before sending, the data is temporarily saved and checked for quality to ensure there are no missing parts. This ensures that the data is uploaded accurately to the server.

[0126] server:

[0127] The server analyzes the image data of the received student answer sheets using OCR and graphic recognition, extracts the student answer data as digital data, and temporarily stores it in memory.

[0128] Data analysis and scoring

[0129] server:

[0130] The server compares the analyzed data with pre-saved correct answers. It uses advanced comparison algorithms to check for partial correct answers. For proof questions in particular, the evaluation focuses on the accuracy of logical steps and conclusions. It identifies errors and partial correct answers and calculates partial points.

[0131] Generate and display grade results

[0132] server:

[0133] The server generates the marking results and sends them to the teacher's device. The results include the score for each question, details of errors, and partial marks. This data is packaged and sent in JSON or XML format.

[0134] Device:

[0135] The device analyzes the received marking results and displays them on a user interface. Errors and partial marks are highlighted. JavaScript and front-end frameworks (e.g., React and Vue.js) can be used as display technologies.

[0136] User (Teacher):

[0137] Teachers can check the marks displayed on the device and provide feedback to students. Errors and partial marks can be visually confirmed, enabling quick and accurate instruction.

[0138] Examples and prompts

[0139] As a concrete example, consider a case where a teacher supports the grading of proof questions in junior high school mathematics. After launching the application and logging in, the teacher selects the correct answer registration mode, photographs the test answer sheet, and sends it to the server. The server analyzes the correct answers and stores them in a database.

[0140] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers and detects errors or partial marks. The grading results are then sent to the teacher's device and displayed with details highlighted. The teacher can then provide feedback to the student based on this information.

[0141] Example prompt sentence:

[0142] Please give us an overview of the system that streamlines grading of mathematical proof problems.

[0143] Please provide a detailed procedure for a system that allows teachers to automatically grade mathematics proof problems.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1: User authentication and login

[0146] User (Teacher):

[0147] Input: Teacher account information (username, password).

[0148] How it works: The teacher launches the application and enters their account information on the login screen.

[0149] The server compares the received account information with the database and performs authentication.

[0150] Output: A successful authentication message or an error message is displayed on the terminal.

[0151] Step 2: Register the correct answer

[0152] User (Teacher):

[0153] Input: Exam answer sheet.

[0154] How it works: After logging in, press the "Register correct answer" button on the main screen that appears and take a photo of the answer sheet with your camera.

[0155] Device:

[0156] Operation: Generates captured image data and performs quality checks and adjustments on the images.

[0157] server:

[0158] Input: Image data sent from the device.

[0159] Operation: Analyzes received image data using an OCR engine and image recognition algorithms, extracts characters and images, and stores them in a database.

[0160] Output: The correct answer data is saved in the database, and a message indicating that the correct answer has been registered is displayed on the terminal.

[0161] Step 3: Reading student answer sheets

[0162] User (Teacher):

[0163] Input: Student answer sheet.

[0164] Operation: Switches to "test marking" mode and takes photos of students' answer sheets one by one with the camera.

[0165] Device:

[0166] Operation: Generates captured image data, temporarily stores it, and checks its quality.

[0167] server:

[0168] Input: Image data of student answer sheets sent from the device.

[0169] How it works: Analyzes image data using OCR and shape recognition to extract characters and shapes. The extracted data is temporarily stored in memory.

[0170] Output: Parsed data is held in memory.

[0171] Step 4: Data analysis and comparison

[0172] server:

[0173] Input: Parsed student answer data, correct answer data in the database.

[0174] How it works: Compares the student's answer data to the correct answer data. A comparison algorithm is used to detect errors and partial correct answers and calculate partial credit.

[0175] Output: Marking results including partial credit and errors.

[0176] Step 5: Generate and transfer the scores

[0177] server:

[0178] Input: Marking results including partial marks and errors.

[0179] Operation: The grading results are packaged in JSON format or similar for transfer to the teacher's device.

[0180] Device:

[0181] How it works: Parses the received marks and displays them in a user interface, highlighting errors and partial marks.

[0182] Output: The grading results are displayed on the teacher's terminal.

[0183] Step 6: Provide feedback

[0184] User (Teacher):

[0185] Input: Detailed marking results (details of each question's score, mistakes, and partial marks).

[0186] How it works: The teacher checks the grading results displayed on the device and provides feedback to the students as needed. For example, the teacher visually checks the errors and partial points for question number "6430" and enters appropriate instructional comments.

[0187] Output: Teacher feedback is provided to students.

[0188] (Application example 1)

[0189] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0190] Traditional test marking in educational settings and quality inspection in factories are time-consuming and labor-intensive, and there is a need for greater efficiency. Complex tasks such as marking mathematical proof problems and inspecting product quality are particularly prone to human error, which can result in a loss of accuracy. This increases the burden on teachers and factory workers and leads to a decline in labor productivity. There is a need for an efficient and accurate system to solve this problem.

[0191] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0192] In this invention, the server includes means for the teacher to register correct answers, means for photographing students' answer sheets and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answers, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for photographing the structure of a product, inspecting the quality of the product based on the recognized character and figure data, and automatically determining whether the product passes or fails, means for transmitting instructions to a robot arm to remove rejected products based on the pass / fail result, means for verifying the accuracy of product dimensions and labels based on the recognized character and figure data, and means for transmitting the quality inspection results to a management system and updating the pass / fail status. This makes it possible to improve the efficiency and accuracy of test marking in educational settings and quality inspection work in factories.

[0193] The "means for registering correct answers by the teacher" is an interface and function that allows the teacher to register answers in the system.

[0194] The "means for photographing student's answer sheet" is a function for photographing the student's answer sheet with a camera and generating image data of the photograph.

[0195] "Image data transmission means" refers to a communication means and protocol for transmitting the generated image data to the server.

[0196] "Character and graphic recognition means" refers to technology that extracts characters and graphics from received image data using optical character recognition (OCR) and graphic recognition algorithms.

[0197] The "answer comparison means" is a function that compares the recognized character and graphic data with correct answer data registered in advance to detect errors and inconsistencies.

[0198] The "partial score calculation means" is a function that calculates scores for the degree of error and partial correct answers based on the comparison results.

[0199] "Calculation result transmission means" refers to the communication means and protocol for transmitting the calculated grading results to the teacher's terminal.

[0200] The "grading result display means" is a user interface and function for visually displaying the calculation results on the teacher's terminal.

[0201] The "product image capturing means" is a function that captures images of products manufactured in a factory using a camera and generates image data of the images.

[0202] The "product quality inspection means" is a function that automatically inspects the quality of a product based on the recognized character and graphic data.

[0203] The "pass / fail judgment means" is a function that judges whether a product passes or fails based on the results of the quality inspection.

[0204] The "rejected product removal instruction means" is a function that sends instructions for the robot arm to remove the rejected product depending on the pass / fail result.

[0205] "Dimension and label accuracy verification means" is a function that verifies the accuracy of product dimensions and labels based on recognized character and graphic data.

[0206] "Management System Transmission Means" refers to the communication means and protocols for transmitting quality inspection results to the management system and updating pass / fail status.

[0207] This invention relates to a system for teachers to efficiently mark tests and a system for automating quality inspections in factories. This system uses a camera to take images of students' answer sheets and products, and analyzes the data using OCR (optical character recognition) and graphic recognition technology to automatically mark and inspect the quality.

[0208] Teacher's correct answer registration

[0209] The teacher launches the application on their own device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is sent from the device to the server. The server uses OCR to extract text from the received image data of the answer sheet and uses a graphic recognition algorithm to detect figures and symbols. This data is then structured and stored in a database.

[0210] Reading and marking student answers

[0211] The teacher switches to "test marking" mode and takes photos of students' answer sheets one by one. The generated image data of the answer sheets is sent from the device to the server. The server receives it and analyzes the answer data using OCR and shape recognition technology. The server compares the analyzed answer data with pre-registered correct answer data to detect errors and partial correct answers. In particular, for mathematical proof questions, the server verifies the accuracy of logical steps and conclusions and calculates partial points.

[0212] Sending and viewing graded results

[0213] The server then sends the calculated marks to the teacher's device, where the marks are displayed in a UI, visually highlighting the score for each question, errors, and partial marks.

[0214] Factory quality inspection

[0215] In the factory, a camera is used to take an image of the product. This image data is sent from the terminal to a server and analyzed using OCR and shape recognition technology. The server verifies the accuracy of the product's label and dimensions and determines whether it passes or fails. Depending on the pass / fail result, a robotic arm is instructed to remove the rejected product. The quality inspection results are sent to the management system, and the pass / fail status is updated.

[0216] Specific examples

[0217] For example, a teacher takes a photo of an exam answer sheet with a camera and sends it to a server. The server analyzes the correct answers using OCR and shape recognition, receives the photo data of the student's answer sheet, and analyzes it in the same way. The server compares the data and generates a score that is sent to the teacher's device. In factories, product labels and dimensions are inspected, and robotic arms remove non-conforming products.

[0218] Example prompt sentence:

[0219] Check if the label in the image is "EXPECTED_LABEL" and check if the product contour length is less than 100

[0220] This will enable more efficient test grading in educational settings and more accurate quality inspections in factories.

[0221] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0222] Step 1: Register the answer sheet

[0223] The user (teacher) starts the application on their device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is generated and sent to the server. The server uses OCR and graphic recognition technology on the received image data to recognize characters and figures, and stores the data in a database as structured data. The input is the image data of the photographed answer sheet, and the output is structured data of the correct answers.

[0224] Step 2: Photograph and submit student answer sheets

[0225] The user (teacher) switches to "test marking" mode and takes photos of students' answer sheets one after another. The generated image data of the answer sheets is sent from the device to the server. The input is the image data of the photographed answer sheets, and the output is the image data of the answer sheets sent to the server.

[0226] Step 3: Analyzing the answer sheet

[0227] The server uses OCR and graphic recognition technology on the image data of the received answer sheet to extract character and graphic data. The input is the image data of the answer sheet, and the output is the analyzed character and graphic data. In this step, the image data is converted into text data and graphic data.

[0228] Step 4: Compare and score answers

[0229] The server compares the analyzed answer data with the correct answer data saved in step 1. It detects errors and partial correct answers and calculates partial points. The input is the analyzed answer data and correct answer data, and the output is the scoring result (including partial points). In this step, the data is compared with the correct answer data and scored.

[0230] Step 5: Submit your results

[0231] The server sends the calculated score to the user's terminal. The input is the score, and the output is the score sent to the user's terminal. In this step, data communication occurs.

[0232] Step 6: View your results

[0233] The user's device displays the received marking results. The displayed information includes the score for each question, details of errors, and partial marks. The input is the received marking results, and the output is the marking results displayed to the user.

[0234] Step 7: Take product images

[0235] The terminal takes images of products manufactured in the factory with a camera. The generated image data is sent to the server. The input is the image data of the photographed product, and the output is the image data sent to the server.

[0236] Step 8: Analyze product quality

[0237] The server uses OCR and shape recognition technology on the received product image data to verify the label and dimensions. The input is the product image data, and the output is the parsed label and dimension data.

[0238] Step 9: Quality assessment and notification of pass / fail

[0239] The server inspects the quality based on the analyzed data and judges whether the product passes or fails. Depending on the pass / fail result, it sends instructions to the robot arm to remove the rejected product. The input is the analyzed label and dimensional data, and the output is the pass / fail judgment result and instructions to the robot arm.

[0240] Step 10: Submit and update test results

[0241] The server sends the quality inspection results to the management system and updates the pass / fail status. The input is the quality inspection result and the output is the updated status of the management system. In this step, data communication with the management system takes place.

[0242] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0243] The present invention is a system designed to improve the efficiency of test grading for school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades tests by having the teacher register correct answers and then photographing students' answers with a camera and sending them to a server. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system enables smoother work. Below, we will explain the form and specific program processing for implementing this system.

[0244] User interface and correct answer registration

[0245] User (Teacher):

[0246] First, the teacher launches the app and logs in. After logging in, the main screen appears and the teacher presses the "Register correct answers" button to activate the camera. The teacher then takes a photo of the test answer sheet and generates image data. The generated image data is sent to the server.

[0247] Device:

[0248] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is later processed for character recognition (OCR) and graphic recognition.

[0249] server:

[0250] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database as correct answer data.

[0251] Reading and grading student responses

[0252] User (Teacher):

[0253] Select the "Test Marking" mode and take photos of each student's answer sheet one by one, which will generate image data for each answer sheet.

[0254] Device:

[0255] The terminal sends the image data of the generated student answer sheet to the server.

[0256] server:

[0257] The server performs character recognition (OCR) and graphic recognition on the received image data, analyzing the student's answers as digital data. This data is then compared with the correct answer data to detect errors and partial correct answers. For proof questions in particular, the accuracy of the logical steps and conclusions is checked. Partial marks are calculated based on errors and partial correct answers, and a final score is generated.

[0258] Introduction of emotion engine and its application

[0259] server:

[0260] An emotion engine is introduced, and the server recognizes the user's (teacher's) emotions. This emotion recognition is performed by analyzing the user's facial expressions and voice using sensors such as cameras and microphones. For example, if an angry or frustrated facial expression is recognized, the system changes the displayed feedback and interface.

[0261] Device:

[0262] The device can then adjust its display based on the results of emotion recognition. For example, if the user is feeling stressed, the system can display a brief summary and possibly offer suggestions for relaxing the user.

[0263] User (Teacher):

[0264] The emotion engine allows teachers to receive appropriate feedback based on their own emotional state, reducing excessive stress during grading and enabling them to work more efficiently.

[0265] Sending and viewing graded results

[0266] server:

[0267] The server sends the generated grading results to the teacher's terminal, and based on the emotion engine, the grading results are displayed to the user in an appropriate format.

[0268] Device:

[0269] The device then displays the received scoring results on the UI. The displayed information includes the score for each question, details of errors, and partial points. Errors and partial points are highlighted based on the recognized emotion.

[0270] User (Teacher):

[0271] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[0272] Specific examples

[0273] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[0274] Next, the teacher takes photos of each student's answer sheet and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where the errors and partial marks are highlighted.

[0275] This system will streamline teachers' marking work, saving them a huge amount of time and effort. The emotion engine will also reduce the mental burden on teachers.

[0276] The processing flow will be explained below.

[0277] Step 1:

[0278] User (teacher): Launch the app and log in by entering your account information on the login screen.

[0279] Step 2:

[0280] On your device: Send your login credentials to the server.

[0281] Step 3:

[0282] Server: Validates the authentication information and returns the authentication result to the terminal. If authentication is successful, it instructs the terminal to display the main screen.

[0283] Step 4:

[0284] User (Teacher): Press the "Register correct answer" button on the main screen to launch the camera.

[0285] Step 5:

[0286] Device: The camera is activated and the teacher takes a picture of the answer sheet to generate image data.

[0287] Step 6:

[0288] Terminal: Sends the image data of the photographed answer sheet to the server.

[0289] Step 7:

[0290] Server: Processes the received image data and performs character recognition (OCR) and shape recognition. A character recognition engine is used to extract text from the image, and a shape recognition algorithm is used to detect figures and symbols.

[0291] Step 8:

[0292] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[0293] Step 9:

[0294] User (Teacher): Select "Test Grading" mode on the main screen.

[0295] Step 10:

[0296] User (teacher): Takes photos of students' answer sheets one after another.

[0297] Step 11:

[0298] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[0299] Step 12:

[0300] Server: Processes the image data of the received answer sheets, performs character recognition (OCR) and graphic recognition, extracts the students' answers and graphics, and analyzes them as digital data.

[0301] Step 13:

[0302] Server: Compares the student's answers with the registered correct answers. It verifies the text and figures, and in the case of proof questions, checks the accuracy of the logical steps and conclusions.

[0303] Step 14:

[0304] Server: Determines errors, partial correct answers, and complete correct answers, highlights errors, and calculates partial points.

[0305] Step 15:

[0306] Server: Generates the scoring results and compiles them into data including the score for each question, the total score, and details of any errors.

[0307] Step 16:

[0308] Server: Sends the grading results to the teacher's device.

[0309] Step 17:

[0310] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[0311] Step 18:

[0312] Server: Using the emotion engine, analyzes the user's (teacher's) facial expressions and voice and recognizes their emotions.

[0313] Step 19:

[0314] Server: Based on the recognized emotions, it adjusts the interface and feedback method, and generates appropriate feedback according to the teacher's emotional state.

[0315] Step 20:

[0316] Device: Adjusts the interface and changes the display content based on the recognized emotion information.

[0317] Step 21:

[0318] User (Teacher): Check the displayed marks and feedback and provide appropriate feedback to the student. Errors are highlighted and partial marks are visually confirmed.

[0319] Example 2

[0320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0321] Conventional test marking systems have difficulty efficiently and accurately marking complex problems, especially those like mathematical proofs. Furthermore, they are unable to adjust the interface to take into account the teacher's emotional state, which can lead to stress during long marking sessions. Furthermore, they lack sufficient feedback to students, preventing them from providing effective learning support.

[0322] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet to generate image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for recognizing the teacher's emotions, and means for adjusting the display method based on the recognized emotions. This enables efficient and accurate grading of complex questions, reduces stress by adjusting the interface according to the teacher's emotional state, and enables effective feedback to students.

[0323] A "teacher" refers to a professional who provides instruction and education to students at an educational institution.

[0324] "Correct answer" refers to the exact or best answer to a particular question.

[0325] A "means" refers to a method or device used to achieve a particular purpose.

[0326] An "answer sheet" refers to the paper on which students write their answers in an exam or test.

[0327] "Image data" refers to image information acquired by a camera or scanner and stored in digital format.

[0328] A "server" refers to a computer system that stores and processes data.

[0329] "Characters" refer to symbols used to represent human languages.

[0330] "Figure" refers to a geometric shape or symbol.

[0331] "Recognition" refers to the process of extracting specific information from an image or sound.

[0332] "Comparison" refers to the process of comparing two or more pieces of data to identify differences and similarities.

[0333] An "error" is an inaccuracy or mistake compared to the correct answer.

[0334] "Partial credit" refers to a mark awarded as a portion of the total score.

[0335] "Calculation" refers to the process of deriving a specific result from numbers and data.

[0336] "Terminal" refers to a device that communicates with a server and sends and receives data.

[0337] "Emotion" refers to a person's psychological state and facial expression.

[0338] "Recognized emotion" refers to the emotional state determined by the system through analysis of the user's facial expressions and voice.

[0339] "Display method" refers to how information or results are presented in the user interface.

[0340] The present invention relates to a system that helps teachers to efficiently grade tests, and in particular, a system that automates grading of complex problems such as mathematical proofs. This system reduces the burden on teachers and enables quick and accurate feedback to students.

[0341] System configuration

[0342] The system consists of the following main components:

[0343] 1. User (Teacher):

[0344] A device that allows teachers to register correct answers and photograph students' answer sheets. Specifically, it uses a device such as a smartphone or tablet.

[0345] 2. Terminal:

[0346] The device used by the teacher takes pictures of the correct answer sheets and the students' answer sheets and sends the image data to the server. This device is equipped with a camera function and an internet connection function.

[0347] 3. Server:

[0348] The server processes the received image data and recognizes characters and figures. Specifically, it uses an OCR (Optical Character Recognition) engine and figure recognition algorithms. The recognized data is compared with a database of correct answers to detect errors and partial correct answers.

[0349] 4. Emotion Engine:

[0350] The emotion engine recognizes the teacher's emotions and adjusts the way the user interface is displayed. Specifically, the system is equipped with a camera and microphone to analyze the teacher's facial expressions and voice.

[0351] Program processing

[0352] 1. Correct Answer Registration:

[0353] The teacher launches the app on their device and logs in. They press the "Register correct answer" button on the main screen to activate the camera and take a picture of the test answer sheet. The captured image data is sent from the device to the server. The server processes the received image data using an OCR engine and a shape recognition algorithm, and stores the text and shape data in a database.

[0354] 2. Scoring Student Responses:

[0355] The teacher selects the "Test Marking" mode and takes a photo of the student's answer sheet. The device sends the captured image data to the server. The server processes the received data with an OCR engine and shape recognition algorithms, and compares the recognized data with the correct answer data. It detects errors and partial correct answers, calculates partial points, and generates the final marking result.

[0356] 3. Emotion recognition:

[0357] The server uses an emotion engine to analyze the teacher's facial expressions and voice to recognize their emotional state. Based on the recognized emotion, the system adjusts the interface and feedback displayed. For example, if the teacher is feeling stressed, the system adjusts to display only brief summary information.

[0358] 4. View your score:

[0359] The server sends the calculated score to the teacher's device. The device displays the received score on the UI, highlighting errors and partial points. The teacher checks the displayed results and provides feedback to the student.

[0360] Specific examples

[0361] For example, to assist with grading proof questions in junior high school mathematics, a teacher first launches the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the answer sheet and stores it in a database. Next, the teacher takes a photo of the student's answer sheet and sends it to the server. The server compares the student's answer with the correct answer and generates a grading result. The final grading result is sent to the teacher's device, and errors and partial points are highlighted.

[0362] Prompt Sentence Examples

[0363] "Please explain your system for streamlining the grading of proof questions in junior high school mathematics. Specifically, please explain how correct answers are registered, how student answers are graded, how an emotion engine is used, and how the final graded results are transmitted and displayed."

[0364] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0365] Step 1:

[0366] The user (teacher) starts the app and logs in. On the login screen, they enter their username and password and press the login button. The entered authentication information is sent from the device to the server.

[0367] Input: Username and Password

[0368] Data processing: The server checks the authentication information against a database

[0369] Output: Authentication token

[0370] Specific operation: After successful login, an authentication token is sent to the device.

[0371] Step 2:

[0372] The user (teacher) presses the "Register correct answer" button on the main screen. When the teacher takes a photo of the test answer sheet using the camera and presses the "Send" button, the device generates image data and sends it to the server.

[0373] Input: Image data of the answer sheet

[0374] Data processing: Encode image data into BASE64 format

[0375] Output: Encoded image data

[0376] Specific operation: The device sends the image data to the server as an HTTP POST request.

[0377] Step 3:

[0378] The server processes the received image data and extracts text and graphics data using an optical character recognition (OCR) engine and graphics recognition algorithms.

[0379] Input: Encoded image data

[0380] Data processing: Extract text and graphics using OCR engine and graphic recognition algorithms

[0381] Output: Recognized text and graphics data

[0382] Specific behavior: Save the extracted data to the database

[0383] Step 4:

[0384] The user (teacher) selects the "Test Marking" mode, takes pictures of students' answer sheets one after another, and presses the "Send" button. The device then sends the captured image data to the server.

[0385] Input: Image data of student answer sheets

[0386] Data processing: Encode image data into BASE64 format

[0387] Output: Encoded image data

[0388] Specific operation: The device sends the image data to the server as an HTTP POST request.

[0389] Step 5:

[0390] The server performs character recognition (OCR) and graphic recognition on the image data of the received student answer sheets, and compares the extracted data with the correct answer data.

[0391] Input: Encoded image data

[0392] Data processing: Extract text and graphics using an OCR engine and graphic recognition algorithms, and compare them with the correct data

[0393] Output: Information about errors and partial answers

[0394] Specific behavior: Calculate partial points based on errors and partial correct answers

[0395] Step 6:

[0396] The server sends the calculated score to the teacher's terminal.

[0397] Input: Information about errors and partial answers

[0398] Data processing: Calculating the score and converting the results to JSON format

[0399] Output: JSON data of the scoring results

[0400] Specific behavior: Sends JSON data as an HTTP response

[0401] Step 7:

[0402] The device then displays the received scoring results on a user interface. The displayed information includes the score for each question, details of errors, and partial points, and the display method is adjusted based on the results of the emotion engine.

[0403] Input: JSON data of the score results

[0404] Data processing: Parsing JSON data and binding it to UI components

[0405] Output: A visual representation of the scoring results

[0406] Specific operation: Errors and partial points are highlighted

[0407] Step 8:

[0408] The server uses an emotion engine to recognize the teacher's emotional state and adjusts the feedback and interface based on the recognized emotion.

[0409] Input: Teacher's facial expressions and voice data

[0410] Data processing: Analysis and emotion determination using an emotion engine

[0411] Output: Recognized emotion information

[0412] Specific behavior: Dynamically adjusting the UI and displaying concise summary information

[0413] Step 9:

[0414] The user (teacher) checks the displayed grading results and provides feedback to the student. Specifically, the user enters additional comments and feedback through the input form and sends it to the student through the system.

[0415] Input: Marking results and additional feedback

[0416] Data processing: Input feedback is saved in a database

[0417] Output: Send feedback

[0418] Specific behavior: The feedback content is notified to the student.

[0419] (Application example 2)

[0420] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] Manufacturing sites are required to simultaneously achieve quality control and reduce the mental burden on workers. In particular, it is important to accurately inspect product quality, provide prompt feedback, and take appropriate measures when workers feel stressed or fatigued. To solve these issues, a system that combines quality control and emotion recognition is required. However, conventional systems have difficulty providing these two functions in an integrated manner, which can lead to a decrease in work efficiency.

[0422] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for registering correct standards for quality control, means for photographing the manufactured product and generating image data, means for transmitting the generated image data to the server and comparing it with the quality standard, means for recognizing the worker's emotions and providing feedback, means for transmitting the recognition results to the worker, and means for displaying the recognition results on the worker's terminal. This enables quality control and reduces the mental burden on the worker.

[0423] The "means for teachers to register correct answers" is a mechanism for teachers to input standard answers set in advance into the system.

[0424] The "means for photographing students' answer sheets and generating image data" refers to a mechanism for capturing students' answers with a photographing device such as a camera and converting the images into digital data.

[0425] The "means for transmitting the generated image data to the server" is a mechanism for sending the captured image to the central processing unit via the network.

[0426] The "means for recognizing characters and figures from received image data" is a mechanism for identifying text and figures contained in the transmitted image data and extracting them as digital data.

[0427] The "means for comparing recognized character and graphic data with correct answers" is a mechanism for comparing extracted character and graphic information with pre-registered correct answer data and performing evaluation.

[0428] The "means for detecting errors based on the comparison results and calculating partial points" is a mechanism for detecting errors from the results of comparing the character and graphic data with the correct data and calculating the partial points accordingly.

[0429] The "means for transmitting the calculation results to the teacher's terminal" is a mechanism for transferring the calculated scores and evaluation results to the terminal used by the educator.

[0430] The "means for displaying the calculation results on the teacher's terminal" is a mechanism for visually presenting the calculated scores and evaluation results on the teacher's terminal.

[0431] "Means for registering correct standards for quality control" is a mechanism for registering standards for quality evaluation in the manufacturing industry in the system.

[0432] The "means for photographing manufactured products and generating image data" is a mechanism for photographing manufactured products with a camera or the like and generating the image as data.

[0433] The "means for transmitting the generated image data to a server and comparing it with quality standards" is a mechanism for transmitting image data of the photographed product to a central processing unit and comparing it with quality evaluation standards.

[0434] The "means for recognizing workers' emotions and providing feedback" is a mechanism for determining the emotions of workers from their facial expressions and voice, and providing appropriate advice and evaluations based on that.

[0435] The "means for transmitting the recognition result to the worker" is a mechanism for transferring the emotion determination result to the terminal used by the worker.

[0436] The "means for displaying the recognition result on the worker's terminal" is a mechanism for visually presenting the emotion determination result on the worker's terminal.

[0437] The present invention provides a system for integrating quality control and worker emotion recognition in educational and manufacturing settings. Specific embodiments for carrying out the invention will be described below.

[0438] System Overview

[0439] This system provides an interface that can be easily operated by educators and workers using terminals, and realizes quality control and emotion recognition functions accurately and quickly.The system consists of the following main components.

[0440] 1. Teachers and quality control personnel:

[0441] The teacher starts the application and logs in. After logging in, the teacher uses the correct answer registration mode to take a photo of the correct answer with the camera and generate image data. This image data is sent to the server and registered as the correct answer data.

[0442] Quality control personnel can register quality standard data in a similar manner.

[0443] 2. Student and Product Management:

[0444] Students' answer sheets and finished products are photographed with a camera, and the image data is sent to a server, where characters and figures are recognized from the received image data and compared with registered correct answer data and quality standard data.

[0445] 3. Server:

[0446] The server uses OCR (optical character recognition) software to extract characters and graphic recognition algorithms to detect shapes and symbols. This data is compared with correct answers and quality benchmark data to detect errors and calculate partial points.

[0447] 4. Emotion Recognition Engine:

[0448] The server is also equipped with an emotion recognition engine that uses sensors such as cameras and microphones to analyze the emotions of users (workers and teachers). For example, if the system detects fatigue or stress in a user, it will provide appropriate feedback.

[0449] 5. Terminal:

[0450] The calculation results and emotion recognition results are displayed in real time on the devices of teachers, quality control personnel, and workers, and the display method can be adjusted according to the user's emotional state.

[0451] Hardware and Software Use

[0452] This system uses the following hardware and software:

[0453] Camera: Used to acquire image data.

[0454] Server: The main component that processes and manages data.

[0455] Terminal (PC, tablet, smartphone, etc.): A device that is operated by the user.

[0456] OCR software (e.g., PyTesseract): Extract character data from images.

[0457] Shape recognition algorithm: An algorithm for recognizing shapes and symbols from images.

[0458] Emotion recognition engine (e.g., Keras-based model): for analyzing emotions from user facial expressions and voice.

[0459] Specific examples

[0460] For example, this system is extremely effective in quality control on automobile production lines. Quality control personnel register product quality standards and take photos of manufactured parts with a camera. The image data is then sent to a server where it is automatically compared with the quality standards. Meanwhile, an emotion recognition engine analyzes the facial expressions and voice of workers, and if fatigue or stress is detected, the system suggests taking a break.

[0461] Prompt Sentence Examples

[0462] An example of an input prompt for the generative AI model associated with this system is shown below.

[0463] "Think of a quality control system on an automobile production line. In the system, an inspection robot takes a picture of the product and compares it with quality standards to automatically inspect and evaluate it. The system also recognizes the emotions of workers and provides appropriate feedback. For workers who appear particularly tired, it suggests that they take a break."

[0464] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0465] Step 1:

[0466] A user (teacher or quality control officer) starts the application and logs in. The input here is the user's identification information (user ID and password), and the output is the authentication result. Once the user is authenticated, the main screen is displayed. During this authentication process, the user information is checked against the database, and login is allowed only if it is correct.

[0467] Step 2:

[0468] The user selects the "Correct Answer Registration" mode or the "Quality Standard Registration" mode and takes a picture of the correct answer sheet or quality standard with the camera. The input here is the captured image, and the output is the generated image data. The device controls the camera and saves the image as digital data. At this stage, the image data is saved locally on the spot.

[0469] Step 3:

[0470] The terminal sends the generated image data to the server. The input here is the image data, and the output is a transmission success message to the server. The terminal uploads the image data to the server via the network and receives a confirmation message if successful.

[0471] Step 4:

[0472] The server recognizes characters and graphics from the image data it receives. The input here is the transmitted image data, and the output is the recognized character and graphic data. Specifically, OCR software (e.g., PyTesseract) and graphic recognition algorithms analyze the image on the server and extract character and graphic information.

[0473] Step 5:

[0474] The server compares the recognized character and graphic data with the correct answer data or quality standard data. The input here is the recognized character and graphic data and the pre-registered correct answer data or quality standard data, and the output is the comparison result. The server collates these data and detects differences.

[0475] Step 6:

[0476] The server detects errors based on the comparison results and calculates partial scores. The input here is the comparison results, and the output is the calculated partial scores and a list of error locations. The server follows a programmed algorithm to determine which parts are correct, which parts are incorrect, and calculate the partial scores.

[0477] Step 7:

[0478] The server sends the calculation results to the user's device. The input here is the calculation result, and the output is a message to the device indicating successful transmission. The server packages the calculation results in JSON format or similar and sends them over the network to the user's device.

[0479] Step 8:

[0480] The terminal displays the calculation results it receives. The input here is the calculation result received, and the output is visual feedback that can be checked by a teacher or quality control officer. The terminal displays the calculation results on the screen, highlighting any errors or partial points to the user.

[0481] Step 9:

[0482] Sensors (camera and microphone) for recognizing the user's emotions capture the user's facial expressions and voice. The input here is the user's facial image and voice data, and the output is the captured sensor data. The device records this data as it goes along and sends it to a server for analysis.

[0483] Step 10:

[0484] The server analyzes the user's emotions using an emotion recognition engine, where the input is the captured sensor data and the output is the recognized emotion classification (e.g., stress, satisfaction, confidence, etc.). The emotion recognition engine (e.g., a Keras-based model) is used to determine the user's emotions in real time.

[0485] Step 11:

[0486] The server provides feedback based on the recognition results and sends the results to the user's device. The input here is the emotion recognition result, and the output is a feedback message to the user. The server creates advice or warnings according to the recognized emotion and sends them to the device.

[0487] Step 12:

[0488] The terminal displays the received recognition results and provides appropriate visual feedback to the user. The input here is the received recognition results, and the output is a feedback display that the user can check. The terminal displays the feedback content in a form that the user can easily understand.

[0489] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0490] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0491] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0492] [Second embodiment]

[0493] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0494] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0495] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0496] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0497] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0498] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0499] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0500] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0501] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0502] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0503] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0504] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0505] The present invention is a system that aims to improve the efficiency of test grading by school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades students' answers by having the teacher register the correct answers and then photographing the students' answers with a camera and sending them to a server. Below, we will explain how to implement this system.

[0506] User interface and correct answer registration

[0507] User (Teacher):

[0508] The teacher launches the app and enters their account information on the login screen. After logging in, the main screen appears, displaying a "Register correct answers" button. Pressing this button activates the camera, takes a picture of the test answer sheet, and generates image data. The generated image data is sent to the server.

[0509] Device:

[0510] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is then processed for character recognition (OCR) and graphic recognition.

[0511] server:

[0512] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database.

[0513] Reading and grading student responses

[0514] User (Teacher):

[0515] The system switches to "Test Marking" mode and begins taking photos of students' answer sheets one by one, generating image data for each answer sheet.

[0516] Device:

[0517] The terminal sends the image data of the generated student answer sheet to the server.

[0518] server:

[0519] The server performs character recognition (OCR) and graphic recognition on the received image data and analyzes the student's answers as digital data. The analyzed data is compared with the correct answer data already saved, and errors and partial correct answers are detected. In particular, for proof questions, the accuracy of the logical steps and conclusions is checked. Partial points are calculated based on errors and partial correct answers, and the final score is generated.

[0520] Sending and viewing graded results

[0521] server:

[0522] The server sends the generated grading results to the teacher's terminal.

[0523] Device:

[0524] The device then displays the received score results on the UI, including the score for each question, any errors made, and details of the partial score.

[0525] User (Teacher):

[0526] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[0527] Specific examples

[0528] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[0529] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where errors and partial marks are highlighted. The teacher then provides feedback to the student based on this.

[0530] This system will streamline teachers' marking work, saving them a huge amount of time and effort.

[0531] The processing flow will be explained below.

[0532] Step 1:

[0533] User (teacher): Start the app, enter your account information, and log in. After logging in, the main screen will appear, and you can press the "Register correct answer" button.

[0534] Step 2:

[0535] Device: When the correct answer registration button is pressed, the camera is activated and the teacher takes a picture of the answer sheet. The captured image data is generated.

[0536] Step 3:

[0537] Terminal: Sends the generated image data to the server.

[0538] Step 4:

[0539] Server: Initially processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols.

[0540] Step 5:

[0541] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[0542] Step 6:

[0543] User (teacher): Selects "Test marking" mode and takes photos of students' answer sheets one after another.

[0544] Step 7:

[0545] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[0546] Step 8:

[0547] Server: Performs character recognition (OCR) and graphic recognition on the received image data. Extracts student answers and graphics and analyzes them as digital data.

[0548] Step 9:

[0549] Server: Compares the student's answers with the registered correct answers. For each problem, matches the text and figures. For proof problems, checks the accuracy of the logical steps and conclusions.

[0550] Step 10:

[0551] Server: Determines errors, partial correct answers, and complete correct answers in detail. Highlights errors and calculates partial points.

[0552] Step 11:

[0553] Server: Generates and compiles the scoring results, including the score for each question, the total score, and details of errors.

[0554] Step 12:

[0555] Server: Sends the scoring results to the device.

[0556] Step 13:

[0557] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[0558] Step 14:

[0559] User (Teacher): Checks the marks and provides feedback to students, highlighting errors and visually checking partial marks details.

[0560] Example 1

[0561] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0562] Grading tests in the past required a great deal of time and effort from teachers, especially for complex problems like mathematical proofs. This increased the burden on teachers and made it difficult to grade quickly and accurately. Additionally, calculating partial marks and identifying errors had to be done manually, further reducing work efficiency.

[0563] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0564] In this invention, the server includes a means for the teacher to register correct answer data, a means for photographing the student's answer sheet to generate image data, and a means for transmitting the generated image data to an information processing device. This reduces the burden on the teacher and enables quick and accurate grading. Furthermore, by including a means for recognizing characters and figures from the received image data and comparing the recognized character and figure data with correct answer data, and a means for detecting errors based on the comparison results and calculating partial points, more advanced grading can be achieved. Furthermore, by including a means for highlighting details of errors and partial points, the teacher can provide efficient and visual feedback.

[0565] "Means for teachers to register correct answer data" refers to a device or application that allows teachers to input or record correct answers to test questions.

[0566] "Means for photographing students' answer sheets and generating image data" refers to means for photographing students' answers with a device such as a camera or scanner and generating image data thereof.

[0567] The "means for transmitting the generated image data to the information processing device" refers to a communication method or protocol for transmitting image data from a terminal to a central information processing device (such as a server).

[0568] "Means for recognizing characters and figures from received image data" refers to technology that extracts characters and figures from image data using OCR (optical character recognition) technology or a figure recognition algorithm.

[0569] "Means for comparing recognized character and graphic data with correct answer data" refers to an algorithm for comparing extracted character and graphic data with pre-registered correct answer data and evaluating the recognition results.

[0570] "Means for detecting errors based on the comparison results and calculating partial points" refers to a process for analyzing the comparison results, detecting errors or partial correct answers, and calculating points based on the results.

[0571] "Means for transmitting the calculation results to the teacher's terminal" refers to a method for communicating the calculation results from the server to the terminal used by the teacher.

[0572] "Means for displaying calculation results on the teacher's terminal" refers to an interface for visually displaying calculation results on the terminal screen.

[0573] "Quality check and communication means for image data by the terminal" refers to a method for verifying that the captured image data is accurate and complete, and then communicating the data.

[0574] "Means for analyzing data in an information processing device" refers to a method by which a central computer such as a server analyzes received data and extracts and processes the necessary information.

[0575] "Means for highlighting details of errors and partial marks" refers to visual devices that highlight the parts of the marking results that require careful checking.

[0576] The present invention is a system that allows school teachers to automatically grade complex problems, such as mathematical proofs. This system automatically grades students' answers by having teachers register the correct answer data and taking photos of the students' answers with a camera and sending them to a server. Specific embodiments for implementing this system are described below.

[0577] User interface and correct answer registration

[0578] User (Teacher):

[0579] The teacher launches the application and logs in by entering their account information on the login screen. After logging in, the main screen appears, and a "Register correct answers" button appears. The teacher presses this button to take a photo of the answer sheet with the camera and generate image data.

[0580] Device:

[0581] The device compresses the generated image data of the answer sheet and checks the image quality before sending it to the server. Possible devices include smartphones and tablets. Wi-Fi and mobile data communications are used as communication methods. Libraries such as OpenCV and Tesseract OCR can be used to process the image data.

[0582] server:

[0583] The server analyzes the received image data and uses an OCR engine to extract text from the image. It also uses a shape recognition algorithm to detect figures and mathematical formulas. This extracted data is structured and stored in a database. MySQL or MongoDB can be used as the database management system.

[0584] Read and submit student responses

[0585] User (Teacher):

[0586] The teacher switches to "test marking" mode and takes pictures of the students' answer sheets one by one with the camera, which generates image data of the students' answer sheets.

[0587] Device:

[0588] The device sends the image data of the generated student answer sheet to the server. Before sending, the data is temporarily saved and checked for quality to ensure there are no missing parts. This ensures that the data is uploaded accurately to the server.

[0589] server:

[0590] The server analyzes the image data of the received student answer sheets using OCR and graphic recognition, extracts the student answer data as digital data, and temporarily stores it in memory.

[0591] Data analysis and scoring

[0592] server:

[0593] The server compares the analyzed data with pre-saved correct answers. It uses advanced comparison algorithms to check for partial correct answers. For proof questions in particular, the evaluation focuses on the accuracy of logical steps and conclusions. It identifies errors and partial correct answers and calculates partial points.

[0594] Generate and display grade results

[0595] server:

[0596] The server generates the marking results and sends them to the teacher's device. The results include the score for each question, details of errors, and partial marks. This data is packaged and sent in JSON or XML format.

[0597] Device:

[0598] The device analyzes the received marking results and displays them on a user interface. Errors and partial marks are highlighted. JavaScript and front-end frameworks (e.g., React and Vue.js) can be used as display technologies.

[0599] User (Teacher):

[0600] Teachers can check the marks displayed on the device and provide feedback to students. Errors and partial marks can be visually confirmed, enabling quick and accurate instruction.

[0601] Examples and prompts

[0602] As a concrete example, consider a case where a teacher supports the grading of proof questions in junior high school mathematics. After launching the application and logging in, the teacher selects the correct answer registration mode, photographs the test answer sheet, and sends it to the server. The server analyzes the correct answers and stores them in a database.

[0603] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers and detects errors or partial marks. The grading results are then sent to the teacher's device and displayed with details highlighted. The teacher can then provide feedback to the student based on this information.

[0604] Example prompt sentence:

[0605] Please give us an overview of the system that streamlines grading of mathematical proof problems.

[0606] Please provide a detailed procedure for a system that allows teachers to automatically grade mathematics proof problems.

[0607] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0608] Step 1: User authentication and login

[0609] User (Teacher):

[0610] Input: Teacher account information (username, password).

[0611] How it works: The teacher launches the application and enters their account information on the login screen.

[0612] The server compares the received account information with the database and performs authentication.

[0613] Output: A successful authentication message or an error message is displayed on the terminal.

[0614] Step 2: Register the correct answer

[0615] User (Teacher):

[0616] Input: Exam answer sheet.

[0617] How it works: After logging in, press the "Register correct answer" button on the main screen that appears and take a photo of the answer sheet with your camera.

[0618] Device:

[0619] Operation: Generates captured image data and performs quality checks and adjustments on the images.

[0620] server:

[0621] Input: Image data sent from the device.

[0622] Operation: Analyzes received image data using an OCR engine and image recognition algorithms, extracts characters and images, and stores them in a database.

[0623] Output: The correct answer data is saved in the database, and a message indicating that the correct answer has been registered is displayed on the terminal.

[0624] Step 3: Reading student answer sheets

[0625] User (Teacher):

[0626] Input: Student answer sheet.

[0627] Operation: Switches to "test marking" mode and takes photos of students' answer sheets one by one with the camera.

[0628] Device:

[0629] Operation: Generates captured image data, temporarily stores it, and checks its quality.

[0630] server:

[0631] Input: Image data of student answer sheets sent from the device.

[0632] How it works: Analyzes image data using OCR and shape recognition to extract characters and shapes. The extracted data is temporarily stored in memory.

[0633] Output: Parsed data is held in memory.

[0634] Step 4: Data analysis and comparison

[0635] server:

[0636] Input: Parsed student answer data, correct answer data in the database.

[0637] How it works: Compares the student's answer data to the correct answer data. A comparison algorithm is used to detect errors and partial correct answers and calculate partial credit.

[0638] Output: Marking results including partial credit and errors.

[0639] Step 5: Generate and transfer the scores

[0640] server:

[0641] Input: Marking results including partial marks and errors.

[0642] Operation: The grading results are packaged in JSON format or similar for transfer to the teacher's device.

[0643] Device:

[0644] How it works: Parses the received marks and displays them in a user interface, highlighting errors and partial marks.

[0645] Output: The grading results are displayed on the teacher's terminal.

[0646] Step 6: Provide feedback

[0647] User (Teacher):

[0648] Input: Detailed marking results (details of each question's score, mistakes, and partial marks).

[0649] How it works: The teacher checks the grading results displayed on the device and provides feedback to the students as needed. For example, the teacher visually checks the errors and partial points for question number "6430" and enters appropriate instructional comments.

[0650] Output: Teacher feedback is provided to students.

[0651] (Application example 1)

[0652] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0653] Traditional test marking in educational settings and quality inspection in factories are time-consuming and labor-intensive, and there is a need for greater efficiency. Complex tasks such as marking mathematical proof problems and inspecting product quality are particularly prone to human error, which can result in a loss of accuracy. This increases the burden on teachers and factory workers and leads to a decline in labor productivity. There is a need for an efficient and accurate system to solve this problem.

[0654] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0655] In this invention, the server includes means for the teacher to register correct answers, means for photographing students' answer sheets and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answers, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for photographing the structure of a product, inspecting the quality of the product based on the recognized character and figure data, and automatically determining whether the product passes or fails, means for transmitting instructions to a robot arm to remove rejected products based on the pass / fail result, means for verifying the accuracy of product dimensions and labels based on the recognized character and figure data, and means for transmitting the quality inspection results to a management system and updating the pass / fail status. This makes it possible to improve the efficiency and accuracy of test marking in educational settings and quality inspection work in factories.

[0656] The "means for registering correct answers by the teacher" is an interface and function that allows the teacher to register answers in the system.

[0657] The "means for photographing student's answer sheet" is a function for photographing the student's answer sheet with a camera and generating image data of the photograph.

[0658] "Image data transmission means" refers to a communication means and protocol for transmitting the generated image data to the server.

[0659] "Character and graphic recognition means" refers to technology that extracts characters and graphics from received image data using optical character recognition (OCR) and graphic recognition algorithms.

[0660] The "answer comparison means" is a function that compares the recognized character and graphic data with correct answer data registered in advance to detect errors and inconsistencies.

[0661] The "partial score calculation means" is a function that calculates scores for the degree of error and partial correct answers based on the comparison results.

[0662] "Calculation result transmission means" refers to the communication means and protocol for transmitting the calculated grading results to the teacher's terminal.

[0663] The "grading result display means" is a user interface and function for visually displaying the calculation results on the teacher's terminal.

[0664] The "product image capturing means" is a function that captures images of products manufactured in a factory using a camera and generates image data of the images.

[0665] The "product quality inspection means" is a function that automatically inspects the quality of a product based on the recognized character and graphic data.

[0666] The "pass / fail judgment means" is a function that judges whether a product passes or fails based on the results of the quality inspection.

[0667] The "rejected product removal instruction means" is a function that sends instructions for the robot arm to remove the rejected product depending on the pass / fail result.

[0668] "Dimension and label accuracy verification means" is a function that verifies the accuracy of product dimensions and labels based on recognized character and graphic data.

[0669] "Management System Transmission Means" refers to the communication means and protocols for transmitting quality inspection results to the management system and updating pass / fail status.

[0670] This invention relates to a system for teachers to efficiently mark tests and a system for automating quality inspections in factories. This system uses a camera to take images of students' answer sheets and products, and analyzes the data using OCR (optical character recognition) and graphic recognition technology to automatically mark and inspect the quality.

[0671] Teacher's correct answer registration

[0672] The teacher launches the application on their own device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is sent from the device to the server. The server uses OCR to extract text from the received image data of the answer sheet and uses a graphic recognition algorithm to detect figures and symbols. This data is then structured and stored in a database.

[0673] Reading and marking student answers

[0674] The teacher switches to "test marking" mode and takes photos of students' answer sheets one by one. The generated image data of the answer sheets is sent from the device to the server. The server receives it and analyzes the answer data using OCR and shape recognition technology. The server compares the analyzed answer data with pre-registered correct answer data to detect errors and partial correct answers. In particular, for mathematical proof questions, the server verifies the accuracy of logical steps and conclusions and calculates partial points.

[0675] Sending and viewing graded results

[0676] The server then sends the calculated marks to the teacher's device, where the marks are displayed in a UI, visually highlighting the score for each question, errors, and partial marks.

[0677] Factory quality inspection

[0678] In the factory, a camera is used to take an image of the product. This image data is sent from the terminal to a server and analyzed using OCR and shape recognition technology. The server verifies the accuracy of the product's label and dimensions and determines whether it passes or fails. Depending on the pass / fail result, a robotic arm is instructed to remove the rejected product. The quality inspection results are sent to the management system, and the pass / fail status is updated.

[0679] Specific examples

[0680] For example, a teacher takes a photo of an exam answer sheet with a camera and sends it to a server. The server analyzes the correct answers using OCR and shape recognition, receives the photo data of the student's answer sheet, and analyzes it in the same way. The server compares the data and generates a score that is sent to the teacher's device. In factories, product labels and dimensions are inspected, and robotic arms remove non-conforming products.

[0681] Example prompt sentence:

[0682] Check if the label in the image is "EXPECTED_LABEL" and check if the product contour length is less than 100

[0683] This will enable more efficient test grading in educational settings and more accurate quality inspections in factories.

[0684] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0685] Step 1: Register the answer sheet

[0686] The user (teacher) starts the application on their device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is generated and sent to the server. The server uses OCR and graphic recognition technology on the received image data to recognize characters and figures, and stores the data in a database as structured data. The input is the image data of the photographed answer sheet, and the output is structured data of the correct answers.

[0687] Step 2: Photograph and submit student answer sheets

[0688] The user (teacher) switches to "test marking" mode and takes photos of students' answer sheets one after another. The generated image data of the answer sheets is sent from the device to the server. The input is the image data of the photographed answer sheets, and the output is the image data of the answer sheets sent to the server.

[0689] Step 3: Analyzing the answer sheet

[0690] The server uses OCR and graphic recognition technology on the image data of the received answer sheet to extract character and graphic data. The input is the image data of the answer sheet, and the output is the analyzed character and graphic data. In this step, the image data is converted into text data and graphic data.

[0691] Step 4: Compare and score answers

[0692] The server compares the analyzed answer data with the correct answer data saved in step 1. It detects errors and partial correct answers and calculates partial points. The input is the analyzed answer data and correct answer data, and the output is the scoring result (including partial points). In this step, the data is compared with the correct answer data and scored.

[0693] Step 5: Submit your results

[0694] The server sends the calculated score to the user's terminal. The input is the score, and the output is the score sent to the user's terminal. In this step, data communication occurs.

[0695] Step 6: View your results

[0696] The user's device displays the received marking results. The displayed information includes the score for each question, details of errors, and partial marks. The input is the received marking results, and the output is the marking results displayed to the user.

[0697] Step 7: Take product images

[0698] The terminal takes images of products manufactured in the factory with a camera. The generated image data is sent to the server. The input is the image data of the photographed product, and the output is the image data sent to the server.

[0699] Step 8: Analyze product quality

[0700] The server uses OCR and shape recognition technology on the received product image data to verify the label and dimensions. The input is the product image data, and the output is the parsed label and dimension data.

[0701] Step 9: Quality assessment and notification of pass / fail

[0702] The server inspects the quality based on the analyzed data and judges whether the product passes or fails. Depending on the pass / fail result, it sends instructions to the robot arm to remove the rejected product. The input is the analyzed label and dimensional data, and the output is the pass / fail judgment result and instructions to the robot arm.

[0703] Step 10: Submit and update test results

[0704] The server sends the quality inspection results to the management system and updates the pass / fail status. The input is the quality inspection result and the output is the updated status of the management system. In this step, data communication with the management system takes place.

[0705] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0706] The present invention is a system designed to improve the efficiency of test grading for school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades tests by having the teacher register correct answers and then photographing students' answers with a camera and sending them to a server. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system enables smoother work. Below, we will explain the form and specific program processing for implementing this system.

[0707] User interface and correct answer registration

[0708] User (Teacher):

[0709] First, the teacher launches the app and logs in. After logging in, the main screen appears and the teacher presses the "Register correct answers" button to activate the camera. The teacher then takes a photo of the test answer sheet and generates image data. The generated image data is sent to the server.

[0710] Device:

[0711] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is later processed for character recognition (OCR) and graphic recognition.

[0712] server:

[0713] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database as correct answer data.

[0714] Reading and grading student responses

[0715] User (Teacher):

[0716] Select the "Test Marking" mode and take photos of each student's answer sheet one by one, which will generate image data for each answer sheet.

[0717] Device:

[0718] The terminal sends the image data of the generated student answer sheet to the server.

[0719] server:

[0720] The server performs character recognition (OCR) and graphic recognition on the received image data, analyzing the student's answers as digital data. This data is then compared with the correct answer data to detect errors and partial correct answers. For proof questions in particular, the accuracy of the logical steps and conclusions is checked. Partial marks are calculated based on errors and partial correct answers, and a final score is generated.

[0721] Introduction of emotion engine and its application

[0722] server:

[0723] An emotion engine is introduced, and the server recognizes the user's (teacher's) emotions. This emotion recognition is performed by analyzing the user's facial expressions and voice using sensors such as cameras and microphones. For example, if an angry or frustrated facial expression is recognized, the system changes the displayed feedback and interface.

[0724] Device:

[0725] The device can then adjust its display based on the results of emotion recognition. For example, if the user is feeling stressed, the system can display a brief summary and possibly offer suggestions for relaxing the user.

[0726] User (Teacher):

[0727] The emotion engine allows teachers to receive appropriate feedback based on their own emotional state, reducing excessive stress during grading and enabling them to work more efficiently.

[0728] Sending and viewing graded results

[0729] server:

[0730] The server sends the generated grading results to the teacher's terminal, and based on the emotion engine, the grading results are displayed to the user in an appropriate format.

[0731] Device:

[0732] The device then displays the received scoring results on the UI. The displayed information includes the score for each question, details of errors, and partial points. Errors and partial points are highlighted based on the recognized emotion.

[0733] User (Teacher):

[0734] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[0735] Specific examples

[0736] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[0737] Next, the teacher takes photos of each student's answer sheet and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where the errors and partial marks are highlighted.

[0738] This system will streamline teachers' marking work, saving them a huge amount of time and effort. The emotion engine will also reduce the mental burden on teachers.

[0739] The processing flow will be explained below.

[0740] Step 1:

[0741] User (teacher): Launch the app and log in by entering your account information on the login screen.

[0742] Step 2:

[0743] On your device: Send your login credentials to the server.

[0744] Step 3:

[0745] Server: Validates the authentication information and returns the authentication result to the terminal. If authentication is successful, it instructs the terminal to display the main screen.

[0746] Step 4:

[0747] User (Teacher): Press the "Register correct answer" button on the main screen to launch the camera.

[0748] Step 5:

[0749] Device: The camera is activated and the teacher takes a picture of the answer sheet to generate image data.

[0750] Step 6:

[0751] Terminal: Sends the image data of the photographed answer sheet to the server.

[0752] Step 7:

[0753] Server: Processes the received image data and performs character recognition (OCR) and shape recognition. A character recognition engine is used to extract text from the image, and a shape recognition algorithm is used to detect figures and symbols.

[0754] Step 8:

[0755] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[0756] Step 9:

[0757] User (Teacher): Select "Test Grading" mode on the main screen.

[0758] Step 10:

[0759] User (teacher): Takes photos of students' answer sheets one after another.

[0760] Step 11:

[0761] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[0762] Step 12:

[0763] Server: Processes the image data of the received answer sheets, performs character recognition (OCR) and graphic recognition, extracts the students' answers and graphics, and analyzes them as digital data.

[0764] Step 13:

[0765] Server: Compares the student's answers with the registered correct answers. It verifies the text and figures, and in the case of proof questions, checks the accuracy of the logical steps and conclusions.

[0766] Step 14:

[0767] Server: Determines errors, partial correct answers, and complete correct answers, highlights errors, and calculates partial points.

[0768] Step 15:

[0769] Server: Generates the scoring results and compiles them into data including the score for each question, the total score, and details of any errors.

[0770] Step 16:

[0771] Server: Sends the grading results to the teacher's device.

[0772] Step 17:

[0773] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[0774] Step 18:

[0775] Server: Using the emotion engine, analyzes the user's (teacher's) facial expressions and voice and recognizes their emotions.

[0776] Step 19:

[0777] Server: Based on the recognized emotions, it adjusts the interface and feedback method, and generates appropriate feedback according to the teacher's emotional state.

[0778] Step 20:

[0779] Device: Adjusts the interface and changes the display content based on the recognized emotion information.

[0780] Step 21:

[0781] User (Teacher): Check the displayed marks and feedback and provide appropriate feedback to the student. Errors are highlighted and partial marks are visually confirmed.

[0782] Example 2

[0783] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0784] Conventional test marking systems have difficulty efficiently and accurately marking complex problems, especially those like mathematical proofs. Furthermore, they are unable to adjust the interface to take into account the teacher's emotional state, which can lead to stress during long marking sessions. Furthermore, they lack sufficient feedback to students, preventing them from providing effective learning support.

[0785] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet to generate image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for recognizing the teacher's emotions, and means for adjusting the display method based on the recognized emotions. This enables efficient and accurate grading of complex questions, reduces stress by adjusting the interface according to the teacher's emotional state, and enables effective feedback to students.

[0786] A "teacher" refers to a professional who provides instruction and education to students at an educational institution.

[0787] "Correct answer" refers to the exact or best answer to a particular question.

[0788] A "means" refers to a method or device used to achieve a particular purpose.

[0789] An "answer sheet" refers to the paper on which students write their answers in an exam or test.

[0790] "Image data" refers to image information acquired by a camera or scanner and stored in digital format.

[0791] A "server" refers to a computer system that stores and processes data.

[0792] "Characters" refer to symbols used to represent human languages.

[0793] "Figure" refers to a geometric shape or symbol.

[0794] "Recognition" refers to the process of extracting specific information from an image or sound.

[0795] "Comparison" refers to the process of comparing two or more pieces of data to identify differences and similarities.

[0796] An "error" is an inaccuracy or mistake compared to the correct answer.

[0797] "Partial credit" refers to a mark awarded as a portion of the total score.

[0798] "Calculation" refers to the process of deriving a specific result from numbers and data.

[0799] "Terminal" refers to a device that communicates with a server and sends and receives data.

[0800] "Emotion" refers to a person's psychological state and facial expression.

[0801] "Recognized emotion" refers to the emotional state determined by the system through analysis of the user's facial expressions and voice.

[0802] "Display method" refers to how information or results are presented in the user interface.

[0803] The present invention relates to a system that helps teachers to efficiently grade tests, and in particular, a system that automates grading of complex problems such as mathematical proofs. This system reduces the burden on teachers and enables quick and accurate feedback to students.

[0804] System configuration

[0805] The system consists of the following main components:

[0806] 1. User (Teacher):

[0807] A device that allows teachers to register correct answers and photograph students' answer sheets. Specifically, it uses a device such as a smartphone or tablet.

[0808] 2. Terminal:

[0809] The device used by the teacher takes pictures of the correct answer sheets and the students' answer sheets and sends the image data to the server. This device is equipped with a camera function and an internet connection function.

[0810] 3. Server:

[0811] The server processes the received image data and recognizes characters and figures. Specifically, it uses an OCR (Optical Character Recognition) engine and figure recognition algorithms. The recognized data is compared with a database of correct answers to detect errors and partial correct answers.

[0812] 4. Emotion Engine:

[0813] The emotion engine recognizes the teacher's emotions and adjusts the way the user interface is displayed. Specifically, the system is equipped with a camera and microphone to analyze the teacher's facial expressions and voice.

[0814] Program processing

[0815] 1. Correct Answer Registration:

[0816] The teacher launches the app on their device and logs in. They press the "Register correct answer" button on the main screen to activate the camera and take a picture of the test answer sheet. The captured image data is sent from the device to the server. The server processes the received image data using an OCR engine and a shape recognition algorithm, and stores the text and shape data in a database.

[0817] 2. Scoring Student Responses:

[0818] The teacher selects the "Test Marking" mode and takes a photo of the student's answer sheet. The device sends the captured image data to the server. The server processes the received data with an OCR engine and shape recognition algorithms, and compares the recognized data with the correct answer data. It detects errors and partial correct answers, calculates partial points, and generates the final marking result.

[0819] 3. Emotion recognition:

[0820] The server uses an emotion engine to analyze the teacher's facial expressions and voice to recognize their emotional state. Based on the recognized emotion, the system adjusts the interface and feedback displayed. For example, if the teacher is feeling stressed, the system adjusts to display only brief summary information.

[0821] 4. View your score:

[0822] The server sends the calculated score to the teacher's device. The device displays the received score on the UI, highlighting errors and partial points. The teacher checks the displayed results and provides feedback to the student.

[0823] Specific examples

[0824] For example, to assist with grading proof questions in junior high school mathematics, a teacher first launches the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the answer sheet and stores it in a database. Next, the teacher takes a photo of the student's answer sheet and sends it to the server. The server compares the student's answer with the correct answer and generates a grading result. The final grading result is sent to the teacher's device, and errors and partial points are highlighted.

[0825] Prompt Sentence Examples

[0826] "Please explain your system for streamlining the grading of proof questions in junior high school mathematics. Specifically, please explain how correct answers are registered, how student answers are graded, how an emotion engine is used, and how the final graded results are transmitted and displayed."

[0827] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0828] Step 1:

[0829] The user (teacher) starts the app and logs in. On the login screen, they enter their username and password and press the login button. The entered authentication information is sent from the device to the server.

[0830] Input: Username and Password

[0831] Data processing: The server checks the authentication information against a database

[0832] Output: Authentication token

[0833] Specific operation: After successful login, an authentication token is sent to the device.

[0834] Step 2:

[0835] The user (teacher) presses the "Register correct answer" button on the main screen. When the teacher takes a photo of the test answer sheet using the camera and presses the "Send" button, the device generates image data and sends it to the server.

[0836] Input: Image data of the answer sheet

[0837] Data processing: Encode image data into BASE64 format

[0838] Output: Encoded image data

[0839] Specific operation: The device sends the image data to the server as an HTTP POST request.

[0840] Step 3:

[0841] The server processes the received image data and extracts text and graphics data using an optical character recognition (OCR) engine and graphics recognition algorithms.

[0842] Input: Encoded image data

[0843] Data processing: Extract text and graphics using OCR engine and graphic recognition algorithms

[0844] Output: Recognized text and graphics data

[0845] Specific behavior: Save the extracted data to the database

[0846] Step 4:

[0847] The user (teacher) selects the "Test Marking" mode, takes pictures of students' answer sheets one after another, and presses the "Send" button. The device then sends the captured image data to the server.

[0848] Input: Image data of student answer sheets

[0849] Data processing: Encode image data into BASE64 format

[0850] Output: Encoded image data

[0851] Specific operation: The device sends the image data to the server as an HTTP POST request.

[0852] Step 5:

[0853] The server performs character recognition (OCR) and graphic recognition on the image data of the received student answer sheets, and compares the extracted data with the correct answer data.

[0854] Input: Encoded image data

[0855] Data processing: Extract text and graphics using an OCR engine and graphic recognition algorithms, and compare them with the correct data

[0856] Output: Information about errors and partial answers

[0857] Specific behavior: Calculate partial points based on errors and partial correct answers

[0858] Step 6:

[0859] The server sends the calculated score to the teacher's terminal.

[0860] Input: Information about errors and partial answers

[0861] Data processing: Calculating the score and converting the results to JSON format

[0862] Output: JSON data of the scoring results

[0863] Specific behavior: Sends JSON data as an HTTP response

[0864] Step 7:

[0865] The device then displays the received scoring results on a user interface. The displayed information includes the score for each question, details of errors, and partial points, and the display method is adjusted based on the results of the emotion engine.

[0866] Input: JSON data of the score results

[0867] Data processing: Parsing JSON data and binding it to UI components

[0868] Output: A visual representation of the scoring results

[0869] Specific operation: Errors and partial points are highlighted

[0870] Step 8:

[0871] The server uses an emotion engine to recognize the teacher's emotional state and adjusts the feedback and interface based on the recognized emotion.

[0872] Input: Teacher's facial expressions and voice data

[0873] Data processing: Analysis and emotion determination using an emotion engine

[0874] Output: Recognized emotion information

[0875] Specific behavior: Dynamically adjusting the UI and displaying concise summary information

[0876] Step 9:

[0877] The user (teacher) checks the displayed grading results and provides feedback to the student. Specifically, the user enters additional comments and feedback through the input form and sends it to the student through the system.

[0878] Input: Marking results and additional feedback

[0879] Data processing: Input feedback is saved in a database

[0880] Output: Send feedback

[0881] Specific behavior: The feedback content is notified to the student.

[0882] (Application example 2)

[0883] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] Manufacturing sites are required to simultaneously achieve quality control and reduce the mental burden on workers. In particular, it is important to accurately inspect product quality, provide prompt feedback, and take appropriate measures when workers feel stressed or fatigued. To solve these issues, a system that combines quality control and emotion recognition is required. However, conventional systems have difficulty providing these two functions in an integrated manner, which can lead to a decrease in work efficiency.

[0885] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for registering correct standards for quality control, means for photographing the manufactured product and generating image data, means for transmitting the generated image data to the server and comparing it with the quality standard, means for recognizing the worker's emotions and providing feedback, means for transmitting the recognition results to the worker, and means for displaying the recognition results on the worker's terminal. This enables quality control and reduces the mental burden on the worker.

[0886] The "means for teachers to register correct answers" is a mechanism for teachers to input standard answers set in advance into the system.

[0887] The "means for photographing students' answer sheets and generating image data" refers to a mechanism for capturing students' answers with a photographing device such as a camera and converting the images into digital data.

[0888] The "means for transmitting the generated image data to the server" is a mechanism for sending the captured image to the central processing unit via the network.

[0889] The "means for recognizing characters and figures from received image data" is a mechanism for identifying text and figures contained in the transmitted image data and extracting them as digital data.

[0890] The "means for comparing recognized character and graphic data with correct answers" is a mechanism for comparing extracted character and graphic information with pre-registered correct answer data and performing evaluation.

[0891] The "means for detecting errors based on the comparison results and calculating partial points" is a mechanism for detecting errors from the results of comparing the character and graphic data with the correct data and calculating the partial points accordingly.

[0892] The "means for transmitting the calculation results to the teacher's terminal" is a mechanism for transferring the calculated scores and evaluation results to the terminal used by the educator.

[0893] The "means for displaying the calculation results on the teacher's terminal" is a mechanism for visually presenting the calculated scores and evaluation results on the teacher's terminal.

[0894] "Means for registering correct standards for quality control" is a mechanism for registering standards for quality evaluation in the manufacturing industry in the system.

[0895] The "means for photographing manufactured products and generating image data" is a mechanism for photographing manufactured products with a camera or the like and generating the image as data.

[0896] The "means for transmitting the generated image data to a server and comparing it with quality standards" is a mechanism for transmitting image data of the photographed product to a central processing unit and comparing it with quality evaluation standards.

[0897] The "means for recognizing workers' emotions and providing feedback" is a mechanism for determining the emotions of workers from their facial expressions and voice, and providing appropriate advice and evaluations based on that.

[0898] The "means for transmitting the recognition result to the worker" is a mechanism for transferring the emotion determination result to the terminal used by the worker.

[0899] The "means for displaying the recognition result on the worker's terminal" is a mechanism for visually presenting the emotion determination result on the worker's terminal.

[0900] The present invention provides a system for integrating quality control and worker emotion recognition in educational and manufacturing settings. Specific embodiments for carrying out the invention will be described below.

[0901] System Overview

[0902] This system provides an interface that can be easily operated by educators and workers using terminals, and realizes quality control and emotion recognition functions accurately and quickly.The system consists of the following main components.

[0903] 1. Teachers and quality control personnel:

[0904] The teacher starts the application and logs in. After logging in, the teacher uses the correct answer registration mode to take a photo of the correct answer with the camera and generate image data. This image data is sent to the server and registered as the correct answer data.

[0905] Quality control personnel can register quality standard data in a similar manner.

[0906] 2. Student and Product Management:

[0907] Students' answer sheets and finished products are photographed with a camera, and the image data is sent to a server, where characters and figures are recognized from the received image data and compared with registered correct answer data and quality standard data.

[0908] 3. Server:

[0909] The server uses OCR (optical character recognition) software to extract characters and graphic recognition algorithms to detect shapes and symbols. This data is compared with correct answers and quality benchmark data to detect errors and calculate partial points.

[0910] 4. Emotion Recognition Engine:

[0911] The server is also equipped with an emotion recognition engine that uses sensors such as cameras and microphones to analyze the emotions of users (workers and teachers). For example, if the system detects fatigue or stress in a user, it will provide appropriate feedback.

[0912] 5. Terminal:

[0913] The calculation results and emotion recognition results are displayed in real time on the devices of teachers, quality control personnel, and workers, and the display method can be adjusted according to the user's emotional state.

[0914] Hardware and Software Use

[0915] This system uses the following hardware and software:

[0916] Camera: Used to acquire image data.

[0917] Server: The main component that processes and manages data.

[0918] Terminal (PC, tablet, smartphone, etc.): A device that is operated by the user.

[0919] OCR software (e.g., PyTesseract): Extract character data from images.

[0920] Shape recognition algorithm: An algorithm for recognizing shapes and symbols from images.

[0921] Emotion recognition engine (e.g., Keras-based model): for analyzing emotions from user facial expressions and voice.

[0922] Specific examples

[0923] For example, this system is extremely effective in quality control on automobile production lines. Quality control personnel register product quality standards and take photos of manufactured parts with a camera. The image data is then sent to a server where it is automatically compared with the quality standards. Meanwhile, an emotion recognition engine analyzes the facial expressions and voice of workers, and if fatigue or stress is detected, the system suggests taking a break.

[0924] Prompt Sentence Examples

[0925] An example of an input prompt for the generative AI model associated with this system is shown below.

[0926] "Think of a quality control system on an automobile production line. In the system, an inspection robot takes a picture of the product and compares it with quality standards to automatically inspect and evaluate it. The system also recognizes the emotions of workers and provides appropriate feedback. For workers who appear particularly tired, it suggests that they take a break."

[0927] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0928] Step 1:

[0929] A user (teacher or quality control officer) starts the application and logs in. The input here is the user's identification information (user ID and password), and the output is the authentication result. Once the user is authenticated, the main screen is displayed. During this authentication process, the user information is checked against the database, and login is allowed only if it is correct.

[0930] Step 2:

[0931] The user selects the "Correct Answer Registration" mode or the "Quality Standard Registration" mode and takes a picture of the correct answer sheet or quality standard with the camera. The input here is the captured image, and the output is the generated image data. The device controls the camera and saves the image as digital data. At this stage, the image data is saved locally on the spot.

[0932] Step 3:

[0933] The terminal sends the generated image data to the server. The input here is the image data, and the output is a transmission success message to the server. The terminal uploads the image data to the server via the network and receives a confirmation message if successful.

[0934] Step 4:

[0935] The server recognizes characters and graphics from the image data it receives. The input here is the transmitted image data, and the output is the recognized character and graphic data. Specifically, OCR software (e.g., PyTesseract) and graphic recognition algorithms analyze the image on the server and extract character and graphic information.

[0936] Step 5:

[0937] The server compares the recognized character and graphic data with the correct answer data or quality standard data. The input here is the recognized character and graphic data and the pre-registered correct answer data or quality standard data, and the output is the comparison result. The server collates these data and detects differences.

[0938] Step 6:

[0939] The server detects errors based on the comparison results and calculates partial scores. The input here is the comparison results, and the output is the calculated partial scores and a list of error locations. The server follows a programmed algorithm to determine which parts are correct, which parts are incorrect, and calculate the partial scores.

[0940] Step 7:

[0941] The server sends the calculation results to the user's device. The input here is the calculation result, and the output is a message to the device indicating successful transmission. The server packages the calculation results in JSON format or similar and sends them over the network to the user's device.

[0942] Step 8:

[0943] The terminal displays the calculation results it receives. The input here is the calculation result received, and the output is visual feedback that can be checked by a teacher or quality control officer. The terminal displays the calculation results on the screen, highlighting any errors or partial points to the user.

[0944] Step 9:

[0945] Sensors (camera and microphone) for recognizing the user's emotions capture the user's facial expressions and voice. The input here is the user's facial image and voice data, and the output is the captured sensor data. The device records this data as it goes along and sends it to a server for analysis.

[0946] Step 10:

[0947] The server analyzes the user's emotions using an emotion recognition engine, where the input is the captured sensor data and the output is the recognized emotion classification (e.g., stress, satisfaction, confidence, etc.). The emotion recognition engine (e.g., a Keras-based model) is used to determine the user's emotions in real time.

[0948] Step 11:

[0949] The server provides feedback based on the recognition results and sends the results to the user's device. The input here is the emotion recognition result, and the output is a feedback message to the user. The server creates advice or warnings according to the recognized emotion and sends them to the device.

[0950] Step 12:

[0951] The terminal displays the received recognition results and provides appropriate visual feedback to the user. The input here is the received recognition results, and the output is a feedback display that the user can check. The terminal displays the feedback content in a form that the user can easily understand.

[0952] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0953] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0954] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0955] [Third embodiment]

[0956] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0957] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0958] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0959] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0960] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0961] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0962] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0963] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0964] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0965] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0966] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0967] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0968] The present invention is a system that aims to improve the efficiency of test grading by school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades students' answers by having the teacher register the correct answers and then photographing the students' answers with a camera and sending them to a server. Below, we will explain how to implement this system.

[0969] User interface and correct answer registration

[0970] User (Teacher):

[0971] The teacher launches the app and enters their account information on the login screen. After logging in, the main screen appears, displaying a "Register correct answers" button. Pressing this button activates the camera, takes a picture of the test answer sheet, and generates image data. The generated image data is sent to the server.

[0972] Device:

[0973] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is then processed for character recognition (OCR) and graphic recognition.

[0974] server:

[0975] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database.

[0976] Reading and grading student responses

[0977] User (Teacher):

[0978] The system switches to "Test Marking" mode and begins taking photos of students' answer sheets one by one, generating image data for each answer sheet.

[0979] Device:

[0980] The terminal sends the image data of the generated student answer sheet to the server.

[0981] server:

[0982] The server performs character recognition (OCR) and graphic recognition on the received image data and analyzes the student's answers as digital data. The analyzed data is compared with the correct answer data already saved, and errors and partial correct answers are detected. In particular, for proof questions, the accuracy of the logical steps and conclusions is checked. Partial points are calculated based on errors and partial correct answers, and the final score is generated.

[0983] Sending and viewing graded results

[0984] server:

[0985] The server sends the generated grading results to the teacher's terminal.

[0986] Device:

[0987] The device then displays the received score results on the UI, including the score for each question, any errors made, and details of the partial score.

[0988] User (Teacher):

[0989] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[0990] Specific examples

[0991] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[0992] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where errors and partial marks are highlighted. The teacher then provides feedback to the student based on this.

[0993] This system will streamline teachers' marking work, saving them a huge amount of time and effort.

[0994] The processing flow will be explained below.

[0995] Step 1:

[0996] User (teacher): Start the app, enter your account information, and log in. After logging in, the main screen will appear, and you can press the "Register correct answer" button.

[0997] Step 2:

[0998] Device: When the correct answer registration button is pressed, the camera is activated and the teacher takes a picture of the answer sheet. The captured image data is generated.

[0999] Step 3:

[1000] Terminal: Sends the generated image data to the server.

[1001] Step 4:

[1002] Server: Initially processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols.

[1003] Step 5:

[1004] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[1005] Step 6:

[1006] User (teacher): Selects "Test marking" mode and takes photos of students' answer sheets one after another.

[1007] Step 7:

[1008] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[1009] Step 8:

[1010] Server: Performs character recognition (OCR) and graphic recognition on the received image data. Extracts student answers and graphics and analyzes them as digital data.

[1011] Step 9:

[1012] Server: Compares the student's answers with the registered correct answers. For each problem, matches the text and figures. For proof problems, checks the accuracy of the logical steps and conclusions.

[1013] Step 10:

[1014] Server: Determines errors, partial correct answers, and complete correct answers in detail. Highlights errors and calculates partial points.

[1015] Step 11:

[1016] Server: Generates and compiles the scoring results, including the score for each question, the total score, and details of errors.

[1017] Step 12:

[1018] Server: Sends the scoring results to the device.

[1019] Step 13:

[1020] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[1021] Step 14:

[1022] User (Teacher): Checks the marks and provides feedback to students, highlighting errors and visually checking partial marks details.

[1023] Example 1

[1024] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1025] Grading tests in the past required a great deal of time and effort from teachers, especially for complex problems like mathematical proofs. This increased the burden on teachers and made it difficult to grade quickly and accurately. Additionally, calculating partial marks and identifying errors had to be done manually, further reducing work efficiency.

[1026] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1027] In this invention, the server includes a means for the teacher to register correct answer data, a means for photographing the student's answer sheet to generate image data, and a means for transmitting the generated image data to an information processing device. This reduces the burden on the teacher and enables quick and accurate grading. Furthermore, by including a means for recognizing characters and figures from the received image data and comparing the recognized character and figure data with correct answer data, and a means for detecting errors based on the comparison results and calculating partial points, more advanced grading can be achieved. Furthermore, by including a means for highlighting details of errors and partial points, the teacher can provide efficient and visual feedback.

[1028] "Means for teachers to register correct answer data" refers to a device or application that allows teachers to input or record correct answers to test questions.

[1029] "Means for photographing students' answer sheets and generating image data" refers to means for photographing students' answers with a device such as a camera or scanner and generating image data thereof.

[1030] The "means for transmitting the generated image data to the information processing device" refers to a communication method or protocol for transmitting image data from a terminal to a central information processing device (such as a server).

[1031] "Means for recognizing characters and figures from received image data" refers to technology that extracts characters and figures from image data using OCR (optical character recognition) technology or a figure recognition algorithm.

[1032] "Means for comparing recognized character and graphic data with correct answer data" refers to an algorithm for comparing extracted character and graphic data with pre-registered correct answer data and evaluating the recognition results.

[1033] "Means for detecting errors based on the comparison results and calculating partial points" refers to a process for analyzing the comparison results, detecting errors or partial correct answers, and calculating points based on the results.

[1034] "Means for transmitting the calculation results to the teacher's terminal" refers to a method for communicating the calculation results from the server to the terminal used by the teacher.

[1035] "Means for displaying calculation results on the teacher's terminal" refers to an interface for visually displaying calculation results on the terminal screen.

[1036] "Quality check and communication means for image data by the terminal" refers to a method for verifying that the captured image data is accurate and complete, and then communicating the data.

[1037] "Means for analyzing data in an information processing device" refers to a method by which a central computer such as a server analyzes received data and extracts and processes the necessary information.

[1038] "Means for highlighting details of errors and partial marks" refers to visual devices that highlight the parts of the marking results that require careful checking.

[1039] The present invention is a system that allows school teachers to automatically grade complex problems, such as mathematical proofs. This system automatically grades students' answers by having teachers register the correct answer data and taking photos of the students' answers with a camera and sending them to a server. Specific embodiments for implementing this system are described below.

[1040] User interface and correct answer registration

[1041] User (Teacher):

[1042] The teacher launches the application and logs in by entering their account information on the login screen. After logging in, the main screen appears, and a "Register correct answers" button appears. The teacher presses this button to take a photo of the answer sheet with the camera and generate image data.

[1043] Device:

[1044] The device compresses the generated image data of the answer sheet and checks the image quality before sending it to the server. Possible devices include smartphones and tablets. Wi-Fi and mobile data communications are used as communication methods. Libraries such as OpenCV and Tesseract OCR can be used to process the image data.

[1045] server:

[1046] The server analyzes the received image data and uses an OCR engine to extract text from the image. It also uses a shape recognition algorithm to detect figures and mathematical formulas. This extracted data is structured and stored in a database. MySQL or MongoDB can be used as the database management system.

[1047] Read and submit student responses

[1048] User (Teacher):

[1049] The teacher switches to "test marking" mode and takes pictures of the students' answer sheets one by one with the camera, which generates image data of the students' answer sheets.

[1050] Device:

[1051] The device sends the image data of the generated student answer sheet to the server. Before sending, the data is temporarily saved and checked for quality to ensure there are no missing parts. This ensures that the data is uploaded accurately to the server.

[1052] server:

[1053] The server analyzes the image data of the received student answer sheets using OCR and graphic recognition, extracts the student answer data as digital data, and temporarily stores it in memory.

[1054] Data analysis and scoring

[1055] server:

[1056] The server compares the analyzed data with pre-saved correct answers. It uses advanced comparison algorithms to check for partial correct answers. For proof questions in particular, the evaluation focuses on the accuracy of logical steps and conclusions. It identifies errors and partial correct answers and calculates partial points.

[1057] Generate and display grade results

[1058] server:

[1059] The server generates the marking results and sends them to the teacher's device. The results include the score for each question, details of errors, and partial marks. This data is packaged and sent in JSON or XML format.

[1060] Device:

[1061] The device analyzes the received marking results and displays them on a user interface. Errors and partial marks are highlighted. JavaScript and front-end frameworks (e.g., React and Vue.js) can be used as display technologies.

[1062] User (Teacher):

[1063] Teachers can check the marks displayed on the device and provide feedback to students. Errors and partial marks can be visually confirmed, enabling quick and accurate instruction.

[1064] Examples and prompts

[1065] As a concrete example, consider a case where a teacher supports the grading of proof questions in junior high school mathematics. After launching the application and logging in, the teacher selects the correct answer registration mode, photographs the test answer sheet, and sends it to the server. The server analyzes the correct answers and stores them in a database.

[1066] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers and detects errors or partial marks. The grading results are then sent to the teacher's device and displayed with details highlighted. The teacher can then provide feedback to the student based on this information.

[1067] Example prompt sentence:

[1068] Please give us an overview of the system that streamlines grading of mathematical proof problems.

[1069] Please provide a detailed procedure for a system that allows teachers to automatically grade mathematics proof problems.

[1070] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1071] Step 1: User authentication and login

[1072] User (Teacher):

[1073] Input: Teacher account information (username, password).

[1074] How it works: The teacher launches the application and enters their account information on the login screen.

[1075] The server compares the received account information with the database and performs authentication.

[1076] Output: A successful authentication message or an error message is displayed on the terminal.

[1077] Step 2: Register the correct answer

[1078] User (Teacher):

[1079] Input: Exam answer sheet.

[1080] How it works: After logging in, press the "Register correct answer" button on the main screen that appears and take a photo of the answer sheet with your camera.

[1081] Device:

[1082] Operation: Generates captured image data and performs quality checks and adjustments on the images.

[1083] server:

[1084] Input: Image data sent from the device.

[1085] Operation: Analyzes received image data using an OCR engine and image recognition algorithms, extracts characters and images, and stores them in a database.

[1086] Output: The correct answer data is saved in the database, and a message indicating that the correct answer has been registered is displayed on the terminal.

[1087] Step 3: Reading student answer sheets

[1088] User (Teacher):

[1089] Input: Student answer sheet.

[1090] Operation: Switches to "test marking" mode and takes photos of students' answer sheets one by one with the camera.

[1091] Device:

[1092] Operation: Generates captured image data, temporarily stores it, and checks its quality.

[1093] server:

[1094] Input: Image data of student answer sheets sent from the device.

[1095] How it works: Analyzes image data using OCR and shape recognition to extract characters and shapes. The extracted data is temporarily stored in memory.

[1096] Output: Parsed data is held in memory.

[1097] Step 4: Data analysis and comparison

[1098] server:

[1099] Input: Parsed student answer data, correct answer data in the database.

[1100] How it works: Compares the student's answer data to the correct answer data. A comparison algorithm is used to detect errors and partial correct answers and calculate partial credit.

[1101] Output: Marking results including partial credit and errors.

[1102] Step 5: Generate and transfer the scores

[1103] server:

[1104] Input: Marking results including partial marks and errors.

[1105] Operation: The grading results are packaged in JSON format or similar for transfer to the teacher's device.

[1106] Device:

[1107] How it works: Parses the received marks and displays them in a user interface, highlighting errors and partial marks.

[1108] Output: The grading results are displayed on the teacher's terminal.

[1109] Step 6: Provide feedback

[1110] User (Teacher):

[1111] Input: Detailed marking results (details of each question's score, mistakes, and partial marks).

[1112] How it works: The teacher checks the grading results displayed on the device and provides feedback to the students as needed. For example, the teacher visually checks the errors and partial points for question number "6430" and enters appropriate instructional comments.

[1113] Output: Teacher feedback is provided to students.

[1114] (Application example 1)

[1115] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1116] Traditional test marking in educational settings and quality inspection in factories are time-consuming and labor-intensive, and there is a need for greater efficiency. Complex tasks such as marking mathematical proof problems and inspecting product quality are particularly prone to human error, which can result in a loss of accuracy. This increases the burden on teachers and factory workers and leads to a decline in labor productivity. There is a need for an efficient and accurate system to solve this problem.

[1117] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1118] In this invention, the server includes means for the teacher to register correct answers, means for photographing students' answer sheets and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answers, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for photographing the structure of a product, inspecting the quality of the product based on the recognized character and figure data, and automatically determining whether the product passes or fails, means for transmitting instructions to a robot arm to remove rejected products based on the pass / fail result, means for verifying the accuracy of product dimensions and labels based on the recognized character and figure data, and means for transmitting the quality inspection results to a management system and updating the pass / fail status. This makes it possible to improve the efficiency and accuracy of test marking in educational settings and quality inspection work in factories.

[1119] The "means for registering correct answers by the teacher" is an interface and function that allows the teacher to register answers in the system.

[1120] The "means for photographing student's answer sheet" is a function for photographing the student's answer sheet with a camera and generating image data of the photograph.

[1121] "Image data transmission means" refers to a communication means and protocol for transmitting the generated image data to the server.

[1122] "Character and graphic recognition means" refers to technology that extracts characters and graphics from received image data using optical character recognition (OCR) and graphic recognition algorithms.

[1123] The "answer comparison means" is a function that compares the recognized character and graphic data with correct answer data registered in advance to detect errors and inconsistencies.

[1124] The "partial score calculation means" is a function that calculates scores for the degree of error and partial correct answers based on the comparison results.

[1125] "Calculation result transmission means" refers to the communication means and protocol for transmitting the calculated grading results to the teacher's terminal.

[1126] The "grading result display means" is a user interface and function for visually displaying the calculation results on the teacher's terminal.

[1127] The "product image capturing means" is a function that captures images of products manufactured in a factory using a camera and generates image data of the images.

[1128] The "product quality inspection means" is a function that automatically inspects the quality of a product based on the recognized character and graphic data.

[1129] The "pass / fail judgment means" is a function that judges whether a product passes or fails based on the results of the quality inspection.

[1130] The "rejected product removal instruction means" is a function that sends instructions for the robot arm to remove the rejected product depending on the pass / fail result.

[1131] "Dimension and label accuracy verification means" is a function that verifies the accuracy of product dimensions and labels based on recognized character and graphic data.

[1132] "Management System Transmission Means" refers to the communication means and protocols for transmitting quality inspection results to the management system and updating pass / fail status.

[1133] This invention relates to a system for teachers to efficiently mark tests and a system for automating quality inspections in factories. This system uses a camera to take images of students' answer sheets and products, and analyzes the data using OCR (optical character recognition) and graphic recognition technology to automatically mark and inspect the quality.

[1134] Teacher's correct answer registration

[1135] The teacher launches the application on their own device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is sent from the device to the server. The server uses OCR to extract text from the received image data of the answer sheet and uses a graphic recognition algorithm to detect figures and symbols. This data is then structured and stored in a database.

[1136] Reading and marking student answers

[1137] The teacher switches to "test marking" mode and takes photos of students' answer sheets one by one. The generated image data of the answer sheets is sent from the device to the server. The server receives it and analyzes the answer data using OCR and shape recognition technology. The server compares the analyzed answer data with pre-registered correct answer data to detect errors and partial correct answers. In particular, for mathematical proof questions, the server verifies the accuracy of logical steps and conclusions and calculates partial points.

[1138] Sending and viewing graded results

[1139] The server then sends the calculated marks to the teacher's device, where the marks are displayed in a UI, visually highlighting the score for each question, errors, and partial marks.

[1140] Factory quality inspection

[1141] In the factory, a camera is used to take an image of the product. This image data is sent from the terminal to a server and analyzed using OCR and shape recognition technology. The server verifies the accuracy of the product's label and dimensions and determines whether it passes or fails. Depending on the pass / fail result, a robotic arm is instructed to remove the rejected product. The quality inspection results are sent to the management system, and the pass / fail status is updated.

[1142] Specific examples

[1143] For example, a teacher takes a photo of an exam answer sheet with a camera and sends it to a server. The server analyzes the correct answers using OCR and shape recognition, receives the photo data of the student's answer sheet, and analyzes it in the same way. The server compares the data and generates a score that is sent to the teacher's device. In factories, product labels and dimensions are inspected, and robotic arms remove non-conforming products.

[1144] Example prompt sentence:

[1145] Check if the label in the image is "EXPECTED_LABEL" and check if the product contour length is less than 100

[1146] This will enable more efficient test grading in educational settings and more accurate quality inspections in factories.

[1147] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1148] Step 1: Register the answer sheet

[1149] The user (teacher) starts the application on their device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is generated and sent to the server. The server uses OCR and graphic recognition technology on the received image data to recognize characters and figures, and stores the data in a database as structured data. The input is the image data of the photographed answer sheet, and the output is structured data of the correct answers.

[1150] Step 2: Photograph and submit student answer sheets

[1151] The user (teacher) switches to "test marking" mode and takes photos of students' answer sheets one after another. The generated image data of the answer sheets is sent from the device to the server. The input is the image data of the photographed answer sheets, and the output is the image data of the answer sheets sent to the server.

[1152] Step 3: Analyzing the answer sheet

[1153] The server uses OCR and graphic recognition technology on the image data of the received answer sheet to extract character and graphic data. The input is the image data of the answer sheet, and the output is the analyzed character and graphic data. In this step, the image data is converted into text data and graphic data.

[1154] Step 4: Compare and score answers

[1155] The server compares the analyzed answer data with the correct answer data saved in step 1. It detects errors and partial correct answers and calculates partial points. The input is the analyzed answer data and correct answer data, and the output is the scoring result (including partial points). In this step, the data is compared with the correct answer data and scored.

[1156] Step 5: Submit your results

[1157] The server sends the calculated score to the user's terminal. The input is the score, and the output is the score sent to the user's terminal. In this step, data communication occurs.

[1158] Step 6: View your results

[1159] The user's device displays the received marking results. The displayed information includes the score for each question, details of errors, and partial marks. The input is the received marking results, and the output is the marking results displayed to the user.

[1160] Step 7: Take product images

[1161] The terminal takes images of products manufactured in the factory with a camera. The generated image data is sent to the server. The input is the image data of the photographed product, and the output is the image data sent to the server.

[1162] Step 8: Analyze product quality

[1163] The server uses OCR and shape recognition technology on the received product image data to verify the label and dimensions. The input is the product image data, and the output is the parsed label and dimension data.

[1164] Step 9: Quality assessment and notification of pass / fail

[1165] The server inspects the quality based on the analyzed data and judges whether the product passes or fails. Depending on the pass / fail result, it sends instructions to the robot arm to remove the rejected product. The input is the analyzed label and dimensional data, and the output is the pass / fail judgment result and instructions to the robot arm.

[1166] Step 10: Submit and update test results

[1167] The server sends the quality inspection results to the management system and updates the pass / fail status. The input is the quality inspection result and the output is the updated status of the management system. In this step, data communication with the management system takes place.

[1168] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1169] The present invention is a system designed to improve the efficiency of test grading for school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades tests by having the teacher register correct answers and then photographing students' answers with a camera and sending them to a server. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system enables smoother work. Below, we will explain the form and specific program processing for implementing this system.

[1170] User interface and correct answer registration

[1171] User (Teacher):

[1172] First, the teacher launches the app and logs in. After logging in, the main screen appears and the teacher presses the "Register correct answers" button to activate the camera. The teacher then takes a photo of the test answer sheet and generates image data. The generated image data is sent to the server.

[1173] Device:

[1174] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is later processed for character recognition (OCR) and graphic recognition.

[1175] server:

[1176] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database as correct answer data.

[1177] Reading and grading student responses

[1178] User (Teacher):

[1179] Select the "Test Marking" mode and take photos of each student's answer sheet one by one, which will generate image data for each answer sheet.

[1180] Device:

[1181] The terminal sends the image data of the generated student answer sheet to the server.

[1182] server:

[1183] The server performs character recognition (OCR) and graphic recognition on the received image data, analyzing the student's answers as digital data. This data is then compared with the correct answer data to detect errors and partial correct answers. For proof questions in particular, the accuracy of the logical steps and conclusions is checked. Partial marks are calculated based on errors and partial correct answers, and a final score is generated.

[1184] Introduction of emotion engine and its application

[1185] server:

[1186] An emotion engine is introduced, and the server recognizes the user's (teacher's) emotions. This emotion recognition is performed by analyzing the user's facial expressions and voice using sensors such as cameras and microphones. For example, if an angry or frustrated facial expression is recognized, the system changes the displayed feedback and interface.

[1187] Device:

[1188] The device can then adjust its display based on the results of emotion recognition. For example, if the user is feeling stressed, the system can display a brief summary and possibly offer suggestions for relaxing the user.

[1189] User (Teacher):

[1190] The emotion engine allows teachers to receive appropriate feedback based on their own emotional state, reducing excessive stress during grading and enabling them to work more efficiently.

[1191] Sending and viewing graded results

[1192] server:

[1193] The server sends the generated grading results to the teacher's terminal, and based on the emotion engine, the grading results are displayed to the user in an appropriate format.

[1194] Device:

[1195] The device then displays the received scoring results on the UI. The displayed information includes the score for each question, details of errors, and partial points. Errors and partial points are highlighted based on the recognized emotion.

[1196] User (Teacher):

[1197] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[1198] Specific examples

[1199] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[1200] Next, the teacher takes photos of each student's answer sheet and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where the errors and partial marks are highlighted.

[1201] This system will streamline teachers' marking work, saving them a huge amount of time and effort. The emotion engine will also reduce the mental burden on teachers.

[1202] The processing flow will be explained below.

[1203] Step 1:

[1204] User (teacher): Launch the app and log in by entering your account information on the login screen.

[1205] Step 2:

[1206] On your device: Send your login credentials to the server.

[1207] Step 3:

[1208] Server: Validates the authentication information and returns the authentication result to the terminal. If authentication is successful, it instructs the terminal to display the main screen.

[1209] Step 4:

[1210] User (Teacher): Press the "Register correct answer" button on the main screen to launch the camera.

[1211] Step 5:

[1212] Device: The camera is activated and the teacher takes a picture of the answer sheet to generate image data.

[1213] Step 6:

[1214] Terminal: Sends the image data of the photographed answer sheet to the server.

[1215] Step 7:

[1216] Server: Processes the received image data and performs character recognition (OCR) and shape recognition. A character recognition engine is used to extract text from the image, and a shape recognition algorithm is used to detect figures and symbols.

[1217] Step 8:

[1218] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[1219] Step 9:

[1220] User (Teacher): Select "Test Grading" mode on the main screen.

[1221] Step 10:

[1222] User (teacher): Takes photos of students' answer sheets one after another.

[1223] Step 11:

[1224] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[1225] Step 12:

[1226] Server: Processes the image data of the received answer sheets, performs character recognition (OCR) and graphic recognition, extracts the students' answers and graphics, and analyzes them as digital data.

[1227] Step 13:

[1228] Server: Compares the student's answers with the registered correct answers. It verifies the text and figures, and in the case of proof questions, checks the accuracy of the logical steps and conclusions.

[1229] Step 14:

[1230] Server: Determines errors, partial correct answers, and complete correct answers, highlights errors, and calculates partial points.

[1231] Step 15:

[1232] Server: Generates the scoring results and compiles them into data including the score for each question, the total score, and details of any errors.

[1233] Step 16:

[1234] Server: Sends the grading results to the teacher's device.

[1235] Step 17:

[1236] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[1237] Step 18:

[1238] Server: Using the emotion engine, analyzes the user's (teacher's) facial expressions and voice and recognizes their emotions.

[1239] Step 19:

[1240] Server: Based on the recognized emotions, it adjusts the interface and feedback method, and generates appropriate feedback according to the teacher's emotional state.

[1241] Step 20:

[1242] Device: Adjusts the interface and changes the display content based on the recognized emotion information.

[1243] Step 21:

[1244] User (Teacher): Check the displayed marks and feedback and provide appropriate feedback to the student. Errors are highlighted and partial marks are visually confirmed.

[1245] Example 2

[1246] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1247] Conventional test marking systems have difficulty efficiently and accurately marking complex problems, especially those like mathematical proofs. Furthermore, they are unable to adjust the interface to take into account the teacher's emotional state, which can lead to stress during long marking sessions. Furthermore, they lack sufficient feedback to students, preventing them from providing effective learning support.

[1248] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet to generate image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for recognizing the teacher's emotions, and means for adjusting the display method based on the recognized emotions. This enables efficient and accurate grading of complex questions, reduces stress by adjusting the interface according to the teacher's emotional state, and enables effective feedback to students.

[1249] A "teacher" refers to a professional who provides instruction and education to students at an educational institution.

[1250] "Correct answer" refers to the exact or best answer to a particular question.

[1251] A "means" refers to a method or device used to achieve a particular purpose.

[1252] An "answer sheet" refers to the paper on which students write their answers in an exam or test.

[1253] "Image data" refers to image information acquired by a camera or scanner and stored in digital format.

[1254] A "server" refers to a computer system that stores and processes data.

[1255] "Characters" refer to symbols used to represent human languages.

[1256] "Figure" refers to a geometric shape or symbol.

[1257] "Recognition" refers to the process of extracting specific information from an image or sound.

[1258] "Comparison" refers to the process of comparing two or more pieces of data to identify differences and similarities.

[1259] An "error" is an inaccuracy or mistake compared to the correct answer.

[1260] "Partial credit" refers to a mark awarded as a portion of the total score.

[1261] "Calculation" refers to the process of deriving a specific result from numbers and data.

[1262] "Terminal" refers to a device that communicates with a server and sends and receives data.

[1263] "Emotion" refers to a person's psychological state and facial expression.

[1264] "Recognized emotion" refers to the emotional state determined by the system through analysis of the user's facial expressions and voice.

[1265] "Display method" refers to how information or results are presented in the user interface.

[1266] The present invention relates to a system that helps teachers to efficiently grade tests, and in particular, a system that automates grading of complex problems such as mathematical proofs. This system reduces the burden on teachers and enables quick and accurate feedback to students.

[1267] System configuration

[1268] The system consists of the following main components:

[1269] 1. User (Teacher):

[1270] A device that allows teachers to register correct answers and photograph students' answer sheets. Specifically, it uses a device such as a smartphone or tablet.

[1271] 2. Terminal:

[1272] The device used by the teacher takes pictures of the correct answer sheets and the students' answer sheets and sends the image data to the server. This device is equipped with a camera function and an internet connection function.

[1273] 3. Server:

[1274] The server processes the received image data and recognizes characters and figures. Specifically, it uses an OCR (Optical Character Recognition) engine and figure recognition algorithms. The recognized data is compared with a database of correct answers to detect errors and partial correct answers.

[1275] 4. Emotion Engine:

[1276] The emotion engine recognizes the teacher's emotions and adjusts the way the user interface is displayed. Specifically, the system is equipped with a camera and microphone to analyze the teacher's facial expressions and voice.

[1277] Program processing

[1278] 1. Correct Answer Registration:

[1279] The teacher launches the app on their device and logs in. They press the "Register correct answer" button on the main screen to activate the camera and take a picture of the test answer sheet. The captured image data is sent from the device to the server. The server processes the received image data using an OCR engine and a shape recognition algorithm, and stores the text and shape data in a database.

[1280] 2. Scoring Student Responses:

[1281] The teacher selects the "Test Marking" mode and takes a photo of the student's answer sheet. The device sends the captured image data to the server. The server processes the received data with an OCR engine and shape recognition algorithms, and compares the recognized data with the correct answer data. It detects errors and partial correct answers, calculates partial points, and generates the final marking result.

[1282] 3. Emotion recognition:

[1283] The server uses an emotion engine to analyze the teacher's facial expressions and voice to recognize their emotional state. Based on the recognized emotion, the system adjusts the interface and feedback displayed. For example, if the teacher is feeling stressed, the system adjusts to display only brief summary information.

[1284] 4. View your score:

[1285] The server sends the calculated score to the teacher's device. The device displays the received score on the UI, highlighting errors and partial points. The teacher checks the displayed results and provides feedback to the student.

[1286] Specific examples

[1287] For example, to assist with grading proof questions in junior high school mathematics, a teacher first launches the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the answer sheet and stores it in a database. Next, the teacher takes a photo of the student's answer sheet and sends it to the server. The server compares the student's answer with the correct answer and generates a grading result. The final grading result is sent to the teacher's device, and errors and partial points are highlighted.

[1288] Prompt Sentence Examples

[1289] "Please explain your system for streamlining the grading of proof questions in junior high school mathematics. Specifically, please explain how correct answers are registered, how student answers are graded, how an emotion engine is used, and how the final graded results are transmitted and displayed."

[1290] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1291] Step 1:

[1292] The user (teacher) starts the app and logs in. On the login screen, they enter their username and password and press the login button. The entered authentication information is sent from the device to the server.

[1293] Input: Username and Password

[1294] Data processing: The server checks the authentication information against a database

[1295] Output: Authentication token

[1296] Specific operation: After successful login, an authentication token is sent to the device.

[1297] Step 2:

[1298] The user (teacher) presses the "Register correct answer" button on the main screen. When the teacher takes a photo of the test answer sheet using the camera and presses the "Send" button, the device generates image data and sends it to the server.

[1299] Input: Image data of the answer sheet

[1300] Data processing: Encode image data into BASE64 format

[1301] Output: Encoded image data

[1302] Specific operation: The device sends the image data to the server as an HTTP POST request.

[1303] Step 3:

[1304] The server processes the received image data and extracts text and graphics data using an optical character recognition (OCR) engine and graphics recognition algorithms.

[1305] Input: Encoded image data

[1306] Data processing: Extract text and graphics using OCR engine and graphic recognition algorithms

[1307] Output: Recognized text and graphics data

[1308] Specific behavior: Save the extracted data to the database

[1309] Step 4:

[1310] The user (teacher) selects the "Test Marking" mode, takes pictures of students' answer sheets one after another, and presses the "Send" button. The device then sends the captured image data to the server.

[1311] Input: Image data of student answer sheets

[1312] Data processing: Encode image data into BASE64 format

[1313] Output: Encoded image data

[1314] Specific operation: The device sends the image data to the server as an HTTP POST request.

[1315] Step 5:

[1316] The server performs character recognition (OCR) and graphic recognition on the image data of the received student answer sheets, and compares the extracted data with the correct answer data.

[1317] Input: Encoded image data

[1318] Data processing: Extract text and graphics using an OCR engine and graphic recognition algorithms, and compare them with the correct data

[1319] Output: Information about errors and partial answers

[1320] Specific behavior: Calculate partial points based on errors and partial correct answers

[1321] Step 6:

[1322] The server sends the calculated score to the teacher's terminal.

[1323] Input: Information about errors and partial answers

[1324] Data processing: Calculating the score and converting the results to JSON format

[1325] Output: JSON data of the scoring results

[1326] Specific behavior: Sends JSON data as an HTTP response

[1327] Step 7:

[1328] The device then displays the received scoring results on a user interface. The displayed information includes the score for each question, details of errors, and partial points, and the display method is adjusted based on the results of the emotion engine.

[1329] Input: JSON data of the score results

[1330] Data processing: Parsing JSON data and binding it to UI components

[1331] Output: A visual representation of the scoring results

[1332] Specific operation: Errors and partial points are highlighted

[1333] Step 8:

[1334] The server uses an emotion engine to recognize the teacher's emotional state and adjusts the feedback and interface based on the recognized emotion.

[1335] Input: Teacher's facial expressions and voice data

[1336] Data processing: Analysis and emotion determination using an emotion engine

[1337] Output: Recognized emotion information

[1338] Specific behavior: Dynamically adjusting the UI and displaying concise summary information

[1339] Step 9:

[1340] The user (teacher) checks the displayed grading results and provides feedback to the student. Specifically, the user enters additional comments and feedback through the input form and sends it to the student through the system.

[1341] Input: Marking results and additional feedback

[1342] Data processing: Input feedback is saved in a database

[1343] Output: Send feedback

[1344] Specific behavior: The feedback content is notified to the student.

[1345] (Application example 2)

[1346] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1347] Manufacturing sites are required to simultaneously achieve quality control and reduce the mental burden on workers. In particular, it is important to accurately inspect product quality, provide prompt feedback, and take appropriate measures when workers feel stressed or fatigued. To solve these issues, a system that combines quality control and emotion recognition is required. However, conventional systems have difficulty providing these two functions in an integrated manner, which can lead to a decrease in work efficiency.

[1348] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for registering correct standards for quality control, means for photographing the manufactured product and generating image data, means for transmitting the generated image data to the server and comparing it with the quality standard, means for recognizing the worker's emotions and providing feedback, means for transmitting the recognition results to the worker, and means for displaying the recognition results on the worker's terminal. This enables quality control and reduces the mental burden on the worker.

[1349] The "means for teachers to register correct answers" is a mechanism for teachers to input standard answers set in advance into the system.

[1350] The "means for photographing students' answer sheets and generating image data" refers to a mechanism for capturing students' answers with a photographing device such as a camera and converting the images into digital data.

[1351] The "means for transmitting the generated image data to the server" is a mechanism for sending the captured image to the central processing unit via the network.

[1352] The "means for recognizing characters and figures from received image data" is a mechanism for identifying text and figures contained in the transmitted image data and extracting them as digital data.

[1353] The "means for comparing recognized character and graphic data with correct answers" is a mechanism for comparing extracted character and graphic information with pre-registered correct answer data and performing evaluation.

[1354] The "means for detecting errors based on the comparison results and calculating partial points" is a mechanism for detecting errors from the results of comparing the character and graphic data with the correct data and calculating the partial points accordingly.

[1355] The "means for transmitting the calculation results to the teacher's terminal" is a mechanism for transferring the calculated scores and evaluation results to the terminal used by the educator.

[1356] The "means for displaying the calculation results on the teacher's terminal" is a mechanism for visually presenting the calculated scores and evaluation results on the teacher's terminal.

[1357] "Means for registering correct standards for quality control" is a mechanism for registering standards for quality evaluation in the manufacturing industry in the system.

[1358] The "means for photographing manufactured products and generating image data" is a mechanism for photographing manufactured products with a camera or the like and generating the image as data.

[1359] The "means for transmitting the generated image data to a server and comparing it with quality standards" is a mechanism for transmitting image data of the photographed product to a central processing unit and comparing it with quality evaluation standards.

[1360] The "means for recognizing workers' emotions and providing feedback" is a mechanism for determining the emotions of workers from their facial expressions and voice, and providing appropriate advice and evaluations based on that.

[1361] The "means for transmitting the recognition result to the worker" is a mechanism for transferring the emotion determination result to the terminal used by the worker.

[1362] The "means for displaying the recognition result on the worker's terminal" is a mechanism for visually presenting the emotion determination result on the worker's terminal.

[1363] The present invention provides a system for integrating quality control and worker emotion recognition in educational and manufacturing settings. Specific embodiments for carrying out the invention will be described below.

[1364] System Overview

[1365] This system provides an interface that can be easily operated by educators and workers using terminals, and realizes quality control and emotion recognition functions accurately and quickly.The system consists of the following main components.

[1366] 1. Teachers and quality control personnel:

[1367] The teacher starts the application and logs in. After logging in, the teacher uses the correct answer registration mode to take a photo of the correct answer with the camera and generate image data. This image data is sent to the server and registered as the correct answer data.

[1368] Quality control personnel can register quality standard data in a similar manner.

[1369] 2. Student and Product Management:

[1370] Students' answer sheets and finished products are photographed with a camera, and the image data is sent to a server, where characters and figures are recognized from the received image data and compared with registered correct answer data and quality standard data.

[1371] 3. Server:

[1372] The server uses OCR (optical character recognition) software to extract characters and graphic recognition algorithms to detect shapes and symbols. This data is compared with correct answers and quality benchmark data to detect errors and calculate partial points.

[1373] 4. Emotion Recognition Engine:

[1374] The server is also equipped with an emotion recognition engine that uses sensors such as cameras and microphones to analyze the emotions of users (workers and teachers). For example, if the system detects fatigue or stress in a user, it will provide appropriate feedback.

[1375] 5. Terminal:

[1376] The calculation results and emotion recognition results are displayed in real time on the devices of teachers, quality control personnel, and workers, and the display method can be adjusted according to the user's emotional state.

[1377] Hardware and Software Use

[1378] This system uses the following hardware and software:

[1379] Camera: Used to acquire image data.

[1380] Server: The main component that processes and manages data.

[1381] Terminal (PC, tablet, smartphone, etc.): A device that is operated by the user.

[1382] OCR software (e.g., PyTesseract): Extract character data from images.

[1383] Shape recognition algorithm: An algorithm for recognizing shapes and symbols from images.

[1384] Emotion recognition engine (e.g., Keras-based model): for analyzing emotions from user facial expressions and voice.

[1385] Specific examples

[1386] For example, this system is extremely effective in quality control on automobile production lines. Quality control personnel register product quality standards and take photos of manufactured parts with a camera. The image data is then sent to a server where it is automatically compared with the quality standards. Meanwhile, an emotion recognition engine analyzes the facial expressions and voice of workers, and if fatigue or stress is detected, the system suggests taking a break.

[1387] Prompt Sentence Examples

[1388] An example of an input prompt for the generative AI model associated with this system is shown below.

[1389] "Think of a quality control system on an automobile production line. In the system, an inspection robot takes a picture of the product and compares it with quality standards to automatically inspect and evaluate it. The system also recognizes the emotions of workers and provides appropriate feedback. For workers who appear particularly tired, it suggests that they take a break."

[1390] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1391] Step 1:

[1392] A user (teacher or quality control officer) starts the application and logs in. The input here is the user's identification information (user ID and password), and the output is the authentication result. Once the user is authenticated, the main screen is displayed. During this authentication process, the user information is checked against the database, and login is allowed only if it is correct.

[1393] Step 2:

[1394] The user selects the "Correct Answer Registration" mode or the "Quality Standard Registration" mode and takes a picture of the correct answer sheet or quality standard with the camera. The input here is the captured image, and the output is the generated image data. The device controls the camera and saves the image as digital data. At this stage, the image data is saved locally on the spot.

[1395] Step 3:

[1396] The terminal sends the generated image data to the server. The input here is the image data, and the output is a transmission success message to the server. The terminal uploads the image data to the server via the network and receives a confirmation message if successful.

[1397] Step 4:

[1398] The server recognizes characters and graphics from the image data it receives. The input here is the transmitted image data, and the output is the recognized character and graphic data. Specifically, OCR software (e.g., PyTesseract) and graphic recognition algorithms analyze the image on the server and extract character and graphic information.

[1399] Step 5:

[1400] The server compares the recognized character and graphic data with the correct answer data or quality standard data. The input here is the recognized character and graphic data and the pre-registered correct answer data or quality standard data, and the output is the comparison result. The server collates these data and detects differences.

[1401] Step 6:

[1402] The server detects errors based on the comparison results and calculates partial scores. The input here is the comparison results, and the output is the calculated partial scores and a list of error locations. The server follows a programmed algorithm to determine which parts are correct, which parts are incorrect, and calculate the partial scores.

[1403] Step 7:

[1404] The server sends the calculation results to the user's device. The input here is the calculation result, and the output is a message to the device indicating successful transmission. The server packages the calculation results in JSON format or similar and sends them over the network to the user's device.

[1405] Step 8:

[1406] The terminal displays the calculation results it receives. The input here is the calculation result received, and the output is visual feedback that can be checked by a teacher or quality control officer. The terminal displays the calculation results on the screen, highlighting any errors or partial points to the user.

[1407] Step 9:

[1408] Sensors (camera and microphone) for recognizing the user's emotions capture the user's facial expressions and voice. The input here is the user's facial image and voice data, and the output is the captured sensor data. The device records this data as it goes along and sends it to a server for analysis.

[1409] Step 10:

[1410] The server analyzes the user's emotions using an emotion recognition engine, where the input is the captured sensor data and the output is the recognized emotion classification (e.g., stress, satisfaction, confidence, etc.). The emotion recognition engine (e.g., a Keras-based model) is used to determine the user's emotions in real time.

[1411] Step 11:

[1412] The server provides feedback based on the recognition results and sends the results to the user's device. The input here is the emotion recognition result, and the output is a feedback message to the user. The server creates advice or warnings according to the recognized emotion and sends them to the device.

[1413] Step 12:

[1414] The terminal displays the received recognition results and provides appropriate visual feedback to the user. The input here is the received recognition results, and the output is a feedback display that the user can check. The terminal displays the feedback content in a form that the user can easily understand.

[1415] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1416] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1417] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1418] [Fourth embodiment]

[1419] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1420] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1421] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1422] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1423] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1425] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1426] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1427] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1428] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1429] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1430] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1431] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1432] The present invention is a system that aims to improve the efficiency of test grading by school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades students' answers by having the teacher register the correct answers and then photographing the students' answers with a camera and sending them to a server. Below, we will explain how to implement this system.

[1433] User interface and correct answer registration

[1434] User (Teacher):

[1435] The teacher launches the app and enters their account information on the login screen. After logging in, the main screen appears, displaying a "Register correct answers" button. Pressing this button activates the camera, takes a picture of the test answer sheet, and generates image data. The generated image data is sent to the server.

[1436] Device:

[1437] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is then processed for character recognition (OCR) and graphic recognition.

[1438] server:

[1439] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database.

[1440] Reading and grading student responses

[1441] User (Teacher):

[1442] The system switches to "Test Marking" mode and begins taking photos of students' answer sheets one by one, generating image data for each answer sheet.

[1443] Device:

[1444] The terminal sends the image data of the generated student answer sheet to the server.

[1445] server:

[1446] The server performs character recognition (OCR) and graphic recognition on the received image data and analyzes the student's answers as digital data. The analyzed data is compared with the correct answer data already saved, and errors and partial correct answers are detected. In particular, for proof questions, the accuracy of the logical steps and conclusions is checked. Partial points are calculated based on errors and partial correct answers, and the final score is generated.

[1447] Sending and viewing graded results

[1448] server:

[1449] The server sends the generated grading results to the teacher's terminal.

[1450] Device:

[1451] The device then displays the received score results on the UI, including the score for each question, any errors made, and details of the partial score.

[1452] User (Teacher):

[1453] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[1454] Specific examples

[1455] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[1456] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where errors and partial marks are highlighted. The teacher then provides feedback to the student based on this.

[1457] This system will streamline teachers' marking work, saving them a huge amount of time and effort.

[1458] The processing flow will be explained below.

[1459] Step 1:

[1460] User (teacher): Start the app, enter your account information, and log in. After logging in, the main screen will appear, and you can press the "Register correct answer" button.

[1461] Step 2:

[1462] Device: When the correct answer registration button is pressed, the camera is activated and the teacher takes a picture of the answer sheet. The captured image data is generated.

[1463] Step 3:

[1464] Terminal: Sends the generated image data to the server.

[1465] Step 4:

[1466] Server: Initially processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols.

[1467] Step 5:

[1468] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[1469] Step 6:

[1470] User (teacher): Selects "Test marking" mode and takes photos of students' answer sheets one after another.

[1471] Step 7:

[1472] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[1473] Step 8:

[1474] Server: Performs character recognition (OCR) and graphic recognition on the received image data. Extracts student answers and graphics and analyzes them as digital data.

[1475] Step 9:

[1476] Server: Compares the student's answers with the registered correct answers. For each problem, matches the text and figures. For proof problems, checks the accuracy of the logical steps and conclusions.

[1477] Step 10:

[1478] Server: Determines errors, partial correct answers, and complete correct answers in detail. Highlights errors and calculates partial points.

[1479] Step 11:

[1480] Server: Generates and compiles the scoring results, including the score for each question, the total score, and details of errors.

[1481] Step 12:

[1482] Server: Sends the scoring results to the device.

[1483] Step 13:

[1484] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[1485] Step 14:

[1486] User (Teacher): Checks the marks and provides feedback to students, highlighting errors and visually checking partial marks details.

[1487] Example 1

[1488] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1489] Grading tests in the past required a great deal of time and effort from teachers, especially for complex problems like mathematical proofs. This increased the burden on teachers and made it difficult to grade quickly and accurately. Additionally, calculating partial marks and identifying errors had to be done manually, further reducing work efficiency.

[1490] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1491] In this invention, the server includes a means for the teacher to register correct answer data, a means for photographing the student's answer sheet to generate image data, and a means for transmitting the generated image data to an information processing device. This reduces the burden on the teacher and enables quick and accurate grading. Furthermore, by including a means for recognizing characters and figures from the received image data and comparing the recognized character and figure data with correct answer data, and a means for detecting errors based on the comparison results and calculating partial points, more advanced grading can be achieved. Furthermore, by including a means for highlighting details of errors and partial points, the teacher can provide efficient and visual feedback.

[1492] "Means for teachers to register correct answer data" refers to a device or application that allows teachers to input or record correct answers to test questions.

[1493] "Means for photographing students' answer sheets and generating image data" refers to means for photographing students' answers with a device such as a camera or scanner and generating image data thereof.

[1494] The "means for transmitting the generated image data to the information processing device" refers to a communication method or protocol for transmitting image data from a terminal to a central information processing device (such as a server).

[1495] "Means for recognizing characters and figures from received image data" refers to technology that extracts characters and figures from image data using OCR (optical character recognition) technology or a figure recognition algorithm.

[1496] "Means for comparing recognized character and graphic data with correct answer data" refers to an algorithm for comparing extracted character and graphic data with pre-registered correct answer data and evaluating the recognition results.

[1497] "Means for detecting errors based on the comparison results and calculating partial points" refers to a process for analyzing the comparison results, detecting errors or partial correct answers, and calculating points based on the results.

[1498] "Means for transmitting the calculation results to the teacher's terminal" refers to a method for communicating the calculation results from the server to the terminal used by the teacher.

[1499] "Means for displaying calculation results on the teacher's terminal" refers to an interface for visually displaying calculation results on the terminal screen.

[1500] "Quality check and communication means for image data by the terminal" refers to a method for verifying that the captured image data is accurate and complete, and then communicating the data.

[1501] "Means for analyzing data in an information processing device" refers to a method by which a central computer such as a server analyzes received data and extracts and processes the necessary information.

[1502] "Means for highlighting details of errors and partial marks" refers to visual devices that highlight the parts of the marking results that require careful checking.

[1503] The present invention is a system that allows school teachers to automatically grade complex problems, such as mathematical proofs. This system automatically grades students' answers by having teachers register the correct answer data and taking photos of the students' answers with a camera and sending them to a server. Specific embodiments for implementing this system are described below.

[1504] User interface and correct answer registration

[1505] User (Teacher):

[1506] The teacher launches the application and logs in by entering their account information on the login screen. After logging in, the main screen appears, and a "Register correct answers" button appears. The teacher presses this button to take a photo of the answer sheet with the camera and generate image data.

[1507] Device:

[1508] The device compresses the generated image data of the answer sheet and checks the image quality before sending it to the server. Possible devices include smartphones and tablets. Wi-Fi and mobile data communications are used as communication methods. Libraries such as OpenCV and Tesseract OCR can be used to process the image data.

[1509] server:

[1510] The server analyzes the received image data and uses an OCR engine to extract text from the image. It also uses a shape recognition algorithm to detect figures and mathematical formulas. This extracted data is structured and stored in a database. MySQL or MongoDB can be used as the database management system.

[1511] Read and submit student responses

[1512] User (Teacher):

[1513] The teacher switches to "test marking" mode and takes pictures of the students' answer sheets one by one with the camera, which generates image data of the students' answer sheets.

[1514] Device:

[1515] The device sends the image data of the generated student answer sheet to the server. Before sending, the data is temporarily saved and checked for quality to ensure there are no missing parts. This ensures that the data is uploaded accurately to the server.

[1516] server:

[1517] The server analyzes the image data of the received student answer sheets using OCR and graphic recognition, extracts the student answer data as digital data, and temporarily stores it in memory.

[1518] Data analysis and scoring

[1519] server:

[1520] The server compares the analyzed data with pre-saved correct answers. It uses advanced comparison algorithms to check for partial correct answers. For proof questions in particular, the evaluation focuses on the accuracy of logical steps and conclusions. It identifies errors and partial correct answers and calculates partial points.

[1521] Generate and display grade results

[1522] server:

[1523] The server generates the marking results and sends them to the teacher's device. The results include the score for each question, details of errors, and partial marks. This data is packaged and sent in JSON or XML format.

[1524] Device:

[1525] The device analyzes the received marking results and displays them on a user interface. Errors and partial marks are highlighted. JavaScript and front-end frameworks (e.g., React and Vue.js) can be used as display technologies.

[1526] User (Teacher):

[1527] Teachers can check the marks displayed on the device and provide feedback to students. Errors and partial marks can be visually confirmed, enabling quick and accurate instruction.

[1528] Examples and prompts

[1529] As a concrete example, consider a case where a teacher supports the grading of proof questions in junior high school mathematics. After launching the application and logging in, the teacher selects the correct answer registration mode, photographs the test answer sheet, and sends it to the server. The server analyzes the correct answers and stores them in a database.

[1530] Next, the teacher takes photos of each student's answer sheet one by one and sends them to the server. The server compares the student's answers with the correct answers and detects errors or partial marks. The grading results are then sent to the teacher's device and displayed with details highlighted. The teacher can then provide feedback to the student based on this information.

[1531] Example prompt sentence:

[1532] Please give us an overview of the system that streamlines grading of mathematical proof problems.

[1533] Please provide a detailed procedure for a system that allows teachers to automatically grade mathematics proof problems.

[1534] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1535] Step 1: User authentication and login

[1536] User (Teacher):

[1537] Input: Teacher account information (username, password).

[1538] How it works: The teacher launches the application and enters their account information on the login screen.

[1539] The server compares the received account information with the database and performs authentication.

[1540] Output: A successful authentication message or an error message is displayed on the terminal.

[1541] Step 2: Register the correct answer

[1542] User (Teacher):

[1543] Input: Exam answer sheet.

[1544] How it works: After logging in, press the "Register correct answer" button on the main screen that appears and take a photo of the answer sheet with your camera.

[1545] Device:

[1546] Operation: Generates captured image data and performs quality checks and adjustments on the images.

[1547] server:

[1548] Input: Image data sent from the device.

[1549] Operation: Analyzes received image data using an OCR engine and image recognition algorithms, extracts characters and images, and stores them in a database.

[1550] Output: The correct answer data is saved in the database, and a message indicating that the correct answer has been registered is displayed on the terminal.

[1551] Step 3: Reading student answer sheets

[1552] User (Teacher):

[1553] Input: Student answer sheet.

[1554] Operation: Switches to "test marking" mode and takes photos of students' answer sheets one by one with the camera.

[1555] Device:

[1556] Operation: Generates captured image data, temporarily stores it, and checks its quality.

[1557] server:

[1558] Input: Image data of student answer sheets sent from the device.

[1559] How it works: Analyzes image data using OCR and shape recognition to extract characters and shapes. The extracted data is temporarily stored in memory.

[1560] Output: Parsed data is held in memory.

[1561] Step 4: Data analysis and comparison

[1562] server:

[1563] Input: Parsed student answer data, correct answer data in the database.

[1564] How it works: Compares the student's answer data to the correct answer data. A comparison algorithm is used to detect errors and partial correct answers and calculate partial credit.

[1565] Output: Marking results including partial credit and errors.

[1566] Step 5: Generate and transfer the scores

[1567] server:

[1568] Input: Marking results including partial marks and errors.

[1569] Operation: The grading results are packaged in JSON format or similar for transfer to the teacher's device.

[1570] Device:

[1571] How it works: Parses the received marks and displays them in a user interface, highlighting errors and partial marks.

[1572] Output: The grading results are displayed on the teacher's terminal.

[1573] Step 6: Provide feedback

[1574] User (Teacher):

[1575] Input: Detailed marking results (details of each question's score, mistakes, and partial marks).

[1576] How it works: The teacher checks the grading results displayed on the device and provides feedback to the students as needed. For example, the teacher visually checks the errors and partial points for question number "6430" and enters appropriate instructional comments.

[1577] Output: Teacher feedback is provided to students.

[1578] (Application example 1)

[1579] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1580] Traditional test marking in educational settings and quality inspection in factories are time-consuming and labor-intensive, and there is a need for greater efficiency. Complex tasks such as marking mathematical proof problems and inspecting product quality are particularly prone to human error, which can result in a loss of accuracy. This increases the burden on teachers and factory workers and leads to a decline in labor productivity. There is a need for an efficient and accurate system to solve this problem.

[1581] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1582] In this invention, the server includes means for the teacher to register correct answers, means for photographing students' answer sheets and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answers, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for photographing the structure of a product, inspecting the quality of the product based on the recognized character and figure data, and automatically determining whether the product passes or fails, means for transmitting instructions to a robot arm to remove rejected products based on the pass / fail result, means for verifying the accuracy of product dimensions and labels based on the recognized character and figure data, and means for transmitting the quality inspection results to a management system and updating the pass / fail status. This makes it possible to improve the efficiency and accuracy of test marking in educational settings and quality inspection work in factories.

[1583] The "means for registering correct answers by the teacher" is an interface and function that allows the teacher to register answers in the system.

[1584] The "means for photographing student's answer sheet" is a function for photographing the student's answer sheet with a camera and generating image data of the photograph.

[1585] "Image data transmission means" refers to a communication means and protocol for transmitting the generated image data to the server.

[1586] "Character and graphic recognition means" refers to technology that extracts characters and graphics from received image data using optical character recognition (OCR) and graphic recognition algorithms.

[1587] The "answer comparison means" is a function that compares the recognized character and graphic data with correct answer data registered in advance to detect errors and inconsistencies.

[1588] The "partial score calculation means" is a function that calculates scores for the degree of error and partial correct answers based on the comparison results.

[1589] "Calculation result transmission means" refers to the communication means and protocol for transmitting the calculated grading results to the teacher's terminal.

[1590] The "grading result display means" is a user interface and function for visually displaying the calculation results on the teacher's terminal.

[1591] The "product image capturing means" is a function that captures images of products manufactured in a factory using a camera and generates image data of the images.

[1592] The "product quality inspection means" is a function that automatically inspects the quality of a product based on the recognized character and graphic data.

[1593] The "pass / fail judgment means" is a function that judges whether a product passes or fails based on the results of the quality inspection.

[1594] The "rejected product removal instruction means" is a function that sends instructions for the robot arm to remove the rejected product depending on the pass / fail result.

[1595] "Dimension and label accuracy verification means" is a function that verifies the accuracy of product dimensions and labels based on recognized character and graphic data.

[1596] "Management System Transmission Means" refers to the communication means and protocols for transmitting quality inspection results to the management system and updating pass / fail status.

[1597] This invention relates to a system for teachers to efficiently mark tests and a system for automating quality inspections in factories. This system uses a camera to take images of students' answer sheets and products, and analyzes the data using OCR (optical character recognition) and graphic recognition technology to automatically mark and inspect the quality.

[1598] Teacher's correct answer registration

[1599] The teacher launches the application on their own device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is sent from the device to the server. The server uses OCR to extract text from the received image data of the answer sheet and uses a graphic recognition algorithm to detect figures and symbols. This data is then structured and stored in a database.

[1600] Reading and marking student answers

[1601] The teacher switches to "test marking" mode and takes photos of students' answer sheets one by one. The generated image data of the answer sheets is sent from the device to the server. The server receives it and analyzes the answer data using OCR and shape recognition technology. The server compares the analyzed answer data with pre-registered correct answer data to detect errors and partial correct answers. In particular, for mathematical proof questions, the server verifies the accuracy of logical steps and conclusions and calculates partial points.

[1602] Sending and viewing graded results

[1603] The server then sends the calculated marks to the teacher's device, where the marks are displayed in a UI, visually highlighting the score for each question, errors, and partial marks.

[1604] Factory quality inspection

[1605] In the factory, a camera is used to take an image of the product. This image data is sent from the terminal to a server and analyzed using OCR and shape recognition technology. The server verifies the accuracy of the product's label and dimensions and determines whether it passes or fails. Depending on the pass / fail result, a robotic arm is instructed to remove the rejected product. The quality inspection results are sent to the management system, and the pass / fail status is updated.

[1606] Specific examples

[1607] For example, a teacher takes a photo of an exam answer sheet with a camera and sends it to a server. The server analyzes the correct answers using OCR and shape recognition, receives the photo data of the student's answer sheet, and analyzes it in the same way. The server compares the data and generates a score that is sent to the teacher's device. In factories, product labels and dimensions are inspected, and robotic arms remove non-conforming products.

[1608] Example prompt sentence:

[1609] Check if the label in the image is "EXPECTED_LABEL" and check if the product contour length is less than 100

[1610] This will enable more efficient test grading in educational settings and more accurate quality inspections in factories.

[1611] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1612] Step 1: Register the answer sheet

[1613] The user (teacher) starts the application on their device and logs in. After logging in, they select the "Correct Answer Registration" mode and take a photo of the test answer sheet with their camera. This captured image data is generated and sent to the server. The server uses OCR and graphic recognition technology on the received image data to recognize characters and figures, and stores the data in a database as structured data. The input is the image data of the photographed answer sheet, and the output is structured data of the correct answers.

[1614] Step 2: Photograph and submit student answer sheets

[1615] The user (teacher) switches to "test marking" mode and takes photos of students' answer sheets one after another. The generated image data of the answer sheets is sent from the device to the server. The input is the image data of the photographed answer sheets, and the output is the image data of the answer sheets sent to the server.

[1616] Step 3: Analyzing the answer sheet

[1617] The server uses OCR and graphic recognition technology on the image data of the received answer sheet to extract character and graphic data. The input is the image data of the answer sheet, and the output is the analyzed character and graphic data. In this step, the image data is converted into text data and graphic data.

[1618] Step 4: Compare and score answers

[1619] The server compares the analyzed answer data with the correct answer data saved in step 1. It detects errors and partial correct answers and calculates partial points. The input is the analyzed answer data and correct answer data, and the output is the scoring result (including partial points). In this step, the data is compared with the correct answer data and scored.

[1620] Step 5: Submit your results

[1621] The server sends the calculated score to the user's terminal. The input is the score, and the output is the score sent to the user's terminal. In this step, data communication occurs.

[1622] Step 6: View your results

[1623] The user's device displays the received marking results. The displayed information includes the score for each question, details of errors, and partial marks. The input is the received marking results, and the output is the marking results displayed to the user.

[1624] Step 7: Take product images

[1625] The terminal takes images of products manufactured in the factory with a camera. The generated image data is sent to the server. The input is the image data of the photographed product, and the output is the image data sent to the server.

[1626] Step 8: Analyze product quality

[1627] The server uses OCR and shape recognition technology on the received product image data to verify the label and dimensions. The input is the product image data, and the output is the parsed label and dimension data.

[1628] Step 9: Quality assessment and notification of pass / fail

[1629] The server inspects the quality based on the analyzed data and judges whether the product passes or fails. Depending on the pass / fail result, it sends instructions to the robot arm to remove the rejected product. The input is the analyzed label and dimensional data, and the output is the pass / fail judgment result and instructions to the robot arm.

[1630] Step 10: Submit and update test results

[1631] The server sends the quality inspection results to the management system and updates the pass / fail status. The input is the quality inspection result and the output is the updated status of the management system. In this step, data communication with the management system takes place.

[1632] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1633] The present invention is a system designed to improve the efficiency of test grading for school teachers, and is particularly specialized for grading complex problems such as mathematical proofs. This system automatically grades tests by having the teacher register correct answers and then photographing students' answers with a camera and sending them to a server. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system enables smoother work. Below, we will explain the form and specific program processing for implementing this system.

[1634] User interface and correct answer registration

[1635] User (Teacher):

[1636] First, the teacher launches the app and logs in. After logging in, the main screen appears and the teacher presses the "Register correct answers" button to activate the camera. The teacher then takes a photo of the test answer sheet and generates image data. The generated image data is sent to the server.

[1637] Device:

[1638] The device generates image data of the answer sheet photographed by the teacher and sends it to the server, where it is later processed for character recognition (OCR) and graphic recognition.

[1639] server:

[1640] The server processes the received image data and performs character recognition (OCR) and graphic recognition. The character recognition engine extracts text from the image, and the graphic recognition algorithm detects figures and symbols. This extracted data is structured and stored in a database as correct answer data.

[1641] Reading and grading student responses

[1642] User (Teacher):

[1643] Select the "Test Marking" mode and take photos of each student's answer sheet one by one, which will generate image data for each answer sheet.

[1644] Device:

[1645] The terminal sends the image data of the generated student answer sheet to the server.

[1646] server:

[1647] The server performs character recognition (OCR) and graphic recognition on the received image data, analyzing the student's answers as digital data. This data is then compared with the correct answer data to detect errors and partial correct answers. For proof questions in particular, the accuracy of the logical steps and conclusions is checked. Partial marks are calculated based on errors and partial correct answers, and a final score is generated.

[1648] Introduction of emotion engine and its application

[1649] server:

[1650] An emotion engine is introduced, and the server recognizes the user's (teacher's) emotions. This emotion recognition is performed by analyzing the user's facial expressions and voice using sensors such as cameras and microphones. For example, if an angry or frustrated facial expression is recognized, the system changes the displayed feedback and interface.

[1651] Device:

[1652] The device can then adjust its display based on the results of emotion recognition. For example, if the user is feeling stressed, the system can display a brief summary and possibly offer suggestions for relaxing the user.

[1653] User (Teacher):

[1654] The emotion engine allows teachers to receive appropriate feedback based on their own emotional state, reducing excessive stress during grading and enabling them to work more efficiently.

[1655] Sending and viewing graded results

[1656] server:

[1657] The server sends the generated grading results to the teacher's terminal, and based on the emotion engine, the grading results are displayed to the user in an appropriate format.

[1658] Device:

[1659] The device then displays the received scoring results on the UI. The displayed information includes the score for each question, details of errors, and partial points. Errors and partial points are highlighted based on the recognized emotion.

[1660] User (Teacher):

[1661] Teachers can view the displayed marks and provide feedback to students, with errors highlighted and partial marks visually displayed.

[1662] Specific examples

[1663] As an example, consider the case of supporting the grading of proof questions in mathematics at a junior high school. First, a teacher starts the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the correct answers and stores them in a database.

[1664] Next, the teacher takes photos of each student's answer sheet and sends them to the server. The server compares the student's answers with the correct answers, detects errors, and calculates partial marks. The grading results are then sent to the teacher's device, where the errors and partial marks are highlighted.

[1665] This system will streamline teachers' marking work, saving them a huge amount of time and effort. The emotion engine will also reduce the mental burden on teachers.

[1666] The processing flow will be explained below.

[1667] Step 1:

[1668] User (teacher): Launch the app and log in by entering your account information on the login screen.

[1669] Step 2:

[1670] On your device: Send your login credentials to the server.

[1671] Step 3:

[1672] Server: Validates the authentication information and returns the authentication result to the terminal. If authentication is successful, it instructs the terminal to display the main screen.

[1673] Step 4:

[1674] User (Teacher): Press the "Register correct answer" button on the main screen to launch the camera.

[1675] Step 5:

[1676] Device: The camera is activated and the teacher takes a picture of the answer sheet to generate image data.

[1677] Step 6:

[1678] Terminal: Sends the image data of the photographed answer sheet to the server.

[1679] Step 7:

[1680] Server: Processes the received image data and performs character recognition (OCR) and shape recognition. A character recognition engine is used to extract text from the image, and a shape recognition algorithm is used to detect figures and symbols.

[1681] Step 8:

[1682] Server: The extracted text and graphic data are structured and stored in a database as correct answer data.

[1683] Step 9:

[1684] User (Teacher): Select "Test Grading" mode on the main screen.

[1685] Step 10:

[1686] User (teacher): Takes photos of students' answer sheets one after another.

[1687] Step 11:

[1688] Terminal: Generates image data of the photographed answer sheet and sends it to the server.

[1689] Step 12:

[1690] Server: Processes the image data of the received answer sheets, performs character recognition (OCR) and graphic recognition, extracts the students' answers and graphics, and analyzes them as digital data.

[1691] Step 13:

[1692] Server: Compares the student's answers with the registered correct answers. It verifies the text and figures, and in the case of proof questions, checks the accuracy of the logical steps and conclusions.

[1693] Step 14:

[1694] Server: Determines errors, partial correct answers, and complete correct answers, highlights errors, and calculates partial points.

[1695] Step 15:

[1696] Server: Generates the scoring results and compiles them into data including the score for each question, the total score, and details of any errors.

[1697] Step 16:

[1698] Server: Sends the grading results to the teacher's device.

[1699] Step 17:

[1700] Device: Display the received scoring results in the UI, color-coding correct and incorrect answers and showing details of partial marks.

[1701] Step 18:

[1702] Server: Using the emotion engine, analyzes the user's (teacher's) facial expressions and voice and recognizes their emotions.

[1703] Step 19:

[1704] Server: Based on the recognized emotions, it adjusts the interface and feedback method, and generates appropriate feedback according to the teacher's emotional state.

[1705] Step 20:

[1706] Device: Adjusts the interface and changes the display content based on the recognized emotion information.

[1707] Step 21:

[1708] User (Teacher): Check the displayed marks and feedback and provide appropriate feedback to the student. Errors are highlighted and partial marks are visually confirmed.

[1709] Example 2

[1710] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1711] Conventional test marking systems have difficulty efficiently and accurately marking complex problems, especially those like mathematical proofs. Furthermore, they are unable to adjust the interface to take into account the teacher's emotional state, which can lead to stress during long marking sessions. Furthermore, they lack sufficient feedback to students, preventing them from providing effective learning support.

[1712] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet to generate image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for recognizing the teacher's emotions, and means for adjusting the display method based on the recognized emotions. This enables efficient and accurate grading of complex questions, reduces stress by adjusting the interface according to the teacher's emotional state, and enables effective feedback to students.

[1713] A "teacher" refers to a professional who provides instruction and education to students at an educational institution.

[1714] "Correct answer" refers to the exact or best answer to a particular question.

[1715] A "means" refers to a method or device used to achieve a particular purpose.

[1716] An "answer sheet" refers to the paper on which students write their answers in an exam or test.

[1717] "Image data" refers to image information acquired by a camera or scanner and stored in digital format.

[1718] A "server" refers to a computer system that stores and processes data.

[1719] "Characters" refer to symbols used to represent human languages.

[1720] "Figure" refers to a geometric shape or symbol.

[1721] "Recognition" refers to the process of extracting specific information from an image or sound.

[1722] "Comparison" refers to the process of comparing two or more pieces of data to identify differences and similarities.

[1723] An "error" is an inaccuracy or mistake compared to the correct answer.

[1724] "Partial credit" refers to a mark awarded as a portion of the total score.

[1725] "Calculation" refers to the process of deriving a specific result from numbers and data.

[1726] "Terminal" refers to a device that communicates with a server and sends and receives data.

[1727] "Emotion" refers to a person's psychological state and facial expression.

[1728] "Recognized emotion" refers to the emotional state determined by the system through analysis of the user's facial expressions and voice.

[1729] "Display method" refers to how information or results are presented in the user interface.

[1730] The present invention relates to a system that helps teachers to efficiently grade tests, and in particular, a system that automates grading of complex problems such as mathematical proofs. This system reduces the burden on teachers and enables quick and accurate feedback to students.

[1731] System configuration

[1732] The system consists of the following main components:

[1733] 1. User (Teacher):

[1734] A device that allows teachers to register correct answers and photograph students' answer sheets. Specifically, it uses a device such as a smartphone or tablet.

[1735] 2. Terminal:

[1736] The device used by the teacher takes pictures of the correct answer sheets and the students' answer sheets and sends the image data to the server. This device is equipped with a camera function and an internet connection function.

[1737] 3. Server:

[1738] The server processes the received image data and recognizes characters and figures. Specifically, it uses an OCR (Optical Character Recognition) engine and figure recognition algorithms. The recognized data is compared with a database of correct answers to detect errors and partial correct answers.

[1739] 4. Emotion Engine:

[1740] The emotion engine recognizes the teacher's emotions and adjusts the way the user interface is displayed. Specifically, the system is equipped with a camera and microphone to analyze the teacher's facial expressions and voice.

[1741] Program processing

[1742] 1. Correct Answer Registration:

[1743] The teacher launches the app on their device and logs in. They press the "Register correct answer" button on the main screen to activate the camera and take a picture of the test answer sheet. The captured image data is sent from the device to the server. The server processes the received image data using an OCR engine and a shape recognition algorithm, and stores the text and shape data in a database.

[1744] 2. Scoring Student Responses:

[1745] The teacher selects the "Test Marking" mode and takes a photo of the student's answer sheet. The device sends the captured image data to the server. The server processes the received data with an OCR engine and shape recognition algorithms, and compares the recognized data with the correct answer data. It detects errors and partial correct answers, calculates partial points, and generates the final marking result.

[1746] 3. Emotion recognition:

[1747] The server uses an emotion engine to analyze the teacher's facial expressions and voice to recognize their emotional state. Based on the recognized emotion, the system adjusts the interface and feedback displayed. For example, if the teacher is feeling stressed, the system adjusts to display only brief summary information.

[1748] 4. View your score:

[1749] The server sends the calculated score to the teacher's device. The device displays the received score on the UI, highlighting errors and partial points. The teacher checks the displayed results and provides feedback to the student.

[1750] Specific examples

[1751] For example, to assist with grading proof questions in junior high school mathematics, a teacher first launches the app and logs in. They select the correct answer registration mode, take a photo of the test answer sheet, and send it to the server. The server analyzes the answer sheet and stores it in a database. Next, the teacher takes a photo of the student's answer sheet and sends it to the server. The server compares the student's answer with the correct answer and generates a grading result. The final grading result is sent to the teacher's device, and errors and partial points are highlighted.

[1752] Prompt Sentence Examples

[1753] "Please explain your system for streamlining the grading of proof questions in junior high school mathematics. Specifically, please explain how correct answers are registered, how student answers are graded, how an emotion engine is used, and how the final graded results are transmitted and displayed."

[1754] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1755] Step 1:

[1756] The user (teacher) starts the app and logs in. On the login screen, they enter their username and password and press the login button. The entered authentication information is sent from the device to the server.

[1757] Input: Username and Password

[1758] Data processing: The server checks the authentication information against a database

[1759] Output: Authentication token

[1760] Specific operation: After successful login, an authentication token is sent to the device.

[1761] Step 2:

[1762] The user (teacher) presses the "Register correct answer" button on the main screen. When the teacher takes a photo of the test answer sheet using the camera and presses the "Send" button, the device generates image data and sends it to the server.

[1763] Input: Image data of the answer sheet

[1764] Data processing: Encode image data into BASE64 format

[1765] Output: Encoded image data

[1766] Specific operation: The device sends the image data to the server as an HTTP POST request.

[1767] Step 3:

[1768] The server processes the received image data and extracts text and graphics data using an optical character recognition (OCR) engine and graphics recognition algorithms.

[1769] Input: Encoded image data

[1770] Data processing: Extract text and graphics using OCR engine and graphic recognition algorithms

[1771] Output: Recognized text and graphics data

[1772] Specific behavior: Save the extracted data to the database

[1773] Step 4:

[1774] The user (teacher) selects the "Test Marking" mode, takes pictures of students' answer sheets one after another, and presses the "Send" button. The device then sends the captured image data to the server.

[1775] Input: Image data of student answer sheets

[1776] Data processing: Encode image data into BASE64 format

[1777] Output: Encoded image data

[1778] Specific operation: The device sends the image data to the server as an HTTP POST request.

[1779] Step 5:

[1780] The server performs character recognition (OCR) and graphic recognition on the image data of the received student answer sheets, and compares the extracted data with the correct answer data.

[1781] Input: Encoded image data

[1782] Data processing: Extract text and graphics using an OCR engine and graphic recognition algorithms, and compare them with the correct data

[1783] Output: Information about errors and partial answers

[1784] Specific behavior: Calculate partial points based on errors and partial correct answers

[1785] Step 6:

[1786] The server sends the calculated score to the teacher's terminal.

[1787] Input: Information about errors and partial answers

[1788] Data processing: Calculating the score and converting the results to JSON format

[1789] Output: JSON data of the scoring results

[1790] Specific behavior: Sends JSON data as an HTTP response

[1791] Step 7:

[1792] The device then displays the received scoring results on a user interface. The displayed information includes the score for each question, details of errors, and partial points, and the display method is adjusted based on the results of the emotion engine.

[1793] Input: JSON data of the score results

[1794] Data processing: Parsing JSON data and binding it to UI components

[1795] Output: A visual representation of the scoring results

[1796] Specific operation: Errors and partial points are highlighted

[1797] Step 8:

[1798] The server uses an emotion engine to recognize the teacher's emotional state and adjusts the feedback and interface based on the recognized emotion.

[1799] Input: Teacher's facial expressions and voice data

[1800] Data processing: Analysis and emotion determination using an emotion engine

[1801] Output: Recognized emotion information

[1802] Specific behavior: Dynamically adjusting the UI and displaying concise summary information

[1803] Step 9:

[1804] The user (teacher) checks the displayed grading results and provides feedback to the student. Specifically, the user enters additional comments and feedback through the input form and sends it to the student through the system.

[1805] Input: Marking results and additional feedback

[1806] Data processing: Input feedback is saved in a database

[1807] Output: Send feedback

[1808] Specific behavior: The feedback content is notified to the student.

[1809] (Application example 2)

[1810] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1811] Manufacturing sites are required to simultaneously achieve quality control and reduce the mental burden on workers. In particular, it is important to accurately inspect product quality, provide prompt feedback, and take appropriate measures when workers feel stressed or fatigued. To solve these issues, a system that combines quality control and emotion recognition is required. However, conventional systems have difficulty providing these two functions in an integrated manner, which can lead to a decrease in work efficiency.

[1812] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for the teacher to register the correct answer, means for photographing the student's answer sheet and generating image data, means for transmitting the generated image data to the server, means for recognizing characters and figures from the received image data, means for comparing the recognized character and figure data with the correct answer, means for detecting errors and calculating partial points based on the comparison results, means for transmitting the calculation results to the teacher's terminal, means for displaying the calculation results on the teacher's terminal, means for registering correct standards for quality control, means for photographing the manufactured product and generating image data, means for transmitting the generated image data to the server and comparing it with the quality standard, means for recognizing the worker's emotions and providing feedback, means for transmitting the recognition results to the worker, and means for displaying the recognition results on the worker's terminal. This enables quality control and reduces the mental burden on the worker.

[1813] The "means for teachers to register correct answers" is a mechanism for teachers to input standard answers set in advance into the system.

[1814] The "means for photographing students' answer sheets and generating image data" refers to a mechanism for capturing students' answers with a photographing device such as a camera and converting the images into digital data.

[1815] The "means for transmitting the generated image data to the server" is a mechanism for sending the captured image to the central processing unit via the network.

[1816] The "means for recognizing characters and figures from received image data" is a mechanism for identifying text and figures contained in the transmitted image data and extracting them as digital data.

[1817] The "means for comparing recognized character and graphic data with correct answers" is a mechanism for comparing extracted character and graphic information with pre-registered correct answer data and performing evaluation.

[1818] The "means for detecting errors based on the comparison results and calculating partial points" is a mechanism for detecting errors from the results of comparing the character and graphic data with the correct data and calculating the partial points accordingly.

[1819] The "means for transmitting the calculation results to the teacher's terminal" is a mechanism for transferring the calculated scores and evaluation results to the terminal used by the educator.

[1820] The "means for displaying the calculation results on the teacher's terminal" is a mechanism for visually presenting the calculated scores and evaluation results on the teacher's terminal.

[1821] "Means for registering correct standards for quality control" is a mechanism for registering standards for quality evaluation in the manufacturing industry in the system.

[1822] The "means for photographing manufactured products and generating image data" is a mechanism for photographing manufactured products with a camera or the like and generating the image as data.

[1823] The "means for transmitting the generated image data to a server and comparing it with quality standards" is a mechanism for transmitting image data of the photographed product to a central processing unit and comparing it with quality evaluation standards.

[1824] The "means for recognizing workers' emotions and providing feedback" is a mechanism for determining the emotions of workers from their facial expressions and voice, and providing appropriate advice and evaluations based on that.

[1825] The "means for transmitting the recognition result to the worker" is a mechanism for transferring the emotion determination result to the terminal used by the worker.

[1826] The "means for displaying the recognition result on the worker's terminal" is a mechanism for visually presenting the emotion determination result on the worker's terminal.

[1827] The present invention provides a system for integrating quality control and worker emotion recognition in educational and manufacturing settings. Specific embodiments for carrying out the invention will be described below.

[1828] System Overview

[1829] This system provides an interface that can be easily operated by educators and workers using terminals, and realizes quality control and emotion recognition functions accurately and quickly.The system consists of the following main components.

[1830] 1. Teachers and quality control personnel:

[1831] The teacher starts the application and logs in. After logging in, the teacher uses the correct answer registration mode to take a photo of the correct answer with the camera and generate image data. This image data is sent to the server and registered as the correct answer data.

[1832] Quality control personnel can register quality standard data in a similar manner.

[1833] 2. Student and Product Management:

[1834] Students' answer sheets and finished products are photographed with a camera, and the image data is sent to a server, where characters and figures are recognized from the received image data and compared with registered correct answer data and quality standard data.

[1835] 3. Server:

[1836] The server uses OCR (optical character recognition) software to extract characters and graphic recognition algorithms to detect shapes and symbols. This data is compared with correct answers and quality benchmark data to detect errors and calculate partial points.

[1837] 4. Emotion Recognition Engine:

[1838] The server is also equipped with an emotion recognition engine that uses sensors such as cameras and microphones to analyze the emotions of users (workers and teachers). For example, if the system detects fatigue or stress in a user, it will provide appropriate feedback.

[1839] 5. Terminal:

[1840] The calculation results and emotion recognition results are displayed in real time on the devices of teachers, quality control personnel, and workers, and the display method can be adjusted according to the user's emotional state.

[1841] Hardware and Software Use

[1842] This system uses the following hardware and software:

[1843] Camera: Used to acquire image data.

[1844] Server: The main component that processes and manages data.

[1845] Terminal (PC, tablet, smartphone, etc.): A device that is operated by the user.

[1846] OCR software (e.g., PyTesseract): Extract character data from images.

[1847] Shape recognition algorithm: An algorithm for recognizing shapes and symbols from images.

[1848] Emotion recognition engine (e.g., Keras-based model): for analyzing emotions from user facial expressions and voice.

[1849] Specific examples

[1850] For example, this system is extremely effective in quality control on automobile production lines. Quality control personnel register product quality standards and take photos of manufactured parts with a camera. The image data is then sent to a server where it is automatically compared with the quality standards. Meanwhile, an emotion recognition engine analyzes the facial expressions and voice of workers, and if fatigue or stress is detected, the system suggests taking a break.

[1851] Prompt Sentence Examples

[1852] An example of an input prompt for the generative AI model associated with this system is shown below.

[1853] "Think of a quality control system on an automobile production line. In the system, an inspection robot takes a picture of the product and compares it with quality standards to automatically inspect and evaluate it. The system also recognizes the emotions of workers and provides appropriate feedback. For workers who appear particularly tired, it suggests that they take a break."

[1854] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1855] Step 1:

[1856] A user (teacher or quality control officer) starts the application and logs in. The input here is the user's identification information (user ID and password), and the output is the authentication result. Once the user is authenticated, the main screen is displayed. During this authentication process, the user information is checked against the database, and login is allowed only if it is correct.

[1857] Step 2:

[1858] The user selects the "Correct Answer Registration" mode or the "Quality Standard Registration" mode and takes a picture of the correct answer sheet or quality standard with the camera. The input here is the captured image, and the output is the generated image data. The device controls the camera and saves the image as digital data. At this stage, the image data is saved locally on the spot.

[1859] Step 3:

[1860] The terminal sends the generated image data to the server. The input here is the image data, and the output is a transmission success message to the server. The terminal uploads the image data to the server via the network and receives a confirmation message if successful.

[1861] Step 4:

[1862] The server recognizes characters and graphics from the image data it receives. The input here is the transmitted image data, and the output is the recognized character and graphic data. Specifically, OCR software (e.g., PyTesseract) and graphic recognition algorithms analyze the image on the server and extract character and graphic information.

[1863] Step 5:

[1864] The server compares the recognized character and graphic data with the correct answer data or quality standard data. The input here is the recognized character and graphic data and the pre-registered correct answer data or quality standard data, and the output is the comparison result. The server collates these data and detects differences.

[1865] Step 6:

[1866] The server detects errors based on the comparison results and calculates partial scores. The input here is the comparison results, and the output is the calculated partial scores and a list of error locations. The server follows a programmed algorithm to determine which parts are correct, which parts are incorrect, and calculate the partial scores.

[1867] Step 7:

[1868] The server sends the calculation results to the user's device. The input here is the calculation result, and the output is a message to the device indicating successful transmission. The server packages the calculation results in JSON format or similar and sends them over the network to the user's device.

[1869] Step 8:

[1870] The terminal displays the calculation results it receives. The input here is the calculation result received, and the output is visual feedback that can be checked by a teacher or quality control officer. The terminal displays the calculation results on the screen, highlighting any errors or partial points to the user.

[1871] Step 9:

[1872] Sensors (camera and microphone) for recognizing the user's emotions capture the user's facial expressions and voice. The input here is the user's facial image and voice data, and the output is the captured sensor data. The device records this data as it goes along and sends it to a server for analysis.

[1873] Step 10:

[1874] The server analyzes the user's emotions using an emotion recognition engine, where the input is the captured sensor data and the output is the recognized emotion classification (e.g., stress, satisfaction, confidence, etc.). The emotion recognition engine (e.g., a Keras-based model) is used to determine the user's emotions in real time.

[1875] Step 11:

[1876] The server provides feedback based on the recognition results and sends the results to the user's device. The input here is the emotion recognition result, and the output is a feedback message to the user. The server creates advice or warnings according to the recognized emotion and sends them to the device.

[1877] Step 12:

[1878] The terminal displays the received recognition results and provides appropriate visual feedback to the user. The input here is the received recognition results, and the output is a feedback display that the user can check. The terminal displays the feedback content in a form that the user can easily understand.

[1879] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1880] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1881] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1882] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1883] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1884] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1885] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1886] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1887] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1888] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1889] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1890] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and ...

Claims

1. a means for the teacher to register the correct answer; A means for taking a photograph of the student's answer sheet to generate image data; means for transmitting the generated image data to a server; means for recognizing characters and figures from the received image data; means for comparing the recognized character and graphic data with a correct answer; means for detecting errors and calculating partial points based on the comparison results; A means for transmitting the calculation results to the teacher's terminal; The system includes a means for displaying the calculation results on the teacher's terminal.

2. 2. The system of claim 1, further comprising means for logically verifying an answer to the proof problem based on the recognized character and graphic data.

3. The system according to claim 1, further comprising means for highlighting errors in the graded results displayed on the teacher's terminal.

Images