System

Abstract

To provide a system for providing an optimum order-made educational experience to each student, and for performing effective guidance while reducing the labor of a teacher.SOLUTION: A means for reading an image of a submission and converting the image into text data, a means for analyzing the text data and recognizing a learning tendency and a personality trait, a means for generating questions of daily homework, a small test, and a regular examination based on the recognized learning tendency and personality trait, a means for distributing the generated questions to a student, a means for evaluating an answer of the student and calculating an output score and a personality score, a means for proposing an optimal course and job title to the student based on the output score and the personality score, and a means for displaying a personality determination result to a teacher based on the score of the student, means for instructing a method of instruction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the current education system, it is extremely difficult for teachers to provide optimal educational content and career guidance for each student. While the burden on teachers increases, students have few opportunities to receive education suited to their learning tendencies and personalities. Furthermore, there are insufficient systems for comprehensively evaluating students' abilities and characteristics when selecting their career paths or employment destinations. A new system that can solve these problems and provide an efficient and effective educational experience is needed. [Means for solving the problem]

[0005] The present invention provides a system that solves the above-mentioned problems by the following means. First, images of submitted materials are scanned and converted into text data using OCR technology. Next, the converted text data is analyzed to recognize learning tendencies and personality traits. Based on this, daily homework, quizzes, and regular exam questions are automatically generated using a generative AI model and distributed to students. Students' answers are evaluated, and output scores and personality scores are calculated. Based on the results, the system suggests the most suitable career path and employment for each student. Furthermore, personality assessment results based on the student's score are displayed to teachers, and guidance on teaching methods is provided. This realizes a system that provides a customized educational experience that is optimal for each student, reducing the burden on teachers while enabling effective instruction.

[0006] "Images of submitted work" refers to image data of assignments and answers submitted by students, photographed or scanned using a digital camera or scanner.

[0007] "OCR technology" is an abbreviation for Optical Character Recognition, and is a technology that converts character information in an image into text data.

[0008] "Text data" is digital data of character information extracted from an image using OCR technology.

[0009] "Learning tendencies" are characteristics that indicate what learning style or area a student is strong or weak in, as revealed by analyzing the content of the student's submissions and answers.

[0010] "Personality traits" are traits that indicate the student's personality and behavioral tendencies, which are inferred based on the student's submitted work and answers.

[0011] A "generative AI model" is an algorithm or machine learning model that uses artificial intelligence to automatically generate questions and tasks.

[0012] "Question generation" refers to automatically creating optimal homework, quiz, and exam questions and assignments based on a student's learning habits and personality traits.

[0013] An "output score" is a numerical assessment of a student's academic performance based on their answers and submissions.

[0014] A "personality score" is a numerical value of a personality trait assessed based on a student's answers and submissions.

[0015] "Career and employment suggestions" means recommending the most suitable educational and employment options for students based on their output and personality scores.

[0016] "Personality assessment results" are the results of an analysis of a student's personality and behavioral tendencies calculated based on the student's scores.

[0017] "Guidance on teaching methods" means providing advice to teachers to help them select the most appropriate teaching methods for students based on the personality assessment results and scores. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system for improving the efficiency of individualized instruction in educational systems. The system reads images of students' submitted work, analyzes their learning tendencies and personality traits from the images, automatically generates personalized homework, quizzes, and regular exam questions based on the results, and evaluates the students' answers to calculate output scores and personality scores. These scores are used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods.

[0040] Image loading and analysis of submissions

[0041] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[0042] Homework, quiz, and exam question generation

[0043] The server automatically generates questions for daily homework, quizzes, and regular exams using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. This enables each student to receive optimal learning content. The device delivers the generated questions to the student, who then answers them.

[0044] Answer analysis and scoring

[0045] Students' answers are sent to the server via their devices. The server evaluates the answers using an automatic scoring system and generates grade data. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This allows each student's learning outcomes and personality traits to be expressed numerically.

[0046] Career and employment suggestions

[0047] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[0048] Teacher feedback and instructional guide

[0049] The server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on teaching methods based on the results. This information is provided to the teacher via their device, and the teacher can use it to determine the best teaching policy for the student.

[0050] Example of a sample

[0051] For example, when a student submits a mathematics assignment, the device takes a photo of the assignment and sends it to the server. The server then uses OCR technology to convert it into text data and analyzes the text, identifying that the student has strengths in mathematics. Based on this information, the server uses a generative AI model to create a challenging mathematics assignment for the student and distributes it to the device. The student then works on the assignment and uploads their answers back to the server via their device. The answers are graded, and the student's output score and personality score are updated. As a result, the student is recommended a career path specializing in mathematics, and the teacher is provided with appropriate teaching methods (e.g., in-depth explanations or additional challenging assignments) via the device.

[0052] As a result, the present invention makes it possible to build a system that provides an educational experience that is optimized for each individual student and realizes effective instruction while reducing the burden on teachers.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] Students take photos of their own work using their smartphones or tablets, which then save the image data and upload it to the server.

[0056] Step 2:

[0057] The server receives the uploaded image data and uses OCR technology to extract text data from the image data.

[0058] Step 3:

[0059] The server then analyzes the extracted text data using natural language processing (NLP) algorithms, and the results of the analysis identify the student's learning tendencies and personality traits.

[0060] Step 4:

[0061] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on identified learning tendencies and personality traits.

[0062] Step 5:

[0063] The server stores the generated questions in a database, and then distributes the stored questions to the device.

[0064] Step 6:

[0065] The terminal displays the delivered questions to the student. The user (student) works on the questions and sends the answers to the server via the terminal.

[0066] Step 7:

[0067] The server evaluates the received answer data with an automatic scoring system, and generates score data as the evaluation result.

[0068] Step 8:

[0069] The server calculates an output score and a personality score based on the generated performance data.

[0070] Step 9:

[0071] The server analyzes the calculated output score and personality score and generates information to suggest the most suitable career path and employment for the student.

[0072] Step 10:

[0073] The server distributes career and employment suggestion information to the terminal, which then displays the suggestion content to students and teachers.

[0074] Step 11:

[0075] The server displays the personality assessment results based on the student's scores to the teacher, and generates guidance information on teaching methods based on the results.

[0076] Step 12:

[0077] The server distributes the generated teaching method guide information for teachers to the terminal, which displays it to the teacher so that the teacher can use it to decide on a teaching policy.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] In the conventional education system, it was difficult to provide individualized instruction tailored to each learner's characteristics and progress, making it difficult to achieve effective learning. It also placed a heavy burden on teachers, making it difficult to find appropriate teaching methods. Furthermore, career and occupation suggestions were not based on the learner's characteristics and abilities, so suggestions optimal for each learner were not made.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily learning assignments based on the recognized learning tendencies and personality traits, means for distributing the generated learning assignments to an educational device, means for evaluating the learner's answers and calculating a learning outcome score and personality trait score, means for suggesting the learner's optimal educational course and career based on the learning outcome score and personality trait score, and means for displaying personality assessment results to instructors based on the learner's score and providing instruction methods. This makes it possible to streamline individual instruction, provide the learner with an optimal educational experience, reduce the instructor's burden, and achieve effective learning instruction.

[0083] "Means for reading images of submitted materials and converting them into text data" refers to the function of the educational system to capture information submitted by learners on paper or digital media as images and convert them into digital text data using character recognition technology.

[0084] "Means for analyzing text data and recognizing learning tendencies and personality traits" refers to a function that analyzes converted text data using natural language processing algorithms, etc., to reveal learners' learning behavior and personality traits.

[0085] "Means for generating daily learning tasks based on learning tendencies and personality traits" refers to a function that automatically creates personalized learning tasks using a generative AI model or the like based on analyzed learning tendencies and personality traits.

[0086] "Means for distributing generated learning tasks to educational devices" refers to a function that sends learning tasks generated on a server to a terminal or device used by a learner, allowing the learner to receive and work on them.

[0087] "Means for evaluating learners' answers and calculating learning achievement scores and personality trait scores" refers to a function that evaluates answers submitted by learners using an automatic scoring system and quantifies the level of learning achievement and personality traits.

[0088] "Means for suggesting the most suitable educational course and career for a learner based on learning outcome scores and personality trait scores" refers to a function for recommending appropriate educational courses and future career choices to a learner based on the calculated scores.

[0089] "A means of displaying personality assessment results based on a learner's score to instructors and providing guidance on teaching methods" refers to a function that allows instructors to understand a learner's score and personality traits and, based on that, present information that will enable them to provide the most appropriate teaching method.

[0090] The present invention relates to an educational system that provides personalized learning. This system reads and analyzes images of submitted work to identify learning tendencies and personality traits, and then generates and distributes daily learning assignments based on that information. The following describes how this system is specifically implemented.

[0091] First, the user (student) takes a photo of their homework or assignment using a device such as a smartphone or tablet. This image data is then uploaded to a server using a dedicated application. This application runs on a typical mobile device and is designed to be intuitive for users to operate.

[0092] The server converts uploaded images into text data using optical character recognition (OCR) technology. Specifically, it can use services such as Google Cloud Vision API. This allows paper submissions to be handled as digital text data.

[0093] The server then analyzes this text data with natural language processing (NLP) algorithms, which can use open-source NLP libraries such as spaCy, to extract the student's learning habits and personality traits and provide a personalized assessment.

[0094] Furthermore, based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. This generative AI model automatically generates appropriate problems and tasks when given a specific prompt as input. An example of a specific prompt is, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[0095] The generated learning assignments are then sent to the user's (student's) device via a dedicated application. The student works on the assignments and enters their answers. These answer data are also uploaded to the server.

[0096] The server evaluates the answers submitted by the students using an automated scoring system. For example, an automated scoring tool such as the Grammarly API can be used here. Based on the graded answers, a learning outcome score and personality trait score are calculated. This quantifies the student's learning progress and characteristics and is used for the next step.

[0097] Finally, the server analyzes the calculated scores and, based on the results, recommends the most suitable educational course or career for the student. These recommendations are based on past data and statistical models, allowing for more accurate and appropriate recommendations. These recommendations are then communicated to students and teachers via a dedicated application.

[0098] The server also has a function to provide teachers with personality assessment results and guidance on teaching methods, allowing them to provide effective instruction tailored to the characteristics of each student.

[0099] This invention will enable personalized education, which is expected to improve learner satisfaction and learning outcomes. It will also reduce the burden on teachers and provide an environment where they can provide effective instruction.

[0100] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0101] Step 1:

[0102] The user (student) takes a photo of their homework or assignments using a smartphone or tablet. The input is the handwritten assignment, and the output is digital image data. Specifically, the user opens the camera app and takes a photo of the homework page. Care is taken to ensure the image is clear.

[0103] Step 2:

[0104] Users upload the images they have taken to the server using a dedicated application. The input is the captured image data, and the output is the image file sent to the server. Specifically, the user opens the application, presses the "Upload" button, selects the captured image, and sends it.

[0105] Step 3:

[0106] The server converts the uploaded image into text data using optical character recognition (OCR). The input is image data and the output is text data. Specifically, the server calls the Google Cloud Vision API, analyzes the text information in the image, and converts it into text data.

[0107] Step 4:

[0108] The server analyzes the converted text data using natural language processing (NLP) algorithms. The input is text data, and the output is data on learning tendencies and personality traits. Specifically, the server analyzes the text data using the spaCy library to extract frequently occurring errors, learning characteristics, vocabulary usage, and so on.

[0109] Step 5:

[0110] Based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. The input is learning tendency data and personality trait data, and the output is personalized learning tasks. Specifically, the server calls the generative AI model and provides a specific prompt as input. For example, a prompt such as, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[0111] Step 6:

[0112] The device (student's smartphone or tablet) delivers the generated learning assignments to the user through an application. The input is the personalized learning assignment, and the output is the learning assignment displayed on the user's device. Specifically, the device's app receives the new learning assignment and notifies the user through a notification.

[0113] Step 7:

[0114] The user works on the delivered learning tasks and inputs their answers. The input is the learning task answered by the user, and the output is the answer data. Specifically, the user enters their answer in the application's input field and presses the "Submit" button to send it.

[0115] Step 8:

[0116] The user sends the answer to the server via the terminal. The input is the answer data, and the output is the answer data sent to the server. In concrete terms, the answer submitted by the user is automatically uploaded to the server.

[0117] Step 9:

[0118] The server evaluates the answers submitted by students using an automated scoring system. The input is the answer data, and the output is the scoring results and grade data. Specifically, the server uses an automated scoring tool such as the Grammarly API to assign a score to each answer.

[0119] Step 10:

[0120] The server calculates learning outcome scores and personality trait scores based on the grading results. The input is the grading results and grade data, and the output is the learning outcome scores and personality trait scores. Specifically, the server calculates the scores for each assessment item using a statistical model.

[0121] Step 11:

[0122] The server then uses the calculated scores to suggest optimal educational courses and careers for students. The input is the learning outcome score and personality trait score, and the output is the suggested educational course and career information. Specifically, the server generates optimal suggestions by referencing past databases and statistical models.

[0123] Step 12:

[0124] The server distributes the proposals to students and teachers via their terminals. The input is the proposed educational course and career information, and the output is the proposal information displayed on the user terminal and the teacher terminal. Specifically, the server sends new proposal information via the notification system.

[0125] Step 13:

[0126] The server presents personality assessment results and information to teachers that guide teaching strategies. The input is learning outcome scores and personality trait scores, and the output is a teaching guide for teachers. Specifically, the server displays the teaching guide on a dashboard or similar, making it accessible to teachers.

[0127] Step 14:

[0128] The device notifies the teacher of this information and helps them decide on the best teaching strategy. The input is a teaching guide for the teacher, and the output is the teaching strategy notified to the teacher. Specifically, the device application displays a notification and provides a link to access the teaching guide.

[0129] The above is the specific processing flow of this system.

[0130] (Application example 1)

[0131] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0132] To improve work efficiency in the field, personalized training tailored to the characteristics and skills of each staff member is necessary. However, the current system requires too much effort to respond to each individual, making it difficult to provide efficient training programs. Furthermore, there are challenges in analyzing staff abilities and characteristics and proposing appropriate guidance and role allocation based on that.

[0133] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0134] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily homework, quizzes, and regular exam questions and training programs based on the recognized learning tendencies and personality traits, means for distributing the generated questions and programs to users, means for evaluating the user's answers and results and calculating an output score and personality score, means for suggesting the user's optimal career path, employment, and work method based on the output score and personality score, and means for displaying personality assessment results to instructors based on the user's score and providing guidance on teaching methods. This allows for the automatic generation of training programs optimized for individual staff members, improving specific work efficiency and enabling appropriate division of roles.

[0135] "Submission Image" refers to image data that a User takes using a camera or other photographic device and uploads to the System.

[0136] "Text data" refers to character string data such as sentences and numerical information extracted from images using OCR technology.

[0137] "Learning tendency" refers to a characteristic such as how a user tends to study, for example, whether they prefer visual information or auditory information.

[0138] "Personality traits" refer to the characteristics of a user's personality and behavioral patterns, such as the level of concentration and communication ability.

[0139] "Homework" refers to study tasks given to users on a daily basis, including questions and materials automatically generated and provided by the system.

[0140] A "quiz" is a test that can be completed in a short amount of time to check the level of understanding of daily learning.

[0141] "Regular exams" refer to tests conducted at regular intervals to assess overall understanding and progress.

[0142] A "training program" refers to specific training content and tasks that are generated according to the characteristics of a user in order to improve the user's work efficiency.

[0143] "Output score" refers to a numerical indicator that evaluates the results of a user's learning or work.

[0144] "Personality score" refers to a numerical index of a user's personality traits and behavioral patterns.

[0145] "Career path" refers to the direction and plans of students and staff regarding their future studies and careers.

[0146] "Employment destinations" refers to companies or workplaces where students and staff may find employment in the future.

[0147] "Work methods" refers to the tasks and procedures performed by staff, including efficient methods that the system optimizes and suggests.

[0148] "Instructor" refers to a teacher, coach, or person in a leadership position engaged in education or training.

[0149] "Instruction method" refers to the specific method or approach that an instructor uses to educate or train a user.

[0150] The term "system" refers to a series of devices and programs realized by combining the above means.

[0151] This invention is a system for improving the efficiency of individual training in education systems and factories. In particular, factory robots can analyze the work tendencies and characteristics of staff and automatically generate training programs based on the results.

[0152] Hardware and software used

[0153] Camera: Used to film staff working.

[0154] Server: Responsible for central information processing such as image analysis, data processing, and evaluation.

[0155] Smart glasses or terminals: Used as a display device to provide training programs and feedback to staff.

[0156] OCR technology: Used to extract text data from images. Specifically, we will use pytesseract as an example.

[0157] NLP algorithms: used to analyze text data and extract learning trends and personality traits. In this example, we use GPT-4.

[0158] Generative AI models (such as GPT-4): Used to automatically generate training programs based on learning tendencies and personality traits.

[0159] Processing steps

[0160] First, staff members use cameras to take pictures of their work and upload the images to a server, which then converts the images into text using OCR technology. The converted text is then analyzed using AI-based natural language processing (NLP) algorithms to extract learning trends and personality traits.

[0161] Next, the server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. The generated program is distributed to staff via smart glasses or other devices, and staff use it to train. The training results and answers are sent back to the server, which evaluates them using an automatic scoring system. An output score and personality score are calculated from the evaluation results.

[0162] The server then proposes optimal work methods and division of roles for each staff member based on the calculated score. This information is provided to the staff member and instructor via their terminal, and the instructor then decides on the optimal training policy for the staff member.

[0163] Specific examples

[0164] For example, let's say a new factory worker is learning how to operate equipment. The worker uses a camera to take a picture of themselves performing the operation and sends the image to a server. The server then uses OCR technology to convert the image data into text data, and runs an NLP algorithm on that data to determine whether the new worker is quick to understand the procedure and has a high level of concentration.

[0165] The server uses the results of this evaluation to automatically generate an appropriate training program for new staff using a generative AI model (GPT-4). For example, it generates training materials detailing operation procedures and specific practical tasks, which it provides to the new staff through smart glasses. The server then sends the new staff's performance and answers to the server, which evaluates them using an automatic scoring system. Based on the results, it calculates an output score and personality score and suggests optimal work methods and role allocations to the new staff.

[0166] Example prompt statement

[0167] An example of a prompt to enter into GPT-4:

[0168] Generate efficient, personalized training programs based on staff analytics including:

[0169] Learning Propensity: Staff members pick up work procedures quickly and have good attention to detail.

[0170] Personality traits: Highly focused and collaborative.

[0171] Analysis results:

[0172] Staff have proven quicker to understand how to operate the new equipment and work better with other staff.

[0173] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0174] Step 1:

[0175] Users use a camera to take pictures of their submissions and upload them to a server, which records the user's operation procedures and work.

[0176] Input: An image of the submission taken with a camera

[0177] Output: Image data uploaded to the server

[0178] Step 2:

[0179] The server converts the uploaded image data into text data using OCR technology (e.g., pytesseract). Through OCR processing, character information in the image is extracted as text.

[0180] Input: Uploaded image data

[0181] Data processing: Conversion to text data using OCR technology

[0182] Output: Converted text data

[0183] Step 3:

[0184] The server then analyzes the converted text data using a natural language processing (NLP) algorithm (e.g., GPT-4) to extract the user's learning tendencies and personality traits. The analysis results include the user's strengths and learning tendencies.

[0185] Input: Converted text data

[0186] Data Computing: Analyzing Learning Tendencies and Personality Traits with NLP Algorithms

[0187] Output: Extracted learning tendencies and personality traits

[0188] Step 4:

[0189] The server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. Specifically, it inputs a prompt sentence to the generative AI model and generates appropriate training content based on it.

[0190] Input: extracted learning tendencies and personality traits

[0191] Data Computing: Generating Training Programs with Generative AI Models

[0192] Output: The generated training program

[0193] Step 5:

[0194] The terminal delivers the generated training program to the user, which includes displaying the training content via smart glasses or other display devices.

[0195] Input: Generated training program

[0196] Action: Displaying a training program on the device

[0197] Output: A user-visible training program

[0198] Step 6:

[0199] The user follows the training program provided through the terminal and reports the progress and results to the server. In this step, the user's answers and progress are sent from the terminal to the server.

[0200] Input: User's actions and answers

[0201] Operation: User sends progress and answers to the server

[0202] Output: Implementation status and answer data sent to the server

[0203] Step 7:

[0204] The server evaluates the received data using an automated scoring system, and calculates an output score and personality score as the evaluation results.

[0205] Input: User's performance and answer data

[0206] Data calculations: Evaluation by an automated scoring system

[0207] Output: Calculated output scores and personality scores

[0208] Step 8:

[0209] The server then proposes optimal work methods and role allocations to the user based on the calculated scores, and provides the proposed information to the user and instructor via their terminals.

[0210] Input: Output scores and personality scores

[0211] Data calculation: Optimizing work methods and division of roles

[0212] Output: optimal recommendations to users and instructors

[0213] Step 9:

[0214] The instructor determines an instruction method for the user based on the received suggestion information and provides appropriate feedback to the user.

[0215] Input: Best Suggestion Information

[0216] Action: Deciding how to teach and providing feedback to the user

[0217] Output: Specific instruction and feedback to the user

[0218] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0219] This invention is a system for improving the efficiency of individual instruction in educational systems, and by combining it with an emotion engine that recognizes user emotions, it achieves even more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are then used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine recognizes students' emotions and provides feedback to the educational content.

[0220] Image loading and analysis of submissions

[0221] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[0222] Introducing the Emotion Engine

[0223] When a user (student) submits an assignment or solves a problem, the device uses a camera and microphone to capture the user's facial expression and voice data. This data is analyzed in real time by an emotion engine to recognize the user's emotional state. For example, facial expression recognition technology and voice analysis technology are used to detect the emotions (e.g., joy, anger, sadness, surprise) that the user is showing.

[0224] Homework, quiz, and exam question generation

[0225] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on the identified learning tendencies and personality traits, as well as the recognized emotional state. For example, if a user shows signs of nervousness or anxiety about a question, the server can generate questions that take their emotional state into account, providing questions of a level of difficulty appropriate for that learning situation. The device then distributes the generated questions to the student, who then answers them.

[0226] Answer analysis and scoring

[0227] Students' answers are sent to the server via their terminal. The server evaluates the received answer data using an automatic scoring system. Grade data is generated as the evaluation results. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This quantifies each student's learning outcomes and personality traits, and also expresses their emotional data. The detailed data obtained in this way is used for analysis.

[0228] Career and employment suggestions

[0229] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[0230] Teacher feedback and instructional guide

[0231] Based on the student's score and analysis results, the server displays the personality assessment results (e.g., MBTI) to the teacher and provides guidance on how to teach based on the results. This feedback information is generated taking into account emotional data. For example, it may include how to respond to specific emotional states and how to provide effective feedback. This information is provided to the teacher via their device, allowing them to determine the best teaching method for the student.

[0232] Example of a sample

[0233] For example, when a student submits a math assignment, the device takes a photo of the assignment and the emotion engine analyzes the student's facial expressions and voice. If the user is nervous, appropriate assistance is provided. The server uses OCR technology to convert the data into text and analyzes the data to extract learning tendencies and personality traits. Based on this, the device generates questions and sends them to the student, who then answers them. Once the answers are sent to the server and graded, the output score and personality score are updated. The student's emotional state is also recorded and used to optimize the learning process. Teachers are then advised on teaching methods based on the student's emotions and scores, improving the quality of individualized instruction.

[0234] As a result, the present invention makes it possible to build a system that provides an educational experience optimized for each individual student, reduces the burden on teachers, and provides effective instruction. Furthermore, by combining it with emotion recognition technology, the quality and effectiveness of learning can be further improved.

[0235] The processing flow will be explained below.

[0236] Step 1:

[0237] Students take photos of their own work using their smartphones or tablets. The devices store the images of the work along with the students' facial expressions and voice data, and upload them to the server.

[0238] Step 2:

[0239] The server receives the uploaded image data and facial expression / voice data, and first extracts text data from the image data using OCR technology.

[0240] Step 3:

[0241] The server analyzes the extracted text data using natural language processing (NLP) algorithms, and the analysis results identify the student's learning tendencies and personality traits.

[0242] Step 4:

[0243] At the same time, the server's emotion engine uses facial expression recognition and voice analysis technologies to analyze the user's emotional state, for example, determining whether the user is nervous or relaxed when taking the image.

[0244] Step 5:

[0245] The server comprehensively evaluates the identified learning tendencies, personality traits, and emotional states, and based on that, generates personalized homework, quizzes, and regular exam questions using a generative AI model (e.g., GPT-4).

[0246] Step 6:

[0247] The server stores the generated questions in a database and then distributes them to the terminals, which then display the questions to the students.

[0248] Step 7:

[0249] The user (student) works on the problem and inputs the answer into the terminal, which then sends the answer data to the server.

[0250] Step 8:

[0251] The server evaluates the received answer data using an automatic scoring system, generates performance data, and calculates an output score and personality score based on the answer content and performance data.

[0252] Step 9:

[0253] The server analyzes the calculated output score and personality score and generates information to suggest the student's optimal career path and employment based on the results, taking into account the user's emotional state.

[0254] Step 10:

[0255] The server distributes information on career and employment suggestions to the terminal, and the terminal displays the suggestions to the students and teachers.

[0256] Step 11:

[0257] The server displays the personality assessment results and emotional state based on the student's scores to the teacher, and generates guidance information for teaching methods based on the results.

[0258] Step 12:

[0259] The server then distributes the generated instructional method guide information to the terminal, which then displays it to the teacher.The teacher then decides on a teaching policy based on this information and provides individual instruction to the student.

[0260] Example 2

[0261] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0262] Traditional educational systems have faced the challenge of making it difficult to provide instruction that fully takes into account each student's learning tendencies and personality traits. Furthermore, there was a lack of a way to grasp students' emotional states in real time and reflect them in the learning content. As a result, the quality of individualized instruction declined, hindering students' motivation to learn and academic improvement.

[0263] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading an image of a submitted work and converting it into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for acquiring the user's facial expression and voice data and analyzing it to recognize their emotional state, and means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies, personality traits, and emotional state. This makes it possible to provide learning content optimized for each student and take into account the student's emotional state, thereby improving learning effectiveness and enhancing the quality of individualized instruction.

[0264] "Image of submission" means a digital image of handwritten or printed material submitted by a student as part of an assignment.

[0265] The "means for converting into text data" is a mechanism for extracting character information in an image as text data using optical character recognition technology.

[0266] "Means for recognizing learning tendencies and personality traits" is a function that analyzes text data and identifies students' learning patterns and personality traits.

[0267] "Means for acquiring a user's facial expression and voice data and analyzing it to recognize their emotional state" refers to a technology that uses a camera and microphone to collect a user's facial expression and voice and then analyzes it to determine their emotional state.

[0268] The "means for generating daily homework, quiz, and exam questions" is a mechanism for creating individually customized study questions based on learning tendencies, personality traits, and emotional state.

[0269] The "means for calculating output scores and personality scores" is a function that automatically calculates evaluation scores for academic performance and personality traits based on the student's answers and grade data.

[0270] The "means for suggesting career paths and employment opportunities" is a system that suggests future career paths and suitable employment opportunities to students based on their output scores and personality scores.

[0271] "Means for providing guidance on teaching methods" is a system that provides teachers with learning data and personality data on students and advises them on effective teaching methods.

[0272] This invention is a system for improving the efficiency of individual instruction in educational systems, particularly by combining an emotion engine that recognizes user emotions to achieve more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are also used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine can recognize students' emotions and provide feedback to the educational content.

[0273] First, the user (student) takes a picture of the submitted work using a smartphone or tablet. This image is displayed on the device, and after the user confirms it, they upload it to the server. The server then converts the received image data into text data using OCR technology (e.g., Tesseract). At this stage, handwritten or printed characters are digitized and proceed to the next analysis step.

[0274] The server analyzes the converted text data using an automatic analysis algorithm (e.g., SpaCy) to extract learning trends and personality traits. Additionally, when users (students) submit their work or solve problems, the device uses a camera and microphone to capture their facial expressions and voice data. This data is sent to the server in real time and analyzed using an emotion engine (e.g., Microsoft Azure Emotion Recognition API).

[0275] The server then uses a generative AI model (e.g., GPT-4) to generate questions for daily homework, quizzes, and regular exams based on the learning tendencies, personality traits, and emotional states. For example, example prompts include:

[0276] "This student tends to be more theoretical in his learning, and in recent exams he has struggled with understanding applied mathematics questions. His current emotional state is one of tension and anxiety. Please create applied mathematics questions of medium difficulty for this student to deepen his understanding. Please include illustrations in the questions and provide thorough explanations."

[0277] The generated questions are sent to students via their devices, who then answer them. The answers are then sent to a server via the device, where they are evaluated using an automated scoring system (e.g., OpenAI Codex). Based on this, output scores and personality scores are calculated, and the server stores these data.

[0278] Furthermore, the server will make career and employment recommendations based on the analysis results. The recommendations are calculated using statistical models to determine the optimal career path and are provided to students and teachers via their devices. The recommendations are best suited to the user's abilities and characteristics.

[0279] Finally, the server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on how to teach based on the results. This information is generated taking into account emotional data and is provided to the teacher via their terminal.

[0280] As a result, a system will be created that improves the quality of education and makes individualized instruction more efficient through learning programs optimized for each student and feedback based on their emotions.

[0281] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0282] Step 1: Take and upload an image of your submission

[0283] The user (student) takes a picture of the submitted work using a smartphone or tablet, and the captured image is displayed on the device screen.

[0284] The terminal uploads the image confirmed by the user to the server.

[0285] Input: A user-taken image of the submission.

[0286] Output: Image data of the submission uploaded to the server.

[0287] Step 2: Convert image data to text using OCR technology

[0288] The server converts the image data of the received submission into text data using OCR technology (e.g., Tesseract).

[0289] The server temporarily stores the converted text data and uses it for subsequent analysis steps.

[0290] Input: Image data of the submission.

[0291] Output: Text data.

[0292] What it does: Converts handwritten or printed text in an image into digital text.

[0293] Step 3: Analysis of learning tendencies and personality traits

[0294] The server analyzes the converted text data using a natural language processing algorithm (e.g., SpaCy).

[0295] The server extracts learning tendencies and personality traits from the text data and stores this information in a database.

[0296] Input: Text data converted by OCR.

[0297] Output: Data on learning tendencies and personality traits.

[0298] Specific Actions: Through text analysis, students' tendency to excel at logical thinking is revealed.

[0299] Step 4: Acquire and analyze emotion data

[0300] When a user (student) solves a problem, the device uses a camera and microphone to capture the user's facial expressions and voice data.

[0301] The terminal transmits the acquired emotion data to the server in real time.

[0302] The server analyzes the data using an emotion engine (e.g., Microsoft Azure Emotion Recognition API) to recognize the user's emotional state.

[0303] Input: User's facial and voice data.

[0304] Output: Data of the user's emotional state.

[0305] Specific operation: Detects emotions such as tension, anxiety, and joy in real time from the user's facial expressions and tone of voice.

[0306] Step 5: Generate homework, quizzes, and exam questions

[0307] The server generates personalized questions using a generative AI model (e.g., GPT-4) based on learning tendencies, personality traits, and emotional states.

[0308] The server automatically generates questions by inputting prompt sentences into a generative AI model.

[0309] The server sends the generated questions to the terminal.

[0310] The terminal distributes the generated questions to the students, who then answer them.

[0311] Input: learning tendencies, personality traits, and emotional state data.

[0312] Output: Personalized homework, quiz and regular exam questions.

[0313] Specific operation: The following prompt is input into the generative AI model to generate a problem: "This student has a theoretical learning tendency, and in recent exams, he has tended to struggle with understanding applied mathematics problems. His current emotional state is showing tension and anxiety. Please create an applied mathematics problem of medium difficulty for this student to deepen his understanding. Please include illustrations and provide detailed explanations in the problem text."

[0314] Step 6: Upload and grade your answers

[0315] The user (student) enters the answer into the terminal and sends it to the server.

[0316] The server evaluates the received answer data using an automatic scoring system (e.g., OpenAI Codex) and generates score data.

[0317] Input: The answer data entered by the student.

[0318] Output: Evaluated performance data.

[0319] Specific operation: Based on the answers, the system determines whether the answers are correct or incorrect, scores them, and automatically generates a grade.

[0320] Step 7: Calculating output and personality scores

[0321] The server automatically calculates the output score and personality score based on the answers and performance data.

[0322] Input: Answers and performance data.

[0323] Output: Output scores and personality scores.

[0324] What it does: Evaluate students' understanding and personality traits and calculate corresponding scores.

[0325] Step 8: Career and employment suggestions

[0326] The server suggests optimal career paths and employment opportunities based on the calculated output score and personality score.

[0327] The server sends the suggestions to the terminals and provides them to students and teachers.

[0328] Inputs: Output scores and personality scores.

[0329] Output: Career and employment suggestions.

[0330] What it does: Uses statistical models to suggest appropriate career paths for students, such as "go to a science and engineering university."

[0331] Step 9: Teacher feedback and instructional guide

[0332] Based on the student's scores and analysis results, the server generates personality assessment results (e.g., MBTI) and information to guide teachers on teaching methods.

[0333] The terminal provides this feedback information to the teacher, who then uses it to determine the teaching policy.

[0334] Input: Student scores and analysis results.

[0335] Output: Personality assessment results and guidance information on teaching methods.

[0336] Specific action: The teacher's device displays information such as, "Student A is analytical and introverted. Rather than praising him, let's suggest a problem-solving approach."

[0337] (Application example 2)

[0338] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] Individualized instruction is extremely important in the modern education system, but providing appropriate learning content to each student and maximizing learning outcomes requires a great deal of time and effort. Furthermore, education that ignores students' emotional states can reduce effectiveness, so there is a need to provide personalized learning experiences using emotion recognition technology. Furthermore, the introduction of virtual classrooms using virtual reality technology is necessary to provide students with new learning methods.

[0340] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0341] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies and personality traits, means for distributing the generated questions to students, means for evaluating students' answers and calculating output scores and personality scores, means for suggesting optimal career paths and employment opportunities for students based on the output scores and personality scores, means for displaying personality assessment results to teachers based on the students' scores and providing instructional guidance, an emotion engine for recognizing students' emotional states using facial expressions and voice data when solving problems, means for personalizing educational content based on the emotional states and generating optimal questions for students, and means for providing a personalized learning experience in a virtual classroom using a virtual reality head-mounted display. This makes it possible to maximize students' learning outcomes, reduce the burden on teachers, and provide effective education tailored to their emotional states.

[0342] "Submissions" refer to assignments and reports that students submit as the result of their studies.

[0343] An "image" is visual information captured as digital data by a camera or scanner.

[0344] "Text data" refers to data that is treated as character information.

[0345] "Learning tendency" refers to the tendency that indicates how a student will effectively proceed with their studies.

[0346] "Personality traits" refer to a student's personality and behavioral characteristics, and are information used to individualize educational methods.

[0347] "Homework," "quizzes," and "regular exams" are questions used to evaluate students' learning status.

[0348] "Delivery" refers to sending the generated questions and feedback to students.

[0349] An "answer" is a solution submitted by a student to a problem.

[0350] "Assessment" means marking students' answers and measuring their performance.

[0351] An "output score" is a score that quantitatively evaluates learning outcomes.

[0352] A "personality score" is a numerical score that represents a student's personality traits.

[0353] "Career path" refers to the direction of education or career that a student should pursue in the future.

[0354] "Employment" refers to the workplace where students plan to work after graduation.

[0355] "Instructional methods" refer to the educational techniques that teachers use with students.

[0356] "Emotional state" refers to the student's emotional or psychological state.

[0357] An "emotion engine" is a technology for recognizing emotional states from data such as facial expressions and voice.

[0358] "Personalization" means customizing something to meet individual needs.

[0359] A "virtual reality head-mounted display" is a visual and auditory device worn to experience a virtual reality space.

[0360] A "virtual classroom" is a virtual educational space recreated using virtual reality technology.

[0361] This invention is a system for individualized instruction that recognizes students' emotional states and uses that information to achieve more sophisticated personalization. To achieve this, the invention mainly uses the following hardware and software:

[0362] Hardware used

[0363] 1. VR Head-Mounted Display (HMD): Used to provide learning experiences within a virtual classroom.

[0364] 2. 360-degree camera: Used by students to film their submissions.

[0365] 3. Microphone: Used to collect student voice data.

[0366] Software used

[0367] 1. OCR technology: Using Google Cloud Vision API, we convert the image of the submitted document into text data.

[0368] 2. Natural Language Processing (NLP): Analyze transformed text data using spaCy and Hugging Face Transformers.

[0369] 3. Emotion recognition technology: Using OpenCV and DeepFace, we identify students' emotional states from facial expressions and voice data.

[0370] 4. Generative AI model: Uses OpenAI GPT-4 to generate personalized homework and test questions.

[0371] 5. Server Technology: We use AWS EC2 and S3 for data storage and processing.

[0372] How to carry out the invention

[0373] 1. Importing and analyzing submitted images

[0374] Students wear a VR head-mounted display and follow instructions in the virtual classroom to take photos of their submissions using a 360-degree camera and upload the images to the server, which then uses the Google Cloud Vision API to convert the image data into text using OCR processing.

[0375] 2. Introducing the Emotion Engine

[0376] While solving problems, the device (camera and microphone of the VR head-mounted display) collects facial and voice data from the student, and uses OpenCV and DeepFace to recognize the student's emotional state in real time. This information is sent to a server and used to personalize the educational content.

[0377] 3. Homework, quiz, and regular exam question generation

[0378] The server uses a generative AI model (GPT-4) to automatically generate questions optimized for each student based on their analyzed learning tendencies and personality traits, as well as their recognized emotional state. This allows a student showing signs of nervousness or anxiety to be provided with appropriate questions to alleviate that state. The generated questions are then distributed to students within the virtual classroom.

[0379] 4. Answer analysis and scoring

[0380] Students enter their answers to questions in the VR space and send them to the server, which then evaluates the answers using an automatic scoring system to calculate output and personality scores.

[0381] 5. Career and employment suggestions

[0382] Based on the calculated scores, the server proposes the best career paths and employment opportunities for students and provides this information within the virtual classroom, allowing students to choose a career path that suits their own abilities and characteristics.

[0383] 6. Teacher feedback and instructional guide

[0384] Teachers can also sit in the same virtual classroom and receive feedback on teaching methods based on students' scores and emotional states, helping teachers to determine the best teaching strategy for each individual student.

[0385] Specific examples

[0386] For example, when a student submits a math assignment, they follow instructions in the VR space and take a photo of the assignment with a 360-degree camera. The server converts the image into text data using OCR technology and then analyzes it with an NLP algorithm to identify learning trends. While the student is solving the problem, an emotion engine analyzes their facial expressions and voice to detect their level of tension. A generative AI model automatically generates a problem of a level of difficulty appropriate for this state and presents it to the student in the VR space. After solving the problem, the answer is automatically graded, and the output score and personality score are updated.

[0387] Example prompt sentence:

[0388] Question: This student has high ability in math but is prone to anxiety before tests. Generate problems designed to help him overcome his anxiety.

[0389] In this way, this invention is a system that can provide each student with an optimal educational experience, reduce the burden on teachers, and maximize learning effectiveness.

[0390] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0391] Step 1:

[0392] The user (student) wears a VR head-mounted display and takes a picture of the submitted work with a 360-degree camera in the virtual classroom. The input is a physical image of the submitted work, and the output is digital image data.

[0393] Step 2:

[0394] The terminal uploads the image data of the photographed submission to the server. The input is digital image data, and the output is image data stored on the server. The server receives and stores this image data.

[0395] Step 3:

[0396] The server calls the Google Cloud Vision API, performs OCR on the uploaded image data, and converts it into text data. The input is the image data of the submitted item, and the output is text data.

[0397] Step 4:

[0398] The server analyzes the converted text data using spaCy and Hugging Face Transformers to extract learning tendencies and personality traits. The input is text data, and the output is analyzed learning tendencies and personality traits data.

[0399] Step 5:

[0400] When the user solves a problem, the device collects the user's facial expression and voice data in real time using the camera and microphone of the VR head-mounted display and sends it to the server. The input is the user's facial expression and voice data, and the output is the emotional data sent to the server.

[0401] Step 6:

[0402] The server analyzes the emotional data using OpenCV and DeepFace to recognize the user's current emotional state. The input is facial expression and voice data, and the output is emotional state data analyzed in real time.

[0403] Step 7:

[0404] The server uses GPT-4 to generate personalized homework and test questions based on the analyzed learning tendencies, personality traits, and emotional state. The generated questions are appropriate for the user's learning situation and emotional state. The input is learning tendency data, personality trait data, and emotional state data, and the output is personalized questions.

[0405] Step 8:

[0406] The server delivers the generated questions to the user's VR head-mounted display. The input is a personalized question, and the output is a question displayed in the user's VR classroom.

[0407] Step 9:

[0408] The user answers questions in the VR classroom and sends the answer data to the server. The input is the user's answer data, and the output is the answer data sent to the server.

[0409] Step 10:

[0410] The server evaluates the received answer data using an automatic scoring system and calculates an output score and a personality score. The input is the user's answer data, and the output is the output score and the personality score.

[0411] Step 11:

[0412] Based on the calculated scores, the server uses a generative AI model to suggest optimal career paths and employment opportunities for users. The inputs are the output score and personality score, and the output is career and employment suggestions.

[0413] Step 12:

[0414] The server also provides teachers with feedback on teaching methods based on the user's score and emotional state. The input is score data and emotional state data, and the output is a teaching guide for the teacher.

[0415] Step 13:

[0416] The device displays the generated teaching guide to the teacher in the VR space, and the teacher then provides optimal instruction to the user based on it. The input is the teaching guide, and the output is the display information to the teacher.

[0417] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0418] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0419] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0420] [Second embodiment]

[0421] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0422] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0423] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0424] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0425] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0426] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0427] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0428] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0429] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0430] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0431] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0432] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0433] This invention is a system for improving the efficiency of individualized instruction in educational systems. The system reads images of students' submitted work, analyzes their learning tendencies and personality traits from the images, automatically generates personalized homework, quizzes, and regular exam questions based on the results, and evaluates the students' answers to calculate output scores and personality scores. These scores are used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods.

[0434] Image loading and analysis of submissions

[0435] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[0436] Homework, quiz, and exam question generation

[0437] The server automatically generates questions for daily homework, quizzes, and regular exams using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. This enables each student to receive optimal learning content. The device delivers the generated questions to the student, who then answers them.

[0438] Answer analysis and scoring

[0439] Students' answers are sent to the server via their devices. The server evaluates the answers using an automatic scoring system and generates grade data. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This allows each student's learning outcomes and personality traits to be expressed numerically.

[0440] Career and employment suggestions

[0441] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[0442] Teacher feedback and instructional guide

[0443] The server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on teaching methods based on the results. This information is provided to the teacher via their device, and the teacher can use it to determine the best teaching policy for the student.

[0444] Example of a sample

[0445] For example, when a student submits a mathematics assignment, the device takes a photo of the assignment and sends it to the server. The server then uses OCR technology to convert it into text data and analyzes the text, identifying that the student has strengths in mathematics. Based on this information, the server uses a generative AI model to create a challenging mathematics assignment for the student and distributes it to the device. The student then works on the assignment and uploads their answers back to the server via their device. The answers are graded, and the student's output score and personality score are updated. As a result, the student is recommended a career path specializing in mathematics, and the teacher is provided with appropriate teaching methods (e.g., in-depth explanations or additional challenging assignments) via the device.

[0446] As a result, the present invention makes it possible to build a system that provides an educational experience that is optimized for each individual student and realizes effective instruction while reducing the burden on teachers.

[0447] The processing flow will be explained below.

[0448] Step 1:

[0449] Students take photos of their own work using their smartphones or tablets, which then save the image data and upload it to the server.

[0450] Step 2:

[0451] The server receives the uploaded image data and uses OCR technology to extract text data from the image data.

[0452] Step 3:

[0453] The server then analyzes the extracted text data using natural language processing (NLP) algorithms, and the results of the analysis identify the student's learning tendencies and personality traits.

[0454] Step 4:

[0455] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on identified learning tendencies and personality traits.

[0456] Step 5:

[0457] The server stores the generated questions in a database, and then distributes the stored questions to the device.

[0458] Step 6:

[0459] The terminal displays the delivered questions to the student. The user (student) works on the questions and sends the answers to the server via the terminal.

[0460] Step 7:

[0461] The server evaluates the received answer data with an automatic scoring system, and generates score data as the evaluation result.

[0462] Step 8:

[0463] The server calculates an output score and a personality score based on the generated performance data.

[0464] Step 9:

[0465] The server analyzes the calculated output score and personality score and generates information to suggest the most suitable career path and employment for the student.

[0466] Step 10:

[0467] The server distributes career and employment suggestion information to the terminal, which then displays the suggestion content to students and teachers.

[0468] Step 11:

[0469] The server displays the personality assessment results based on the student's scores to the teacher, and generates guidance information on teaching methods based on the results.

[0470] Step 12:

[0471] The server distributes the generated teaching method guide information for teachers to the terminal, which displays it to the teacher so that the teacher can use it to decide on a teaching policy.

[0472] Example 1

[0473] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0474] In the conventional education system, it was difficult to provide individualized instruction tailored to each learner's characteristics and progress, making it difficult to achieve effective learning. It also placed a heavy burden on teachers, making it difficult to find appropriate teaching methods. Furthermore, career and occupation suggestions were not based on the learner's characteristics and abilities, so suggestions optimal for each learner were not made.

[0475] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0476] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily learning assignments based on the recognized learning tendencies and personality traits, means for distributing the generated learning assignments to an educational device, means for evaluating the learner's answers and calculating a learning outcome score and personality trait score, means for suggesting the learner's optimal educational course and career based on the learning outcome score and personality trait score, and means for displaying personality assessment results to instructors based on the learner's score and providing instruction methods. This makes it possible to streamline individual instruction, provide the learner with an optimal educational experience, reduce the instructor's burden, and achieve effective learning instruction.

[0477] "Means for reading images of submitted materials and converting them into text data" refers to the function of the educational system to capture information submitted by learners on paper or digital media as images and convert them into digital text data using character recognition technology.

[0478] "Means for analyzing text data and recognizing learning tendencies and personality traits" refers to a function that analyzes converted text data using natural language processing algorithms, etc., to reveal learners' learning behavior and personality traits.

[0479] "Means for generating daily learning tasks based on learning tendencies and personality traits" refers to a function that automatically creates personalized learning tasks using a generative AI model or the like based on analyzed learning tendencies and personality traits.

[0480] "Means for distributing generated learning tasks to educational devices" refers to a function that sends learning tasks generated on a server to a terminal or device used by a learner, allowing the learner to receive and work on them.

[0481] "Means for evaluating learners' answers and calculating learning achievement scores and personality trait scores" refers to a function that evaluates answers submitted by learners using an automatic scoring system and quantifies the level of learning achievement and personality traits.

[0482] "Means for suggesting the most suitable educational course and career for a learner based on learning outcome scores and personality trait scores" refers to a function for recommending appropriate educational courses and future career choices to a learner based on the calculated scores.

[0483] "A means of displaying personality assessment results based on a learner's score to instructors and providing guidance on teaching methods" refers to a function that allows instructors to understand a learner's score and personality traits and, based on that, present information that will enable them to provide the most appropriate teaching method.

[0484] The present invention relates to an educational system that provides personalized learning. This system reads and analyzes images of submitted work to identify learning tendencies and personality traits, and then generates and distributes daily learning assignments based on that information. The following describes how this system is specifically implemented.

[0485] First, the user (student) takes a photo of their homework or assignment using a device such as a smartphone or tablet. This image data is then uploaded to a server using a dedicated application. This application runs on a typical mobile device and is designed to be intuitive for users to operate.

[0486] The server converts uploaded images into text data using optical character recognition (OCR) technology. Specifically, it can use services such as Google Cloud Vision API. This allows paper submissions to be handled as digital text data.

[0487] The server then analyzes this text data with natural language processing (NLP) algorithms, which can use open-source NLP libraries such as spaCy, to extract the student's learning habits and personality traits and provide a personalized assessment.

[0488] Furthermore, based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. This generative AI model automatically generates appropriate problems and tasks when given a specific prompt as input. An example of a specific prompt is, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[0489] The generated learning assignments are then sent to the user's (student's) device via a dedicated application. The student works on the assignments and enters their answers. These answer data are also uploaded to the server.

[0490] The server evaluates the answers submitted by the students using an automated scoring system. For example, an automated scoring tool such as the Grammarly API can be used here. Based on the graded answers, a learning outcome score and personality trait score are calculated. This quantifies the student's learning progress and characteristics and is used for the next step.

[0491] Finally, the server analyzes the calculated scores and, based on the results, recommends the most suitable educational course or career for the student. These recommendations are based on past data and statistical models, allowing for more accurate and appropriate recommendations. These recommendations are then communicated to students and teachers via a dedicated application.

[0492] The server also has a function to provide teachers with personality assessment results and guidance on teaching methods, allowing them to provide effective instruction tailored to the characteristics of each student.

[0493] This invention will enable personalized education, which is expected to improve learner satisfaction and learning outcomes. It will also reduce the burden on teachers and provide an environment where they can provide effective instruction.

[0494] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0495] Step 1:

[0496] The user (student) takes a photo of their homework or assignments using a smartphone or tablet. The input is the handwritten assignment, and the output is digital image data. Specifically, the user opens the camera app and takes a photo of the homework page. Care is taken to ensure the image is clear.

[0497] Step 2:

[0498] Users upload the images they have taken to the server using a dedicated application. The input is the captured image data, and the output is the image file sent to the server. Specifically, the user opens the application, presses the "Upload" button, selects the captured image, and sends it.

[0499] Step 3:

[0500] The server converts the uploaded image into text data using optical character recognition (OCR). The input is image data and the output is text data. Specifically, the server calls the Google Cloud Vision API, analyzes the text information in the image, and converts it into text data.

[0501] Step 4:

[0502] The server analyzes the converted text data using natural language processing (NLP) algorithms. The input is text data, and the output is data on learning tendencies and personality traits. Specifically, the server analyzes the text data using the spaCy library to extract frequently occurring errors, learning characteristics, vocabulary usage, and so on.

[0503] Step 5:

[0504] Based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. The input is learning tendency data and personality trait data, and the output is personalized learning tasks. Specifically, the server calls the generative AI model and provides a specific prompt as input. For example, a prompt such as, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[0505] Step 6:

[0506] The device (student's smartphone or tablet) delivers the generated learning assignments to the user through an application. The input is the personalized learning assignment, and the output is the learning assignment displayed on the user's device. Specifically, the device's app receives the new learning assignment and notifies the user through a notification.

[0507] Step 7:

[0508] The user works on the delivered learning tasks and inputs their answers. The input is the learning task answered by the user, and the output is the answer data. Specifically, the user enters their answer in the application's input field and presses the "Submit" button to send it.

[0509] Step 8:

[0510] The user sends the answer to the server via the terminal. The input is the answer data, and the output is the answer data sent to the server. In concrete terms, the answer submitted by the user is automatically uploaded to the server.

[0511] Step 9:

[0512] The server evaluates the answers submitted by students using an automated scoring system. The input is the answer data, and the output is the scoring results and grade data. Specifically, the server uses an automated scoring tool such as the Grammarly API to assign a score to each answer.

[0513] Step 10:

[0514] The server calculates learning outcome scores and personality trait scores based on the grading results. The input is the grading results and grade data, and the output is the learning outcome scores and personality trait scores. Specifically, the server calculates the scores for each assessment item using a statistical model.

[0515] Step 11:

[0516] The server then uses the calculated scores to suggest optimal educational courses and careers for students. The input is the learning outcome score and personality trait score, and the output is the suggested educational course and career information. Specifically, the server generates optimal suggestions by referencing past databases and statistical models.

[0517] Step 12:

[0518] The server distributes the proposals to students and teachers via their terminals. The input is the proposed educational course and career information, and the output is the proposal information displayed on the user terminal and the teacher terminal. Specifically, the server sends new proposal information via the notification system.

[0519] Step 13:

[0520] The server presents personality assessment results and information to teachers that guide teaching strategies. The input is learning outcome scores and personality trait scores, and the output is a teaching guide for teachers. Specifically, the server displays the teaching guide on a dashboard or similar, making it accessible to teachers.

[0521] Step 14:

[0522] The device notifies the teacher of this information and helps them decide on the best teaching strategy. The input is a teaching guide for the teacher, and the output is the teaching strategy notified to the teacher. Specifically, the device application displays a notification and provides a link to access the teaching guide.

[0523] The above is the specific processing flow of this system.

[0524] (Application example 1)

[0525] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0526] To improve work efficiency in the field, personalized training tailored to the characteristics and skills of each staff member is necessary. However, the current system requires too much effort to respond to each individual, making it difficult to provide efficient training programs. Furthermore, there are challenges in analyzing staff abilities and characteristics and proposing appropriate guidance and role allocation based on that.

[0527] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0528] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily homework, quizzes, and regular exam questions and training programs based on the recognized learning tendencies and personality traits, means for distributing the generated questions and programs to users, means for evaluating the user's answers and results and calculating an output score and personality score, means for suggesting the user's optimal career path, employment, and work method based on the output score and personality score, and means for displaying personality assessment results to instructors based on the user's score and providing guidance on teaching methods. This allows for the automatic generation of training programs optimized for individual staff members, improving specific work efficiency and enabling appropriate division of roles.

[0529] "Submission Image" refers to image data that a User takes using a camera or other photographic device and uploads to the System.

[0530] "Text data" refers to character string data such as sentences and numerical information extracted from images using OCR technology.

[0531] "Learning tendency" refers to a characteristic such as how a user tends to study, for example, whether they prefer visual information or auditory information.

[0532] "Personality traits" refer to the characteristics of a user's personality and behavioral patterns, such as the level of concentration and communication ability.

[0533] "Homework" refers to study tasks given to users on a daily basis, including questions and materials automatically generated and provided by the system.

[0534] A "quiz" is a test that can be completed in a short amount of time to check the level of understanding of daily learning.

[0535] "Regular exams" refer to tests conducted at regular intervals to assess overall understanding and progress.

[0536] A "training program" refers to specific training content and tasks that are generated according to the characteristics of a user in order to improve the user's work efficiency.

[0537] "Output score" refers to a numerical indicator that evaluates the results of a user's learning or work.

[0538] "Personality score" refers to a numerical index of a user's personality traits and behavioral patterns.

[0539] "Career path" refers to the direction and plans of students and staff regarding their future studies and careers.

[0540] "Employment destinations" refers to companies or workplaces where students and staff may find employment in the future.

[0541] "Work methods" refers to the tasks and procedures performed by staff, including efficient methods that the system optimizes and suggests.

[0542] "Instructor" refers to a teacher, coach, or person in a leadership position engaged in education or training.

[0543] "Instruction method" refers to the specific method or approach that an instructor uses to educate or train a user.

[0544] The term "system" refers to a series of devices and programs realized by combining the above means.

[0545] This invention is a system for improving the efficiency of individual training in education systems and factories. In particular, factory robots can analyze the work tendencies and characteristics of staff and automatically generate training programs based on the results.

[0546] Hardware and software used

[0547] Camera: Used to film staff working.

[0548] Server: Responsible for central information processing such as image analysis, data processing, and evaluation.

[0549] Smart glasses or terminals: Used as a display device to provide training programs and feedback to staff.

[0550] OCR technology: Used to extract text data from images. Specifically, we will use pytesseract as an example.

[0551] NLP algorithms: used to analyze text data and extract learning trends and personality traits. In this example, we use GPT-4.

[0552] Generative AI models (such as GPT-4): Used to automatically generate training programs based on learning tendencies and personality traits.

[0553] Processing steps

[0554] First, staff members use cameras to take pictures of their work and upload the images to a server, which then converts the images into text using OCR technology. The converted text is then analyzed using AI-based natural language processing (NLP) algorithms to extract learning trends and personality traits.

[0555] Next, the server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. The generated program is distributed to staff via smart glasses or other devices, and staff use it to train. The training results and answers are sent back to the server, which evaluates them using an automatic scoring system. An output score and personality score are calculated from the evaluation results.

[0556] The server then proposes optimal work methods and division of roles for each staff member based on the calculated score. This information is provided to the staff member and instructor via their terminal, and the instructor then decides on the optimal training policy for the staff member.

[0557] Specific examples

[0558] For example, let's say a new factory worker is learning how to operate equipment. The worker uses a camera to take a picture of themselves performing the operation and sends the image to a server. The server then uses OCR technology to convert the image data into text data, and runs an NLP algorithm on that data to determine whether the new worker is quick to understand the procedure and has a high level of concentration.

[0559] The server uses the results of this evaluation to automatically generate an appropriate training program for new staff using a generative AI model (GPT-4). For example, it generates training materials detailing operation procedures and specific practical tasks, which it provides to the new staff through smart glasses. The server then sends the new staff's performance and answers to the server, which evaluates them using an automatic scoring system. Based on the results, it calculates an output score and personality score and suggests optimal work methods and role allocations to the new staff.

[0560] Example prompt statement

[0561] An example of a prompt to enter into GPT-4:

[0562] Generate efficient, personalized training programs based on staff analytics including:

[0563] Learning Propensity: Staff members pick up work procedures quickly and have good attention to detail.

[0564] Personality traits: Highly focused and collaborative.

[0565] Analysis results:

[0566] Staff have proven quicker to understand how to operate the new equipment and work better with other staff.

[0567] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0568] Step 1:

[0569] Users use a camera to take pictures of their submissions and upload them to a server, which records the user's operation procedures and work.

[0570] Input: An image of the submission taken with a camera

[0571] Output: Image data uploaded to the server

[0572] Step 2:

[0573] The server converts the uploaded image data into text data using OCR technology (e.g., pytesseract). Through OCR processing, character information in the image is extracted as text.

[0574] Input: Uploaded image data

[0575] Data processing: Conversion to text data using OCR technology

[0576] Output: Converted text data

[0577] Step 3:

[0578] The server then analyzes the converted text data using a natural language processing (NLP) algorithm (e.g., GPT-4) to extract the user's learning tendencies and personality traits. The analysis results include the user's strengths and learning tendencies.

[0579] Input: Converted text data

[0580] Data Computing: Analyzing Learning Tendencies and Personality Traits with NLP Algorithms

[0581] Output: Extracted learning tendencies and personality traits

[0582] Step 4:

[0583] The server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. Specifically, it inputs a prompt sentence to the generative AI model and generates appropriate training content based on it.

[0584] Input: extracted learning tendencies and personality traits

[0585] Data Computing: Generating Training Programs with Generative AI Models

[0586] Output: The generated training program

[0587] Step 5:

[0588] The terminal delivers the generated training program to the user, which includes displaying the training content via smart glasses or other display devices.

[0589] Input: Generated training program

[0590] Action: Displaying a training program on the device

[0591] Output: A user-visible training program

[0592] Step 6:

[0593] The user follows the training program provided through the terminal and reports the progress and results to the server. In this step, the user's answers and progress are sent from the terminal to the server.

[0594] Input: User's actions and answers

[0595] Operation: User sends progress and answers to the server

[0596] Output: Implementation status and answer data sent to the server

[0597] Step 7:

[0598] The server evaluates the received data using an automated scoring system, and calculates an output score and personality score as the evaluation results.

[0599] Input: User's performance and answer data

[0600] Data calculations: Evaluation by an automated scoring system

[0601] Output: Calculated output scores and personality scores

[0602] Step 8:

[0603] The server then proposes optimal work methods and role allocations to the user based on the calculated scores, and provides the proposed information to the user and instructor via their terminals.

[0604] Input: Output scores and personality scores

[0605] Data calculation: Optimizing work methods and division of roles

[0606] Output: optimal recommendations to users and instructors

[0607] Step 9:

[0608] The instructor determines an instruction method for the user based on the received suggestion information and provides appropriate feedback to the user.

[0609] Input: Best Suggestion Information

[0610] Action: Deciding how to teach and providing feedback to the user

[0611] Output: Specific instruction and feedback to the user

[0612] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0613] This invention is a system for improving the efficiency of individual instruction in educational systems, and by combining it with an emotion engine that recognizes user emotions, it achieves even more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are then used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine recognizes students' emotions and provides feedback to the educational content.

[0614] Image loading and analysis of submissions

[0615] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[0616] Introducing the Emotion Engine

[0617] When a user (student) submits an assignment or solves a problem, the device uses a camera and microphone to capture the user's facial expression and voice data. This data is analyzed in real time by an emotion engine to recognize the user's emotional state. For example, facial expression recognition technology and voice analysis technology are used to detect the emotions (e.g., joy, anger, sadness, surprise) that the user is showing.

[0618] Homework, quiz, and exam question generation

[0619] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on the identified learning tendencies and personality traits, as well as the recognized emotional state. For example, if a user shows signs of nervousness or anxiety about a question, the server can generate questions that take their emotional state into account, providing questions of a level of difficulty appropriate for that learning situation. The device then distributes the generated questions to the student, who then answers them.

[0620] Answer analysis and scoring

[0621] Students' answers are sent to the server via their terminal. The server evaluates the received answer data using an automatic scoring system. Grade data is generated as the evaluation results. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This quantifies each student's learning outcomes and personality traits, and also expresses their emotional data. The detailed data obtained in this way is used for analysis.

[0622] Career and employment suggestions

[0623] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[0624] Teacher feedback and instructional guide

[0625] Based on the student's score and analysis results, the server displays the personality assessment results (e.g., MBTI) to the teacher and provides guidance on how to teach based on the results. This feedback information is generated taking into account emotional data. For example, it may include how to respond to specific emotional states and how to provide effective feedback. This information is provided to the teacher via their device, allowing them to determine the best teaching method for the student.

[0626] Example of a sample

[0627] For example, when a student submits a math assignment, the device takes a photo of the assignment and the emotion engine analyzes the student's facial expressions and voice. If the user is nervous, appropriate assistance is provided. The server uses OCR technology to convert the data into text and analyzes the data to extract learning tendencies and personality traits. Based on this, the device generates questions and sends them to the student, who then answers them. Once the answers are sent to the server and graded, the output score and personality score are updated. The student's emotional state is also recorded and used to optimize the learning process. Teachers are then advised on teaching methods based on the student's emotions and scores, improving the quality of individualized instruction.

[0628] As a result, the present invention makes it possible to build a system that provides an educational experience optimized for each individual student, reduces the burden on teachers, and provides effective instruction. Furthermore, by combining it with emotion recognition technology, the quality and effectiveness of learning can be further improved.

[0629] The processing flow will be explained below.

[0630] Step 1:

[0631] Students take photos of their own work using their smartphones or tablets. The devices store the images of the work along with the students' facial expressions and voice data, and upload them to the server.

[0632] Step 2:

[0633] The server receives the uploaded image data and facial expression / voice data, and first extracts text data from the image data using OCR technology.

[0634] Step 3:

[0635] The server analyzes the extracted text data using natural language processing (NLP) algorithms, and the analysis results identify the student's learning tendencies and personality traits.

[0636] Step 4:

[0637] At the same time, the server's emotion engine uses facial expression recognition and voice analysis technologies to analyze the user's emotional state, for example, determining whether the user is nervous or relaxed when taking the image.

[0638] Step 5:

[0639] The server comprehensively evaluates the identified learning tendencies, personality traits, and emotional states, and based on that, generates personalized homework, quizzes, and regular exam questions using a generative AI model (e.g., GPT-4).

[0640] Step 6:

[0641] The server stores the generated questions in a database and then distributes them to the terminals, which then display the questions to the students.

[0642] Step 7:

[0643] The user (student) works on the problem and inputs the answer into the terminal, which then sends the answer data to the server.

[0644] Step 8:

[0645] The server evaluates the received answer data using an automatic scoring system, generates performance data, and calculates an output score and personality score based on the answer content and performance data.

[0646] Step 9:

[0647] The server analyzes the calculated output score and personality score and generates information to suggest the student's optimal career path and employment based on the results, taking into account the user's emotional state.

[0648] Step 10:

[0649] The server distributes information on career and employment suggestions to the terminal, and the terminal displays the suggestions to the students and teachers.

[0650] Step 11:

[0651] The server displays the personality assessment results and emotional state based on the student's scores to the teacher, and generates guidance information for teaching methods based on the results.

[0652] Step 12:

[0653] The server then distributes the generated instructional method guide information to the terminal, which then displays it to the teacher.The teacher then decides on a teaching policy based on this information and provides individual instruction to the student.

[0654] Example 2

[0655] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0656] Traditional educational systems have faced the challenge of making it difficult to provide instruction that fully takes into account each student's learning tendencies and personality traits. Furthermore, there was a lack of a way to grasp students' emotional states in real time and reflect them in the learning content. As a result, the quality of individualized instruction declined, hindering students' motivation to learn and academic improvement.

[0657] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading an image of a submitted work and converting it into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for acquiring the user's facial expression and voice data and analyzing it to recognize their emotional state, and means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies, personality traits, and emotional state. This makes it possible to provide learning content optimized for each student and take into account the student's emotional state, thereby improving learning effectiveness and enhancing the quality of individualized instruction.

[0658] "Image of submission" means a digital image of handwritten or printed material submitted by a student as part of an assignment.

[0659] The "means for converting into text data" is a mechanism for extracting character information in an image as text data using optical character recognition technology.

[0660] "Means for recognizing learning tendencies and personality traits" is a function that analyzes text data and identifies students' learning patterns and personality traits.

[0661] "Means for acquiring a user's facial expression and voice data and analyzing it to recognize their emotional state" refers to a technology that uses a camera and microphone to collect a user's facial expression and voice and then analyzes it to determine their emotional state.

[0662] The "means for generating daily homework, quiz, and exam questions" is a mechanism for creating individually customized study questions based on learning tendencies, personality traits, and emotional state.

[0663] The "means for calculating output scores and personality scores" is a function that automatically calculates evaluation scores for academic performance and personality traits based on the student's answers and grade data.

[0664] The "means for suggesting career paths and employment opportunities" is a system that suggests future career paths and suitable employment opportunities to students based on their output scores and personality scores.

[0665] "Means for providing guidance on teaching methods" is a system that provides teachers with learning data and personality data on students and advises them on effective teaching methods.

[0666] This invention is a system for improving the efficiency of individual instruction in educational systems, particularly by combining an emotion engine that recognizes user emotions to achieve more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are also used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine can recognize students' emotions and provide feedback to the educational content.

[0667] First, the user (student) takes a picture of the submitted work using a smartphone or tablet. This image is displayed on the device, and after the user confirms it, they upload it to the server. The server then converts the received image data into text data using OCR technology (e.g., Tesseract). At this stage, handwritten or printed characters are digitized and proceed to the next analysis step.

[0668] The server analyzes the converted text data using an automatic analysis algorithm (e.g., SpaCy) to extract learning trends and personality traits. Additionally, when users (students) submit their work or solve problems, the device uses a camera and microphone to capture their facial expressions and voice data. This data is sent to the server in real time and analyzed using an emotion engine (e.g., Microsoft Azure Emotion Recognition API).

[0669] The server then uses a generative AI model (e.g., GPT-4) to generate questions for daily homework, quizzes, and regular exams based on the learning tendencies, personality traits, and emotional states. For example, example prompts include:

[0670] "This student tends to be more theoretical in his learning, and in recent exams he has struggled with understanding applied mathematics questions. His current emotional state is one of tension and anxiety. Please create applied mathematics questions of medium difficulty for this student to deepen his understanding. Please include illustrations in the questions and provide thorough explanations."

[0671] The generated questions are sent to students via their devices, who then answer them. The answers are then sent to a server via the device, where they are evaluated using an automated scoring system (e.g., OpenAI Codex). Based on this, output scores and personality scores are calculated, and the server stores these data.

[0672] Furthermore, the server will make career and employment recommendations based on the analysis results. The recommendations are calculated using statistical models to determine the optimal career path and are provided to students and teachers via their devices. The recommendations are best suited to the user's abilities and characteristics.

[0673] Finally, the server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on how to teach based on the results. This information is generated taking into account emotional data and is provided to the teacher via their terminal.

[0674] As a result, a system will be created that improves the quality of education and makes individualized instruction more efficient through learning programs optimized for each student and feedback based on their emotions.

[0675] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0676] Step 1: Take and upload an image of your submission

[0677] The user (student) takes a picture of the submitted work using a smartphone or tablet, and the captured image is displayed on the device screen.

[0678] The terminal uploads the image confirmed by the user to the server.

[0679] Input: A user-taken image of the submission.

[0680] Output: Image data of the submission uploaded to the server.

[0681] Step 2: Convert image data to text using OCR technology

[0682] The server converts the image data of the received submission into text data using OCR technology (e.g., Tesseract).

[0683] The server temporarily stores the converted text data and uses it for subsequent analysis steps.

[0684] Input: Image data of the submission.

[0685] Output: Text data.

[0686] What it does: Converts handwritten or printed text in an image into digital text.

[0687] Step 3: Analysis of learning tendencies and personality traits

[0688] The server analyzes the converted text data using a natural language processing algorithm (e.g., SpaCy).

[0689] The server extracts learning tendencies and personality traits from the text data and stores this information in a database.

[0690] Input: Text data converted by OCR.

[0691] Output: Data on learning tendencies and personality traits.

[0692] Specific Actions: Through text analysis, students' tendency to excel at logical thinking is revealed.

[0693] Step 4: Acquire and analyze emotion data

[0694] When a user (student) solves a problem, the device uses a camera and microphone to capture the user's facial expressions and voice data.

[0695] The terminal transmits the acquired emotion data to the server in real time.

[0696] The server analyzes the data using an emotion engine (e.g., Microsoft Azure Emotion Recognition API) to recognize the user's emotional state.

[0697] Input: User's facial and voice data.

[0698] Output: Data of the user's emotional state.

[0699] Specific operation: Detects emotions such as tension, anxiety, and joy in real time from the user's facial expressions and tone of voice.

[0700] Step 5: Generate homework, quizzes, and exam questions

[0701] The server generates personalized questions using a generative AI model (e.g., GPT-4) based on learning tendencies, personality traits, and emotional states.

[0702] The server automatically generates questions by inputting prompt sentences into a generative AI model.

[0703] The server sends the generated questions to the terminal.

[0704] The terminal distributes the generated questions to the students, who then answer them.

[0705] Input: learning tendencies, personality traits, and emotional state data.

[0706] Output: Personalized homework, quiz and regular exam questions.

[0707] Specific operation: The following prompt is input into the generative AI model to generate a problem: "This student has a theoretical learning tendency, and in recent exams, he has tended to struggle with understanding applied mathematics problems. His current emotional state is showing tension and anxiety. Please create an applied mathematics problem of medium difficulty for this student to deepen his understanding. Please include illustrations and provide detailed explanations in the problem text."

[0708] Step 6: Upload and grade your answers

[0709] The user (student) enters the answer into the terminal and sends it to the server.

[0710] The server evaluates the received answer data using an automatic scoring system (e.g., OpenAI Codex) and generates score data.

[0711] Input: The answer data entered by the student.

[0712] Output: Evaluated performance data.

[0713] Specific operation: Based on the answers, the system determines whether the answers are correct or incorrect, scores them, and automatically generates a grade.

[0714] Step 7: Calculating output and personality scores

[0715] The server automatically calculates the output score and personality score based on the answers and performance data.

[0716] Input: Answers and performance data.

[0717] Output: Output scores and personality scores.

[0718] What it does: Evaluate students' understanding and personality traits and calculate corresponding scores.

[0719] Step 8: Career and employment suggestions

[0720] The server suggests optimal career paths and employment opportunities based on the calculated output score and personality score.

[0721] The server sends the suggestions to the terminals and provides them to students and teachers.

[0722] Inputs: Output scores and personality scores.

[0723] Output: Career and employment suggestions.

[0724] What it does: Uses statistical models to suggest appropriate career paths for students, such as "go to a science and engineering university."

[0725] Step 9: Teacher feedback and instructional guide

[0726] Based on the student's scores and analysis results, the server generates personality assessment results (e.g., MBTI) and information to guide teachers on teaching methods.

[0727] The terminal provides this feedback information to the teacher, who then uses it to determine the teaching policy.

[0728] Input: Student scores and analysis results.

[0729] Output: Personality assessment results and guidance information on teaching methods.

[0730] Specific action: The teacher's device displays information such as, "Student A is analytical and introverted. Rather than praising him, let's suggest a problem-solving approach."

[0731] (Application example 2)

[0732] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0733] Individualized instruction is extremely important in the modern education system, but providing appropriate learning content to each student and maximizing learning outcomes requires a great deal of time and effort. Furthermore, education that ignores students' emotional states can reduce effectiveness, so there is a need to provide personalized learning experiences using emotion recognition technology. Furthermore, the introduction of virtual classrooms using virtual reality technology is necessary to provide students with new learning methods.

[0734] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0735] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies and personality traits, means for distributing the generated questions to students, means for evaluating students' answers and calculating output scores and personality scores, means for suggesting optimal career paths and employment opportunities for students based on the output scores and personality scores, means for displaying personality assessment results to teachers based on the students' scores and providing instructional guidance, an emotion engine for recognizing students' emotional states using facial expressions and voice data when solving problems, means for personalizing educational content based on the emotional states and generating optimal questions for students, and means for providing a personalized learning experience in a virtual classroom using a virtual reality head-mounted display. This makes it possible to maximize students' learning outcomes, reduce the burden on teachers, and provide effective education tailored to their emotional states.

[0736] "Submissions" refer to assignments and reports that students submit as the result of their studies.

[0737] An "image" is visual information captured as digital data by a camera or scanner.

[0738] "Text data" refers to data that is treated as character information.

[0739] "Learning tendency" refers to the tendency that indicates how a student will effectively proceed with their studies.

[0740] "Personality traits" refer to a student's personality and behavioral characteristics, and are information used to individualize educational methods.

[0741] "Homework," "quizzes," and "regular exams" are questions used to evaluate students' learning status.

[0742] "Delivery" refers to sending the generated questions and feedback to students.

[0743] An "answer" is a solution submitted by a student to a problem.

[0744] "Assessment" means marking students' answers and measuring their performance.

[0745] An "output score" is a score that quantitatively evaluates learning outcomes.

[0746] A "personality score" is a numerical score that represents a student's personality traits.

[0747] "Career path" refers to the direction of education or career that a student should pursue in the future.

[0748] "Employment" refers to the workplace where students plan to work after graduation.

[0749] "Instructional methods" refer to the educational techniques that teachers use with students.

[0750] "Emotional state" refers to the student's emotional or psychological state.

[0751] An "emotion engine" is a technology for recognizing emotional states from data such as facial expressions and voice.

[0752] "Personalization" means customizing something to meet individual needs.

[0753] A "virtual reality head-mounted display" is a visual and auditory device worn to experience a virtual reality space.

[0754] A "virtual classroom" is a virtual educational space recreated using virtual reality technology.

[0755] This invention is a system for individualized instruction that recognizes students' emotional states and uses that information to achieve more sophisticated personalization. To achieve this, the invention mainly uses the following hardware and software:

[0756] Hardware used

[0757] 1. VR Head-Mounted Display (HMD): Used to provide learning experiences within a virtual classroom.

[0758] 2. 360-degree camera: Used by students to film their submissions.

[0759] 3. Microphone: Used to collect student voice data.

[0760] Software used

[0761] 1. OCR technology: Using Google Cloud Vision API, we convert the image of the submitted document into text data.

[0762] 2. Natural Language Processing (NLP): Analyze transformed text data using spaCy and Hugging Face Transformers.

[0763] 3. Emotion recognition technology: Using OpenCV and DeepFace, we identify students' emotional states from facial expressions and voice data.

[0764] 4. Generative AI model: Uses OpenAI GPT-4 to generate personalized homework and test questions.

[0765] 5. Server Technology: We use AWS EC2 and S3 for data storage and processing.

[0766] How to carry out the invention

[0767] 1. Importing and analyzing submitted images

[0768] Students wear a VR head-mounted display and follow instructions in the virtual classroom to take photos of their submissions using a 360-degree camera and upload the images to the server, which then uses the Google Cloud Vision API to convert the image data into text using OCR processing.

[0769] 2. Introducing the Emotion Engine

[0770] While solving problems, the device (camera and microphone of the VR head-mounted display) collects facial and voice data from the student, and uses OpenCV and DeepFace to recognize the student's emotional state in real time. This information is sent to a server and used to personalize the educational content.

[0771] 3. Homework, quiz, and regular exam question generation

[0772] The server uses a generative AI model (GPT-4) to automatically generate questions optimized for each student based on their analyzed learning tendencies and personality traits, as well as their recognized emotional state. This allows a student showing signs of nervousness or anxiety to be provided with appropriate questions to alleviate that state. The generated questions are then distributed to students within the virtual classroom.

[0773] 4. Answer analysis and scoring

[0774] Students enter their answers to questions in the VR space and send them to the server, which then evaluates the answers using an automatic scoring system to calculate output and personality scores.

[0775] 5. Career and employment suggestions

[0776] Based on the calculated scores, the server proposes the best career paths and employment opportunities for students and provides this information within the virtual classroom, allowing students to choose a career path that suits their own abilities and characteristics.

[0777] 6. Teacher feedback and instructional guide

[0778] Teachers can also sit in the same virtual classroom and receive feedback on teaching methods based on students' scores and emotional states, helping teachers to determine the best teaching strategy for each individual student.

[0779] Specific examples

[0780] For example, when a student submits a math assignment, they follow instructions in the VR space and take a photo of the assignment with a 360-degree camera. The server converts the image into text data using OCR technology and then analyzes it with an NLP algorithm to identify learning trends. While the student is solving the problem, an emotion engine analyzes their facial expressions and voice to detect their level of tension. A generative AI model automatically generates a problem of a level of difficulty appropriate for this state and presents it to the student in the VR space. After solving the problem, the answer is automatically graded, and the output score and personality score are updated.

[0781] Example prompt sentence:

[0782] Question: This student has high ability in math but is prone to anxiety before tests. Generate problems designed to help him overcome his anxiety.

[0783] In this way, this invention is a system that can provide each student with an optimal educational experience, reduce the burden on teachers, and maximize learning effectiveness.

[0784] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0785] Step 1:

[0786] The user (student) wears a VR head-mounted display and takes a picture of the submitted work with a 360-degree camera in the virtual classroom. The input is a physical image of the submitted work, and the output is digital image data.

[0787] Step 2:

[0788] The terminal uploads the image data of the photographed submission to the server. The input is digital image data, and the output is image data stored on the server. The server receives and stores this image data.

[0789] Step 3:

[0790] The server calls the Google Cloud Vision API, performs OCR on the uploaded image data, and converts it into text data. The input is the image data of the submitted item, and the output is text data.

[0791] Step 4:

[0792] The server analyzes the converted text data using spaCy and Hugging Face Transformers to extract learning tendencies and personality traits. The input is text data, and the output is analyzed learning tendencies and personality traits data.

[0793] Step 5:

[0794] When the user solves a problem, the device collects the user's facial expression and voice data in real time using the camera and microphone of the VR head-mounted display and sends it to the server. The input is the user's facial expression and voice data, and the output is the emotional data sent to the server.

[0795] Step 6:

[0796] The server analyzes the emotional data using OpenCV and DeepFace to recognize the user's current emotional state. The input is facial expression and voice data, and the output is emotional state data analyzed in real time.

[0797] Step 7:

[0798] The server uses GPT-4 to generate personalized homework and test questions based on the analyzed learning tendencies, personality traits, and emotional state. The generated questions are appropriate for the user's learning situation and emotional state. The input is learning tendency data, personality trait data, and emotional state data, and the output is personalized questions.

[0799] Step 8:

[0800] The server delivers the generated questions to the user's VR head-mounted display. The input is a personalized question, and the output is a question displayed in the user's VR classroom.

[0801] Step 9:

[0802] The user answers questions in the VR classroom and sends the answer data to the server. The input is the user's answer data, and the output is the answer data sent to the server.

[0803] Step 10:

[0804] The server evaluates the received answer data using an automatic scoring system and calculates an output score and a personality score. The input is the user's answer data, and the output is the output score and the personality score.

[0805] Step 11:

[0806] Based on the calculated scores, the server uses a generative AI model to suggest optimal career paths and employment opportunities for users. The inputs are the output score and personality score, and the output is career and employment suggestions.

[0807] Step 12:

[0808] The server also provides teachers with feedback on teaching methods based on the user's score and emotional state. The input is score data and emotional state data, and the output is a teaching guide for the teacher.

[0809] Step 13:

[0810] The device displays the generated teaching guide to the teacher in the VR space, and the teacher then provides optimal instruction to the user based on it. The input is the teaching guide, and the output is the display information to the teacher.

[0811] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0812] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0813] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0814] [Third embodiment]

[0815] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0816] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0817] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0818] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0819] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0820] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0821] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0822] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0823] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0824] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0825] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0826] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0827] This invention is a system for improving the efficiency of individualized instruction in educational systems. The system reads images of students' submitted work, analyzes their learning tendencies and personality traits from the images, automatically generates personalized homework, quizzes, and regular exam questions based on the results, and evaluates the students' answers to calculate output scores and personality scores. These scores are used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods.

[0828] Image loading and analysis of submissions

[0829] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[0830] Homework, quiz, and exam question generation

[0831] The server automatically generates questions for daily homework, quizzes, and regular exams using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. This enables each student to receive optimal learning content. The device delivers the generated questions to the student, who then answers them.

[0832] Answer analysis and scoring

[0833] Students' answers are sent to the server via their devices. The server evaluates the answers using an automatic scoring system and generates grade data. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This allows each student's learning outcomes and personality traits to be expressed numerically.

[0834] Career and employment suggestions

[0835] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[0836] Teacher feedback and instructional guide

[0837] The server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on teaching methods based on the results. This information is provided to the teacher via their device, and the teacher can use it to determine the best teaching policy for the student.

[0838] Example of a sample

[0839] For example, when a student submits a mathematics assignment, the device takes a photo of the assignment and sends it to the server. The server then uses OCR technology to convert it into text data and analyzes the text, identifying that the student has strengths in mathematics. Based on this information, the server uses a generative AI model to create a challenging mathematics assignment for the student and distributes it to the device. The student then works on the assignment and uploads their answers back to the server via their device. The answers are graded, and the student's output score and personality score are updated. As a result, the student is recommended a career path specializing in mathematics, and the teacher is provided with appropriate teaching methods (e.g., in-depth explanations or additional challenging assignments) via the device.

[0840] As a result, the present invention makes it possible to build a system that provides an educational experience that is optimized for each individual student and realizes effective instruction while reducing the burden on teachers.

[0841] The processing flow will be explained below.

[0842] Step 1:

[0843] Students take photos of their own work using their smartphones or tablets, which then save the image data and upload it to the server.

[0844] Step 2:

[0845] The server receives the uploaded image data and uses OCR technology to extract text data from the image data.

[0846] Step 3:

[0847] The server then analyzes the extracted text data using natural language processing (NLP) algorithms, and the results of the analysis identify the student's learning tendencies and personality traits.

[0848] Step 4:

[0849] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on identified learning tendencies and personality traits.

[0850] Step 5:

[0851] The server stores the generated questions in a database, and then distributes the stored questions to the device.

[0852] Step 6:

[0853] The terminal displays the delivered questions to the student. The user (student) works on the questions and sends the answers to the server via the terminal.

[0854] Step 7:

[0855] The server evaluates the received answer data with an automatic scoring system, and generates score data as the evaluation result.

[0856] Step 8:

[0857] The server calculates an output score and a personality score based on the generated performance data.

[0858] Step 9:

[0859] The server analyzes the calculated output score and personality score and generates information to suggest the most suitable career path and employment for the student.

[0860] Step 10:

[0861] The server distributes career and employment suggestion information to the terminal, which then displays the suggestion content to students and teachers.

[0862] Step 11:

[0863] The server displays the personality assessment results based on the student's scores to the teacher, and generates guidance information on teaching methods based on the results.

[0864] Step 12:

[0865] The server distributes the generated teaching method guide information for teachers to the terminal, which displays it to the teacher so that the teacher can use it to decide on a teaching policy.

[0866] Example 1

[0867] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0868] In the conventional education system, it was difficult to provide individualized instruction tailored to each learner's characteristics and progress, making it difficult to achieve effective learning. It also placed a heavy burden on teachers, making it difficult to find appropriate teaching methods. Furthermore, career and occupation suggestions were not based on the learner's characteristics and abilities, so suggestions optimal for each learner were not made.

[0869] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0870] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily learning assignments based on the recognized learning tendencies and personality traits, means for distributing the generated learning assignments to an educational device, means for evaluating the learner's answers and calculating a learning outcome score and personality trait score, means for suggesting the learner's optimal educational course and career based on the learning outcome score and personality trait score, and means for displaying personality assessment results to instructors based on the learner's score and providing instruction methods. This makes it possible to streamline individual instruction, provide the learner with an optimal educational experience, reduce the instructor's burden, and achieve effective learning instruction.

[0871] "Means for reading images of submitted materials and converting them into text data" refers to the function of the educational system to capture information submitted by learners on paper or digital media as images and convert them into digital text data using character recognition technology.

[0872] "Means for analyzing text data and recognizing learning tendencies and personality traits" refers to a function that analyzes converted text data using natural language processing algorithms, etc., to reveal learners' learning behavior and personality traits.

[0873] "Means for generating daily learning tasks based on learning tendencies and personality traits" refers to a function that automatically creates personalized learning tasks using a generative AI model or the like based on analyzed learning tendencies and personality traits.

[0874] "Means for distributing generated learning tasks to educational devices" refers to a function that sends learning tasks generated on a server to a terminal or device used by a learner, allowing the learner to receive and work on them.

[0875] "Means for evaluating learners' answers and calculating learning achievement scores and personality trait scores" refers to a function that evaluates answers submitted by learners using an automatic scoring system and quantifies the level of learning achievement and personality traits.

[0876] "Means for suggesting the most suitable educational course and career for a learner based on learning outcome scores and personality trait scores" refers to a function for recommending appropriate educational courses and future career choices to a learner based on the calculated scores.

[0877] "A means of displaying personality assessment results based on a learner's score to instructors and providing guidance on teaching methods" refers to a function that allows instructors to understand a learner's score and personality traits and, based on that, present information that will enable them to provide the most appropriate teaching method.

[0878] The present invention relates to an educational system that provides personalized learning. This system reads and analyzes images of submitted work to identify learning tendencies and personality traits, and then generates and distributes daily learning assignments based on that information. The following describes how this system is specifically implemented.

[0879] First, the user (student) takes a photo of their homework or assignment using a device such as a smartphone or tablet. This image data is then uploaded to a server using a dedicated application. This application runs on a typical mobile device and is designed to be intuitive for users to operate.

[0880] The server converts uploaded images into text data using optical character recognition (OCR) technology. Specifically, it can use services such as Google Cloud Vision API. This allows paper submissions to be handled as digital text data.

[0881] The server then analyzes this text data with natural language processing (NLP) algorithms, which can use open-source NLP libraries such as spaCy, to extract the student's learning habits and personality traits and provide a personalized assessment.

[0882] Furthermore, based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. This generative AI model automatically generates appropriate problems and tasks when given a specific prompt as input. An example of a specific prompt is, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[0883] The generated learning assignments are then sent to the user's (student's) device via a dedicated application. The student works on the assignments and enters their answers. These answer data are also uploaded to the server.

[0884] The server evaluates the answers submitted by the students using an automated scoring system. For example, an automated scoring tool such as the Grammarly API can be used here. Based on the graded answers, a learning outcome score and personality trait score are calculated. This quantifies the student's learning progress and characteristics and is used for the next step.

[0885] Finally, the server analyzes the calculated scores and, based on the results, recommends the most suitable educational course or career for the student. These recommendations are based on past data and statistical models, allowing for more accurate and appropriate recommendations. These recommendations are then communicated to students and teachers via a dedicated application.

[0886] The server also has a function to provide teachers with personality assessment results and guidance on teaching methods, allowing them to provide effective instruction tailored to the characteristics of each student.

[0887] This invention will enable personalized education, which is expected to improve learner satisfaction and learning outcomes. It will also reduce the burden on teachers and provide an environment where they can provide effective instruction.

[0888] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0889] Step 1:

[0890] The user (student) takes a photo of their homework or assignments using a smartphone or tablet. The input is the handwritten assignment, and the output is digital image data. Specifically, the user opens the camera app and takes a photo of the homework page. Care is taken to ensure the image is clear.

[0891] Step 2:

[0892] Users upload the images they have taken to the server using a dedicated application. The input is the captured image data, and the output is the image file sent to the server. Specifically, the user opens the application, presses the "Upload" button, selects the captured image, and sends it.

[0893] Step 3:

[0894] The server converts the uploaded image into text data using optical character recognition (OCR). The input is image data and the output is text data. Specifically, the server calls the Google Cloud Vision API, analyzes the text information in the image, and converts it into text data.

[0895] Step 4:

[0896] The server analyzes the converted text data using natural language processing (NLP) algorithms. The input is text data, and the output is data on learning tendencies and personality traits. Specifically, the server analyzes the text data using the spaCy library to extract frequently occurring errors, learning characteristics, vocabulary usage, and so on.

[0897] Step 5:

[0898] Based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. The input is learning tendency data and personality trait data, and the output is personalized learning tasks. Specifically, the server calls the generative AI model and provides a specific prompt as input. For example, a prompt such as, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[0899] Step 6:

[0900] The device (student's smartphone or tablet) delivers the generated learning assignments to the user through an application. The input is the personalized learning assignment, and the output is the learning assignment displayed on the user's device. Specifically, the device's app receives the new learning assignment and notifies the user through a notification.

[0901] Step 7:

[0902] The user works on the delivered learning tasks and inputs their answers. The input is the learning task answered by the user, and the output is the answer data. Specifically, the user enters their answer in the application's input field and presses the "Submit" button to send it.

[0903] Step 8:

[0904] The user sends the answer to the server via the terminal. The input is the answer data, and the output is the answer data sent to the server. In concrete terms, the answer submitted by the user is automatically uploaded to the server.

[0905] Step 9:

[0906] The server evaluates the answers submitted by students using an automated scoring system. The input is the answer data, and the output is the scoring results and grade data. Specifically, the server uses an automated scoring tool such as the Grammarly API to assign a score to each answer.

[0907] Step 10:

[0908] The server calculates learning outcome scores and personality trait scores based on the grading results. The input is the grading results and grade data, and the output is the learning outcome scores and personality trait scores. Specifically, the server calculates the scores for each assessment item using a statistical model.

[0909] Step 11:

[0910] The server then uses the calculated scores to suggest optimal educational courses and careers for students. The input is the learning outcome score and personality trait score, and the output is the suggested educational course and career information. Specifically, the server generates optimal suggestions by referencing past databases and statistical models.

[0911] Step 12:

[0912] The server distributes the proposals to students and teachers via their terminals. The input is the proposed educational course and career information, and the output is the proposal information displayed on the user terminal and the teacher terminal. Specifically, the server sends new proposal information via the notification system.

[0913] Step 13:

[0914] The server presents personality assessment results and information to teachers that guide teaching strategies. The input is learning outcome scores and personality trait scores, and the output is a teaching guide for teachers. Specifically, the server displays the teaching guide on a dashboard or similar, making it accessible to teachers.

[0915] Step 14:

[0916] The device notifies the teacher of this information and helps them decide on the best teaching strategy. The input is a teaching guide for the teacher, and the output is the teaching strategy notified to the teacher. Specifically, the device application displays a notification and provides a link to access the teaching guide.

[0917] The above is the specific processing flow of this system.

[0918] (Application example 1)

[0919] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0920] To improve work efficiency in the field, personalized training tailored to the characteristics and skills of each staff member is necessary. However, the current system requires too much effort to respond to each individual, making it difficult to provide efficient training programs. Furthermore, there are challenges in analyzing staff abilities and characteristics and proposing appropriate guidance and role allocation based on that.

[0921] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0922] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily homework, quizzes, and regular exam questions and training programs based on the recognized learning tendencies and personality traits, means for distributing the generated questions and programs to users, means for evaluating the user's answers and results and calculating an output score and personality score, means for suggesting the user's optimal career path, employment, and work method based on the output score and personality score, and means for displaying personality assessment results to instructors based on the user's score and providing guidance on teaching methods. This allows for the automatic generation of training programs optimized for individual staff members, improving specific work efficiency and enabling appropriate division of roles.

[0923] "Submission Image" refers to image data that a User takes using a camera or other photographic device and uploads to the System.

[0924] "Text data" refers to character string data such as sentences and numerical information extracted from images using OCR technology.

[0925] "Learning tendency" refers to a characteristic such as how a user tends to study, for example, whether they prefer visual information or auditory information.

[0926] "Personality traits" refer to the characteristics of a user's personality and behavioral patterns, such as the level of concentration and communication ability.

[0927] "Homework" refers to study tasks given to users on a daily basis, including questions and materials automatically generated and provided by the system.

[0928] A "quiz" is a test that can be completed in a short amount of time to check the level of understanding of daily learning.

[0929] "Regular exams" refer to tests conducted at regular intervals to assess overall understanding and progress.

[0930] A "training program" refers to specific training content and tasks that are generated according to the characteristics of a user in order to improve the user's work efficiency.

[0931] "Output score" refers to a numerical indicator that evaluates the results of a user's learning or work.

[0932] "Personality score" refers to a numerical index of a user's personality traits and behavioral patterns.

[0933] "Career path" refers to the direction and plans of students and staff regarding their future studies and careers.

[0934] "Employment destinations" refers to companies or workplaces where students and staff may find employment in the future.

[0935] "Work methods" refers to the tasks and procedures performed by staff, including efficient methods that the system optimizes and suggests.

[0936] "Instructor" refers to a teacher, coach, or person in a leadership position engaged in education or training.

[0937] "Instruction method" refers to the specific method or approach that an instructor uses to educate or train a user.

[0938] The term "system" refers to a series of devices and programs realized by combining the above means.

[0939] This invention is a system for improving the efficiency of individual training in education systems and factories. In particular, factory robots can analyze the work tendencies and characteristics of staff and automatically generate training programs based on the results.

[0940] Hardware and software used

[0941] Camera: Used to film staff working.

[0942] Server: Responsible for central information processing such as image analysis, data processing, and evaluation.

[0943] Smart glasses or terminals: Used as a display device to provide training programs and feedback to staff.

[0944] OCR technology: Used to extract text data from images. Specifically, we will use pytesseract as an example.

[0945] NLP algorithms: used to analyze text data and extract learning trends and personality traits. In this example, we use GPT-4.

[0946] Generative AI models (such as GPT-4): Used to automatically generate training programs based on learning tendencies and personality traits.

[0947] Processing steps

[0948] First, staff members use cameras to take pictures of their work and upload the images to a server, which then converts the images into text using OCR technology. The converted text is then analyzed using AI-based natural language processing (NLP) algorithms to extract learning trends and personality traits.

[0949] Next, the server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. The generated program is distributed to staff via smart glasses or other devices, and staff use it to train. The training results and answers are sent back to the server, which evaluates them using an automatic scoring system. An output score and personality score are calculated from the evaluation results.

[0950] The server then proposes optimal work methods and division of roles for each staff member based on the calculated score. This information is provided to the staff member and instructor via their terminal, and the instructor then decides on the optimal training policy for the staff member.

[0951] Specific examples

[0952] For example, let's say a new factory worker is learning how to operate equipment. The worker uses a camera to take a picture of themselves performing the operation and sends the image to a server. The server then uses OCR technology to convert the image data into text data, and runs an NLP algorithm on that data to determine whether the new worker is quick to understand the procedure and has a high level of concentration.

[0953] The server uses the results of this evaluation to automatically generate an appropriate training program for new staff using a generative AI model (GPT-4). For example, it generates training materials detailing operation procedures and specific practical tasks, which it provides to the new staff through smart glasses. The server then sends the new staff's performance and answers to the server, which evaluates them using an automatic scoring system. Based on the results, it calculates an output score and personality score and suggests optimal work methods and role allocations to the new staff.

[0954] Example prompt statement

[0955] An example of a prompt to enter into GPT-4:

[0956] Generate efficient, personalized training programs based on staff analytics including:

[0957] Learning Propensity: Staff members pick up work procedures quickly and have good attention to detail.

[0958] Personality traits: Highly focused and collaborative.

[0959] Analysis results:

[0960] Staff have proven quicker to understand how to operate the new equipment and work better with other staff.

[0961] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0962] Step 1:

[0963] Users use a camera to take pictures of their submissions and upload them to a server, which records the user's operation procedures and work.

[0964] Input: An image of the submission taken with a camera

[0965] Output: Image data uploaded to the server

[0966] Step 2:

[0967] The server converts the uploaded image data into text data using OCR technology (e.g., pytesseract). Through OCR processing, character information in the image is extracted as text.

[0968] Input: Uploaded image data

[0969] Data processing: Conversion to text data using OCR technology

[0970] Output: Converted text data

[0971] Step 3:

[0972] The server then analyzes the converted text data using a natural language processing (NLP) algorithm (e.g., GPT-4) to extract the user's learning tendencies and personality traits. The analysis results include the user's strengths and learning tendencies.

[0973] Input: Converted text data

[0974] Data Computing: Analyzing Learning Tendencies and Personality Traits with NLP Algorithms

[0975] Output: Extracted learning tendencies and personality traits

[0976] Step 4:

[0977] The server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. Specifically, it inputs a prompt sentence to the generative AI model and generates appropriate training content based on it.

[0978] Input: extracted learning tendencies and personality traits

[0979] Data Computing: Generating Training Programs with Generative AI Models

[0980] Output: The generated training program

[0981] Step 5:

[0982] The terminal delivers the generated training program to the user, which includes displaying the training content via smart glasses or other display devices.

[0983] Input: Generated training program

[0984] Action: Displaying a training program on the device

[0985] Output: A user-visible training program

[0986] Step 6:

[0987] The user follows the training program provided through the terminal and reports the progress and results to the server. In this step, the user's answers and progress are sent from the terminal to the server.

[0988] Input: User's actions and answers

[0989] Operation: User sends progress and answers to the server

[0990] Output: Implementation status and answer data sent to the server

[0991] Step 7:

[0992] The server evaluates the received data using an automated scoring system, and calculates an output score and personality score as the evaluation results.

[0993] Input: User's performance and answer data

[0994] Data calculations: Evaluation by an automated scoring system

[0995] Output: Calculated output scores and personality scores

[0996] Step 8:

[0997] The server then proposes optimal work methods and role allocations to the user based on the calculated scores, and provides the proposed information to the user and instructor via their terminals.

[0998] Input: Output scores and personality scores

[0999] Data calculation: Optimizing work methods and division of roles

[1000] Output: optimal recommendations to users and instructors

[1001] Step 9:

[1002] The instructor determines an instruction method for the user based on the received suggestion information and provides appropriate feedback to the user.

[1003] Input: Best Suggestion Information

[1004] Action: Deciding how to teach and providing feedback to the user

[1005] Output: Specific instruction and feedback to the user

[1006] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1007] This invention is a system for improving the efficiency of individual instruction in educational systems, and by combining it with an emotion engine that recognizes user emotions, it achieves even more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are then used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine recognizes students' emotions and provides feedback to the educational content.

[1008] Image loading and analysis of submissions

[1009] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[1010] Introducing the Emotion Engine

[1011] When a user (student) submits an assignment or solves a problem, the device uses a camera and microphone to capture the user's facial expression and voice data. This data is analyzed in real time by an emotion engine to recognize the user's emotional state. For example, facial expression recognition technology and voice analysis technology are used to detect the emotions (e.g., joy, anger, sadness, surprise) that the user is showing.

[1012] Homework, quiz, and exam question generation

[1013] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on the identified learning tendencies and personality traits, as well as the recognized emotional state. For example, if a user shows signs of nervousness or anxiety about a question, the server can generate questions that take their emotional state into account, providing questions of a level of difficulty appropriate for that learning situation. The device then distributes the generated questions to the student, who then answers them.

[1014] Answer analysis and scoring

[1015] Students' answers are sent to the server via their terminal. The server evaluates the received answer data using an automatic scoring system. Grade data is generated as the evaluation results. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This quantifies each student's learning outcomes and personality traits, and also expresses their emotional data. The detailed data obtained in this way is used for analysis.

[1016] Career and employment suggestions

[1017] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[1018] Teacher feedback and instructional guide

[1019] Based on the student's score and analysis results, the server displays the personality assessment results (e.g., MBTI) to the teacher and provides guidance on how to teach based on the results. This feedback information is generated taking into account emotional data. For example, it may include how to respond to specific emotional states and how to provide effective feedback. This information is provided to the teacher via their device, allowing them to determine the best teaching method for the student.

[1020] Example of a sample

[1021] For example, when a student submits a math assignment, the device takes a photo of the assignment and the emotion engine analyzes the student's facial expressions and voice. If the user is nervous, appropriate assistance is provided. The server uses OCR technology to convert the data into text and analyzes the data to extract learning tendencies and personality traits. Based on this, the device generates questions and sends them to the student, who then answers them. Once the answers are sent to the server and graded, the output score and personality score are updated. The student's emotional state is also recorded and used to optimize the learning process. Teachers are then advised on teaching methods based on the student's emotions and scores, improving the quality of individualized instruction.

[1022] As a result, the present invention makes it possible to build a system that provides an educational experience optimized for each individual student, reduces the burden on teachers, and provides effective instruction. Furthermore, by combining it with emotion recognition technology, the quality and effectiveness of learning can be further improved.

[1023] The processing flow will be explained below.

[1024] Step 1:

[1025] Students take photos of their own work using their smartphones or tablets. The devices store the images of the work along with the students' facial expressions and voice data, and upload them to the server.

[1026] Step 2:

[1027] The server receives the uploaded image data and facial expression / voice data, and first extracts text data from the image data using OCR technology.

[1028] Step 3:

[1029] The server analyzes the extracted text data using natural language processing (NLP) algorithms, and the analysis results identify the student's learning tendencies and personality traits.

[1030] Step 4:

[1031] At the same time, the server's emotion engine uses facial expression recognition and voice analysis technologies to analyze the user's emotional state, for example, determining whether the user is nervous or relaxed when taking the image.

[1032] Step 5:

[1033] The server comprehensively evaluates the identified learning tendencies, personality traits, and emotional states, and based on that, generates personalized homework, quizzes, and regular exam questions using a generative AI model (e.g., GPT-4).

[1034] Step 6:

[1035] The server stores the generated questions in a database and then distributes them to the terminals, which then display the questions to the students.

[1036] Step 7:

[1037] The user (student) works on the problem and inputs the answer into the terminal, which then sends the answer data to the server.

[1038] Step 8:

[1039] The server evaluates the received answer data using an automatic scoring system, generates performance data, and calculates an output score and personality score based on the answer content and performance data.

[1040] Step 9:

[1041] The server analyzes the calculated output score and personality score and generates information to suggest the student's optimal career path and employment based on the results, taking into account the user's emotional state.

[1042] Step 10:

[1043] The server distributes information on career and employment suggestions to the terminal, and the terminal displays the suggestions to the students and teachers.

[1044] Step 11:

[1045] The server displays the personality assessment results and emotional state based on the student's scores to the teacher, and generates guidance information for teaching methods based on the results.

[1046] Step 12:

[1047] The server then distributes the generated instructional method guide information to the terminal, which then displays it to the teacher.The teacher then decides on a teaching policy based on this information and provides individual instruction to the student.

[1048] Example 2

[1049] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] Traditional educational systems have faced the challenge of making it difficult to provide instruction that fully takes into account each student's learning tendencies and personality traits. Furthermore, there was a lack of a way to grasp students' emotional states in real time and reflect them in the learning content. As a result, the quality of individualized instruction declined, hindering students' motivation to learn and academic improvement.

[1051] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading an image of a submitted work and converting it into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for acquiring the user's facial expression and voice data and analyzing it to recognize their emotional state, and means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies, personality traits, and emotional state. This makes it possible to provide learning content optimized for each student and take into account the student's emotional state, thereby improving learning effectiveness and enhancing the quality of individualized instruction.

[1052] "Image of submission" means a digital image of handwritten or printed material submitted by a student as part of an assignment.

[1053] The "means for converting into text data" is a mechanism for extracting character information in an image as text data using optical character recognition technology.

[1054] "Means for recognizing learning tendencies and personality traits" is a function that analyzes text data and identifies students' learning patterns and personality traits.

[1055] "Means for acquiring a user's facial expression and voice data and analyzing it to recognize their emotional state" refers to a technology that uses a camera and microphone to collect a user's facial expression and voice and then analyzes it to determine their emotional state.

[1056] The "means for generating daily homework, quiz, and exam questions" is a mechanism for creating individually customized study questions based on learning tendencies, personality traits, and emotional state.

[1057] The "means for calculating output scores and personality scores" is a function that automatically calculates evaluation scores for academic performance and personality traits based on the student's answers and grade data.

[1058] The "means for suggesting career paths and employment opportunities" is a system that suggests future career paths and suitable employment opportunities to students based on their output scores and personality scores.

[1059] "Means for providing guidance on teaching methods" is a system that provides teachers with learning data and personality data on students and advises them on effective teaching methods.

[1060] This invention is a system for improving the efficiency of individual instruction in educational systems, particularly by combining an emotion engine that recognizes user emotions to achieve more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are also used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine can recognize students' emotions and provide feedback to the educational content.

[1061] First, the user (student) takes a picture of the submitted work using a smartphone or tablet. This image is displayed on the device, and after the user confirms it, they upload it to the server. The server then converts the received image data into text data using OCR technology (e.g., Tesseract). At this stage, handwritten or printed characters are digitized and proceed to the next analysis step.

[1062] The server analyzes the converted text data using an automatic analysis algorithm (e.g., SpaCy) to extract learning trends and personality traits. Additionally, when users (students) submit their work or solve problems, the device uses a camera and microphone to capture their facial expressions and voice data. This data is sent to the server in real time and analyzed using an emotion engine (e.g., Microsoft Azure Emotion Recognition API).

[1063] The server then uses a generative AI model (e.g., GPT-4) to generate questions for daily homework, quizzes, and regular exams based on the learning tendencies, personality traits, and emotional states. For example, example prompts include:

[1064] "This student tends to be more theoretical in his learning, and in recent exams he has struggled with understanding applied mathematics questions. His current emotional state is one of tension and anxiety. Please create applied mathematics questions of medium difficulty for this student to deepen his understanding. Please include illustrations in the questions and provide thorough explanations."

[1065] The generated questions are sent to students via their devices, who then answer them. The answers are then sent to a server via the device, where they are evaluated using an automated scoring system (e.g., OpenAI Codex). Based on this, output scores and personality scores are calculated, and the server stores these data.

[1066] Furthermore, the server will make career and employment recommendations based on the analysis results. The recommendations are calculated using statistical models to determine the optimal career path and are provided to students and teachers via their devices. The recommendations are best suited to the user's abilities and characteristics.

[1067] Finally, the server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on how to teach based on the results. This information is generated taking into account emotional data and is provided to the teacher via their terminal.

[1068] As a result, a system will be created that improves the quality of education and makes individualized instruction more efficient through learning programs optimized for each student and feedback based on their emotions.

[1069] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1070] Step 1: Take and upload an image of your submission

[1071] The user (student) takes a picture of the submitted work using a smartphone or tablet, and the captured image is displayed on the device screen.

[1072] The terminal uploads the image confirmed by the user to the server.

[1073] Input: A user-taken image of the submission.

[1074] Output: Image data of the submission uploaded to the server.

[1075] Step 2: Convert image data to text using OCR technology

[1076] The server converts the image data of the received submission into text data using OCR technology (e.g., Tesseract).

[1077] The server temporarily stores the converted text data and uses it for subsequent analysis steps.

[1078] Input: Image data of the submission.

[1079] Output: Text data.

[1080] What it does: Converts handwritten or printed text in an image into digital text.

[1081] Step 3: Analysis of learning tendencies and personality traits

[1082] The server analyzes the converted text data using a natural language processing algorithm (e.g., SpaCy).

[1083] The server extracts learning tendencies and personality traits from the text data and stores this information in a database.

[1084] Input: Text data converted by OCR.

[1085] Output: Data on learning tendencies and personality traits.

[1086] Specific Actions: Through text analysis, students' tendency to excel at logical thinking is revealed.

[1087] Step 4: Acquire and analyze emotion data

[1088] When a user (student) solves a problem, the device uses a camera and microphone to capture the user's facial expressions and voice data.

[1089] The terminal transmits the acquired emotion data to the server in real time.

[1090] The server analyzes the data using an emotion engine (e.g., Microsoft Azure Emotion Recognition API) to recognize the user's emotional state.

[1091] Input: User's facial and voice data.

[1092] Output: Data of the user's emotional state.

[1093] Specific operation: Detects emotions such as tension, anxiety, and joy in real time from the user's facial expressions and tone of voice.

[1094] Step 5: Generate homework, quizzes, and exam questions

[1095] The server generates personalized questions using a generative AI model (e.g., GPT-4) based on learning tendencies, personality traits, and emotional states.

[1096] The server automatically generates questions by inputting prompt sentences into a generative AI model.

[1097] The server sends the generated questions to the terminal.

[1098] The terminal distributes the generated questions to the students, who then answer them.

[1099] Input: learning tendencies, personality traits, and emotional state data.

[1100] Output: Personalized homework, quiz and regular exam questions.

[1101] Specific operation: The following prompt is input into the generative AI model to generate a problem: "This student has a theoretical learning tendency, and in recent exams, he has tended to struggle with understanding applied mathematics problems. His current emotional state is showing tension and anxiety. Please create an applied mathematics problem of medium difficulty for this student to deepen his understanding. Please include illustrations and provide detailed explanations in the problem text."

[1102] Step 6: Upload and grade your answers

[1103] The user (student) enters the answer into the terminal and sends it to the server.

[1104] The server evaluates the received answer data using an automatic scoring system (e.g., OpenAI Codex) and generates score data.

[1105] Input: The answer data entered by the student.

[1106] Output: Evaluated performance data.

[1107] Specific operation: Based on the answers, the system determines whether the answers are correct or incorrect, scores them, and automatically generates a grade.

[1108] Step 7: Calculating output and personality scores

[1109] The server automatically calculates the output score and personality score based on the answers and performance data.

[1110] Input: Answers and performance data.

[1111] Output: Output scores and personality scores.

[1112] What it does: Evaluate students' understanding and personality traits and calculate corresponding scores.

[1113] Step 8: Career and employment suggestions

[1114] The server suggests optimal career paths and employment opportunities based on the calculated output score and personality score.

[1115] The server sends the suggestions to the terminals and provides them to students and teachers.

[1116] Inputs: Output scores and personality scores.

[1117] Output: Career and employment suggestions.

[1118] What it does: Uses statistical models to suggest appropriate career paths for students, such as "go to a science and engineering university."

[1119] Step 9: Teacher feedback and instructional guide

[1120] Based on the student's scores and analysis results, the server generates personality assessment results (e.g., MBTI) and information to guide teachers on teaching methods.

[1121] The terminal provides this feedback information to the teacher, who then uses it to determine the teaching policy.

[1122] Input: Student scores and analysis results.

[1123] Output: Personality assessment results and guidance information on teaching methods.

[1124] Specific action: The teacher's device displays information such as, "Student A is analytical and introverted. Rather than praising him, let's suggest a problem-solving approach."

[1125] (Application example 2)

[1126] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1127] Individualized instruction is extremely important in the modern education system, but providing appropriate learning content to each student and maximizing learning outcomes requires a great deal of time and effort. Furthermore, education that ignores students' emotional states can reduce effectiveness, so there is a need to provide personalized learning experiences using emotion recognition technology. Furthermore, the introduction of virtual classrooms using virtual reality technology is necessary to provide students with new learning methods.

[1128] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1129] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies and personality traits, means for distributing the generated questions to students, means for evaluating students' answers and calculating output scores and personality scores, means for suggesting optimal career paths and employment opportunities for students based on the output scores and personality scores, means for displaying personality assessment results to teachers based on the students' scores and providing instructional guidance, an emotion engine for recognizing students' emotional states using facial expressions and voice data when solving problems, means for personalizing educational content based on the emotional states and generating optimal questions for students, and means for providing a personalized learning experience in a virtual classroom using a virtual reality head-mounted display. This makes it possible to maximize students' learning outcomes, reduce the burden on teachers, and provide effective education tailored to their emotional states.

[1130] "Submissions" refer to assignments and reports that students submit as the result of their studies.

[1131] An "image" is visual information captured as digital data by a camera or scanner.

[1132] "Text data" refers to data that is treated as character information.

[1133] "Learning tendency" refers to the tendency that indicates how a student will effectively proceed with their studies.

[1134] "Personality traits" refer to a student's personality and behavioral characteristics, and are information used to individualize educational methods.

[1135] "Homework," "quizzes," and "regular exams" are questions used to evaluate students' learning status.

[1136] "Delivery" refers to sending the generated questions and feedback to students.

[1137] An "answer" is a solution submitted by a student to a problem.

[1138] "Assessment" means marking students' answers and measuring their performance.

[1139] An "output score" is a score that quantitatively evaluates learning outcomes.

[1140] A "personality score" is a numerical score that represents a student's personality traits.

[1141] "Career path" refers to the direction of education or career that a student should pursue in the future.

[1142] "Employment" refers to the workplace where students plan to work after graduation.

[1143] "Instructional methods" refer to the educational techniques that teachers use with students.

[1144] "Emotional state" refers to the student's emotional or psychological state.

[1145] An "emotion engine" is a technology for recognizing emotional states from data such as facial expressions and voice.

[1146] "Personalization" means customizing something to meet individual needs.

[1147] A "virtual reality head-mounted display" is a visual and auditory device worn to experience a virtual reality space.

[1148] A "virtual classroom" is a virtual educational space recreated using virtual reality technology.

[1149] This invention is a system for individualized instruction that recognizes students' emotional states and uses that information to achieve more sophisticated personalization. To achieve this, the invention mainly uses the following hardware and software:

[1150] Hardware used

[1151] 1. VR Head-Mounted Display (HMD): Used to provide learning experiences within a virtual classroom.

[1152] 2. 360-degree camera: Used by students to film their submissions.

[1153] 3. Microphone: Used to collect student voice data.

[1154] Software used

[1155] 1. OCR technology: Using Google Cloud Vision API, we convert the image of the submitted document into text data.

[1156] 2. Natural Language Processing (NLP): Analyze transformed text data using spaCy and Hugging Face Transformers.

[1157] 3. Emotion recognition technology: Using OpenCV and DeepFace, we identify students' emotional states from facial expressions and voice data.

[1158] 4. Generative AI model: Uses OpenAI GPT-4 to generate personalized homework and test questions.

[1159] 5. Server Technology: We use AWS EC2 and S3 for data storage and processing.

[1160] How to carry out the invention

[1161] 1. Importing and analyzing submitted images

[1162] Students wear a VR head-mounted display and follow instructions in the virtual classroom to take photos of their submissions using a 360-degree camera and upload the images to the server, which then uses the Google Cloud Vision API to convert the image data into text using OCR processing.

[1163] 2. Introducing the Emotion Engine

[1164] While solving problems, the device (camera and microphone of the VR head-mounted display) collects facial and voice data from the student, and uses OpenCV and DeepFace to recognize the student's emotional state in real time. This information is sent to a server and used to personalize the educational content.

[1165] 3. Homework, quiz, and regular exam question generation

[1166] The server uses a generative AI model (GPT-4) to automatically generate questions optimized for each student based on their analyzed learning tendencies and personality traits, as well as their recognized emotional state. This allows a student showing signs of nervousness or anxiety to be provided with appropriate questions to alleviate that state. The generated questions are then distributed to students within the virtual classroom.

[1167] 4. Answer analysis and scoring

[1168] Students enter their answers to questions in the VR space and send them to the server, which then evaluates the answers using an automatic scoring system to calculate output and personality scores.

[1169] 5. Career and employment suggestions

[1170] Based on the calculated scores, the server proposes the best career paths and employment opportunities for students and provides this information within the virtual classroom, allowing students to choose a career path that suits their own abilities and characteristics.

[1171] 6. Teacher feedback and instructional guide

[1172] Teachers can also sit in the same virtual classroom and receive feedback on teaching methods based on students' scores and emotional states, helping teachers to determine the best teaching strategy for each individual student.

[1173] Specific examples

[1174] For example, when a student submits a math assignment, they follow instructions in the VR space and take a photo of the assignment with a 360-degree camera. The server converts the image into text data using OCR technology and then analyzes it with an NLP algorithm to identify learning trends. While the student is solving the problem, an emotion engine analyzes their facial expressions and voice to detect their level of tension. A generative AI model automatically generates a problem of a level of difficulty appropriate for this state and presents it to the student in the VR space. After solving the problem, the answer is automatically graded, and the output score and personality score are updated.

[1175] Example prompt sentence:

[1176] Question: This student has high ability in math but is prone to anxiety before tests. Generate problems designed to help him overcome his anxiety.

[1177] In this way, this invention is a system that can provide each student with an optimal educational experience, reduce the burden on teachers, and maximize learning effectiveness.

[1178] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1179] Step 1:

[1180] The user (student) wears a VR head-mounted display and takes a picture of the submitted work with a 360-degree camera in the virtual classroom. The input is a physical image of the submitted work, and the output is digital image data.

[1181] Step 2:

[1182] The terminal uploads the image data of the photographed submission to the server. The input is digital image data, and the output is image data stored on the server. The server receives and stores this image data.

[1183] Step 3:

[1184] The server calls the Google Cloud Vision API, performs OCR on the uploaded image data, and converts it into text data. The input is the image data of the submitted item, and the output is text data.

[1185] Step 4:

[1186] The server analyzes the converted text data using spaCy and Hugging Face Transformers to extract learning tendencies and personality traits. The input is text data, and the output is analyzed learning tendencies and personality traits data.

[1187] Step 5:

[1188] When the user solves a problem, the device collects the user's facial expression and voice data in real time using the camera and microphone of the VR head-mounted display and sends it to the server. The input is the user's facial expression and voice data, and the output is the emotional data sent to the server.

[1189] Step 6:

[1190] The server analyzes the emotional data using OpenCV and DeepFace to recognize the user's current emotional state. The input is facial expression and voice data, and the output is emotional state data analyzed in real time.

[1191] Step 7:

[1192] The server uses GPT-4 to generate personalized homework and test questions based on the analyzed learning tendencies, personality traits, and emotional state. The generated questions are appropriate for the user's learning situation and emotional state. The input is learning tendency data, personality trait data, and emotional state data, and the output is personalized questions.

[1193] Step 8:

[1194] The server delivers the generated questions to the user's VR head-mounted display. The input is a personalized question, and the output is a question displayed in the user's VR classroom.

[1195] Step 9:

[1196] The user answers questions in the VR classroom and sends the answer data to the server. The input is the user's answer data, and the output is the answer data sent to the server.

[1197] Step 10:

[1198] The server evaluates the received answer data using an automatic scoring system and calculates an output score and a personality score. The input is the user's answer data, and the output is the output score and the personality score.

[1199] Step 11:

[1200] Based on the calculated scores, the server uses a generative AI model to suggest optimal career paths and employment opportunities for users. The inputs are the output score and personality score, and the output is career and employment suggestions.

[1201] Step 12:

[1202] The server also provides teachers with feedback on teaching methods based on the user's score and emotional state. The input is score data and emotional state data, and the output is a teaching guide for the teacher.

[1203] Step 13:

[1204] The device displays the generated teaching guide to the teacher in the VR space, and the teacher then provides optimal instruction to the user based on it. The input is the teaching guide, and the output is the display information to the teacher.

[1205] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1206] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1207] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1208] [Fourth embodiment]

[1209] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1210] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1211] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1212] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1213] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1215] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1216] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1217] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1218] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1219] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1220] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1221] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] This invention is a system for improving the efficiency of individualized instruction in educational systems. The system reads images of students' submitted work, analyzes their learning tendencies and personality traits from the images, automatically generates personalized homework, quizzes, and regular exam questions based on the results, and evaluates the students' answers to calculate output scores and personality scores. These scores are used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods.

[1223] Image loading and analysis of submissions

[1224] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[1225] Homework, quiz, and exam question generation

[1226] The server automatically generates questions for daily homework, quizzes, and regular exams using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. This enables each student to receive optimal learning content. The device delivers the generated questions to the student, who then answers them.

[1227] Answer analysis and scoring

[1228] Students' answers are sent to the server via their devices. The server evaluates the answers using an automatic scoring system and generates grade data. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This allows each student's learning outcomes and personality traits to be expressed numerically.

[1229] Career and employment suggestions

[1230] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[1231] Teacher feedback and instructional guide

[1232] The server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on teaching methods based on the results. This information is provided to the teacher via their device, and the teacher can use it to determine the best teaching policy for the student.

[1233] Example of a sample

[1234] For example, when a student submits a mathematics assignment, the device takes a photo of the assignment and sends it to the server. The server then uses OCR technology to convert it into text data and analyzes the text, identifying that the student has strengths in mathematics. Based on this information, the server uses a generative AI model to create a challenging mathematics assignment for the student and distributes it to the device. The student then works on the assignment and uploads their answers back to the server via their device. The answers are graded, and the student's output score and personality score are updated. As a result, the student is recommended a career path specializing in mathematics, and the teacher is provided with appropriate teaching methods (e.g., in-depth explanations or additional challenging assignments) via the device.

[1235] As a result, the present invention makes it possible to build a system that provides an educational experience that is optimized for each individual student and realizes effective instruction while reducing the burden on teachers.

[1236] The processing flow will be explained below.

[1237] Step 1:

[1238] Students take photos of their own work using their smartphones or tablets, which then save the image data and upload it to the server.

[1239] Step 2:

[1240] The server receives the uploaded image data and uses OCR technology to extract text data from the image data.

[1241] Step 3:

[1242] The server then analyzes the extracted text data using natural language processing (NLP) algorithms, and the results of the analysis identify the student's learning tendencies and personality traits.

[1243] Step 4:

[1244] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on identified learning tendencies and personality traits.

[1245] Step 5:

[1246] The server stores the generated questions in a database, and then distributes the stored questions to the device.

[1247] Step 6:

[1248] The terminal displays the delivered questions to the student. The user (student) works on the questions and sends the answers to the server via the terminal.

[1249] Step 7:

[1250] The server evaluates the received answer data with an automatic scoring system, and generates score data as the evaluation result.

[1251] Step 8:

[1252] The server calculates an output score and a personality score based on the generated performance data.

[1253] Step 9:

[1254] The server analyzes the calculated output score and personality score and generates information to suggest the most suitable career path and employment for the student.

[1255] Step 10:

[1256] The server distributes career and employment suggestion information to the terminal, which then displays the suggestion content to students and teachers.

[1257] Step 11:

[1258] The server displays the personality assessment results based on the student's scores to the teacher, and generates guidance information on teaching methods based on the results.

[1259] Step 12:

[1260] The server distributes the generated teaching method guide information for teachers to the terminal, which displays it to the teacher so that the teacher can use it to decide on a teaching policy.

[1261] Example 1

[1262] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1263] In the conventional education system, it was difficult to provide individualized instruction tailored to each learner's characteristics and progress, making it difficult to achieve effective learning. It also placed a heavy burden on teachers, making it difficult to find appropriate teaching methods. Furthermore, career and occupation suggestions were not based on the learner's characteristics and abilities, so suggestions optimal for each learner were not made.

[1264] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1265] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily learning assignments based on the recognized learning tendencies and personality traits, means for distributing the generated learning assignments to an educational device, means for evaluating the learner's answers and calculating a learning outcome score and personality trait score, means for suggesting the learner's optimal educational course and career based on the learning outcome score and personality trait score, and means for displaying personality assessment results to instructors based on the learner's score and providing instruction methods. This makes it possible to streamline individual instruction, provide the learner with an optimal educational experience, reduce the instructor's burden, and achieve effective learning instruction.

[1266] "Means for reading images of submitted materials and converting them into text data" refers to the function of the educational system to capture information submitted by learners on paper or digital media as images and convert them into digital text data using character recognition technology.

[1267] "Means for analyzing text data and recognizing learning tendencies and personality traits" refers to a function that analyzes converted text data using natural language processing algorithms, etc., to reveal learners' learning behavior and personality traits.

[1268] "Means for generating daily learning tasks based on learning tendencies and personality traits" refers to a function that automatically creates personalized learning tasks using a generative AI model or the like based on analyzed learning tendencies and personality traits.

[1269] "Means for distributing generated learning tasks to educational devices" refers to a function that sends learning tasks generated on a server to a terminal or device used by a learner, allowing the learner to receive and work on them.

[1270] "Means for evaluating learners' answers and calculating learning achievement scores and personality trait scores" refers to a function that evaluates answers submitted by learners using an automatic scoring system and quantifies the level of learning achievement and personality traits.

[1271] "Means for suggesting the most suitable educational course and career for a learner based on learning outcome scores and personality trait scores" refers to a function for recommending appropriate educational courses and future career choices to a learner based on the calculated scores.

[1272] "A means of displaying personality assessment results based on a learner's score to instructors and providing guidance on teaching methods" refers to a function that allows instructors to understand a learner's score and personality traits and, based on that, present information that will enable them to provide the most appropriate teaching method.

[1273] The present invention relates to an educational system that provides personalized learning. This system reads and analyzes images of submitted work to identify learning tendencies and personality traits, and then generates and distributes daily learning assignments based on that information. The following describes how this system is specifically implemented.

[1274] First, the user (student) takes a photo of their homework or assignment using a device such as a smartphone or tablet. This image data is then uploaded to a server using a dedicated application. This application runs on a typical mobile device and is designed to be intuitive for users to operate.

[1275] The server converts uploaded images into text data using optical character recognition (OCR) technology. Specifically, it can use services such as Google Cloud Vision API. This allows paper submissions to be handled as digital text data.

[1276] The server then analyzes this text data with natural language processing (NLP) algorithms, which can use open-source NLP libraries such as spaCy, to extract the student's learning habits and personality traits and provide a personalized assessment.

[1277] Furthermore, based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. This generative AI model automatically generates appropriate problems and tasks when given a specific prompt as input. An example of a specific prompt is, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[1278] The generated learning assignments are then sent to the user's (student's) device via a dedicated application. The student works on the assignments and enters their answers. These answer data are also uploaded to the server.

[1279] The server evaluates the answers submitted by the students using an automated scoring system. For example, an automated scoring tool such as the Grammarly API can be used here. Based on the graded answers, a learning outcome score and personality trait score are calculated. This quantifies the student's learning progress and characteristics and is used for the next step.

[1280] Finally, the server analyzes the calculated scores and, based on the results, recommends the most suitable educational course or career for the student. These recommendations are based on past data and statistical models, allowing for more accurate and appropriate recommendations. These recommendations are then communicated to students and teachers via a dedicated application.

[1281] The server also has a function to provide teachers with personality assessment results and guidance on teaching methods, allowing them to provide effective instruction tailored to the characteristics of each student.

[1282] This invention will enable personalized education, which is expected to improve learner satisfaction and learning outcomes. It will also reduce the burden on teachers and provide an environment where they can provide effective instruction.

[1283] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1284] Step 1:

[1285] The user (student) takes a photo of their homework or assignments using a smartphone or tablet. The input is the handwritten assignment, and the output is digital image data. Specifically, the user opens the camera app and takes a photo of the homework page. Care is taken to ensure the image is clear.

[1286] Step 2:

[1287] Users upload the images they have taken to the server using a dedicated application. The input is the captured image data, and the output is the image file sent to the server. Specifically, the user opens the application, presses the "Upload" button, selects the captured image, and sends it.

[1288] Step 3:

[1289] The server converts the uploaded image into text data using optical character recognition (OCR). The input is image data and the output is text data. Specifically, the server calls the Google Cloud Vision API, analyzes the text information in the image, and converts it into text data.

[1290] Step 4:

[1291] The server analyzes the converted text data using natural language processing (NLP) algorithms. The input is text data, and the output is data on learning tendencies and personality traits. Specifically, the server analyzes the text data using the spaCy library to extract frequently occurring errors, learning characteristics, vocabulary usage, and so on.

[1292] Step 5:

[1293] Based on these analysis results, the server uses a generative AI model (e.g., GPT-4) to create daily learning tasks. The input is learning tendency data and personality trait data, and the output is personalized learning tasks. Specifically, the server calls the generative AI model and provides a specific prompt as input. For example, a prompt such as, "This student has a high ability to understand mathematics. Please generate the next high-difficulty algebra problem."

[1294] Step 6:

[1295] The device (student's smartphone or tablet) delivers the generated learning assignments to the user through an application. The input is the personalized learning assignment, and the output is the learning assignment displayed on the user's device. Specifically, the device's app receives the new learning assignment and notifies the user through a notification.

[1296] Step 7:

[1297] The user works on the delivered learning tasks and inputs their answers. The input is the learning task answered by the user, and the output is the answer data. Specifically, the user enters their answer in the application's input field and presses the "Submit" button to send it.

[1298] Step 8:

[1299] The user sends the answer to the server via the terminal. The input is the answer data, and the output is the answer data sent to the server. In concrete terms, the answer submitted by the user is automatically uploaded to the server.

[1300] Step 9:

[1301] The server evaluates the answers submitted by students using an automated scoring system. The input is the answer data, and the output is the scoring results and grade data. Specifically, the server uses an automated scoring tool such as the Grammarly API to assign a score to each answer.

[1302] Step 10:

[1303] The server calculates learning outcome scores and personality trait scores based on the grading results. The input is the grading results and grade data, and the output is the learning outcome scores and personality trait scores. Specifically, the server calculates the scores for each assessment item using a statistical model.

[1304] Step 11:

[1305] The server then uses the calculated scores to suggest optimal educational courses and careers for students. The input is the learning outcome score and personality trait score, and the output is the suggested educational course and career information. Specifically, the server generates optimal suggestions by referencing past databases and statistical models.

[1306] Step 12:

[1307] The server distributes the proposals to students and teachers via their terminals. The input is the proposed educational course and career information, and the output is the proposal information displayed on the user terminal and the teacher terminal. Specifically, the server sends new proposal information via the notification system.

[1308] Step 13:

[1309] The server presents personality assessment results and information to teachers that guide teaching strategies. The input is learning outcome scores and personality trait scores, and the output is a teaching guide for teachers. Specifically, the server displays the teaching guide on a dashboard or similar, making it accessible to teachers.

[1310] Step 14:

[1311] The device notifies the teacher of this information and helps them decide on the best teaching strategy. The input is a teaching guide for the teacher, and the output is the teaching strategy notified to the teacher. Specifically, the device application displays a notification and provides a link to access the teaching guide.

[1312] The above is the specific processing flow of this system.

[1313] (Application example 1)

[1314] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1315] To improve work efficiency in the field, personalized training tailored to the characteristics and skills of each staff member is necessary. However, the current system requires too much effort to respond to each individual, making it difficult to provide efficient training programs. Furthermore, there are challenges in analyzing staff abilities and characteristics and proposing appropriate guidance and role allocation based on that.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1317] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating daily homework, quizzes, and regular exam questions and training programs based on the recognized learning tendencies and personality traits, means for distributing the generated questions and programs to users, means for evaluating the user's answers and results and calculating an output score and personality score, means for suggesting the user's optimal career path, employment, and work method based on the output score and personality score, and means for displaying personality assessment results to instructors based on the user's score and providing guidance on teaching methods. This allows for the automatic generation of training programs optimized for individual staff members, improving specific work efficiency and enabling appropriate division of roles.

[1318] "Submission Image" refers to image data that a User takes using a camera or other photographic device and uploads to the System.

[1319] "Text data" refers to character string data such as sentences and numerical information extracted from images using OCR technology.

[1320] "Learning tendency" refers to a characteristic such as how a user tends to study, for example, whether they prefer visual information or auditory information.

[1321] "Personality traits" refer to the characteristics of a user's personality and behavioral patterns, such as the level of concentration and communication ability.

[1322] "Homework" refers to study tasks given to users on a daily basis, including questions and materials automatically generated and provided by the system.

[1323] A "quiz" is a test that can be completed in a short amount of time to check the level of understanding of daily learning.

[1324] "Regular exams" refer to tests conducted at regular intervals to assess overall understanding and progress.

[1325] A "training program" refers to specific training content and tasks that are generated according to the characteristics of a user in order to improve the user's work efficiency.

[1326] "Output score" refers to a numerical indicator that evaluates the results of a user's learning or work.

[1327] "Personality score" refers to a numerical index of a user's personality traits and behavioral patterns.

[1328] "Career path" refers to the direction and plans of students and staff regarding their future studies and careers.

[1329] "Employment destinations" refers to companies or workplaces where students and staff may find employment in the future.

[1330] "Work methods" refers to the tasks and procedures performed by staff, including efficient methods that the system optimizes and suggests.

[1331] "Instructor" refers to a teacher, coach, or person in a leadership position engaged in education or training.

[1332] "Instruction method" refers to the specific method or approach that an instructor uses to educate or train a user.

[1333] The term "system" refers to a series of devices and programs realized by combining the above means.

[1334] This invention is a system for improving the efficiency of individual training in education systems and factories. In particular, factory robots can analyze the work tendencies and characteristics of staff and automatically generate training programs based on the results.

[1335] Hardware and software used

[1336] Camera: Used to film staff working.

[1337] Server: Responsible for central information processing such as image analysis, data processing, and evaluation.

[1338] Smart glasses or terminals: Used as a display device to provide training programs and feedback to staff.

[1339] OCR technology: Used to extract text data from images. Specifically, we will use pytesseract as an example.

[1340] NLP algorithms: used to analyze text data and extract learning trends and personality traits. In this example, we use GPT-4.

[1341] Generative AI models (such as GPT-4): Used to automatically generate training programs based on learning tendencies and personality traits.

[1342] Processing steps

[1343] First, staff members use cameras to take pictures of their work and upload the images to a server, which then converts the images into text using OCR technology. The converted text is then analyzed using AI-based natural language processing (NLP) algorithms to extract learning trends and personality traits.

[1344] Next, the server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. The generated program is distributed to staff via smart glasses or other devices, and staff use it to train. The training results and answers are sent back to the server, which evaluates them using an automatic scoring system. An output score and personality score are calculated from the evaluation results.

[1345] The server then proposes optimal work methods and division of roles for each staff member based on the calculated score. This information is provided to the staff member and instructor via their terminal, and the instructor then decides on the optimal training policy for the staff member.

[1346] Specific examples

[1347] For example, let's say a new factory worker is learning how to operate equipment. The worker uses a camera to take a picture of themselves performing the operation and sends the image to a server. The server then uses OCR technology to convert the image data into text data, and runs an NLP algorithm on that data to determine whether the new worker is quick to understand the procedure and has a high level of concentration.

[1348] The server uses the results of this evaluation to automatically generate an appropriate training program for new staff using a generative AI model (GPT-4). For example, it generates training materials detailing operation procedures and specific practical tasks, which it provides to the new staff through smart glasses. The server then sends the new staff's performance and answers to the server, which evaluates them using an automatic scoring system. Based on the results, it calculates an output score and personality score and suggests optimal work methods and role allocations to the new staff.

[1349] Example prompt statement

[1350] An example of a prompt to enter into GPT-4:

[1351] Generate efficient, personalized training programs based on staff analytics including:

[1352] Learning Propensity: Staff members pick up work procedures quickly and have good attention to detail.

[1353] Personality traits: Highly focused and collaborative.

[1354] Analysis results:

[1355] Staff have proven quicker to understand how to operate the new equipment and work better with other staff.

[1356] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1357] Step 1:

[1358] Users use a camera to take pictures of their submissions and upload them to a server, which records the user's operation procedures and work.

[1359] Input: An image of the submission taken with a camera

[1360] Output: Image data uploaded to the server

[1361] Step 2:

[1362] The server converts the uploaded image data into text data using OCR technology (e.g., pytesseract). Through OCR processing, character information in the image is extracted as text.

[1363] Input: Uploaded image data

[1364] Data processing: Conversion to text data using OCR technology

[1365] Output: Converted text data

[1366] Step 3:

[1367] The server then analyzes the converted text data using a natural language processing (NLP) algorithm (e.g., GPT-4) to extract the user's learning tendencies and personality traits. The analysis results include the user's strengths and learning tendencies.

[1368] Input: Converted text data

[1369] Data Computing: Analyzing Learning Tendencies and Personality Traits with NLP Algorithms

[1370] Output: Extracted learning tendencies and personality traits

[1371] Step 4:

[1372] The server automatically generates a training program using a generative AI model (e.g., GPT-4) based on the extracted learning tendencies and personality traits. Specifically, it inputs a prompt sentence to the generative AI model and generates appropriate training content based on it.

[1373] Input: extracted learning tendencies and personality traits

[1374] Data Computing: Generating Training Programs with Generative AI Models

[1375] Output: The generated training program

[1376] Step 5:

[1377] The terminal delivers the generated training program to the user, which includes displaying the training content via smart glasses or other display devices.

[1378] Input: Generated training program

[1379] Action: Displaying a training program on the device

[1380] Output: A user-visible training program

[1381] Step 6:

[1382] The user follows the training program provided through the terminal and reports the progress and results to the server. In this step, the user's answers and progress are sent from the terminal to the server.

[1383] Input: User's actions and answers

[1384] Operation: User sends progress and answers to the server

[1385] Output: Implementation status and answer data sent to the server

[1386] Step 7:

[1387] The server evaluates the received data using an automated scoring system, and calculates an output score and personality score as the evaluation results.

[1388] Input: User's performance and answer data

[1389] Data calculations: Evaluation by an automated scoring system

[1390] Output: Calculated output scores and personality scores

[1391] Step 8:

[1392] The server then proposes optimal work methods and role allocations to the user based on the calculated scores, and provides the proposed information to the user and instructor via their terminals.

[1393] Input: Output scores and personality scores

[1394] Data calculation: Optimizing work methods and division of roles

[1395] Output: optimal recommendations to users and instructors

[1396] Step 9:

[1397] The instructor determines an instruction method for the user based on the received suggestion information and provides appropriate feedback to the user.

[1398] Input: Best Suggestion Information

[1399] Action: Deciding how to teach and providing feedback to the user

[1400] Output: Specific instruction and feedback to the user

[1401] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1402] This invention is a system for improving the efficiency of individual instruction in educational systems, and by combining it with an emotion engine that recognizes user emotions, it achieves even more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are then used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine recognizes students' emotions and provides feedback to the educational content.

[1403] Image loading and analysis of submissions

[1404] First, the user (student) takes a photo of their submission with a smartphone or tablet and uploads the image to the server. The server then converts this image data into text data using OCR technology. The converted text data is then analyzed using an AI-based natural language processing (NLP) algorithm to extract learning trends and personality traits.

[1405] Introducing the Emotion Engine

[1406] When a user (student) submits an assignment or solves a problem, the device uses a camera and microphone to capture the user's facial expression and voice data. This data is analyzed in real time by an emotion engine to recognize the user's emotional state. For example, facial expression recognition technology and voice analysis technology are used to detect the emotions (e.g., joy, anger, sadness, surprise) that the user is showing.

[1407] Homework, quiz, and exam question generation

[1408] The server uses a generative AI model (e.g., GPT-4) to generate personalized homework, quiz, and exam questions based on the identified learning tendencies and personality traits, as well as the recognized emotional state. For example, if a user shows signs of nervousness or anxiety about a question, the server can generate questions that take their emotional state into account, providing questions of a level of difficulty appropriate for that learning situation. The device then distributes the generated questions to the student, who then answers them.

[1409] Answer analysis and scoring

[1410] Students' answers are sent to the server via their terminal. The server evaluates the received answer data using an automatic scoring system. Grade data is generated as the evaluation results. Furthermore, output scores and personality scores are calculated based on the answers and grade data. This quantifies each student's learning outcomes and personality traits, and also expresses their emotional data. The detailed data obtained in this way is used for analysis.

[1411] Career and employment suggestions

[1412] The server analyzes the calculated output and personality scores and uses the results to suggest the best career and employment options for students. These suggestions are made using past data and statistical models to help students choose a path that best suits their abilities and characteristics. The suggestions are then distributed to students and teachers via their devices.

[1413] Teacher feedback and instructional guide

[1414] Based on the student's score and analysis results, the server displays the personality assessment results (e.g., MBTI) to the teacher and provides guidance on how to teach based on the results. This feedback information is generated taking into account emotional data. For example, it may include how to respond to specific emotional states and how to provide effective feedback. This information is provided to the teacher via their device, allowing them to determine the best teaching method for the student.

[1415] Example of a sample

[1416] For example, when a student submits a math assignment, the device takes a photo of the assignment and the emotion engine analyzes the student's facial expressions and voice. If the user is nervous, appropriate assistance is provided. The server uses OCR technology to convert the data into text and analyzes the data to extract learning tendencies and personality traits. Based on this, the device generates questions and sends them to the student, who then answers them. Once the answers are sent to the server and graded, the output score and personality score are updated. The student's emotional state is also recorded and used to optimize the learning process. Teachers are then advised on teaching methods based on the student's emotions and scores, improving the quality of individualized instruction.

[1417] As a result, the present invention makes it possible to build a system that provides an educational experience optimized for each individual student, reduces the burden on teachers, and provides effective instruction. Furthermore, by combining it with emotion recognition technology, the quality and effectiveness of learning can be further improved.

[1418] The processing flow will be explained below.

[1419] Step 1:

[1420] Students take photos of their own work using their smartphones or tablets. The devices store the images of the work along with the students' facial expressions and voice data, and upload them to the server.

[1421] Step 2:

[1422] The server receives the uploaded image data and facial expression / voice data, and first extracts text data from the image data using OCR technology.

[1423] Step 3:

[1424] The server analyzes the extracted text data using natural language processing (NLP) algorithms, and the analysis results identify the student's learning tendencies and personality traits.

[1425] Step 4:

[1426] At the same time, the server's emotion engine uses facial expression recognition and voice analysis technologies to analyze the user's emotional state, for example, determining whether the user is nervous or relaxed when taking the image.

[1427] Step 5:

[1428] The server comprehensively evaluates the identified learning tendencies, personality traits, and emotional states, and based on that, generates personalized homework, quizzes, and regular exam questions using a generative AI model (e.g., GPT-4).

[1429] Step 6:

[1430] The server stores the generated questions in a database and then distributes them to the terminals, which then display the questions to the students.

[1431] Step 7:

[1432] The user (student) works on the problem and inputs the answer into the terminal, which then sends the answer data to the server.

[1433] Step 8:

[1434] The server evaluates the received answer data using an automatic scoring system, generates performance data, and calculates an output score and personality score based on the answer content and performance data.

[1435] Step 9:

[1436] The server analyzes the calculated output score and personality score and generates information to suggest the student's optimal career path and employment based on the results, taking into account the user's emotional state.

[1437] Step 10:

[1438] The server distributes information on career and employment suggestions to the terminal, and the terminal displays the suggestions to the students and teachers.

[1439] Step 11:

[1440] The server displays the personality assessment results and emotional state based on the student's scores to the teacher, and generates guidance information for teaching methods based on the results.

[1441] Step 12:

[1442] The server then distributes the generated instructional method guide information to the terminal, which then displays it to the teacher.The teacher then decides on a teaching policy based on this information and provides individual instruction to the student.

[1443] Example 2

[1444] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1445] Traditional educational systems have faced the challenge of making it difficult to provide instruction that fully takes into account each student's learning tendencies and personality traits. Furthermore, there was a lack of a way to grasp students' emotional states in real time and reflect them in the learning content. As a result, the quality of individualized instruction declined, hindering students' motivation to learn and academic improvement.

[1446] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for reading an image of a submitted work and converting it into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for acquiring the user's facial expression and voice data and analyzing it to recognize their emotional state, and means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies, personality traits, and emotional state. This makes it possible to provide learning content optimized for each student and take into account the student's emotional state, thereby improving learning effectiveness and enhancing the quality of individualized instruction.

[1447] "Image of submission" means a digital image of handwritten or printed material submitted by a student as part of an assignment.

[1448] The "means for converting into text data" is a mechanism for extracting character information in an image as text data using optical character recognition technology.

[1449] "Means for recognizing learning tendencies and personality traits" is a function that analyzes text data and identifies students' learning patterns and personality traits.

[1450] "Means for acquiring a user's facial expression and voice data and analyzing it to recognize their emotional state" refers to a technology that uses a camera and microphone to collect a user's facial expression and voice and then analyzes it to determine their emotional state.

[1451] The "means for generating daily homework, quiz, and exam questions" is a mechanism for creating individually customized study questions based on learning tendencies, personality traits, and emotional state.

[1452] The "means for calculating output scores and personality scores" is a function that automatically calculates evaluation scores for academic performance and personality traits based on the student's answers and grade data.

[1453] The "means for suggesting career paths and employment opportunities" is a system that suggests future career paths and suitable employment opportunities to students based on their output scores and personality scores.

[1454] "Means for providing guidance on teaching methods" is a system that provides teachers with learning data and personality data on students and advises them on effective teaching methods.

[1455] This invention is a system for improving the efficiency of individual instruction in educational systems, particularly by combining an emotion engine that recognizes user emotions to achieve more sophisticated personalization. This system reads images of students' submitted work, analyzes the images to determine their learning tendencies and personality traits, automatically generates personalized homework, quizzes, and regular exam questions based on the analysis, and then evaluates the students' answers to calculate output scores and personality scores. These scores are also used to suggest optimal career paths and employment opportunities for students and to provide guidance to teachers on teaching methods. Additionally, the emotion engine can recognize students' emotions and provide feedback to the educational content.

[1456] First, the user (student) takes a picture of the submitted work using a smartphone or tablet. This image is displayed on the device, and after the user confirms it, they upload it to the server. The server then converts the received image data into text data using OCR technology (e.g., Tesseract). At this stage, handwritten or printed characters are digitized and proceed to the next analysis step.

[1457] The server analyzes the converted text data using an automatic analysis algorithm (e.g., SpaCy) to extract learning trends and personality traits. Additionally, when users (students) submit their work or solve problems, the device uses a camera and microphone to capture their facial expressions and voice data. This data is sent to the server in real time and analyzed using an emotion engine (e.g., Microsoft Azure Emotion Recognition API).

[1458] The server then uses a generative AI model (e.g., GPT-4) to generate questions for daily homework, quizzes, and regular exams based on the learning tendencies, personality traits, and emotional states. For example, example prompts include:

[1459] "This student tends to be more theoretical in his learning, and in recent exams he has struggled with understanding applied mathematics questions. His current emotional state is one of tension and anxiety. Please create applied mathematics questions of medium difficulty for this student to deepen his understanding. Please include illustrations in the questions and provide thorough explanations."

[1460] The generated questions are sent to students via their devices, who then answer them. The answers are then sent to a server via the device, where they are evaluated using an automated scoring system (e.g., OpenAI Codex). Based on this, output scores and personality scores are calculated, and the server stores these data.

[1461] Furthermore, the server will make career and employment recommendations based on the analysis results. The recommendations are calculated using statistical models to determine the optimal career path and are provided to students and teachers via their devices. The recommendations are best suited to the user's abilities and characteristics.

[1462] Finally, the server displays the personality assessment results (e.g. MBTI) to the teacher based on the student's scores and analysis results, and provides guidance on how to teach based on the results. This information is generated taking into account emotional data and is provided to the teacher via their terminal.

[1463] As a result, a system will be created that improves the quality of education and makes individualized instruction more efficient through learning programs optimized for each student and feedback based on their emotions.

[1464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1465] Step 1: Take and upload an image of your submission

[1466] The user (student) takes a picture of the submitted work using a smartphone or tablet, and the captured image is displayed on the device screen.

[1467] The terminal uploads the image confirmed by the user to the server.

[1468] Input: A user-taken image of the submission.

[1469] Output: Image data of the submission uploaded to the server.

[1470] Step 2: Convert image data to text using OCR technology

[1471] The server converts the image data of the received submission into text data using OCR technology (e.g., Tesseract).

[1472] The server temporarily stores the converted text data and uses it for subsequent analysis steps.

[1473] Input: Image data of the submission.

[1474] Output: Text data.

[1475] What it does: Converts handwritten or printed text in an image into digital text.

[1476] Step 3: Analysis of learning tendencies and personality traits

[1477] The server analyzes the converted text data using a natural language processing algorithm (e.g., SpaCy).

[1478] The server extracts learning tendencies and personality traits from the text data and stores this information in a database.

[1479] Input: Text data converted by OCR.

[1480] Output: Data on learning tendencies and personality traits.

[1481] Specific Actions: Through text analysis, students' tendency to excel at logical thinking is revealed.

[1482] Step 4: Acquire and analyze emotion data

[1483] When a user (student) solves a problem, the device uses a camera and microphone to capture the user's facial expressions and voice data.

[1484] The terminal transmits the acquired emotion data to the server in real time.

[1485] The server analyzes the data using an emotion engine (e.g., Microsoft Azure Emotion Recognition API) to recognize the user's emotional state.

[1486] Input: User's facial and voice data.

[1487] Output: Data of the user's emotional state.

[1488] Specific operation: Detects emotions such as tension, anxiety, and joy in real time from the user's facial expressions and tone of voice.

[1489] Step 5: Generate homework, quizzes, and exam questions

[1490] The server generates personalized questions using a generative AI model (e.g., GPT-4) based on learning tendencies, personality traits, and emotional states.

[1491] The server automatically generates questions by inputting prompt sentences into a generative AI model.

[1492] The server sends the generated questions to the terminal.

[1493] The terminal distributes the generated questions to the students, who then answer them.

[1494] Input: learning tendencies, personality traits, and emotional state data.

[1495] Output: Personalized homework, quiz and regular exam questions.

[1496] Specific operation: The following prompt is input into the generative AI model to generate a problem: "This student has a theoretical learning tendency, and in recent exams, he has tended to struggle with understanding applied mathematics problems. His current emotional state is showing tension and anxiety. Please create an applied mathematics problem of medium difficulty for this student to deepen his understanding. Please include illustrations and provide detailed explanations in the problem text."

[1497] Step 6: Upload and grade your answers

[1498] The user (student) enters the answer into the terminal and sends it to the server.

[1499] The server evaluates the received answer data using an automatic scoring system (e.g., OpenAI Codex) and generates score data.

[1500] Input: The answer data entered by the student.

[1501] Output: Evaluated performance data.

[1502] Specific operation: Based on the answers, the system determines whether the answers are correct or incorrect, scores them, and automatically generates a grade.

[1503] Step 7: Calculating output and personality scores

[1504] The server automatically calculates the output score and personality score based on the answers and performance data.

[1505] Input: Answers and performance data.

[1506] Output: Output scores and personality scores.

[1507] What it does: Evaluate students' understanding and personality traits and calculate corresponding scores.

[1508] Step 8: Career and employment suggestions

[1509] The server suggests optimal career paths and employment opportunities based on the calculated output score and personality score.

[1510] The server sends the suggestions to the terminals and provides them to students and teachers.

[1511] Inputs: Output scores and personality scores.

[1512] Output: Career and employment suggestions.

[1513] What it does: Uses statistical models to suggest appropriate career paths for students, such as "go to a science and engineering university."

[1514] Step 9: Teacher feedback and instructional guide

[1515] Based on the student's scores and analysis results, the server generates personality assessment results (e.g., MBTI) and information to guide teachers on teaching methods.

[1516] The terminal provides this feedback information to the teacher, who then uses it to determine the teaching policy.

[1517] Input: Student scores and analysis results.

[1518] Output: Personality assessment results and guidance information on teaching methods.

[1519] Specific action: The teacher's device displays information such as, "Student A is analytical and introverted. Rather than praising him, let's suggest a problem-solving approach."

[1520] (Application example 2)

[1521] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1522] Individualized instruction is extremely important in the modern education system, but providing appropriate learning content to each student and maximizing learning outcomes requires a great deal of time and effort. Furthermore, education that ignores students' emotional states can reduce effectiveness, so there is a need to provide personalized learning experiences using emotion recognition technology. Furthermore, the introduction of virtual classrooms using virtual reality technology is necessary to provide students with new learning methods.

[1523] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1524] In this invention, the server includes means for reading images of submitted materials and converting them into text data, means for analyzing the text data and recognizing learning tendencies and personality traits, means for generating questions for daily homework, quizzes, and regular exams based on the recognized learning tendencies and personality traits, means for distributing the generated questions to students, means for evaluating students' answers and calculating output scores and personality scores, means for suggesting optimal career paths and employment opportunities for students based on the output scores and personality scores, means for displaying personality assessment results to teachers based on the students' scores and providing instructional guidance, an emotion engine for recognizing students' emotional states using facial expressions and voice data when solving problems, means for personalizing educational content based on the emotional states and generating optimal questions for students, and means for providing a personalized learning experience in a virtual classroom using a virtual reality head-mounted display. This makes it possible to maximize students' learning outcomes, reduce the burden on teachers, and provide effective education tailored to their emotional states.

[1525] "Submissions" refer to assignments and reports that students submit as the result of their studies.

[1526] An "image" is visual information captured as digital data by a camera or scanner.

[1527] "Text data" refers to data that is treated as character information.

[1528] "Learning tendency" refers to the tendency that indicates how a student will effectively proceed with their studies.

[1529] "Personality traits" refer to a student's personality and behavioral characteristics, and are information used to individualize educational methods.

[1530] "Homework," "quizzes," and "regular exams" are questions used to evaluate students' learning status.

[1531] "Delivery" refers to sending the generated questions and feedback to students.

[1532] An "answer" is a solution submitted by a student to a problem.

[1533] "Assessment" means marking students' answers and measuring their performance.

[1534] An "output score" is a score that quantitatively evaluates learning outcomes.

[1535] A "personality score" is a numerical score that represents a student's personality traits.

[1536] "Career path" refers to the direction of education or career that a student should pursue in the future.

[1537] "Employment" refers to the workplace where students plan to work after graduation.

[1538] "Instructional methods" refer to the educational techniques that teachers use with students.

[1539] "Emotional state" refers to the student's emotional or psychological state.

[1540] An "emotion engine" is a technology for recognizing emotional states from data such as facial expressions and voice.

[1541] "Personalization" means customizing something to meet individual needs.

[1542] A "virtual reality head-mounted display" is a visual and auditory device worn to experience a virtual reality space.

[1543] A "virtual classroom" is a virtual educational space recreated using virtual reality technology.

[1544] This invention is a system for individualized instruction that recognizes students' emotional states and uses that information to achieve more sophisticated personalization. To achieve this, the invention mainly uses the following hardware and software:

[1545] Hardware used

[1546] 1. VR Head-Mounted Display (HMD): Used to provide learning experiences within a virtual classroom.

[1547] 2. 360-degree camera: Used by students to film their submissions.

[1548] 3. Microphone: Used to collect student voice data.

[1549] Software used

[1550] 1. OCR technology: Using Google Cloud Vision API, we convert the image of the submitted document into text data.

[1551] 2. Natural Language Processing (NLP): Analyze transformed text data using spaCy and Hugging Face Transformers.

[1552] 3. Emotion recognition technology: Using OpenCV and DeepFace, we identify students' emotional states from facial expressions and voice data.

[1553] 4. Generative AI model: Uses OpenAI GPT-4 to generate personalized homework and test questions.

[1554] 5. Server Technology: We use AWS EC2 and S3 for data storage and processing.

[1555] How to carry out the invention

[1556] 1. Importing and analyzing submitted images

[1557] Students wear a VR head-mounted display and follow instructions in the virtual classroom to take photos of their submissions using a 360-degree camera and upload the images to the server, which then uses the Google Cloud Vision API to convert the image data into text using OCR processing.

[1558] 2. Introducing the Emotion Engine

[1559] While solving problems, the device (camera and microphone of the VR head-mounted display) collects facial and voice data from the student, and uses OpenCV and DeepFace to recognize the student's emotional state in real time. This information is sent to a server and used to personalize the educational content.

[1560] 3. Homework, quiz, and regular exam question generation

[1561] The server uses a generative AI model (GPT-4) to automatically generate questions optimized for each student based on their analyzed learning tendencies and personality traits, as well as their recognized emotional state. This allows a student showing signs of nervousness or anxiety to be provided with appropriate questions to alleviate that state. The generated questions are then distributed to students within the virtual classroom.

[1562] 4. Answer analysis and scoring

[1563] Students enter their answers to questions in the VR space and send them to the server, which then evaluates the answers using an automatic scoring system to calculate output and personality scores.

[1564] 5. Career and employment suggestions

[1565] Based on the calculated scores, the server proposes the best career paths and employment opportunities for students and provides this information within the virtual classroom, allowing students to choose a career path that suits their own abilities and characteristics.

[1566] 6. Teacher feedback and instructional guide

[1567] Teachers can also sit in the same virtual classroom and receive feedback on teaching methods based on students' scores and emotional states, helping teachers to determine the best teaching strategy for each individual student.

[1568] Specific examples

[1569] For example, when a student submits a math assignment, they follow instructions in the VR space and take a photo of the assignment with a 360-degree camera. The server converts the image into text data using OCR technology and then analyzes it with an NLP algorithm to identify learning trends. While the student is solving the problem, an emotion engine analyzes their facial expressions and voice to detect their level of tension. A generative AI model automatically generates a problem of a level of difficulty appropriate for this state and presents it to the student in the VR space. After solving the problem, the answer is automatically graded, and the output score and personality score are updated.

[1570] Example prompt sentence:

[1571] Question: This student has high ability in math but is prone to anxiety before tests. Generate problems designed to help him overcome his anxiety.

[1572] In this way, this invention is a system that can provide each student with an optimal educational experience, reduce the burden on teachers, and maximize learning effectiveness.

[1573] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1574] Step 1:

[1575] The user (student) wears a VR head-mounted display and takes a picture of the submitted work with a 360-degree camera in the virtual classroom. The input is a physical image of the submitted work, and the output is digital image data.

[1576] Step 2:

[1577] The terminal uploads the image data of the photographed submission to the server. The input is digital image data, and the output is image data stored on the server. The server receives and stores this image data.

[1578] Step 3:

[1579] The server calls the Google Cloud Vision API, performs OCR on the uploaded image data, and converts it into text data. The input is the image data of the submitted item, and the output is text data.

[1580] Step 4:

[1581] The server analyzes the converted text data using spaCy and Hugging Face Transformers to extract learning tendencies and personality traits. The input is text data, and the output is analyzed learning tendencies and personality traits data.

[1582] Step 5:

[1583] When the user solves a problem, the device collects the user's facial expression and voice data in real time using the camera and microphone of the VR head-mounted display and sends it to the server. The input is the user's facial expression and voice data, and the output is the emotional data sent to the server.

[1584] Step 6:

[1585] The server analyzes the emotional data using OpenCV and DeepFace to recognize the user's current emotional state. The input is facial expression and voice data, and the output is emotional state data analyzed in real time.

[1586] Step 7:

[1587] The server uses GPT-4 to generate personalized homework and test questions based on the analyzed learning tendencies, personality traits, and emotional state. The generated questions are appropriate for the user's learning situation and emotional state. The input is learning tendency data, personality trait data, and emotional state data, and the output is personalized questions.

[1588] Step 8:

[1589] The server delivers the generated questions to the user's VR head-mounted display. The input is a personalized question, and the output is a question displayed in the user's VR classroom.

[1590] Step 9:

[1591] The user answers questions in the VR classroom and sends the answer data to the server. The input is the user's answer data, and the output is the answer data sent to the server.

[1592] Step 10:

[1593] The server evaluates the received answer data using an automatic scoring system and calculates an output score and a personality score. The input is the user's answer data, and the output is the output score and the personality score.

[1594] Step 11:

[1595] Based on the calculated scores, the server uses a generative AI model to suggest optimal career paths and employment opportunities for users. The inputs are the output score and personality score, and the output is career and employment suggestions.

[1596] Step 12:

[1597] The server also provides teachers with feedback on teaching methods based on the user's score and emotional state. The input is score data and emotional state data, and the output is a teaching guide for the teacher.

[1598] Step 13:

[1599] The device displays the generated teaching guide to the teacher in the VR space, and the teacher then provides optimal instruction to the user based on it. The input is the teaching guide, and the output is the display information to the teacher.

[1600] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1601] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1602] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1603] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1604] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1605] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1606] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1607] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1608] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1609] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1610] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1611] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1612] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1613] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1614] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1615] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1616] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1617] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1618] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1619] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the...

Claims

1. A means to read the image of the submitted document and convert it into text data; means for analyzing the text data and recognizing learning tendencies and personality traits; means for generating daily homework, quiz, and exam questions based on said identified learning tendencies and personality traits; means for distributing the generated questions to students; a means for evaluating student responses and calculating output and personality scores; A means for suggesting the most suitable career path and employment for a student based on the output score and personality score; A means to display personality assessment results based on students' scores to teachers and provide guidance on teaching methods; A system including:

2. The system of claim 1, wherein the image of the submission is converted into text data using OCR technology.

3. The system of claim 1 , wherein the generated problems are automatically generated using a generative AI model.

Images