Learning support system

Abstract

To provide a system.SOLUTION: A learning support system includes: first means for acquiring teaching material information that is data of characters and images contained in a teaching material engaging a student in learning, and student information that is personal data of the student's age, gender, past test results, deviation, learning objective and target achievement time limit; second means for specifying a learning content that the student is learning, on the basis of the acquired teaching material information; third means for generating a first prompt sentence for outputting advice related to the learning content, on the basis of the specified learning content and the acquired student information; fourth means for generating the advice by inputting the generated first prompt sentence to a generative AI model; fifth means for transmitting the generated advice to smart spectacles used by the student; and sixth means for overlay-displaying the received advice in the teaching material by using an augmented reality technique.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a learning support system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional learning methods, it is difficult for students to fully understand a topic just by using textbooks, notes, and other study materials, and it is also difficult for students to receive appropriate advice tailored to their own learning goals. [Means for solving the problem]

[0005] The learning support system of the present disclosure includes an identification unit that identifies learning content based on learning material information related to the learning material that the student is studying, a generation unit that generates advice based on the learning content identified by the identification unit and the student's information and learning goals, a control unit that overlays the advice generated by the generation unit on the learning material using AR (Augmented Reality) technology, and a learning unit that receives questions and requests related to learning from the student and uses them as learning data. As a result, the learning support system can support the student's learning, allowing the student to study more effectively. In addition, the learning support system can provide support tailored to individual learning needs by receiving questions and requests related to learning from the student and using them as learning data.

[0006] The identification unit identifies the learning content that the student is working on, for example, based on visual information acquired via the smart glasses.

[0007] The generation unit uses natural language processing technology, such as generative AI, to provide advice based on the learning content.

[0008] The control unit, for example, uses the AR function of the smart glasses to overlay learning-related videos or visual explanations on learning materials such as textbooks or notes, allowing students to deepen their understanding of the learning content through visual supplementation.

[0009] The learning unit receives, for example, questions and requests sent by students through the smart glasses, and stores and uses them as learning data. This makes it possible to provide support tailored to the individual learning needs of students. [Brief description of the drawings]

[0010] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] 4 is a sequence diagram showing a process flow of the data processing system according to the first embodiment. FIG. [Figure 12] 11 is a sequence diagram showing a process flow of the data processing system in application example 1. FIG. [Figure 13] FIG. 11 is a sequence diagram showing the flow of processing of the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in application example 2 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0012] First, the terms used in the following description will be explained.

[0013] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0014] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0015] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0016] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0017] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0018] [First embodiment]

[0019] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0020] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0022] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0023] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0024] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0025] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0026] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0027] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0028] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0029] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0030] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0031] This embodiment includes the following elements: The data processing system 10 is an example of a "learning support system" in this disclosure.

[0032] 1. Smart glasses: A device that allows students to view learning materials such as textbooks and notes using smart glasses. The smart glasses have the function of acquiring visual information such as text and images of learning materials acquired as learning material information, and displaying them as an overlay using AR.

[0033] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs prompts containing learning content and student information into the generative AI and obtains appropriate advice.

[0034] 3. Generative AI: A natural language processing model that generates appropriate advice based on prompts containing learning content and student information. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[0035] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[0036] 5. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[0037] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[0038] The process flow will be explained below.

[0039] Step 1: Processing from the smart glasses

[0040] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[0041] The terminal transmits the obtained visibility information to the server.

[0042] Step 2: Server processing

[0043] The server analyzes the received visual information and identifies the learning content.

[0044] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[0045] The server sends the generated prompts to the generative AI to obtain appropriate advice.

[0046] Step 3: Generative AI processing

[0047] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[0048] The generative AI returns the generated advice to the server.

[0049] Step 4: Processing the response from the server

[0050] The server obtains the advice received from the generative AI.

[0051] The server overlays advice to the student via the smart glasses.

[0052] Step 5: Progression of learning and handling of student questions and requests

[0053] Students use the smart glasses' AR capabilities to watch relevant videos and visual explanations as they learn.

[0054] If a student has any questions or requests, they send them to the server through their smart glasses.

[0055] Step 6: Collect training data and fine-tune generative AI

[0056] The server receives questions and requests from students and stores them as learning data.

[0057] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[0058] Example 1

[0059] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0060] Conventional learning support systems could not effectively utilize information from teaching materials or individual student information, making it difficult to provide appropriate advice and learning content in real time. As a result, students' learning efficiency decreased and achievement of learning goals was often delayed. In addition, there was a problem that students' motivation to learn decreased because they could not respond quickly to their questions and requests.

[0061] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0062] In this invention, the server includes: a means for acquiring learning material information on the learning material that the student is studying; a means for identifying the learning content based on the learning material information; a means for generating a prompt sentence based on the learning content and the student's information; a means for inputting the prompt sentence into a generative AI model to generate advice; a means for overlaying the generated advice on the learning material using AR technology; a means for referring to a database that stores information such as the student's age, sex, past test results, deviation score, learning goal, and goal achievement deadline; and a means for transmitting the generated advice and related learning content to the smart glasses. This allows the student to efficiently progress through their studies while receiving personalized advice in real time.

[0063] 1. "Materials information" refers to data such as text and images contained in the materials students use for their studies.

[0064] 2. “Content” refers to information about a particular subject or topic that a student is studying.

[0065] 3. "Student information" refers to personal data such as a student's age, gender, past test results, academic ability score, learning goals, and deadlines for achieving those goals.

[0066] 4. “Prompt” refers to an instruction to be input into a generative AI model based on the learning content and student information.

[0067] 5. “Generative AI model” means an artificial intelligence model that uses natural language processing to generate advice or information based on a prompt.

[0068] 6. “Advice” refers to advice or suggestions for learning support that are generated by the generative AI model based on the prompt.

[0069] 7. "AR technology" refers to augmented reality technology that overlays digital information on the actual field of vision.

[0070] 8. "Database" means a database management system for storing information such as students' age, gender, past test results, deviation scores, learning goals, and deadlines for achieving goals.

[0071] 9. "Smart glasses" means a wearable device for displaying digital information using augmented reality technology.

[0072] 10. “Learning content” means digital information, such as video or visual explanations, that supplements student learning.

[0073] The present invention relates to a system for supporting student learning, and more particularly to a system for providing appropriate advice and learning content in real time by utilizing teaching material information and student information. A specific embodiment for carrying out the present invention will be described.

[0074] System Overview

[0075] The system mainly consists of smart glasses, a server, a generative AI model, a database, and learning content.

[0076] Smart Glasses

[0077] Smart glasses are wearable devices that students use when viewing textbooks, notes, and other study materials. The glasses are equipped with a built-in camera and have the ability to capture visual information such as text and images from study materials. They also have a function that uses AR (Augmented Reality) technology to overlay advice generated based on the captured information.

[0078] server

[0079] The server receives the visual information sent from the smart glasses and identifies the learning content. The server accesses the database to obtain individual information about the students. It also generates a prompt sentence based on the learning content and the student information, and inputs the prompt sentence into the generative AI model.

[0080] Generative AI Models

[0081] The generative AI model uses natural language processing to generate advice based on the prompt sentence. For example, ChatGPT (registered trademark) is used for this AI model. It receives the prompt sentence as input, generates appropriate advice, and returns it to the server.

[0082] Database

[0083] The database stores personal information such as the student's age, gender, past test results, deviation score, learning goals, deadline for achieving the goal, etc. The server accesses this database to obtain the necessary information and uses it to generate prompt sentences.

[0084] Learning Content

[0085] Learning content is digital information to supplement students' learning, such as videos and visual explanations. The server transmits the learning content to the smart glasses as needed, allowing students to view it in real time.

[0086] Examples

[0087] Students solving math problems

[0088] 1. Viewing the materials:

[0089] A user opens a math textbook and begins looking at a problem through the smart glasses, which use their built-in camera to capture the mathematical equation "y = ax^2 + bx + c."

[0090] 2. Transmission of information:

[0091] The smart glasses transmit the captured mathematical formula information to a server.

[0092] 3. Generate prompt:

[0093] The server retrieves student information from the database and generates a prompt statement such as, "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem."

[0094] 4. Generate advice:

[0095] The server inputs a prompt sentence into the generative AI model, which then generates advice such as, "First, use the basic form of the quadratic function, y = ax^2 + bx + c, to confirm the values ​​of a, b, and c. Then, I will explain how to solve it using the solution formula."

[0096] 5. Display advice:

[0097] The server sends the generated advice to the smart glasses, which overlay it on the user's field of view.

[0098] 6. Provision of learning content:

[0099] If the user wants to understand further, the server will send the relevant explanatory video to the smart glasses so that the user can watch the video.

[0100] Examples of prompt statements

[0101] "A 17-year-old male student A is trying to solve a quadratic function problem. I would like some specific advice on how to solve quadratic functions."

[0102] In this way, the system of the present invention can effectively support students' learning and improve their learning efficiency.

[0103] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0104] Step 1:

[0105] The user browses the learning materials through smart glasses. The glasses use a built-in camera to capture visual information such as text and images from the learning materials. Optical character recognition (OCR) software is then used to extract text data from the captured information.

[0106] Input: Visual information from textbooks and notes

[0107] Output: Extracted text data (e.g. "y = ax^2 + bx + c")

[0108] Step 2:

[0109] The smart glasses transmit the extracted text data to a server using known communication means, and the server receives the data and begins analyzing it.

[0110] Input: Text data (e.g. "y = ax^2 + bx + c")

[0111] Output: Received text data

[0112] Step 3:

[0113] The server accesses the database and obtains information about the student (age, gender, past test results, deviation score, learning goals, deadline for achieving goals, etc.). Based on the obtained information, a prompt sentence is generated according to the current learning situation. The prompt sentence is a specific instruction sentence to be passed to the generative AI model.

[0114] Input: Received text data, student information stored in database

[0115] Output: Prompt statement (e.g. "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem.")

[0116] Step 4:

[0117] The server inputs the generated prompt sentence into a generative AI model (e.g., ChatGPT). The generative AI model generates appropriate advice based on the prompt sentence. The generative AI model analyzes information about the learning content and creates advice in a form that is easy for the user to understand.

[0118] Input: Prompt statement

[0119] Output: Generated advice (e.g. "First, use the basic form of the quadratic function y = ax^2 + bx + c to find the values ​​of a, b, and c. Then, use the solution formula to explain how to solve the problem.")

[0120] Step 5:

[0121] The server sends the generated advice to the smart glasses, which overlay the advice on the user's field of view. The displayed advice is integrated into the field of view in a natural way using AR technology.

[0122] Input: Generated advice

[0123] Output: Advice overlaid on the user's view

[0124] Step 6:

[0125] If necessary, the server sends relevant learning content (videos and visual explanations) to the smart glasses, which use their AR capabilities to enable the user to watch this learning content in real time.

[0126] Input: User request, related learning content

[0127] Output: Learning content overlaid on the user's field of view

[0128] (Application example 1)

[0129] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] Current factory robots often lack immediate support for work procedures and problem solving when performing complex tasks or troubleshooting. This leads to issues such as reduced work efficiency and increased downtime. Conventional methods require robot operators to refer to the manual, which can take time to troubleshoot. There is a need to improve this situation and increase work efficiency.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0132] In this invention, the server includes: a specifying means for specifying a task content based on task information related to a task to be performed by the robot; a generating means for generating advice based on the task content specified by the specifying means, the robot information, and a task goal; a control means for overlaying the advice generated by the generating means on the task using an augmented reality (AR) technology; and a learning means for receiving questions and requests related to the task from the robot and using them as task data. This enables the robot to receive necessary advice in real time, making it possible to dramatically improve task efficiency.

[0133] A "robot" is a mechanical device that automates tasks in a factory.

[0134] "Work information" is information relating to the specific work content that the robot will undertake.

[0135] The "identification means" is a function for identifying the work content based on the work information.

[0136] "Work content" refers to the series of work procedures and specific work instructions that the robot must perform.

[0137] "Robot Information" means information including a Robot's identification, current status, and performance data.

[0138] "Task goals" are information that sets specific goals and deadlines that the robot must achieve.

[0139] The "generation means" is a function for generating advice based on the work content identified by the identification means and other information.

[0140] "Advice" is any instruction or recommendation given to assist in the progress of work.

[0141] "AR technology" is a general term for technology that displays virtual information overlaid on real visual information.

[0142] The "control means" is a function for displaying the advice generated by the generation means using AR technology.

[0143] The "learning means" is a function that receives questions and requests from the robot regarding tasks and uses them to improve the system and generate advice.

[0144] The present invention is a system for improving the work efficiency of factory robots. This system identifies the work content based on work information related to the work performed by the robot and provides advice in real time, thereby significantly improving the efficiency of work stagnation and troubleshooting.

[0145] System Configuration

[0146] The system configuration is as follows.

[0147] 1. Robot:

[0148] Robots are installed in factories to perform tasks automatically. They are equipped with cameras that can capture images of the tasks they perform.

[0149] 2. Server:

[0150] The server has the following means:

[0151] Identification method: Analyze images obtained from the robot's camera and other work information to identify the current work content.

[0152] Generation means: Generates appropriate advice based on the task content identified by the identification means and the robot information (identification information, status information, etc.). Here, advice is generated using a generative AI model.

[0153] Control means: The generated advice is overlaid on the robot's field of view using AR technology.

[0154] Learning method: Receive questions and requests from the robot regarding the task, and use them to improve the system and generate advice.

[0155] 3. AR technology:

[0156] Using augmented reality (AR) technology, the generated advice is superimposed on the robot's field of vision, allowing the robot to obtain the necessary information in real time.

[0157] Processing flow

[0158] 1. The robot's camera captures footage of the work.

[0159] 2. The acquired video is sent to the server.

[0160] 3. The server's identification means analyzes the video and identifies the work content.

[0161] 4. The server's generation means generates advice based on the identified work content and robot information.

[0162] 5. The generated advice is overlaid on the robot's field of view using AR technology via the server's control means.

[0163] 6. Receive questions and requests from the robot, store them in a database as a learning tool, and use them to generate advice next time.

[0164] Hardware and Software Used

[0165] Hardware: Robots, cameras, servers

[0166] Software: Generative AI model (generation means), OpenCV (registered trademark) (image analysis), requests (HTTP requests), AR module (AR technology)

[0167] Specific examples

[0168] As a concrete example, consider a scenario in which a robot is repairing a machine in a factory. In this case, if the robot does not know the location of a part, the image of the area around the part captured by the camera is sent to the server. The server analyzes the image and identifies the location of the part, then generates advice on how to handle the part and the repair procedure, which it displays in the robot's field of view. This advice display is done using AR technology, so the robot can proceed with the work while receiving visual support.

[0169] Examples of prompt statements

[0170] Below are examples of prompts to be input to the generative AI model.

[0171] "Machine X is missing part Y. The current work step is step 3."

[0172] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0173] Step 1:

[0174] The robot's camera captures footage of the work.

[0175] Input: Robot working environment

[0176] Output: Camera image

[0177] How it works: The robot's camera films the work area and captures footage in real time.

[0178] Step 2:

[0179] The captured image is sent to the server.

[0180] Input: Camera image

[0181] Output: Camera image data sent to the server

[0182] Operation: Images captured by the camera are sent to a server via the Internet or local network.

[0183] Step 3:

[0184] The server's identification means analyzes the video and identifies the work content.

[0185] Input: Camera video data sent to the server

[0186] Output: Identified work content information

[0187] Operation: The server uses video analysis software (e.g. OpenCV) to recognize objects and text information in the video and identify the current task.

[0188] Step 4:

[0189] A generating means of the server generates advice based on the identified work content and information on the robot.

[0190] Input: Identified task information and robot information (identification information, status information, etc.)

[0191] Output: Generated advice

[0192] How it works: The server inputs a prompt sentence into the generative AI model, which then generates appropriate advice. Example of a prompt sentence: "Part Y of machine X is in short supply. The current work procedure is step 3."

[0193] Step 5:

[0194] The generated advice is overlaid on the robot's field of view using AR technology by the server's control means.

[0195] Input: Generated advice

[0196] Output: Advice overlaid on the robot's field of vision.

[0197] Operation: The server's control means uses the AR module to superimpose the generated advice onto the robot's camera image and display it.

[0198] Step 6:

[0199] The system receives questions and requests from the robot, stores them in a database as a learning tool, and uses them to generate advice the next time.

[0200] Input: Question or request from the robot

[0201] Output: Question and request data stored in a database

[0202] How it works: The robot sends any additional information, questions or requests it needs via the user interface to the server, which then stores the data in a database, where it can be used as learning data to generate advice the next time.

[0203] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0204] This embodiment is composed of the following elements.

[0205] 1. Smart glasses: A device that allows students to view textbooks, notes, and other educational materials using smart glasses. The smart glasses have the ability to acquire visual information and display it as an AR overlay.

[0206] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs learning content, student information, and prompts including the student's emotions recognized by the emotion engine into the generative AI to obtain appropriate advice.

[0207] 3. Generative AI: A natural language processing model to generate appropriate advice based on prompts including learning content, student information, and student sentiment. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[0208] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[0209] 5. Emotion Engine: An engine for recognizing and analyzing student emotions. The emotion engine uses student emotions as learning data and is used to adjust the difficulty of learning and the style of advice.

[0210] 6. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[0211] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[0212] The process flow will be explained below.

[0213] Step 1: Processing from the smart glasses

[0214] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[0215] The terminal transmits the acquired visual information and the student's emotions to the emotion engine for emotion recognition and analysis.

[0216] The terminal transmits the results received from the emotion engine to the server.

[0217] Step 2: Server processing

[0218] The server analyzes the received visual information and identifies the learning content.

[0219] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[0220] The server sends the generated prompts and results from the emotion engine to the generative AI to obtain appropriate advice.

[0221] Step 3: Generative AI processing

[0222] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[0223] The generative AI returns the generated advice to the server.

[0224] Step 4: Processing the response from the server

[0225] The server obtains the advice received from the generative AI.

[0226] The server overlays advice to the student via the smart glasses.

[0227] Step 5: Progression of learning and handling of student questions and requests

[0228] Students can use the AR function to view learning content and progress through their studies.

[0229] If a student has any questions or requests, they send them to the server through their smart glasses.

[0230] Step 6: Collect training data and fine-tune generative AI

[0231] The server receives questions and requests from students and stores them as learning data.

[0232] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[0233] In the data processing system 10, when a student uses the smart glasses to view textbooks, notes, and other study materials, relevant videos and visual commentaries are overlaid in AR. The processor used in the data processing system 10 identifies the study content based on the visual information obtained from the smart glasses, and then combines it with data such as the student's age, gender, past test results, deviation score, study goals, and deadline for achieving the goals to query the generative AI for advice on how to proceed with the study. The generative AI provides appropriate content in response to the student's questions and requests, and uses the questions and requests from the student as learning data to fine-tune the generative AI. This data processing system 10 helps to deepen the understanding of the study content through visual complementation.

[0234] According to the data processing system 10, for example, when a student uses the smart glasses to look at a mathematics textbook, the processor analyzes information from the textbook and identifies the mathematics content that the student should work on. The processor then communicates the learning content and the student's information to the generative AI and asks for advice on how to proceed with the learning. Based on the learning content and the student's information, the generative AI provides advice such as "This chapter is difficult, so we recommend that you thoroughly understand the basics." The student can then proceed with the learning based on the advice and deepen their understanding of mathematics by watching related videos and visual explanations using the AR function of the smart glasses.

[0235] Example 2

[0236] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0237] Conventional learning support systems have difficulty in providing individual advice to students when they are studying. In addition, they are unable to grasp the emotional state of students in real time, and therefore are unable to provide appropriate advice or adjust the difficulty of learning accordingly. This leads to a problem of a decrease in overall learning efficiency.

[0238] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for identifying learning contents based on learning material information related to the learning materials that the student is studying, a means for generating advice based on the learning contents identified by the identification means and the student's information and learning goals, a means for overlaying the advice generated by the generation means on the learning materials using augmented reality technology, a means for recognizing the student's emotions and adjusting the difficulty of the learning and the style of the advice, and a means for receiving questions and requests related to the learning from the student and using them as learning data. This makes it possible to provide appropriate advice according to the individual learning situation and emotional state of the student, thereby improving learning efficiency.

[0239] A "student" is a person who studies certain teaching materials or learning content.

[0240] "Learning material information" refers to information related to learning materials such as textbooks and notebooks used by students.

[0241] "Learning content" refers to specific learning items or themes that are specified based on the learning material information.

[0242] "Student information" means individual data about a student, such as the student's age, gender, past test results, and learning history.

[0243] A "learning objective" is a specific learning purpose or goal that a student wants to achieve.

[0244] "Advice" refers to the guidance or advice provided to assist a student in their learning.

[0245] A "generative model" is an algorithm or software that generates information or advice based on input data.

[0246] "Augmented reality technology" is a technology that displays digital information overlaid on the real world.

[0247] "Means of recognizing emotions" refers to technology or devices for understanding a student's emotional state from their facial expressions, voice, and other sensor information.

[0248] "Learning data" is data obtained during the course of a student's studies, and is information that can be analyzed to provide learning support.

[0249] In order to implement this invention, it is necessary to construct a system including the following elements:

[0250] Hardware and Software Used

[0251] 1. Smart Glasses

[0252] Smart glasses are devices that allow users to view learning materials such as textbooks and notes. The smart glasses are equipped with a camera function to acquire visual information about the learning materials. In addition, the glasses use AR (Augmented Reality) technology to overlay appropriate learning content based on the acquired information.

[0253] 2. Server

[0254] The server receives visual information obtained from the smart glasses and identifies the learning content. Specifically, it uses OCR technology to analyze the text information of the teaching materials and combines it with the student information in the database and the student's emotion data recognized by the emotion engine. As a result, it automatically generates prompt sentences to input into the generative AI model.

[0255] 3. Generative AI Models

[0256] The generative AI model uses natural language processing techniques to generate appropriate advice from prompts, and is fine-tuned using training data to provide customized advice based on learning content and student requests.

[0257] 4. Database

[0258] The database contains information about students, such as their age, gender, past test results, academic ability scores, learning goals, and deadlines for achieving those goals. The server refers to this database and uses it as auxiliary information when generating prompt sentences.

[0259] 5. Emotion Engine

[0260] The emotion engine is software for analyzing students' emotional state in real time. It is used to adjust the difficulty of learning and style of advising, and recognizes emotions from students' facial expressions and voice.

[0261] 6. Learning Content

[0262] Learning content is digital content that complements the learning content, such as relevant videos and visual explanations, and is visually presented to students using the AR capabilities of the smart glasses.

[0263] Examples

[0264] Suppose a user wears smart glasses and is studying trigonometric formulas in a math textbook. The glasses capture the image data of the study material and send it to the server. The server uses OCR to extract text information from the image data and identifies the study content based on this. At the same time, the emotion engine analyzes the student's facial expression and obtains emotion data such as confusion.

[0265] The server combines the teaching material information, student information in the database, and emotion data to generate a prompt sentence like the following:

[0266] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[0267] When this prompt is fed into a generative AI model, the model generates advice based on it, like this:

[0268] "Trigonometric formulas are easier to understand when they are represented visually like this. Try solving the following exercise..."

[0269] The server sends this advice to the smart glasses, which then use AR technology to overlay the advice on the student.

[0270] Examples of prompt statements

[0271] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[0272] This will improve students' learning efficiency and provide customized support tailored to their individual learning situations.

[0273] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0274] Explain the flow of system program processing by dividing it into processing steps

[0275] Step 1:

[0276] Data acquisition using smart glasses

[0277] input:

[0278] The textbook, notes, or other educational material that the user is viewing.

[0279] Specific behavior:

[0280] The camera in the smart glasses captures image data of the teaching materials.

[0281] Data processing:

[0282] Image processing software is used to perform text recognition and image analysis to identify visible information in the teaching materials.

[0283] output:

[0284] Image data and text information of captured teaching materials.

[0285] Step 2:

[0286] Data received by the server

[0287] input:

[0288] Image data and text information of educational materials from smart glasses.

[0289] Specific behavior:

[0290] The smart glasses transmit the data to a server via Wi-Fi.

[0291] The server receives this data on the specified port.

[0292] Data Calculation:

[0293] The server uses OCR technology (Optical Character Recognition) to extract text information from the image data.

[0294] output:

[0295] Text information analyzed by OCR and the original image data.

[0296] Step 3:

[0297] Server determines what to learn

[0298] input:

[0299] Text information extracted by OCR.

[0300] Specific behavior:

[0301] The server analyzes the text information and again queries the database to identify the educational material content.

[0302] Calculate the degree of match between learning history and teaching materials.

[0303] Data Calculation:

[0304] Database query operations and text analysis using natural language processing.

[0305] output:

[0306] Identified learning content.

[0307] Step 4:

[0308] Emotion recognition by emotion engine

[0309] input:

[0310] Facial and voice data of students.

[0311] Specific behavior:

[0312] The emotion engine captures students' facial expressions and voice data through sensors in the smart glasses.

[0313] Data Calculation:

[0314] Uses facial recognition and voice analysis algorithms to assess student emotions in real time.

[0315] output:

[0316] Perceived emotion data (e.g., confusion, excitement, happiness).

[0317] Step 5:

[0318] Server-generated prompts

[0319] input:

[0320] Identified learning content, student information from the database, and sentiment data.

[0321] Specific behavior:

[0322] The server generates a prompt based on this data.

[0323] Data Calculation:

[0324] Use data integration and prompt generation algorithms to create appropriate prompts.

[0325] output:

[0326] Generated prompt text. Example: "My student is having trouble understanding trigonometric formulas, so please generate a visual explanation and some simple exercises."

[0327] Step 6:

[0328] Advice generation using generative AI models

[0329] input:

[0330] The prompt text sent by the server.

[0331] Specific behavior:

[0332] The generative AI model performs natural language processing based on the prompt sentence and generates learning advice.

[0333] Data Calculation:

[0334] Text generation using the model's natural language processing algorithms.

[0335] output:

[0336] Generated advice sentence. Example: "It's a good idea to remember trigonometric formulas like this."

[0337] Step 7:

[0338] Server-based advice sending

[0339] input:

[0340] Advice sentences from a generative AI model.

[0341] Specific behavior:

[0342] The server appropriately formats the generated text advice and transmits it to the smart glasses.

[0343] Data Calculation:

[0344] Text formatting adjustments, data submission via HTTP POST requests.

[0345] output:

[0346] Advice data sent to smart glasses.

[0347] Step 8:

[0348] Data display for smart glasses

[0349] input:

[0350] Advice data sent by the server.

[0351] Specific behavior:

[0352] The smart glasses use an AR engine to provide a visual overlay of the advice received.

[0353] Data Calculation:

[0354] AR engine for visual enhancement and overlay display processing.

[0355] output:

[0356] Advice overlaid on students' field of view.

[0357] Step 9:

[0358] Server updates the database

[0359] input:

[0360] Learning session information (material information, advice, sentiment data, etc.).

[0361] Specific behavior:

[0362] The server records the learning session information and updates a database.

[0363] Data Calculation:

[0364] SQL query operations to create new records and insert data.

[0365] output:

[0366] Updated database.

[0367] The above is an explanation of the system's program processing, broken down into specific steps. It clarifies what data is input, what data processing is performed, and what output is obtained at each step.

[0368] (Application example 2)

[0369] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0370] Conventional work support systems in factories have difficulty providing effective support that is tailored to the worker's situation and emotions, and as a result, they have been unable to fully improve work efficiency and productivity.In addition, workers' anxieties and doubts when attempting new tasks remain unresolved, and training efficiency remains low.

[0371] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes an identification means for identifying the work content based on the work information for a factory worker to work on the work, a generation means for generating advice based on the work content identified by the identification means and the information and work goal of the worker, a control means for overlaying the advice generated by the generation means on the work information using augmented reality technology, a learning means for receiving questions and requests regarding the work from the worker and using them as learning data, and an emotion recognition means for recognizing the emotion of the worker and adjusting the advice generated by the generation means. This makes it possible to provide appropriate support that matches the situation and emotion of the worker in real time, improving work efficiency and making training more efficient.

[0372] A "factory worker" is a person who operates various machines and equipment within a factory and carries out tasks related to the production process.

[0373] "Work information" refers to information regarding the specific work content, procedures, and objects that factory workers must undertake.

[0374] The "specifying means" is a means for specifying the specific work content based on the work information for the factory worker to undertake the work.

[0375] The "generation means" is a means for generating appropriate advice based on the identified work content, the information of the factory worker, and the work goals.

[0376] The "control means" refers to a means for overlaying and displaying the generated advice on the task information using augmented reality technology.

[0377] A "learning means" is a means for receiving questions or requests about work from factory workers and using them as learning data.

[0378] The "emotion recognition means" is a means for recognizing the emotions of a factory worker and adjusting the advice generated by the generation means.

[0379] "Augmented reality technology" is a technology that overlays digital information onto the real world.

[0380] The system for carrying out the present invention provides support in real time for factory workers to improve their work efficiency. The specific configuration and processing of the system will be described below.

[0381] Hardware and Software

[0382] 1. Smart glasses:

[0383] Specific examples: Google(R) Glass(R), Microsoft(R) HoloLens(R)

[0384] Use: A device that obtains visual information from factory workers and displays advice using augmented reality technology.

[0385] 2. Factory robots:

[0386] Uses: Mechanical equipment for automating and supporting work.

[0387] 3. Server:

[0388] A concrete example: Cloud-based servers

[0389] Use: A device that receives visual information and emotion data obtained from smart glasses, analyzes it, and generates prompts for the generative AI model.

[0390] 4. Generative AI models:

[0391] Specific example: OpenAI (registered trademark) ChatGPT

[0392] Use: Natural language processing models to generate appropriate advice based on prompts.

[0393] 5. Emotion Engine:

[0394] Use: An engine for recognizing emotions from the facial expressions of factory workers and tailoring the advice provided by generative AI.

[0395] 6. Database:

[0396] Specific example: MySQL (registered trademark)

[0397] Use: A system for storing information and past learning data of factory workers.

[0398] Processing Description

[0399] 1. Data collection:

[0400] The smart glasses collect visual information and facial expression data from factory workers in real time, making it possible to understand what task they are currently performing and how they are feeling at the time.

[0401] 2. Data processing:

[0402] The server receives visual information and emotion data from the smart glasses, refers to the aforementioned database to identify the task, and simultaneously recognizes emotions using facial expression data and sends prompts based on that information to the generative AI.

[0403] 3. Advice Generation:

[0404] Generative AI models (e.g., OpenAI ChatGPT) generate appropriate technical support and advice for workers based on prompts, addressing their specific questions and concerns.

[0405] 4. Information display:

[0406] The server sends the generated advice to the smart glasses, which use augmented reality technology to overlay it on the worker's field of vision, allowing the worker to see the information he or she needs in real time.

[0407] Examples

[0408] For example, when a worker is setting up new robotic equipment in a factory, the smart glasses will display the operating procedure in real time. Furthermore, if the worker feels anxious, the emotion engine will recognize this and prompt the generative AI to provide appropriate advice.

[0409] Examples of prompt statements

[0410] Task: Setting up a new robotic device. Worker's emotions: Anxiety. Past experience: New to setup. Provide the necessary assistance based on the information displayed in the worker's field of view.

[0411] In this way, the system can provide real-time support that is tailored to the worker's situation and emotions, resulting in improved work efficiency and productivity.

[0412] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0413] Step 1:

[0414] The smart glasses collect visual information and facial expression data from factory workers in real time and send it to a server.

[0415] Input: Factory worker's visual field information, facial expression data

[0416] Output: Vision information and facial expression data sent to the server

[0417] Specific operation: The camera and sensors installed in the smart glasses collect visual information and facial expression data. The collected data is sent to a server using known communication methods.

[0418] Step 2:

[0419] The server analyzes the visual information and facial expression data it receives to identify the task and emotion.

[0420] Input: Vision information and facial expression data sent to the server

[0421] Output: Identified tasks and emotion data

[0422] Specific operation: Using the image analysis algorithm and emotion engine in the server, the task content is analyzed from the field of vision information, and emotions are analyzed from the facial expression data. The field of vision information is compared with the factory environment and equipment to identify the task content, and the facial expression data is analyzed by the emotion engine.

[0423] Step 3:

[0424] The server matches the identified task content and emotion data against a database to extract relevant information and generate prompts.

[0425] Input: Identified task and emotion data

[0426] Output: Generated prompt

[0427] Specific operation: The server compares the information of factory workers stored in the database with past work data and generates prompt sentences to input into the generative AI model.

[0428] Step 4:

[0429] A generative AI model generates appropriate advice based on the prompts.

[0430] Input: Generated prompt

[0431] Output: The generated advice.

[0432] How it works: A generative AI model (e.g., OpenAI ChatGPT) processes the prompt and generates appropriate advice in natural language for the factory worker.

[0433] Step 5:

[0434] The server sends the generated advice to the smart glasses, where it is overlaid on the visual information using augmented reality technology.

[0435] Input: Generated advice

[0436] Output: Advice displayed on smart glasses

[0437] Specific operation: The server sends the generated advice to the smart glasses, which use the augmented reality technology to overlay the advice within the factory worker's field of view.

[0438] Step 6:

[0439] The worker accepts the advice and continues working, and if he has any questions or requests, he sends them to the server via the smart glasses.

[0440] Input: Questions and requests from workers

[0441] Output: The query or request sent to the server.

[0442] How it works: The worker uses voice or gesture input on the smart glasses to send questions or requests to the server, which receives them and records them as learning data.

[0443] Through these steps, the system provides appropriate assistance to factory workers in real time, improving work efficiency and safety.

[0444] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0445] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0446] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0447] [Second embodiment]

[0448] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0449] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0450] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0451] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0452] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0453] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0454] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0455] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0456] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0457] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0458] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0459] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0460] This embodiment is composed of the following elements.

[0461] 1. Smart glasses: A device that allows students to view textbooks, notes, and other educational materials using smart glasses. The smart glasses have the ability to acquire visual information and display it as an AR overlay.

[0462] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs prompts containing learning content and student information into the generative AI and obtains appropriate advice.

[0463] 3. Generative AI: A natural language processing model that generates appropriate advice based on prompts containing learning content and student information. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[0464] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[0465] 5. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[0466] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[0467] The process flow will be explained below.

[0468] Step 1: Processing from the smart glasses

[0469] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[0470] The terminal transmits the obtained visibility information to the server.

[0471] Step 2: Server processing

[0472] The server analyzes the received visual information and identifies the learning content.

[0473] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[0474] The server sends the generated prompts to the generative AI to obtain appropriate advice.

[0475] Step 3: Generative AI processing

[0476] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[0477] The generative AI returns the generated advice to the server.

[0478] Step 4: Processing the response from the server

[0479] The server obtains the advice received from the generative AI.

[0480] The server overlays advice to the student via the smart glasses.

[0481] Step 5: Progression of learning and handling of student questions and requests

[0482] Students use the smart glasses' AR capabilities to watch relevant videos and visual explanations as they learn.

[0483] If a student has any questions or requests, they send them to the server through their smart glasses.

[0484] Step 6: Collect training data and fine-tune generative AI

[0485] The server receives questions and requests from students and stores them as learning data.

[0486] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[0487] Example 1

[0488] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0489] Conventional learning support systems could not effectively utilize information from teaching materials or individual student information, making it difficult to provide appropriate advice and learning content in real time. As a result, students' learning efficiency decreased and achievement of learning goals was often delayed. In addition, there was a problem that students' motivation to learn decreased because they could not respond quickly to their questions and requests.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0491] In this invention, the server includes: a means for acquiring learning material information on the learning material that the student is studying; a means for identifying the learning content based on the learning material information; a means for generating a prompt sentence based on the learning content and the student's information; a means for inputting the prompt sentence into a generative AI model to generate advice; a means for overlaying the generated advice on the learning material using AR technology; a means for referring to a database that stores information such as the student's age, sex, past test results, deviation score, learning goal, and goal achievement deadline; and a means for transmitting the generated advice and related learning content to the smart glasses. This allows the student to efficiently progress through their studies while receiving personalized advice in real time.

[0492] 1. "Materials information" refers to data such as text and images contained in the materials students use for their studies.

[0493] 2. “Content” refers to information about a particular subject or topic that a student is studying.

[0494] 3. "Student information" refers to personal data such as a student's age, gender, past test results, academic ability score, learning goals, and deadlines for achieving those goals.

[0495] 4. “Prompt” refers to an instruction to be input into a generative AI model based on the learning content and student information.

[0496] 5. “Generative AI model” means an artificial intelligence model that uses natural language processing to generate advice or information based on a prompt.

[0497] 6. “Advice” refers to advice or suggestions for learning support that are generated by the generative AI model based on the prompt.

[0498] 7. "AR technology" refers to augmented reality technology that overlays digital information on the actual field of vision.

[0499] 8. "Database" means a database management system for storing information such as students' age, gender, past test results, deviation scores, learning goals, and deadlines for achieving goals.

[0500] 9. "Smart glasses" means a wearable device for displaying digital information using augmented reality technology.

[0501] 10. “Learning content” means digital information, such as video or visual explanations, that supplements student learning.

[0502] The present invention relates to a system for supporting student learning, and more particularly to a system for providing appropriate advice and learning content in real time by utilizing teaching material information and student information. A specific embodiment for carrying out the present invention will be described.

[0503] System Overview

[0504] The system mainly consists of smart glasses, a server, a generative AI model, a database, and learning content.

[0505] Smart Glasses

[0506] Smart glasses are wearable devices that students use when viewing textbooks, notes, and other study materials. The glasses are equipped with a built-in camera and have the ability to capture visual information such as text and images from study materials. They also have a function that uses AR (Augmented Reality) technology to overlay advice generated based on the captured information.

[0507] server

[0508] The server receives the visual information sent from the smart glasses and identifies the learning content. The server accesses the database to obtain individual information about the students. It also generates a prompt sentence based on the learning content and the student information, and inputs the prompt sentence into the generative AI model.

[0509] Generative AI Models

[0510] The generative AI model uses natural language processing to generate advice based on the prompt sentence. For example, ChatGPT is used for this AI model. It receives the prompt sentence as input, generates appropriate advice, and returns it to the server.

[0511] Database

[0512] The database stores personal information such as the student's age, gender, past test results, deviation score, learning goals, deadline for achieving the goal, etc. The server accesses this database to obtain the necessary information and uses it to generate prompt sentences.

[0513] Learning Content

[0514] Learning content is digital information to supplement students' learning, such as videos and visual explanations. The server transmits the learning content to the smart glasses as needed, allowing students to view it in real time.

[0515] Examples

[0516] Students solving math problems

[0517] 1. Viewing the materials:

[0518] A user opens a math textbook and begins looking at a problem through the smart glasses, which use their built-in camera to capture the mathematical equation "y = ax^2 + bx + c."

[0519] 2. Transmission of information:

[0520] The smart glasses transmit the captured mathematical formula information to a server.

[0521] 3. Generate prompt:

[0522] The server retrieves student information from the database and generates a prompt statement such as, "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem."

[0523] 4. Generate advice:

[0524] The server inputs a prompt sentence into the generative AI model, which then generates advice such as, "First, use the basic form of the quadratic function, y = ax^2 + bx + c, to confirm the values ​​of a, b, and c. Then, I will explain how to solve it using the solution formula."

[0525] 5. Display Advice:

[0526] The server sends the generated advice to the smart glasses, which overlay it on the user's field of view.

[0527] 6. Provision of learning content:

[0528] If the user wants to understand further, the server will send the relevant explanatory video to the smart glasses so that the user can watch the video.

[0529] Examples of prompt statements

[0530] "A 17-year-old male student A is trying to solve a quadratic function problem. I would like some specific advice on how to solve quadratic functions."

[0531] In this way, the system of the present invention can effectively support students' learning and improve their learning efficiency.

[0532] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0533] Step 1:

[0534] The user browses the learning materials through smart glasses. The glasses use a built-in camera to capture visual information such as text and images from the learning materials. Optical character recognition (OCR) software is then used to extract text data from the captured information.

[0535] Input: Visual information from textbooks and notes

[0536] Output: Extracted text data (e.g. "y = ax^2 + bx + c")

[0537] Step 2:

[0538] The smart glasses transmit the extracted text data to a server using known communication means, and the server receives the data and begins analyzing it.

[0539] Input: Text data (e.g. "y = ax^2 + bx + c")

[0540] Output: Received text data

[0541] Step 3:

[0542] The server accesses the database and obtains information about the student (age, gender, past test results, deviation score, learning goals, deadline for achieving goals, etc.). Based on the obtained information, a prompt sentence is generated according to the current learning situation. The prompt sentence is a specific instruction sentence to be passed to the generative AI model.

[0543] Input: Received text data, student information stored in database

[0544] Output: Prompt statement (e.g. "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem.")

[0545] Step 4:

[0546] The server inputs the generated prompt sentence into a generative AI model (e.g., ChatGPT). The generative AI model generates appropriate advice based on the prompt sentence. The generative AI model analyzes information about the learning content and creates advice in a form that is easy for the user to understand.

[0547] Input: Prompt statement

[0548] Output: Generated advice (e.g. "First, use the basic form of the quadratic function y = ax^2 + bx + c to find the values ​​of a, b, and c. Then, use the solution formula to explain how to solve the problem.")

[0549] Step 5:

[0550] The server sends the generated advice to the smart glasses, which overlay the advice on the user's field of view. The displayed advice is integrated into the field of view in a natural way using AR technology.

[0551] Input: Generated advice

[0552] Output: Advice overlaid on the user's view

[0553] Step 6:

[0554] If necessary, the server sends relevant learning content (videos and visual explanations) to the smart glasses, which use their AR capabilities to enable the user to watch this learning content in real time.

[0555] Input: User request, related learning content

[0556] Output: Learning content overlaid on the user's field of view

[0557] (Application example 1)

[0558] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0559] Current factory robots often lack immediate support for work procedures and problem solving when performing complex tasks or troubleshooting. This leads to issues such as reduced work efficiency and increased downtime. Conventional methods require robot operators to refer to the manual, which can take time to troubleshoot. There is a need to improve this situation and increase work efficiency.

[0560] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0561] In this invention, the server includes: a specifying means for specifying a task content based on task information related to a task to be performed by the robot; a generating means for generating advice based on the task content specified by the specifying means, the robot information, and a task goal; a control means for overlaying the advice generated by the generating means on the task using an augmented reality (AR) technology; and a learning means for receiving questions and requests related to the task from the robot and using them as task data. This enables the robot to receive necessary advice in real time, making it possible to dramatically improve task efficiency.

[0562] A "robot" is a mechanical device that automates tasks in a factory.

[0563] "Work information" is information relating to the specific work content that the robot will undertake.

[0564] The "identification means" is a function for identifying the work content based on the work information.

[0565] "Work content" refers to the series of work procedures and specific work instructions that the robot must perform.

[0566] "Robot Information" means information about a robot, including its identity, current status, and performance data.

[0567] "Task goals" are information that sets specific goals and deadlines that the robot must achieve.

[0568] The "generation means" is a function for generating advice based on the work content identified by the identification means and other information.

[0569] "Advice" is any instruction or recommendation given to assist in the progress of work.

[0570] "AR technology" is a general term for technology that displays virtual information overlaid on real visual information.

[0571] The "control means" is a function for displaying the advice generated by the generation means using AR technology.

[0572] The "learning means" is a function that receives questions and requests from the robot regarding tasks and uses them to improve the system and generate advice.

[0573] The present invention is a system for improving the work efficiency of factory robots. This system identifies the work content based on work information related to the work performed by the robot and provides advice in real time, thereby significantly improving the efficiency of work stagnation and troubleshooting.

[0574] System Configuration

[0575] The system configuration is as follows.

[0576] 1. Robot:

[0577] Robots are installed in factories to perform tasks automatically. They are equipped with cameras that can capture images of the tasks they perform.

[0578] 2. Server:

[0579] The server has the following means:

[0580] Identification method: Analyze images obtained from the robot's camera and other work information to identify the current work content.

[0581] Generation means: Generates appropriate advice based on the task content identified by the identification means and the robot information (identification information, status information, etc.). Here, advice is generated using a generative AI model.

[0582] Control means: The generated advice is overlaid on the robot's field of view using AR technology.

[0583] Learning method: Receive questions and requests from the robot regarding the task, and use them to improve the system and generate advice.

[0584] 3. AR technology:

[0585] Using augmented reality (AR) technology, the generated advice is superimposed on the robot's field of vision, allowing the robot to obtain the necessary information in real time.

[0586] Processing flow

[0587] 1. The robot's camera captures footage of the work being done.

[0588] 2. The acquired video is sent to the server.

[0589] 3. The server's identification means analyzes the video and identifies the work content.

[0590] 4. The server's generation means generates advice based on the identified work content and robot information.

[0591] 5. The generated advice is overlaid on the robot's field of view using AR technology via the server's control means.

[0592] 6. Receive questions and requests from the robot, store them in a database as a learning tool, and use them to generate advice next time.

[0593] Hardware and Software Used

[0594] Hardware: Robots, cameras, servers

[0595] Software: Generative AI model (generation method), OpenCV (image analysis), requests (HTTP requests), AR module (AR technology)

[0596] Specific examples

[0597] As a concrete example, consider a scenario in which a robot is repairing a machine in a factory. In this case, if the robot does not know the location of a part, the image of the area around the part captured by the camera is sent to the server. The server analyzes the image and identifies the location of the part, then generates advice on how to handle the part and the repair procedure, which it displays in the robot's field of view. This advice display is done using AR technology, so the robot can proceed with the work while receiving visual support.

[0598] Examples of prompt statements

[0599] Below are examples of prompts to be input to the generative AI model.

[0600] "Machine X is missing part Y. The current work step is step 3."

[0601] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0602] Step 1:

[0603] The robot's camera captures footage of the work.

[0604] Input: Robot working environment

[0605] Output: Camera image

[0606] How it works: The robot's camera films the work area and captures footage in real time.

[0607] Step 2:

[0608] The captured image is sent to the server.

[0609] Input: Camera image

[0610] Output: Camera image data sent to the server

[0611] Operation: Images captured by the camera are sent to a server via the Internet or local network.

[0612] Step 3:

[0613] The server's identification means analyzes the video and identifies the work content.

[0614] Input: Camera video data sent to the server

[0615] Output: Identified work content information

[0616] Operation: The server uses video analysis software (e.g. OpenCV) to recognize objects and text information in the video and identify the current task.

[0617] Step 4:

[0618] A generating means of the server generates advice based on the identified work content and information on the robot.

[0619] Input: Identified task information and robot information (identification information, status information, etc.)

[0620] Output: Generated advice

[0621] How it works: The server inputs a prompt sentence into the generative AI model, which then generates appropriate advice. Example of a prompt sentence: "Part Y of machine X is in short supply. The current work procedure is step 3."

[0622] Step 5:

[0623] The generated advice is overlaid on the robot's field of view using AR technology by the server's control means.

[0624] Input: Generated advice

[0625] Output: Advice overlaid on the robot's field of vision

[0626] Operation: The server's control means uses the AR module to superimpose the generated advice onto the robot's camera image and display it.

[0627] Step 6:

[0628] The system receives questions and requests from the robot, stores them in a database as a learning tool, and uses them to generate advice the next time.

[0629] Input: Question or request from the robot

[0630] Output: Question and request data stored in a database

[0631] How it works: The robot sends any additional information, questions or requests it needs via the user interface to the server, which then stores the data in a database, where it can be used as learning data to generate advice the next time.

[0632] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0633] This embodiment is composed of the following elements.

[0634] 1. Smart glasses: A device that allows students to view textbooks, notes, and other educational materials using smart glasses. The smart glasses have the ability to acquire visual information and display it as an AR overlay.

[0635] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs learning content, student information, and prompts including the student's emotions recognized by the emotion engine into the generative AI to obtain appropriate advice.

[0636] 3. Generative AI: A natural language processing model to generate appropriate advice based on prompts including learning content, student information, and student sentiment. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[0637] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[0638] 5. Emotion Engine: An engine for recognizing and analyzing student emotions. The emotion engine uses student emotions as learning data and is used to adjust the difficulty of learning and the style of advice.

[0639] 6. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[0640] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[0641] The process flow will be explained below.

[0642] Step 1: Processing from the smart glasses

[0643] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[0644] The terminal transmits the acquired visual information and the student's emotions to the emotion engine for emotion recognition and analysis.

[0645] The terminal transmits the results received from the emotion engine to the server.

[0646] Step 2: Server processing

[0647] The server analyzes the received visual information and identifies the learning content.

[0648] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[0649] The server sends the generated prompts and results from the emotion engine to the generative AI to obtain appropriate advice.

[0650] Step 3: Generative AI processing

[0651] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[0652] The generative AI returns the generated advice to the server.

[0653] Step 4: Processing the response from the server

[0654] The server obtains the advice received from the generative AI.

[0655] The server overlays advice to the student via the smart glasses.

[0656] Step 5: Progression of learning and handling of student questions and requests

[0657] Students can use the AR function to view learning content and progress through their studies.

[0658] If a student has any questions or requests, they send them to the server through their smart glasses.

[0659] Step 6: Collect training data and fine-tune generative AI

[0660] The server receives questions and requests from students and stores them as learning data.

[0661] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[0662] Example 2

[0663] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0664] Conventional learning support systems have difficulty in providing individual advice to students when they are studying. In addition, they are unable to grasp the emotional state of students in real time, and therefore are unable to provide appropriate advice or adjust the difficulty of learning accordingly. This leads to a problem of a decrease in overall learning efficiency.

[0665] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for identifying learning contents based on learning material information related to the learning materials that the student is studying, a means for generating advice based on the learning contents identified by the identification means and the student's information and learning goals, a means for overlaying the advice generated by the generation means on the learning materials using augmented reality technology, a means for recognizing the student's emotions and adjusting the difficulty of the learning and the style of the advice, and a means for receiving questions and requests related to the learning from the student and using them as learning data. This makes it possible to provide appropriate advice according to the individual learning situation and emotional state of the student, thereby improving learning efficiency.

[0666] A "student" is a person who studies certain teaching materials or learning content.

[0667] "Learning material information" refers to information related to learning materials such as textbooks and notebooks used by students.

[0668] "Learning content" refers to specific learning items or themes that are specified based on the learning material information.

[0669] "Student information" means individual data about a student, such as the student's age, gender, past test results, and learning history.

[0670] A "learning objective" is a specific learning purpose or goal that a student wants to achieve.

[0671] "Advice" refers to the guidance or advice provided to assist a student in their learning.

[0672] A "generative model" is an algorithm or software that generates information or advice based on input data.

[0673] "Augmented reality technology" is a technology that displays digital information overlaid on the real world.

[0674] "Means of recognizing emotions" refers to technology or devices for understanding a student's emotional state from their facial expressions, voice, and other sensor information.

[0675] "Learning data" is data obtained during the course of a student's studies, and is information that can be analyzed to provide learning support.

[0676] In order to implement this invention, it is necessary to construct a system including the following elements:

[0677] Hardware and Software Used

[0678] 1. Smart Glasses

[0679] Smart glasses are devices that allow users to view learning materials such as textbooks and notes. The smart glasses are equipped with a camera function to acquire visual information about the learning materials. They also use AR (Augmented Reality) technology to overlay appropriate learning content based on the acquired information.

[0680] 2. Server

[0681] The server receives visual information obtained from the smart glasses and identifies the learning content. Specifically, it uses OCR technology to analyze the text information of the teaching materials and combines it with the student information in the database and the student's emotion data recognized by the emotion engine. As a result, it automatically generates prompt sentences to input into the generative AI model.

[0682] 3. Generative AI Models

[0683] The generative AI model uses natural language processing techniques to generate appropriate advice from prompts, and is fine-tuned using training data to provide customized advice based on learning content and student requests.

[0684] 4. Database

[0685] The database contains information about students, such as their age, gender, past test results, academic ability scores, learning goals, and deadlines for achieving those goals. The server refers to this database and uses it as auxiliary information when generating prompt sentences.

[0686] 5. Emotion Engine

[0687] The emotion engine is software for analyzing students' emotional state in real time. It is used to adjust the difficulty of learning and style of advising, and recognizes emotions from students' facial expressions and voice.

[0688] 6. Learning Content

[0689] Learning content is digital content that complements the learning content, such as relevant videos and visual explanations, and is visually presented to students using the AR capabilities of the smart glasses.

[0690] Examples

[0691] Suppose a user wears smart glasses and is studying trigonometric formulas in a math textbook. The glasses capture the image data of the study material and send it to the server. The server uses OCR to extract text information from the image data and identifies the study content based on this. At the same time, the emotion engine analyzes the student's facial expression and obtains emotion data such as confusion.

[0692] The server combines the teaching material information, student information in the database, and emotion data to generate a prompt sentence like the following:

[0693] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[0694] When this prompt is fed into a generative AI model, the model generates advice based on it, like this:

[0695] "Trigonometric formulas are easier to understand when they are represented visually like this. Try solving the following exercise..."

[0696] The server sends this advice to the smart glasses, which then use AR technology to overlay the advice on the student.

[0697] Examples of prompt statements

[0698] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[0699] This will improve students' learning efficiency and provide customized support tailored to their individual learning situations.

[0700] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0701] Explain the flow of system program processing by dividing it into processing steps

[0702] Step 1:

[0703] Data acquisition using smart glasses

[0704] input:

[0705] The textbook, notes, or other educational material that the user is viewing.

[0706] Specific behavior:

[0707] The camera in the smart glasses captures image data of the teaching materials.

[0708] Data processing:

[0709] Image processing software is used to perform text recognition and image analysis to identify visible information in the teaching materials.

[0710] output:

[0711] Image data and text information of captured teaching materials.

[0712] Step 2:

[0713] Data received by the server

[0714] input:

[0715] Image data and text information of educational materials from smart glasses.

[0716] Specific behavior:

[0717] The smart glasses transmit the data to a server via Wi-Fi.

[0718] The server receives this data on the specified port.

[0719] Data Calculation:

[0720] The server uses OCR technology (Optical Character Recognition) to extract text information from the image data.

[0721] output:

[0722] Text information analyzed by OCR and the original image data.

[0723] Step 3:

[0724] Server determines what to learn

[0725] input:

[0726] Text information extracted by OCR.

[0727] Specific behavior:

[0728] The server analyzes the text information and again queries the database to identify the educational material content.

[0729] Calculate the degree of match between learning history and teaching materials.

[0730] Data Calculation:

[0731] Database query operations and text analysis using natural language processing.

[0732] output:

[0733] Identified learning content.

[0734] Step 4:

[0735] Emotion recognition by emotion engine

[0736] input:

[0737] Facial and voice data of students.

[0738] Specific behavior:

[0739] The emotion engine captures students' facial expressions and voice data through sensors in the smart glasses.

[0740] Data Calculation:

[0741] Uses facial recognition and voice analysis algorithms to assess student emotions in real time.

[0742] output:

[0743] Perceived emotion data (e.g., confusion, excitement, happiness).

[0744] Step 5:

[0745] Server-generated prompts

[0746] input:

[0747] Identified learning content, student information from the database, and sentiment data.

[0748] Specific behavior:

[0749] The server generates a prompt based on this data.

[0750] Data Calculation:

[0751] Use data integration and prompt generation algorithms to create appropriate prompts.

[0752] output:

[0753] Generated prompt sentence. Example: "My student is having trouble understanding trigonometric formulas, so please generate a visual explanation and some simple exercises."

[0754] Step 6:

[0755] Advice generation using generative AI models

[0756] input:

[0757] The prompt text sent by the server.

[0758] Specific behavior:

[0759] The generative AI model performs natural language processing based on the prompt sentence and generates learning advice.

[0760] Data Calculation:

[0761] Text generation using the model's natural language processing algorithms.

[0762] output:

[0763] Generated advice sentence. Example: "It's a good idea to remember trigonometric formulas like this."

[0764] Step 7:

[0765] Server-based advice sending

[0766] input:

[0767] Advice sentences from a generative AI model.

[0768] Specific behavior:

[0769] The server appropriately formats the generated text advice and transmits it to the smart glasses.

[0770] Data Calculation:

[0771] Text formatting adjustments, data submission via HTTP POST requests.

[0772] output:

[0773] Advice data sent to smart glasses.

[0774] Step 8:

[0775] Data display in smart glasses

[0776] input:

[0777] Advice data sent by the server.

[0778] Specific behavior:

[0779] The smart glasses use an AR engine to provide a visual overlay of the advice received.

[0780] Data Calculation:

[0781] AR engine for visual enhancement and overlay display processing.

[0782] output:

[0783] Advice overlaid on students' field of view.

[0784] Step 9:

[0785] Server updates the database

[0786] input:

[0787] Learning session information (material information, advice, sentiment data, etc.).

[0788] Specific behavior:

[0789] The server records the learning session information and updates a database.

[0790] Data Calculation:

[0791] SQL query operations to create new records and insert data.

[0792] output:

[0793] Updated database.

[0794] The above is an explanation of the system's program processing divided into specific steps. It clarifies what data is input, what data processing is performed, and what output is obtained at each step.

[0795] (Application example 2)

[0796] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0797] Conventional work support systems in factories have difficulty providing effective support that is tailored to the worker's situation and emotions, and as a result, they have been unable to fully improve work efficiency and productivity.In addition, workers' anxieties and doubts when attempting new tasks remain unresolved, and training efficiency remains low.

[0798] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes an identification means for identifying the work content based on the work information for a factory worker to work on the work, a generation means for generating advice based on the work content identified by the identification means and the information and work goal of the worker, a control means for overlaying the advice generated by the generation means on the work information using augmented reality technology, a learning means for receiving questions and requests regarding the work from the worker and using them as learning data, and an emotion recognition means for recognizing the emotion of the worker and adjusting the advice generated by the generation means. This makes it possible to provide appropriate support that matches the situation and emotion of the worker in real time, improving work efficiency and making training more efficient.

[0799] A "factory worker" is a person who operates various machines and equipment within a factory and carries out tasks related to the production process.

[0800] "Work information" refers to information regarding the specific work content, procedures, and objects that factory workers must perform.

[0801] The "specifying means" is a means for specifying the specific work content based on the work information for the factory worker to undertake the work.

[0802] The "generation means" is a means for generating appropriate advice based on the identified work content, information on the factory worker, and work goals.

[0803] The "control means" refers to a means for overlaying and displaying the generated advice on the task information using augmented reality technology.

[0804] A "learning means" is a means for receiving questions or requests about work from factory workers and using them as learning data.

[0805] The "emotion recognition means" is a means for recognizing the emotions of a factory worker and adjusting the advice generated by the generation means.

[0806] "Augmented reality technology" is a technology that overlays digital information onto the real world.

[0807] The system for carrying out the present invention provides support in real time for factory workers to improve their work efficiency. The specific configuration and processing of the system will be described below.

[0808] Hardware and Software

[0809] 1. Smart glasses:

[0810] Use: A device that obtains visual information from factory workers and displays advice using augmented reality technology.

[0811] 2. Factory robots:

[0812] Uses: Mechanical equipment for automating and supporting work.

[0813] 3. Server:

[0814] A concrete example: Cloud-based servers

[0815] Use: A device that receives visual information and emotion data obtained from smart glasses, analyzes it, and generates prompts for the generative AI model.

[0816] 4. Generative AI models:

[0817] Specific example: OpenAI ChatGPT

[0818] Use: Natural language processing models to generate appropriate advice based on prompts.

[0819] 5. Emotion Engine:

[0820] Use: An engine for recognizing emotions from the facial expressions of factory workers and tailoring the advice provided by generative AI.

[0821] 6. Database:

[0822] Specific example: MySQL

[0823] Use: A system for storing information and past learning data of factory workers.

[0824] Processing Description

[0825] 1. Data collection:

[0826] The smart glasses collect visual information and facial expression data from factory workers in real time, making it possible to understand what task they are currently performing and how they are feeling at the time.

[0827] 2. Data processing:

[0828] The server receives visual information and emotion data from the smart glasses, refers to the aforementioned database to identify the task, and simultaneously recognizes emotions using facial expression data and sends prompts based on that information to the generative AI.

[0829] 3. Advice Generation:

[0830] Generative AI models (e.g., OpenAI ChatGPT) generate appropriate technical support and advice for workers based on prompts, addressing their specific questions and concerns.

[0831] 4. Information display:

[0832] The server sends the generated advice to the smart glasses, which use augmented reality technology to overlay it on the worker's field of vision, allowing the worker to see the information he or she needs in real time.

[0833] Examples

[0834] For example, when a worker is setting up new robotic equipment in a factory, the smart glasses will display the operating procedure in real time. Furthermore, if the worker feels anxious, the emotion engine will recognize this and prompt the generative AI for appropriate advice.

[0835] Examples of prompt statements

[0836] Task: Setting up a new robotic device. Worker's emotions: Anxiety. Past experience: New to setup. Provide the necessary assistance based on the information displayed in the worker's field of view.

[0837] In this way, the system can provide real-time support that is tailored to the worker's situation and emotions, resulting in improved work efficiency and productivity.

[0838] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0839] Step 1:

[0840] The smart glasses collect visual information and facial expression data from factory workers in real time and send it to a server.

[0841] Input: Factory worker's visual field information, facial expression data

[0842] Output: Vision information and facial expression data sent to the server

[0843] Specific operation: The camera and sensors installed in the smart glasses collect visual information and facial expression data. The collected data is sent to a server using known communication methods.

[0844] Step 2:

[0845] The server analyzes the visual information and facial expression data it receives to identify the task and emotion.

[0846] Input: Vision information and facial expression data sent to the server

[0847] Output: Identified tasks and emotion data

[0848] Specific operation: Using the image analysis algorithm and emotion engine in the server, the task content is analyzed from the field of vision information, and emotions are analyzed from the facial expression data. The field of vision information is compared with the factory environment and equipment to identify the task content, and the facial expression data is analyzed by the emotion engine.

[0849] Step 3:

[0850] The server matches the identified task content and emotion data against a database to extract relevant information and generate prompts.

[0851] Input: Identified task and emotion data

[0852] Output: Generated prompt

[0853] Specific operation: The server compares the information of factory workers stored in the database with past work data and generates prompt sentences to input into the generative AI model.

[0854] Step 4:

[0855] A generative AI model generates appropriate advice based on the prompts.

[0856] Input: Generated prompt

[0857] Output: The generated advice.

[0858] How it works: A generative AI model (e.g., OpenAI ChatGPT) processes the prompt and generates appropriate advice in natural language for the factory worker.

[0859] Step 5:

[0860] The server sends the generated advice to the smart glasses, where it is overlaid on the visual information using augmented reality technology.

[0861] Input: Generated advice

[0862] Output: Advice displayed on smart glasses

[0863] Specific operation: The server sends the generated advice to the smart glasses, which use the augmented reality technology to overlay the advice within the factory worker's field of view.

[0864] Step 6:

[0865] The worker accepts the advice and continues working, and if he has any questions or requests, he sends them to the server via the smart glasses.

[0866] Input: Questions and requests from workers

[0867] Output: The query or request sent to the server.

[0868] How it works: The worker uses voice or gesture input on the smart glasses to send questions or requests to the server, which receives them and records them as learning data.

[0869] Through these steps, the system provides appropriate assistance to factory workers in real time, improving work efficiency and safety.

[0870] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0871] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0872] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0873] [Third embodiment]

[0874] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0875] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0876] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0877] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0878] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0879] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0880] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0881] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0882] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0883] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0884] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0885] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server", and the headset type terminal 314 will be referred to as the "terminal".

[0886] This embodiment is composed of the following elements.

[0887] 1. Smart glasses: A device that allows students to view textbooks, notes, and other educational materials using smart glasses. The smart glasses have the ability to acquire visual information and display it as an AR overlay.

[0888] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs prompts containing learning content and student information into the generative AI and obtains appropriate advice.

[0889] 3. Generative AI: A natural language processing model that generates appropriate advice based on prompts containing learning content and student information. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[0890] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[0891] 5. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[0892] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[0893] The process flow will be explained below.

[0894] Step 1: Processing from the smart glasses

[0895] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[0896] The terminal transmits the obtained visibility information to the server.

[0897] Step 2: Server processing

[0898] The server analyzes the received visual information and identifies the learning content.

[0899] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[0900] The server sends the generated prompts to the generative AI to obtain appropriate advice.

[0901] Step 3: Generative AI processing

[0902] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[0903] The generative AI returns the generated advice to the server.

[0904] Step 4: Processing the response from the server

[0905] The server obtains the advice received from the generative AI.

[0906] The server overlays advice to the student via the smart glasses.

[0907] Step 5: Progression of learning and handling of student questions and requests

[0908] Students use the smart glasses' AR capabilities to watch relevant videos and visual explanations as they learn.

[0909] If a student has any questions or requests, they send them to the server through their smart glasses.

[0910] Step 6: Collect training data and fine-tune generative AI

[0911] The server receives questions and requests from students and stores them as learning data.

[0912] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[0913] Example 1

[0914] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0915] Conventional learning support systems could not effectively utilize information from teaching materials or individual student information, making it difficult to provide appropriate advice and learning content in real time. As a result, students' learning efficiency decreased and achievement of learning goals was often delayed. In addition, there was a problem that students' motivation to learn decreased because they could not respond quickly to their questions and requests.

[0916] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0917] In this invention, the server includes: a means for acquiring learning material information on the learning material that the student is studying; a means for identifying the learning content based on the learning material information; a means for generating a prompt sentence based on the learning content and the student's information; a means for inputting the prompt sentence into a generative AI model to generate advice; a means for overlaying the generated advice on the learning material using AR technology; a means for referring to a database that stores information such as the student's age, sex, past test results, deviation score, learning goal, and goal achievement deadline; and a means for transmitting the generated advice and related learning content to the smart glasses. This allows the student to efficiently progress through their studies while receiving personalized advice in real time.

[0918] 1. "Materials information" refers to data such as text and images contained in the materials students use for their studies.

[0919] 2. “Content” refers to information about a particular subject or topic that a student is studying.

[0920] 3. "Student information" refers to personal data such as a student's age, gender, past test results, academic ability score, learning goals, and deadlines for achieving those goals.

[0921] 4. “Prompt” refers to an instruction to be input into a generative AI model based on the learning content and student information.

[0922] 5. “Generative AI model” means an artificial intelligence model that uses natural language processing to generate advice or information based on a prompt.

[0923] 6. “Advice” refers to advice or suggestions for learning support that are generated by the generative AI model based on the prompt.

[0924] 7. "AR technology" refers to augmented reality technology that overlays digital information on the actual field of vision.

[0925] 8. "Database" means a database management system for storing information such as students' age, gender, past test results, deviation scores, learning goals, and deadlines for achieving goals.

[0926] 9. "Smart glasses" means a wearable device for displaying digital information using augmented reality technology.

[0927] 10. “Learning content” means digital information, such as video or visual explanations, that supplements student learning.

[0928] The present invention relates to a system for supporting student learning, and more particularly to a system for providing appropriate advice and learning content in real time by utilizing teaching material information and student information. A specific embodiment for carrying out the present invention will be described.

[0929] System Overview

[0930] The system mainly consists of smart glasses, a server, a generative AI model, a database, and learning content.

[0931] Smart Glasses

[0932] Smart glasses are wearable devices that students use when viewing textbooks, notes, and other study materials. The glasses are equipped with a built-in camera and have the ability to capture visual information such as text and images from study materials. They also have a function that uses AR (Augmented Reality) technology to overlay advice generated based on the captured information.

[0933] server

[0934] The server receives the visual information sent from the smart glasses and identifies the learning content. The server accesses the database to obtain individual information about the students. It also generates a prompt sentence based on the learning content and the student information, and inputs the prompt sentence into the generative AI model.

[0935] Generative AI Models

[0936] The generative AI model uses natural language processing to generate advice based on the prompt sentence. For example, ChatGPT is used for this AI model. It receives the prompt sentence as input, generates appropriate advice, and returns it to the server.

[0937] Database

[0938] The database stores personal information such as the student's age, gender, past test results, deviation score, learning goals, deadline for achieving the goal, etc. The server accesses this database to obtain the necessary information and uses it to generate prompt sentences.

[0939] Learning Content

[0940] Learning content is digital information to supplement students' learning, such as videos and visual explanations. The server transmits the learning content to the smart glasses as needed, allowing students to view it in real time.

[0941] Examples

[0942] Students solving math problems

[0943] 1. Viewing the materials:

[0944] A user opens a math textbook and begins looking at a problem through the smart glasses, which use their built-in camera to capture the mathematical equation "y = ax^2 + bx + c."

[0945] 2. Transmission of information:

[0946] The smart glasses transmit the captured mathematical formula information to a server.

[0947] 3. Generate prompt:

[0948] The server retrieves student information from the database and generates a prompt statement such as, "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem."

[0949] 4. Generate advice:

[0950] The server inputs a prompt sentence into the generative AI model, which then generates advice such as, "First, use the basic form of the quadratic function, y = ax^2 + bx + c, to confirm the values ​​of a, b, and c. Then, I will explain how to solve it using the solution formula."

[0951] 5. Display Advice:

[0952] The server sends the generated advice to the smart glasses, which overlay it on the user's field of view.

[0953] 6. Provision of learning content:

[0954] If the user wants to understand further, the server will send the relevant explanatory video to the smart glasses so that the user can watch the video.

[0955] Examples of prompt statements

[0956] "A 17-year-old male student A is trying to solve a quadratic function problem. I would like some specific advice on how to solve quadratic functions."

[0957] In this way, the system of the present invention can effectively support students' learning and improve their learning efficiency.

[0958] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0959] Step 1:

[0960] The user browses the learning materials through smart glasses. The glasses use a built-in camera to capture visual information such as text and images from the learning materials. Optical character recognition (OCR) software is then used to extract text data from the captured information.

[0961] Input: Visual information from textbooks and notes

[0962] Output: Extracted text data (e.g. "y = ax^2 + bx + c")

[0963] Step 2:

[0964] The smart glasses transmit the extracted text data to a server using known communication means, and the server receives the data and begins analyzing it.

[0965] Input: Text data (e.g. "y = ax^2 + bx + c")

[0966] Output: Received text data

[0967] Step 3:

[0968] The server accesses the database and obtains information about the student (age, gender, past test results, deviation score, learning goals, deadline for achieving goals, etc.). Based on the obtained information, a prompt sentence is generated according to the current learning situation. The prompt sentence is a specific instruction sentence to be passed to the generative AI model.

[0969] Input: Received text data, student information stored in database

[0970] Output: Prompt statement (e.g. "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem.")

[0971] Step 4:

[0972] The server inputs the generated prompt sentence into a generative AI model (e.g., ChatGPT). The generative AI model generates appropriate advice based on the prompt sentence. The generative AI model analyzes information about the learning content and creates advice in a form that is easy for the user to understand.

[0973] Input: Prompt statement

[0974] Output: Generated advice (e.g. "First, use the basic form of the quadratic function y = ax^2 + bx + c to find the values ​​of a, b, and c. Then, use the solution formula to explain how to solve the problem.")

[0975] Step 5:

[0976] The server sends the generated advice to the smart glasses, which overlay the advice on the user's field of view. The displayed advice is integrated into the field of view in a natural way using AR technology.

[0977] Input: Generated advice

[0978] Output: Advice overlaid on the user's view

[0979] Step 6:

[0980] If necessary, the server sends relevant learning content (videos and visual explanations) to the smart glasses, which use their AR capabilities to enable the user to watch this learning content in real time.

[0981] Input: User request, related learning content

[0982] Output: Learning content overlaid on the user's field of view

[0983] (Application example 1)

[0984] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0985] Current factory robots often lack immediate support for work procedures and problem solving when performing complex tasks or troubleshooting. This leads to issues such as reduced work efficiency and increased downtime. Conventional methods require robot operators to refer to the manual, which can take time to troubleshoot. There is a need to improve this situation and increase work efficiency.

[0986] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0987] In this invention, the server includes: a specifying means for specifying a task content based on task information related to a task to be performed by the robot; a generating means for generating advice based on the task content specified by the specifying means, the robot information, and a task goal; a control means for overlaying the advice generated by the generating means on the task using an augmented reality (AR) technology; and a learning means for receiving questions and requests related to the task from the robot and using them as task data. This enables the robot to receive necessary advice in real time, making it possible to dramatically improve task efficiency.

[0988] A "robot" is a mechanical device that automates tasks in a factory.

[0989] "Work information" is information relating to the specific work content that the robot will undertake.

[0990] The "identification means" is a function for identifying the work content based on the work information.

[0991] "Work content" refers to the series of work procedures and specific work instructions that the robot must perform.

[0992] "Robot Information" means information about a robot, including its identity, current status, and performance data.

[0993] "Task goals" are information that sets specific goals and deadlines that the robot must achieve.

[0994] The "generation means" is a function for generating advice based on the work content identified by the identification means and other information.

[0995] "Advice" is any instruction or recommendation given to assist in the progress of work.

[0996] "AR technology" is a general term for technology that displays virtual information overlaid on real visual information.

[0997] The "control means" is a function for displaying the advice generated by the generation means using AR technology.

[0998] The "learning means" is a function that receives questions and requests from the robot regarding tasks and uses them to improve the system and generate advice.

[0999] The present invention is a system for improving the work efficiency of factory robots. This system identifies the work content based on work information related to the work performed by the robot and provides advice in real time, thereby significantly improving the efficiency of work stagnation and troubleshooting.

[1000] System Configuration

[1001] The system configuration is as follows.

[1002] 1. Robot:

[1003] Robots are installed in factories to perform tasks automatically. They are equipped with cameras that can capture images of the tasks they perform.

[1004] 2. Server:

[1005] The server has the following means:

[1006] Identification method: Analyze images obtained from the robot's camera and other work information to identify the current work content.

[1007] Generation means: Generates appropriate advice based on the task content identified by the identification means and the robot information (identification information, status information, etc.). Here, advice is generated using a generative AI model.

[1008] Control means: The generated advice is overlaid on the robot's field of view using AR technology.

[1009] Learning method: Receive questions and requests from the robot regarding the task, and use them to improve the system and generate advice.

[1010] 3. AR technology:

[1011] Using augmented reality (AR) technology, the generated advice is superimposed on the robot's field of vision, allowing the robot to obtain the necessary information in real time.

[1012] Processing flow

[1013] 1. The robot's camera captures footage of the work being done.

[1014] 2. The acquired video is sent to the server.

[1015] 3. The server's identification means analyzes the video and identifies the work content.

[1016] 4. The server's generation means generates advice based on the identified work content and robot information.

[1017] 5. The generated advice is overlaid on the robot's field of view using AR technology via the server's control means.

[1018] 6. Receive questions and requests from the robot, store them in a database as a learning tool, and use them to generate advice next time.

[1019] Hardware and Software Used

[1020] Hardware: Robots, cameras, servers

[1021] Software: Generative AI model (generation method), OpenCV (image analysis), requests (HTTP requests), AR module (AR technology)

[1022] Specific examples

[1023] As a concrete example, consider a scenario in which a robot is repairing a machine in a factory. In this case, if the robot does not know the location of a part, the image of the area around the part captured by the camera is sent to the server. The server analyzes the image and identifies the location of the part, then generates advice on how to handle the part and the repair procedure, which it displays in the robot's field of view. This advice display is done using AR technology, so the robot can proceed with the work while receiving visual support.

[1024] Examples of prompt statements

[1025] Below are examples of prompts to be input to the generative AI model.

[1026] "Machine X is missing part Y. The current work step is step 3."

[1027] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1028] Step 1:

[1029] The robot's camera captures footage of the work.

[1030] Input: Robot working environment

[1031] Output: Camera image

[1032] How it works: The robot's camera films the work area and captures footage in real time.

[1033] Step 2:

[1034] The captured image is sent to the server.

[1035] Input: Camera image

[1036] Output: Camera image data sent to the server

[1037] Operation: Images captured by the camera are sent to a server via the Internet or local network.

[1038] Step 3:

[1039] The server's identification means analyzes the video and identifies the work content.

[1040] Input: Camera video data sent to the server

[1041] Output: Identified work content information

[1042] Operation: The server uses video analysis software (e.g. OpenCV) to recognize objects and text information in the video and identify the current task.

[1043] Step 4:

[1044] A generating means of the server generates advice based on the identified work content and information on the robot.

[1045] Input: Identified task information and robot information (identification information, status information, etc.)

[1046] Output: Generated advice

[1047] How it works: The server inputs a prompt sentence into the generative AI model, which then generates appropriate advice. Example of a prompt sentence: "Part Y of machine X is in short supply. The current work procedure is step 3."

[1048] Step 5:

[1049] The generated advice is overlaid on the robot's field of view using AR technology by the server's control means.

[1050] Input: Generated advice

[1051] Output: Advice overlaid on the robot's field of vision

[1052] Operation: The server's control means uses the AR module to superimpose the generated advice onto the robot's camera image and display it.

[1053] Step 6:

[1054] The system receives questions and requests from the robot, stores them in a database as a learning tool, and uses them to generate advice the next time.

[1055] Input: Question or request from the robot

[1056] Output: Question and request data stored in a database

[1057] How it works: The robot sends any additional information, questions or requests it needs via the user interface to the server, which then stores the data in a database, where it can be used as learning data to generate advice the next time.

[1058] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1059] This embodiment is composed of the following elements.

[1060] 1. Smart glasses: A device that allows students to view textbooks, notes, and other educational materials using smart glasses. The smart glasses have the ability to acquire visual information and display it as an AR overlay.

[1061] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs learning content, student information, and prompts including the student's emotions recognized by the emotion engine into the generative AI to obtain appropriate advice.

[1062] 3. Generative AI: A natural language processing model to generate appropriate advice based on prompts including learning content, student information, and student sentiment. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[1063] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[1064] 5. Emotion Engine: An engine for recognizing and analyzing student emotions. The emotion engine uses student emotions as learning data and is used to adjust the difficulty of learning and the style of advice.

[1065] 6. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[1066] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[1067] The process flow will be explained below.

[1068] Step 1: Processing from the smart glasses

[1069] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[1070] The terminal transmits the acquired visual information and the student's emotions to the emotion engine for emotion recognition and analysis.

[1071] The terminal transmits the results received from the emotion engine to the server.

[1072] Step 2: Server processing

[1073] The server analyzes the received visual information and identifies the learning content.

[1074] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[1075] The server sends the generated prompts and results from the emotion engine to the generative AI to obtain appropriate advice.

[1076] Step 3: Generative AI processing

[1077] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[1078] The generative AI returns the generated advice to the server.

[1079] Step 4: Processing the response from the server

[1080] The server obtains the advice received from the generative AI.

[1081] The server overlays advice to the student via the smart glasses.

[1082] Step 5: Progression of learning and handling of student questions and requests

[1083] Students can use the AR function to view learning content and progress through their studies.

[1084] If a student has any questions or requests, they send them to the server through their smart glasses.

[1085] Step 6: Collect training data and fine-tune generative AI

[1086] The server receives questions and requests from students and stores them as learning data.

[1087] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[1088] Example 2

[1089] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1090] Conventional learning support systems have difficulty in providing individual advice to students when they are studying. In addition, they are unable to grasp the emotional state of students in real time, and therefore are unable to provide appropriate advice or adjust the difficulty of learning accordingly. This leads to a problem of a decrease in overall learning efficiency.

[1091] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for identifying learning contents based on learning material information related to the learning materials that the student is studying, a means for generating advice based on the learning contents identified by the identification means and the student's information and learning goals, a means for overlaying the advice generated by the generation means on the learning materials using augmented reality technology, a means for recognizing the student's emotions and adjusting the difficulty of the learning and the style of the advice, and a means for receiving questions and requests related to the learning from the student and using them as learning data. This makes it possible to provide appropriate advice according to the individual learning situation and emotional state of the student, thereby improving learning efficiency.

[1092] A "student" is a person who studies certain teaching materials or learning content.

[1093] "Learning material information" refers to information related to learning materials such as textbooks and notebooks used by students.

[1094] "Learning content" refers to specific learning items or themes that are specified based on the learning material information.

[1095] "Student information" means individual data about a student, such as the student's age, gender, past test results, and learning history.

[1096] A "learning objective" is a specific learning purpose or goal that a student wants to achieve.

[1097] "Advice" refers to the guidance or advice provided to assist a student in their learning.

[1098] A "generative model" is an algorithm or software that generates information or advice based on input data.

[1099] "Augmented reality technology" is a technology that displays digital information overlaid on the real world.

[1100] "Means of recognizing emotions" refers to technology or devices for understanding a student's emotional state from their facial expressions, voice, and other sensor information.

[1101] "Learning data" is data obtained during the course of a student's studies, and is information that can be analyzed to provide learning support.

[1102] In order to implement this invention, it is necessary to construct a system including the following elements:

[1103] Hardware and Software Used

[1104] 1. Smart Glasses

[1105] Smart glasses are devices that allow users to view learning materials such as textbooks and notes. The smart glasses are equipped with a camera function to acquire visual information about the learning materials. They also use AR (Augmented Reality) technology to overlay appropriate learning content based on the acquired information.

[1106] 2. Server

[1107] The server receives visual information obtained from the smart glasses and identifies the learning content. Specifically, it uses OCR technology to analyze the text information of the teaching materials and combines it with the student information in the database and the student's emotion data recognized by the emotion engine. As a result, it automatically generates prompt sentences to input into the generative AI model.

[1108] 3. Generative AI Models

[1109] The generative AI model uses natural language processing techniques to generate appropriate advice from prompts, and is fine-tuned using training data to provide customized advice based on learning content and student requests.

[1110] 4. Database

[1111] The database contains information about students, such as their age, gender, past test results, academic ability scores, learning goals, and deadlines for achieving those goals. The server refers to this database and uses it as auxiliary information when generating prompt sentences.

[1112] 5. Emotion Engine

[1113] The emotion engine is software for analyzing students' emotional state in real time. It is used to adjust the difficulty of learning and style of advising, and recognizes emotions from students' facial expressions and voice.

[1114] 6. Learning Content

[1115] Learning content is digital content that complements the learning content, such as relevant videos and visual explanations, and is visually presented to students using the AR capabilities of the smart glasses.

[1116] Examples

[1117] Suppose a user wears smart glasses and is studying trigonometric formulas in a math textbook. The glasses capture the image data of the study material and send it to the server. The server uses OCR to extract text information from the image data and identifies the study content based on this. At the same time, the emotion engine analyzes the student's facial expression and obtains emotion data such as confusion.

[1118] The server combines the teaching material information, student information in the database, and emotion data to generate a prompt sentence like the following:

[1119] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[1120] When this prompt is fed into a generative AI model, the model generates advice based on it, like this:

[1121] "Trigonometric formulas are easier to understand when they are represented visually like this. Try solving the following exercise..."

[1122] The server sends this advice to the smart glasses, which then use AR technology to overlay the advice on the student.

[1123] Examples of prompt statements

[1124] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[1125] This will improve students' learning efficiency and provide customized support tailored to their individual learning situations.

[1126] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1127] Explain the flow of system program processing by dividing it into processing steps

[1128] Step 1:

[1129] Data acquisition using smart glasses

[1130] input:

[1131] The textbook, notes, or other educational material that the user is viewing.

[1132] Specific behavior:

[1133] The camera in the smart glasses captures image data of the teaching materials.

[1134] Data processing:

[1135] Image processing software is used to perform text recognition and image analysis to identify visible information in the teaching materials.

[1136] output:

[1137] Image data and text information of captured teaching materials.

[1138] Step 2:

[1139] Data received by the server

[1140] input:

[1141] Image data and text information of educational materials from smart glasses.

[1142] Specific behavior:

[1143] The smart glasses transmit the data to a server via Wi-Fi.

[1144] The server receives this data on the specified port.

[1145] Data Calculation:

[1146] The server uses OCR technology (Optical Character Recognition) to extract text information from the image data.

[1147] output:

[1148] Text information analyzed by OCR and the original image data.

[1149] Step 3:

[1150] Server determines what to learn

[1151] input:

[1152] Text information extracted by OCR.

[1153] Specific behavior:

[1154] The server analyzes the text information and again queries the database to identify the educational material content.

[1155] Calculate the degree of match between learning history and teaching materials.

[1156] Data Calculation:

[1157] Database query operations and text analysis using natural language processing.

[1158] output:

[1159] Identified learning content.

[1160] Step 4:

[1161] Emotion recognition by emotion engine

[1162] input:

[1163] Facial and voice data of students.

[1164] Specific behavior:

[1165] The emotion engine captures students' facial expressions and voice data through sensors in the smart glasses.

[1166] Data Calculation:

[1167] Uses facial recognition and voice analysis algorithms to assess student emotions in real time.

[1168] output:

[1169] Perceived emotion data (e.g., confusion, excitement, happiness).

[1170] Step 5:

[1171] Server-generated prompts

[1172] input:

[1173] Identified learning content, student information from the database, and sentiment data.

[1174] Specific behavior:

[1175] The server generates a prompt based on this data.

[1176] Data Calculation:

[1177] Use data integration and prompt generation algorithms to create appropriate prompts.

[1178] output:

[1179] Generated prompt sentence. Example: "My student is having trouble understanding trigonometric formulas, so please generate a visual explanation and some simple exercises."

[1180] Step 6:

[1181] Advice generation using generative AI models

[1182] input:

[1183] The prompt text sent by the server.

[1184] Specific behavior:

[1185] The generative AI model performs natural language processing based on the prompt sentence and generates learning advice.

[1186] Data Calculation:

[1187] Text generation using the model's natural language processing algorithms.

[1188] output:

[1189] Generated advice sentence. Example: "It's a good idea to remember trigonometric formulas like this."

[1190] Step 7:

[1191] Server-based advice sending

[1192] input:

[1193] Advice sentences from a generative AI model.

[1194] Specific behavior:

[1195] The server appropriately formats the generated text advice and transmits it to the smart glasses.

[1196] Data Calculation:

[1197] Text formatting adjustments, data submission via HTTP POST requests.

[1198] output:

[1199] Advice data sent to smart glasses.

[1200] Step 8:

[1201] Data display in smart glasses

[1202] input:

[1203] Advice data sent by the server.

[1204] Specific behavior:

[1205] The smart glasses use an AR engine to provide a visual overlay of the advice received.

[1206] Data Calculation:

[1207] AR engine for visual enhancement and overlay display processing.

[1208] output:

[1209] Advice overlaid on students' field of view.

[1210] Step 9:

[1211] Server updates the database

[1212] input:

[1213] Learning session information (material information, advice, sentiment data, etc.).

[1214] Specific behavior:

[1215] The server records the learning session information and updates a database.

[1216] Data Calculation:

[1217] SQL query operations to create new records and insert data.

[1218] output:

[1219] Updated database.

[1220] The above is an explanation of the system's program processing divided into specific steps. It clarifies what data is input, what data processing is performed, and what output is obtained at each step.

[1221] (Application example 2)

[1222] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1223] Conventional work support systems in factories have difficulty providing effective support that is tailored to the worker's situation and emotions, and as a result, they have been unable to fully improve work efficiency and productivity.In addition, workers' anxieties and doubts when attempting new tasks remain unresolved, and training efficiency remains low.

[1224] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes an identification means for identifying the work content based on the work information for a factory worker to work on the work, a generation means for generating advice based on the work content identified by the identification means and the information and work goal of the worker, a control means for overlaying the advice generated by the generation means on the work information using augmented reality technology, a learning means for receiving questions and requests regarding the work from the worker and using them as learning data, and an emotion recognition means for recognizing the emotion of the worker and adjusting the advice generated by the generation means. This makes it possible to provide appropriate support that matches the situation and emotion of the worker in real time, improving work efficiency and making training more efficient.

[1225] A "factory worker" is a person who operates various machines and equipment within a factory and carries out tasks related to the production process.

[1226] "Work information" refers to information regarding the specific work content, procedures, and objects that factory workers must perform.

[1227] The "specifying means" is a means for specifying the specific work content based on the work information for the factory worker to undertake the work.

[1228] The "generation means" is a means for generating appropriate advice based on the identified work content, information on the factory worker, and work goals.

[1229] The "control means" refers to a means for overlaying and displaying the generated advice on the task information using augmented reality technology.

[1230] A "learning means" is a means for receiving questions or requests about work from factory workers and using them as learning data.

[1231] The "emotion recognition means" is a means for recognizing the emotions of a factory worker and adjusting the advice generated by the generation means.

[1232] "Augmented reality technology" is a technology that overlays digital information onto the real world.

[1233] The system for carrying out the present invention provides support in real time for factory workers to improve their work efficiency. The specific configuration and processing of the system will be described below.

[1234] Hardware and Software

[1235] 1. Smart glasses:

[1236] Use: A device that obtains visual information from factory workers and displays advice using augmented reality technology.

[1237] 2. Factory robots:

[1238] Uses: Mechanical equipment for automating and supporting work.

[1239] 3. Server:

[1240] A concrete example: Cloud-based servers

[1241] Use: A device that receives visual information and emotion data obtained from smart glasses, analyzes it, and generates prompts for the generative AI model.

[1242] 4. Generative AI models:

[1243] Specific example: OpenAI ChatGPT

[1244] Use: Natural language processing models to generate appropriate advice based on prompts.

[1245] 5. Emotion Engine:

[1246] Use: An engine for recognizing emotions from the facial expressions of factory workers and tailoring the advice provided by generative AI.

[1247] 6. Database:

[1248] Specific example: MySQL

[1249] Use: A system for storing information and past learning data of factory workers.

[1250] Processing Description

[1251] 1. Data collection:

[1252] The smart glasses collect visual information and facial expression data from factory workers in real time, making it possible to understand what task they are currently performing and how they are feeling at the time.

[1253] 2. Data processing:

[1254] The server receives visual information and emotion data from the smart glasses, refers to the aforementioned database to identify the task, and simultaneously recognizes emotions using facial expression data and sends prompts based on that information to the generative AI.

[1255] 3. Advice Generation:

[1256] Generative AI models (e.g., OpenAI ChatGPT) generate appropriate technical support and advice for workers based on prompts, addressing their specific questions and concerns.

[1257] 4. Information display:

[1258] The server sends the generated advice to the smart glasses, which use augmented reality technology to overlay it on the worker's field of vision, allowing the worker to see the information he or she needs in real time.

[1259] Examples

[1260] For example, when a worker is setting up new robotic equipment in a factory, the smart glasses will display the operating procedure in real time. Furthermore, if the worker feels anxious, the emotion engine will recognize this and prompt the generative AI for appropriate advice.

[1261] Examples of prompt statements

[1262] Task: Setting up a new robotic device. Worker's emotions: Anxiety. Past experience: New to setup. Provide the necessary assistance based on the information displayed in the worker's field of view.

[1263] In this way, the system can provide real-time support that is tailored to the worker's situation and emotions, resulting in improved work efficiency and productivity.

[1264] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1265] Step 1:

[1266] The smart glasses collect visual information and facial expression data from factory workers in real time and send it to a server.

[1267] Input: Factory worker's visual field information, facial expression data

[1268] Output: Vision information and facial expression data sent to the server

[1269] Specific operation: The camera and sensors installed in the smart glasses collect visual information and facial expression data. The collected data is sent to a server using known communication methods.

[1270] Step 2:

[1271] The server analyzes the visual information and facial expression data it receives to identify the task and emotion.

[1272] Input: Vision information and facial expression data sent to the server

[1273] Output: Identified tasks and emotion data

[1274] Specific operation: Using the image analysis algorithm and emotion engine in the server, the task content is analyzed from the field of vision information, and emotions are analyzed from the facial expression data. The field of vision information is compared with the factory environment and equipment to identify the task content, and the facial expression data is analyzed by the emotion engine.

[1275] Step 3:

[1276] The server matches the identified task content and emotion data against a database to extract relevant information and generate prompts.

[1277] Input: Identified task and emotion data

[1278] Output: Generated prompt

[1279] Specific operation: The server compares the information of factory workers stored in the database with past work data and generates prompt sentences to input into the generative AI model.

[1280] Step 4:

[1281] A generative AI model generates appropriate advice based on the prompts.

[1282] Input: Generated prompt

[1283] Output: The generated advice.

[1284] How it works: A generative AI model (e.g., OpenAI ChatGPT) processes the prompt and generates appropriate advice in natural language for the factory worker.

[1285] Step 5:

[1286] The server sends the generated advice to the smart glasses, where it is overlaid on the visual information using augmented reality technology.

[1287] Input: Generated advice

[1288] Output: Advice displayed on smart glasses

[1289] Specific operation: The server sends the generated advice to the smart glasses, which use the augmented reality technology to overlay the advice within the factory worker's field of view.

[1290] Step 6:

[1291] The worker accepts the advice and continues working, and if he has any questions or requests, he sends them to the server via the smart glasses.

[1292] Input: Questions and requests from workers

[1293] Output: The query or request sent to the server.

[1294] How it works: The worker uses voice or gesture input on the smart glasses to send questions or requests to the server, which receives them and records them as learning data.

[1295] Through these steps, the system provides appropriate assistance to factory workers in real time, improving work efficiency and safety.

[1296] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1297] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1298] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1299] [Fourth embodiment]

[1300] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1301] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1302] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1303] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1304] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1305] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1306] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1307] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1308] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1309] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1310] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1311] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1312] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1313] This embodiment is composed of the following elements.

[1314] 1. Smart glasses: A device that allows students to view textbooks, notes, and other educational materials using smart glasses. The smart glasses have the ability to acquire visual information and display it as an AR overlay.

[1315] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs prompts containing learning content and student information into the generative AI and obtains appropriate advice.

[1316] 3. Generative AI: A natural language processing model that generates appropriate advice based on prompts containing learning content and student information. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[1317] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[1318] 5. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[1319] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[1320] The process flow will be explained below.

[1321] Step 1: Processing from the smart glasses

[1322] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[1323] The terminal transmits the obtained visibility information to the server.

[1324] Step 2: Server processing

[1325] The server analyzes the received visual information and identifies the learning content.

[1326] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[1327] The server sends the generated prompts to the generative AI to obtain appropriate advice.

[1328] Step 3: Generative AI processing

[1329] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[1330] The generative AI returns the generated advice to the server.

[1331] Step 4: Processing the response from the server

[1332] The server obtains the advice received from the generative AI.

[1333] The server overlays advice to the student via the smart glasses.

[1334] Step 5: Progression of learning and handling of student questions and requests

[1335] Students use the smart glasses' AR capabilities to watch relevant videos and visual explanations as they learn.

[1336] If a student has any questions or requests, they send them to the server through their smart glasses.

[1337] Step 6: Collect training data and fine-tune generative AI

[1338] The server receives questions and requests from students and stores them as learning data.

[1339] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[1340] Example 1

[1341] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1342] Conventional learning support systems could not effectively utilize information from teaching materials or individual student information, making it difficult to provide appropriate advice and learning content in real time. As a result, students' learning efficiency decreased and achievement of learning goals was often delayed. In addition, there was a problem that students' motivation to learn decreased because they could not respond quickly to their questions and requests.

[1343] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1344] In this invention, the server includes: a means for acquiring learning material information on the learning material that the student is studying; a means for identifying the learning content based on the learning material information; a means for generating a prompt sentence based on the learning content and the student's information; a means for inputting the prompt sentence into a generative AI model to generate advice; a means for overlaying the generated advice on the learning material using AR technology; a means for referring to a database that stores information such as the student's age, sex, past test results, deviation score, learning goal, and goal achievement deadline; and a means for transmitting the generated advice and related learning content to the smart glasses. This allows the student to efficiently progress through their studies while receiving personalized advice in real time.

[1345] 1. "Materials information" refers to data such as text and images contained in the materials students use for their studies.

[1346] 2. “Content” refers to information about a particular subject or topic that a student is studying.

[1347] 3. "Student information" refers to personal data such as a student's age, gender, past test results, academic ability score, learning goals, and deadlines for achieving those goals.

[1348] 4. “Prompt” refers to an instruction to be input into a generative AI model based on the learning content and student information.

[1349] 5. “Generative AI model” means an artificial intelligence model that uses natural language processing to generate advice or information based on a prompt.

[1350] 6. “Advice” refers to advice or suggestions for learning support that are generated by the generative AI model based on the prompt.

[1351] 7. "AR technology" refers to augmented reality technology that overlays digital information on the actual field of vision.

[1352] 8. "Database" means a database management system for storing information such as students' age, gender, past test results, deviation scores, learning goals, and deadlines for achieving goals.

[1353] 9. "Smart glasses" means a wearable device for displaying digital information using augmented reality technology.

[1354] 10. “Learning content” means digital information, such as video or visual explanations, that supplements student learning.

[1355] The present invention relates to a system for supporting student learning, and more particularly to a system for providing appropriate advice and learning content in real time by utilizing teaching material information and student information. A specific embodiment for carrying out the present invention will be described.

[1356] System Overview

[1357] The system mainly consists of smart glasses, a server, a generative AI model, a database, and learning content.

[1358] Smart Glasses

[1359] Smart glasses are wearable devices that students use when viewing textbooks, notes, and other study materials. The glasses are equipped with a built-in camera and have the ability to capture visual information such as text and images from study materials. They also have a function that uses AR (Augmented Reality) technology to overlay advice generated based on the captured information.

[1360] server

[1361] The server receives the visual information sent from the smart glasses and identifies the learning content. The server accesses the database to obtain individual information about the students. It also generates a prompt sentence based on the learning content and the student information, and inputs the prompt sentence into the generative AI model.

[1362] Generative AI Models

[1363] The generative AI model uses natural language processing to generate advice based on the prompt sentence. For example, ChatGPT is used for this AI model. It receives the prompt sentence as input, generates appropriate advice, and returns it to the server.

[1364] Database

[1365] The database stores personal information such as the student's age, gender, past test results, deviation score, learning goals, deadline for achieving the goal, etc. The server accesses this database to obtain the necessary information and uses it to generate prompt sentences.

[1366] Learning Content

[1367] Learning content is digital information to supplement students' learning, such as videos and visual explanations. The server transmits the learning content to the smart glasses as needed, allowing students to view it in real time.

[1368] Examples

[1369] Students solving math problems

[1370] 1. Viewing the materials:

[1371] A user opens a math textbook and begins looking at a problem through the smart glasses, which use their built-in camera to capture the mathematical equation "y = ax^2 + bx + c."

[1372] 2. Transmission of information:

[1373] The smart glasses transmit the captured mathematical formula information to a server.

[1374] 3. Generate prompt:

[1375] The server retrieves student information from the database and generates a prompt statement such as, "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem."

[1376] 4. Generate advice:

[1377] The server inputs a prompt sentence into the generative AI model, which then generates advice such as, "First, use the basic form of the quadratic function, y = ax^2 + bx + c, to confirm the values ​​of a, b, and c. Then, I will explain how to solve it using the solution formula."

[1378] 5. Display Advice:

[1379] The server sends the generated advice to the smart glasses, which overlay it on the user's field of view.

[1380] 6. Provision of learning content:

[1381] If the user wants to understand further, the server will send the relevant explanatory video to the smart glasses so that the user can watch the video.

[1382] Examples of prompt statements

[1383] "A 17-year-old male student A is trying to solve a quadratic function problem. I would like some specific advice on how to solve quadratic functions."

[1384] In this way, the system of the present invention can effectively support students' learning and improve their learning efficiency.

[1385] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1386] Step 1:

[1387] The user browses the learning materials through smart glasses. The glasses use a built-in camera to capture visual information such as text and images from the learning materials. Optical character recognition (OCR) software is then used to extract text data from the captured information.

[1388] Input: Visual information from textbooks and notes

[1389] Output: Extracted text data (e.g. "y = ax^2 + bx + c")

[1390] Step 2:

[1391] The smart glasses transmit the extracted text data to a server using known communication means, and the server receives the data and begins analyzing it.

[1392] Input: Text data (e.g. "y = ax^2 + bx + c")

[1393] Output: Received text data

[1394] Step 3:

[1395] The server accesses the database and obtains information about the student (age, gender, past test results, deviation score, learning goals, deadline for achieving goals, etc.). Based on the obtained information, a prompt sentence is generated according to the current learning situation. The prompt sentence is a specific instruction sentence to be passed to the generative AI model.

[1396] Input: Received text data, student information stored in database

[1397] Output: Prompt statement (e.g. "A 17-year-old male student A is trying to solve the quadratic function problem y = ax^2 + bx + c. Please give some advice on how to solve this problem.")

[1398] Step 4:

[1399] The server inputs the generated prompt sentence into a generative AI model (e.g., ChatGPT). The generative AI model generates appropriate advice based on the prompt sentence. The generative AI model analyzes information about the learning content and creates advice in a form that is easy for the user to understand.

[1400] Input: Prompt statement

[1401] Output: Generated advice (e.g. "First, use the basic form of the quadratic function y = ax^2 + bx + c to find the values ​​of a, b, and c. Then, use the solution formula to explain how to solve the problem.")

[1402] Step 5:

[1403] The server sends the generated advice to the smart glasses, which overlay the advice on the user's field of view. The displayed advice is integrated into the field of view in a natural way using AR technology.

[1404] Input: Generated advice

[1405] Output: Advice overlaid on the user's view

[1406] Step 6:

[1407] If necessary, the server sends relevant learning content (videos and visual explanations) to the smart glasses, which use their AR capabilities to enable the user to watch this learning content in real time.

[1408] Input: User request, related learning content

[1409] Output: Learning content overlaid on the user's field of view

[1410] (Application example 1)

[1411] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1412] Current factory robots often lack immediate support for work procedures and problem solving when performing complex tasks or troubleshooting. This leads to issues such as reduced work efficiency and increased downtime. Conventional methods require robot operators to refer to the manual, which can take time to troubleshoot. There is a need to improve this situation and increase work efficiency.

[1413] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1414] In this invention, the server includes: a specifying means for specifying a task content based on task information related to a task to be performed by the robot; a generating means for generating advice based on the task content specified by the specifying means, the robot information, and a task goal; a control means for overlaying the advice generated by the generating means on the task using an augmented reality (AR) technology; and a learning means for receiving questions and requests related to the task from the robot and using them as task data. This enables the robot to receive necessary advice in real time, making it possible to dramatically improve task efficiency.

[1415] A "robot" is a mechanical device that automates tasks in a factory.

[1416] "Work information" is information relating to the specific work content that the robot will undertake.

[1417] The "identification means" is a function for identifying the work content based on the work information.

[1418] "Work content" refers to the series of work procedures and specific work instructions that the robot must perform.

[1419] "Robot Information" means information about a robot, including its identity, current status, and performance data.

[1420] "Task goals" are information that sets specific goals and deadlines that the robot must achieve.

[1421] The "generation means" is a function for generating advice based on the work content identified by the identification means and other information.

[1422] "Advice" is any instruction or recommendation given to assist in the progress of work.

[1423] "AR technology" is a general term for technology that displays virtual information overlaid on real visual information.

[1424] The "control means" is a function for displaying the advice generated by the generation means using AR technology.

[1425] The "learning means" is a function that receives questions and requests from the robot regarding tasks and uses them to improve the system and generate advice.

[1426] The present invention is a system for improving the work efficiency of factory robots. This system identifies the work content based on work information related to the work performed by the robot and provides advice in real time, thereby significantly improving the efficiency of work stagnation and troubleshooting.

[1427] System Configuration

[1428] The system configuration is as follows.

[1429] 1. Robot:

[1430] Robots are installed in factories to perform tasks automatically. They are equipped with cameras that can capture images of the tasks they perform.

[1431] 2. Server:

[1432] The server has the following means:

[1433] Identification method: Analyze images obtained from the robot's camera and other work information to identify the current work content.

[1434] Generation means: Generates appropriate advice based on the task content identified by the identification means and the robot information (identification information, status information, etc.). Here, advice is generated using a generative AI model.

[1435] Control means: The generated advice is overlaid on the robot's field of view using AR technology.

[1436] Learning method: Receive questions and requests from the robot regarding the task, and use them to improve the system and generate advice.

[1437] 3. AR technology:

[1438] Using augmented reality (AR) technology, the generated advice is superimposed on the robot's field of vision, allowing the robot to obtain the necessary information in real time.

[1439] Processing flow

[1440] 1. The robot's camera captures footage of the work being done.

[1441] 2. The acquired video is sent to the server.

[1442] 3. The server's identification means analyzes the video and identifies the work content.

[1443] 4. The server's generation means generates advice based on the identified work content and robot information.

[1444] 5. The generated advice is overlaid on the robot's field of view using AR technology via the server's control means.

[1445] 6. Receive questions and requests from the robot, store them in a database as a learning tool, and use them to generate advice next time.

[1446] Hardware and Software Used

[1447] Hardware: Robots, cameras, servers

[1448] Software: Generative AI model (generation method), OpenCV (image analysis), requests (HTTP requests), AR module (AR technology)

[1449] Specific examples

[1450] As a concrete example, consider a scenario in which a robot is repairing a machine in a factory. In this case, if the robot does not know the location of a part, the image of the area around the part captured by the camera is sent to the server. The server analyzes the image and identifies the location of the part, then generates advice on how to handle the part and the repair procedure, which it displays in the robot's field of view. This advice display is done using AR technology, so the robot can proceed with the work while receiving visual support.

[1451] Examples of prompt statements

[1452] Below are examples of prompts to be input to the generative AI model.

[1453] "Machine X is missing part Y. The current work step is step 3."

[1454] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1455] Step 1:

[1456] The robot's camera captures footage of the work.

[1457] Input: Robot working environment

[1458] Output: Camera image

[1459] How it works: The robot's camera films the work area and captures footage in real time.

[1460] Step 2:

[1461] The captured image is sent to the server.

[1462] Input: Camera image

[1463] Output: Camera image data sent to the server

[1464] Operation: Images captured by the camera are sent to a server via the Internet or local network.

[1465] Step 3:

[1466] The server's identification means analyzes the video and identifies the work content.

[1467] Input: Camera video data sent to the server

[1468] Output: Identified work content information

[1469] Operation: The server uses video analysis software (e.g. OpenCV) to recognize objects and text information in the video and identify the current task.

[1470] Step 4:

[1471] A generating means of the server generates advice based on the identified work content and information on the robot.

[1472] Input: Identified task information and robot information (identification information, status information, etc.)

[1473] Output: Generated advice

[1474] How it works: The server inputs a prompt sentence into the generative AI model, which then generates appropriate advice. Example of a prompt sentence: "Part Y of machine X is in short supply. The current work procedure is step 3."

[1475] Step 5:

[1476] The generated advice is overlaid on the robot's field of view using AR technology by the server's control means.

[1477] Input: Generated advice

[1478] Output: Advice overlaid on the robot's field of vision

[1479] Operation: The server's control means uses the AR module to superimpose the generated advice onto the robot's camera image and display it.

[1480] Step 6:

[1481] The system receives questions and requests from the robot, stores them in a database as a learning tool, and uses them to generate advice the next time.

[1482] Input: Question or request from the robot

[1483] Output: Question and request data stored in a database

[1484] How it works: The robot sends any additional information, questions or requests it needs via the user interface to the server, which then stores the data in a database, where it can be used as learning data to generate advice the next time.

[1485] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1486] This embodiment is composed of the following elements.

[1487] 1. Smart glasses: A device that allows students to view textbooks, notes, and other educational materials using smart glasses. The smart glasses have the ability to acquire visual information and display it as an AR overlay.

[1488] 2. Server: A device that receives visual information from the smart glasses and performs processing to identify learning content and generate advice. The server inputs learning content, student information, and prompts including the student's emotions recognized by the emotion engine into the generative AI to obtain appropriate advice.

[1489] 3. Generative AI: A natural language processing model to generate appropriate advice based on prompts including learning content, student information, and student sentiment. Generative AI is fine-tuned using training data to provide appropriate content in response to student questions and requests.

[1490] 4. Database: A database for storing information about students, such as their age, gender, past test results, deviation scores, learning goals, deadlines for achieving goals, etc. The server generates advice by referring to the information stored in the database.

[1491] 5. Emotion Engine: An engine for recognizing and analyzing student emotions. The emotion engine uses student emotions as learning data and is used to adjust the difficulty of learning and the style of advice.

[1492] 6. Learning content: Content that complements the learning content, such as relevant videos and visual explanations. Students can watch and listen to the learning content using the AR function of the smart glasses.

[1493] The data processing system 10 of this embodiment can be implemented by combining the above elements. The specific process flow and operation procedures will be described later.

[1494] The process flow will be explained below.

[1495] Step 1: Processing from the smart glasses

[1496] The terminal (smart glasses) scans educational materials such as textbooks and notebooks and acquires visual information.

[1497] The terminal transmits the acquired visual information and the student's emotions to the emotion engine for emotion recognition and analysis.

[1498] The terminal transmits the results received from the emotion engine to the server.

[1499] Step 2: Server processing

[1500] The server analyzes the received visual information and identifies the learning content.

[1501] The server combines student information (age, gender, past test results, academic ability scores, learning goals, deadlines for achieving goals, etc.) with the identified learning content to generate a prompt to query the generative AI.

[1502] The server sends the generated prompts and results from the emotion engine to the generative AI to obtain appropriate advice.

[1503] Step 3: Generative AI processing

[1504] The generative AI analyzes the prompts it receives and generates appropriate advice for the student.

[1505] The generative AI returns the generated advice to the server.

[1506] Step 4: Processing the response from the server

[1507] The server obtains the advice received from the generative AI.

[1508] The server overlays advice to the student via the smart glasses.

[1509] Step 5: Progression of learning and handling of student questions and requests

[1510] Students can use the AR function to view learning content and progress through their studies.

[1511] If a student has any questions or requests, they send them to the server through their smart glasses.

[1512] Step 6: Collect training data and fine-tune generative AI

[1513] The server receives questions and requests from students and stores them as learning data.

[1514] The server uses the learning data to fine-tune the generative AI so that it can generate more appropriate advice.

[1515] Example 2

[1516] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1517] Conventional learning support systems have difficulty in providing individual advice to students when they are studying. In addition, they are unable to grasp the emotional state of students in real time, and therefore are unable to provide appropriate advice or adjust the difficulty of learning accordingly. This leads to a problem of a decrease in overall learning efficiency.

[1518] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for identifying learning contents based on learning material information related to the learning materials that the student is studying, a means for generating advice based on the learning contents identified by the identification means and the student's information and learning goals, a means for overlaying the advice generated by the generation means on the learning materials using augmented reality technology, a means for recognizing the student's emotions and adjusting the difficulty of the learning and the style of the advice, and a means for receiving questions and requests related to the learning from the student and using them as learning data. This makes it possible to provide appropriate advice according to the individual learning situation and emotional state of the student, thereby improving learning efficiency.

[1519] A "student" is a person who studies certain teaching materials or learning content.

[1520] "Learning material information" refers to information related to learning materials such as textbooks and notebooks used by students.

[1521] "Learning content" refers to specific learning items or themes that are specified based on the learning material information.

[1522] "Student information" means individual data about a student, such as the student's age, gender, past test results, and learning history.

[1523] A "learning objective" is a specific learning purpose or goal that a student wants to achieve.

[1524] "Advice" refers to the guidance or advice provided to assist a student in their learning.

[1525] A "generative model" is an algorithm or software that generates information or advice based on input data.

[1526] "Augmented reality technology" is a technology that displays digital information overlaid on the real world.

[1527] "Means of recognizing emotions" refers to technology or devices for understanding a student's emotional state from their facial expressions, voice, and other sensor information.

[1528] "Learning data" is data obtained during the course of a student's studies, and is information that can be analyzed to provide learning support.

[1529] In order to implement this invention, it is necessary to construct a system including the following elements:

[1530] Hardware and Software Used

[1531] 1. Smart Glasses

[1532] Smart glasses are devices that allow users to view learning materials such as textbooks and notes. The smart glasses are equipped with a camera function to acquire visual information about the learning materials. They also use AR (Augmented Reality) technology to overlay appropriate learning content based on the acquired information.

[1533] 2. Server

[1534] The server receives visual information obtained from the smart glasses and identifies the learning content. Specifically, it uses OCR technology to analyze the text information of the teaching materials and combines it with the student information in the database and the student's emotion data recognized by the emotion engine. As a result, it automatically generates prompt sentences to input into the generative AI model.

[1535] 3. Generative AI Models

[1536] The generative AI model uses natural language processing techniques to generate appropriate advice from prompts, and is fine-tuned using training data to provide customized advice based on learning content and student requests.

[1537] 4. Database

[1538] The database contains information about students, such as their age, gender, past test results, academic ability scores, learning goals, and deadlines for achieving those goals. The server refers to this database and uses it as auxiliary information when generating prompt sentences.

[1539] 5. Emotion Engine

[1540] The emotion engine is software for analyzing students' emotional state in real time. It is used to adjust the difficulty of learning and style of advising, and recognizes emotions from students' facial expressions and voice.

[1541] 6. Learning Content

[1542] Learning content is digital content that complements the learning content, such as relevant videos and visual explanations, and is visually presented to students using the AR capabilities of the smart glasses.

[1543] Examples

[1544] Suppose a user wears smart glasses and is studying trigonometric formulas in a math textbook. The glasses capture the image data of the study material and send it to the server. The server uses OCR to extract text information from the image data and identifies the study content based on this. At the same time, the emotion engine analyzes the student's facial expression and obtains emotion data such as confusion.

[1545] The server combines the teaching material information, student information in the database, and emotion data to generate a prompt sentence like the following:

[1546] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[1547] When this prompt is fed into a generative AI model, the model generates advice based on it, like this:

[1548] "Trigonometric formulas are easier to understand when they are represented visually like this. Try solving the following exercise..."

[1549] The server sends this advice to the smart glasses, which then use AR technology to overlay the advice on the student.

[1550] Examples of prompt statements

[1551] "Students can't understand trigonometric formulas, so generate visual explanations and simple exercises."

[1552] This will improve students' learning efficiency and provide customized support tailored to their individual learning situations.

[1553] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1554] Explain the flow of system program processing by dividing it into processing steps

[1555] Step 1:

[1556] Data acquisition using smart glasses

[1557] input:

[1558] The textbook, notes, or other educational material that the user is viewing.

[1559] Specific behavior:

[1560] The camera in the smart glasses captures image data of the teaching materials.

[1561] Data processing:

[1562] Image processing software is used to perform text recognition and image analysis to identify visible information in the teaching materials.

[1563] output:

[1564] Image data and text information of captured teaching materials.

[1565] Step 2:

[1566] Data received by the server

[1567] input:

[1568] Image data and text information of educational materials from smart glasses.

[1569] Specific behavior:

[1570] The smart glasses transmit the data to a server via Wi-Fi.

[1571] The server receives this data on the specified port.

[1572] Data Calculation:

[1573] The server uses OCR technology (Optical Character Recognition) to extract text information from the image data.

[1574] output:

[1575] Text information analyzed by OCR and the original image data.

[1576] Step 3:

[1577] Server determines what to learn

[1578] input:

[1579] Text information extracted by OCR.

[1580] Specific behavior:

[1581] The server analyzes the text information and again queries the database to identify the educational material content.

[1582] Calculate the degree of match between learning history and teaching materials.

[1583] Data Calculation:

[1584] Database query operations and text analysis using natural language processing.

[1585] output:

[1586] Identified learning content.

[1587] Step 4:

[1588] Emotion recognition by emotion engine

[1589] input:

[1590] Facial and voice data of students.

[1591] Specific behavior:

[1592] The emotion engine captures students' facial expressions and voice data through sensors in the smart glasses.

[1593] Data Calculation:

[1594] Uses facial recognition and voice analysis algorithms to assess student emotions in real time.

[1595] output:

[1596] Perceived emotion data (e.g., confusion, excitement, happiness).

[1597] Step 5:

[1598] Server-generated prompts

[1599] input:

[1600] Identified learning content, student information from the database, and sentiment data.

[1601] Specific behavior:

[1602] The server generates a prompt based on this data.

[1603] Data Calculation:

[1604] Use data integration and prompt generation algorithms to create appropriate prompts.

[1605] output:

[1606] Generated prompt sentence. Example: "My student is having trouble understanding trigonometric formulas, so please generate a visual explanation and some simple exercises."

[1607] Step 6:

[1608] Advice generation using generative AI models

[1609] input:

[1610] The prompt text sent by the server.

[1611] Specific behavior:

[1612] The generative AI model performs natural language processing based on the prompt sentence and generates learning advice.

[1613] Data Calculation:

[1614] Text generation using the model's natural language processing algorithms.

[1615] output:

[1616] Generated advice sentence. Example: "It's a good idea to remember trigonometric formulas like this."

[1617] Step 7:

[1618] Server-based advice sending

[1619] input:

[1620] Advice sentences from a generative AI model.

[1621] Specific behavior:

[1622] The server appropriately formats the generated text advice and transmits it to the smart glasses.

[1623] Data Calculation:

[1624] Text formatting adjustments, data submission via HTTP POST requests.

[1625] output:

[1626] Advice data sent to smart glasses.

[1627] Step 8:

[1628] Data display in smart glasses

[1629] input:

[1630] Advice data sent by the server.

[1631] Specific behavior:

[1632] The smart glasses use an AR engine to provide a visual overlay of the advice received.

[1633] Data Calculation:

[1634] AR engine for visual enhancement and overlay display processing.

[1635] output:

[1636] Advice overlaid on students' field of view.

[1637] Step 9:

[1638] Server updates the database

[1639] input:

[1640] Learning session information (material information, advice, sentiment data, etc.).

[1641] Specific behavior:

[1642] The server records the learning session information and updates a database.

[1643] Data Calculation:

[1644] SQL query operations to create new records and insert data.

[1645] output:

[1646] Updated database.

[1647] The above is an explanation of the system's program processing divided into specific steps. It clarifies what data is input, what data processing is performed, and what output is obtained at each step.

[1648] (Application example 2)

[1649] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1650] Conventional work support systems in factories have difficulty providing effective support that is tailored to the worker's situation and emotions, and as a result, they have been unable to fully improve work efficiency and productivity.In addition, workers' anxieties and doubts when attempting new tasks remain unresolved, and training efficiency remains low.

[1651] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes an identification means for identifying the work content based on the work information for a factory worker to work on the work, a generation means for generating advice based on the work content identified by the identification means and the information and work goal of the worker, a control means for overlaying the advice generated by the generation means on the work information using augmented reality technology, a learning means for receiving questions and requests regarding the work from the worker and using them as learning data, and an emotion recognition means for recognizing the emotion of the worker and adjusting the advice generated by the generation means. This makes it possible to provide appropriate support that matches the situation and emotion of the worker in real time, improving work efficiency and making training more efficient.

[1652] A "factory worker" is a person who operates various machines and equipment within a factory and carries out tasks related to the production process.

[1653] "Work information" refers to information regarding the specific work content, procedures, and objects that factory workers must perform.

[1654] The "specifying means" is a means for specifying the specific work content based on the work information for the factory worker to undertake the work.

[1655] The "generation means" is a means for generating appropriate advice based on the identified work content, information on the factory worker, and work goals.

[1656] The "control means" refers to a means for overlaying and displaying the generated advice on the task information using augmented reality technology.

[1657] A "learning means" is a means for receiving questions or requests about work from factory workers and using them as learning data.

[1658] The "emotion recognition means" is a means for recognizing the emotions of a factory worker and adjusting the advice generated by the generation means.

[1659] "Augmented reality technology" is a technology that overlays digital information onto the real world.

[1660] The system for carrying out the present invention provides support in real time for factory workers to improve their work efficiency. The specific configuration and processing of the system will be described below.

[1661] Hardware and Software

[1662] 1. Smart glasses:

[1663] Use: A device that obtains visual information from factory workers and displays advice using augmented reality technology.

[1664] 2. Factory robots:

[1665] Uses: Mechanical equipment for automating and supporting work.

[1666] 3. Server:

[1667] A concrete example: Cloud-based servers

[1668] Use: A device that receives visual information and emotion data obtained from smart glasses, analyzes it, and generates prompts for the generative AI model.

[1669] 4. Generative AI models:

[1670] Specific example: OpenAI ChatGPT

[1671] Use: Natural language processing models to generate appropriate advice based on prompts.

[1672] 5. Emotion Engine:

[1673] Use: An engine for recognizing emotions from the facial expressions of factory workers and tailoring the advice provided by generative AI.

[1674] 6. Database:

[1675] Specific example: MySQL

[1676] Use: A system for storing information and past learning data of factory workers.

[1677] Processing Description

[1678] 1. Data collection:

[1679] The smart glasses collect visual information and facial expression data from factory workers in real time, making it possible to understand what task they are currently performing and how they are feeling at the time.

[1680] 2. Data processing:

[1681] The server receives visual information and emotion data from the smart glasses, refers to the aforementioned database to identify the task, and simultaneously recognizes emotions using facial expression data and sends prompts based on that information to the generative AI.

[1682] 3. Advice Generation:

[1683] Generative AI models (e.g., OpenAI ChatGPT) generate appropriate technical support and advice for workers based on prompts, addressing their specific questions and concerns.

[1684] 4. Information display:

[1685] The server sends the generated advice to the smart glasses, which use augmented reality technology to overlay it on the worker's field of vision, allowing the worker to see the information he or she needs in real time.

[1686] Examples

[1687] For example, when a worker is setting up new robotic equipment in a factory, the smart glasses will display the operating procedure in real time. Furthermore, if the worker feels anxious, the emotion engine will recognize this and prompt the generative AI for appropriate advice.

[1688] Examples of prompt statements

[1689] Task: Setting up a new robotic device. Worker's emotions: Anxiety. Past experience: New to setup. Provide the necessary assistance based on the information displayed in the worker's field of view.

[1690] In this way, the system can provide real-time support that is tailored to the worker's situation and emotions, resulting in improved work efficiency and productivity.

[1691] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1692] Step 1:

[1693] The smart glasses collect visual information and facial expression data from factory workers in real time and send it to a server.

[1694] Input: Factory worker's visual field information, facial expression data

[1695] Output: Vision information and facial expression data sent to the server

[1696] Specific operation: The camera and sensors installed in the smart glasses collect visual information and facial expression data. The collected data is sent to a server using known communication methods.

[1697] Step 2:

[1698] The server analyzes the visual information and facial expression data it receives to identify the task and emotion.

[1699] Input: Vision information and facial expression data sent to the server

[1700] Output: Identified tasks and emotion data

[1701] Specific operation: Using the image analysis algorithm and emotion engine in the server, the task content is analyzed from the field of vision information, and emotions are analyzed from the facial expression data. The field of vision information is compared with the factory environment and equipment to identify the task content, and the facial expression data is analyzed by the emotion engine.

[1702] Step 3:

[1703] The server matches the identified task content and emotion data against a database to extract relevant information and generate prompts.

[1704] Input: Identified task and emotion data

[1705] Output: Generated prompt

[1706] Specific operation: The server compares the information of factory workers stored in the database with past work data and generates prompt sentences to input into the generative AI model.

[1707] Step 4:

[1708] A generative AI model generates appropriate advice based on the prompts.

[1709] Input: Generated prompt

[1710] Output: The generated advice.

[1711] How it works: A generative AI model (e.g., OpenAI ChatGPT) processes the prompt and generates appropriate advice in natural language for the factory worker.

[1712] Step 5:

[1713] The server sends the generated advice to the smart glasses, where it is overlaid on the visual information using augmented reality technology.

[1714] Input: Generated advice

[1715] Output: Advice displayed on smart glasses

[1716] Specific operation: The server sends the generated advice to the smart glasses, which use the augmented reality technology to overlay the advice within the factory worker's field of view.

[1717] Step 6:

[1718] The worker accepts the advice and continues working, and if he has any questions or requests, he sends them to the server via the smart glasses.

[1719] Input: Questions and requests from workers

[1720] Output: The query or request sent to the server.

[1721] How it works: The worker uses voice or gesture input on the smart glasses to send questions or requests to the server, which receives them and records them as learning data.

[1722] Through these steps, the system provides appropriate assistance to factory workers in real time, improving work efficiency and safety.

[1723] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1724] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1725] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the robot 414.

[1726] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[1727] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1728] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1729] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out you go on emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1730] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[1731] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[1732] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[1733] Although the system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[1734] In the above embodiment, an example is given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to input data.

[1735] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1736] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[1737] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1738] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[1739] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[1740] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1741] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[1742] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[1743] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[1744] The following is further disclosed regarding the above embodiment.

[1745] (1) An identification unit that identifies learning content based on learning material information regarding learning materials that students are studying; a generating unit that generates advice based on the learning content identified by the identifying unit and information and learning goals of the student; a control unit that overlays the advice generated by the generation unit on the teaching material using an AR (Augmented Reality) technology; A learning unit that receives questions and requests from the students regarding their studies and uses them as learning data; A learning support system including:

[1746] (2) The identification unit identifies the learning content based on visual information of the learning material acquired as the learning material information via the smart glasses. (1) A learning support system according to the present invention

[1747] (3) the generation unit generates the advice according to the learning content by using a generative model that generates information according to input data. A learning support system according to (1) or (2).

[1748] (4) an emotion engine that recognizes the emotion of the student and reflects the recognized emotion of the student in the advice generated by the generation unit; A learning support system according to (1), (2), or (3).

[1749] "Example 1"

[1750] (Claim 1) a means for obtaining instructional material information regarding the material the student is working on; A means for identifying learning contents based on the teaching material information; means for generating prompt sentences based on the learning content and student information; A means for inputting the prompt sentence into a generative AI model to generate advice; a means for overlaying and displaying the generated advice on the teaching material using AR technology; A means for referencing a database that stores information such as the student's age, sex, past test results, deviation score, learning goals, and deadline for achieving the goals; means for transmitting the generated advice and associated learning content to the smart glasses; A system including:

[1751] (Claim 2) The system of claim 1 , wherein the educational material information is acquired through smart glasses.

[1752] (Claim 3) The system of claim 1, further comprising a generative AI model that generates information according to input data to generate the prompt sentence.

[1753] "Application example 1"

[1754] (Claim 1) An identification means for identifying a task content based on task information related to a task to be performed by the robot; a generating means for generating advice based on the task content identified by the identifying means, information about the robot, and a task goal; A control means for overlaying and displaying the advice generated by the generation means on the work using an AR (Augmented Reality) technology; A learning means for receiving questions and requests from the robot regarding a task and using them as task data; A system including:

[1755] (Claim 2) The identification means identifies the work content based on visible information of the work acquired as the work information via a camera. 2. The system of claim 1.

[1756] (Claim 3) the generating means generates the advice according to the work content by using a generative model that generates information according to input data. 2. The system of claim 1.

[1757] "Example 2 of combining emotion engines"

[1758] (Claim 1) a means for identifying the learning content based on instructional material information about the material the student is working on; a means for generating advice based on the learning content identified by the identifying means and information and learning goals of the student; a means for overlaying and displaying the advice generated by the generating means on the teaching material using an augmented reality technology; means for recognizing said student's emotions and adjusting the difficulty of the learning or style of advising; A means for receiving questions and requests from the students regarding their studies and using them as study data; A learning support system including:

[1759] (Claim 2) the identifying means identifies the learning content based on visual information of the teaching material acquired as the teaching material information via a display device. 2. The learning support system according to claim 1.

[1760] (Claim 3) the generation means generates the advice according to the learning content by using a generative model that generates information according to input data. 2. The learning support system according to claim 1.

[1761] "Application example 2 when combining emotion engines"

[1762] (Claim 1) An identification means for identifying a task content based on task information for a factory worker to perform the task; a generating means for generating advice based on the work content identified by the identifying means, information about the worker, and a work goal; a control means for overlaying and displaying the advice generated by the generation means on the task information by using an augmented reality technology; A learning means for receiving questions and requests from the worker regarding the work and using them as learning data; an emotion recognition means for recognizing the emotion of the worker and adjusting the advice generated by the generation means; A system including:

[1763] (Claim 2) The system according to claim 1 , wherein the identifying means identifies the work content based on visual information of the work acquired as the work information via smart glasses.

[1764] (Claim 3) 2. The system according to claim 1, wherein the generating means generates the advice according to the work content using a generative model that generates information according to input data. [Explanation of symbols]

[1765] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A first means for acquiring learning material information, which is data of characters and images included in learning materials that a student is studying, and student information, which is personal data of the student, such as the student's age, sex, past test results, deviation score, learning goal, and goal achievement deadline; A second means for identifying the learning content that the student is studying based on the acquired learning material information; a third means for generating a first prompt sentence for outputting advice on the learning content based on the identified learning content and the acquired student information; a fourth means for inputting the generated first prompt sentence into a generative AI model to generate the advice; A fifth means for transmitting the generated advice to smart glasses used by the student; a sixth means for overlaying and displaying the received advice on the teaching material by using an augmented reality technology; A learning support system including:

2. The fifth means transmits learning content, which is digital information for complementing the learning content, together with the advice to the smart glasses. The learning support system according to claim 1 .

3. a seventh means for recognizing the emotion of the student from the facial expression and voice of the student; The third means generates a second prompt sentence for outputting specific advice on the learning content according to the emotional state of the student, based on the learning content, the student information, and the student's emotion; The fourth means inputs the generated second prompt sentence into a generative AI model to generate the specific advice. The learning support system according to claim 1 or 2.

Images