System

Abstract

To provide a system.SOLUTION: A system includes: means for converting speech voice into a text; means for analyzing the converted text to determine whether information included in the speech voice is correct or not; means for generating, when an error is included in the information included in the speech voice, a prompt text for generating correct information, and inputting the prompt text to a generative AI; and means for outputting correct information generated by the generative AI.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] There is a lack of effective means in the educational field to enable students to actively learn in lessons. [Means for solving the problem]

[0005] We provide an educational robot system that combines speech recognition and generative AI. When a learner gives an incorrect answer, this system generates and speaks information to correct the answer. This allows the learner to recognize their own mistake and obtain the correct information. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17]FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] The educational robot system of the present invention comprises a voice recognition means and a generative AI means. The voice recognition means recognizes the learner's utterance and judges whether the utterance is incorrect. The generative AI means generates correct information for the learner's incorrect utterance and speaks that information. Specifically, if the learner says "The Earth is flat," the voice recognition means recognizes this utterance, and the generative AI means generates and speaks the information "That is incorrect. The Earth is spherical."

[0029] "Example 2"

[0030] The educational robot system of the present invention provides an effective means for implementing lesson development in which learners actively learn. Specifically, when a learner gives a wrong answer, the educational robot system generates and speaks information to correct it. This allows the learner to recognize his or her own mistake and obtain the correct information. For example, if a learner says, "The capital of Japan is Osaka," the educational robot system generates and speaks the information, "That is incorrect. The capital of Japan is Tokyo."

[0031] The process flow of each embodiment will be described below.

[0032] "Example 1"

[0033] Step 1: The learner makes an utterance, for example, "The Earth is flat."

[0034] Step 2: The speech recognition means recognizes the learner's utterance. In this example, it recognizes the utterance "The Earth is flat."

[0035] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information "That is incorrect. The Earth is spherical." Step 4: The generative AI means speaks out the generated information. In this example, it speaks out "That is incorrect. The Earth is spherical."

[0036] "Example 2"

[0037] Step 1: A learner makes an utterance. For example, the learner may say, "The capital of Japan is Osaka." Step 2: A speech recognition means recognizes the learner's utterance. In this example, the speech recognition means recognizes the utterance, "The capital of Japan is Osaka."

[0038] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information, "That is incorrect. The capital of Japan is Tokyo."

[0039] Step 4: The generative AI means speaks the generated information. In this example, it speaks, "That is incorrect. The capital of Japan is Tokyo."

[0040] Example 1

[0041] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0042] In conventional educational systems, it was difficult to determine in real time whether what a learner said was correct and to immediately provide the correct information. In addition, if a learner said incorrect information, there was a lack of means to provide appropriate feedback to correct the mistake. This resulted in problems such as learners not deepening their understanding and preventing effective learning.

[0043] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0044] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, a means for a learner to speak an incorrect answer and to generate and speak information for correcting the answer, a means for converting speech to text, a means for analyzing the text and determining the error, a means for generating a prompt sentence for generating correct information, and a means for converting the generated correct information into speech. This makes it possible to recognize a learner's speech in real time, immediately determine an error, and provide correct information.

[0045] A "voice recognition means" is a device or software for converting voice data into text data.

[0046] A "generative artificial intelligence means" is an artificial intelligence technique for generating new information based on input data.

[0047] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information in response to it, and provides feedback in the form of voice.

[0048] A "speech-to-text conversion means" is a technology for analyzing speech data and converting it into corresponding text data.

[0049] A "means for analyzing text and determining errors" is an algorithm or set of rules for analyzing text data and determining whether its content is correct.

[0050] "Means for generating a prompt sentence for generating correct information" refers to a technology for creating an instruction sentence for causing a generative artificial intelligence to generate correct information.

[0051] The "means for converting the generated correct information into voice" refers to a device or software for converting text data into voice data.

[0052] The present invention provides a technology for an educational robot system that recognizes a learner's utterances in real time, immediately detects mistakes, and provides correct information. A specific embodiment of this system will be described below.

[0053] First, the user speaks to the educational robot. For example, they say, "The Earth is flat." The device captures this speech with a microphone and saves it as voice data.

[0054] Next, the terminal transmits the acquired voice data to a voice recognition means. A general voice recognition API can be used as this voice recognition means. For example, Google (registered trademark) Cloud Speech-to-Text API is used. This API converts the voice data into text data. The converted text data becomes "The earth is flat."

[0055] The server receives the text data sent from the device and analyzes the text using a predefined set of rules that includes a list of true and false statements. The server determines that the statement "The Earth is flat" is false.

[0056] Next, the server uses a generative artificial intelligence means to generate correct information. A general generative AI model can be used as this generative artificial intelligence means. For example, the GPT-4 (registered trademark) model from OpenAI (registered trademark) is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical."

[0057] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A general speech synthesis API can be used as the speech synthesis means. For example, the Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical."

[0058] In this way, the educational robot system can recognize what the learner says in real time, immediately identify mistakes, and provide correct information, which will help the learner deepen their understanding and learn more effectively.

[0059] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0060] Step 1:

[0061] The user speaks to the educational robot. For example, "The Earth is flat." The device captures this speech with a microphone and obtains it as voice data. The input is the user's voice, and the output is the captured voice data.

[0062] Step 2:

[0063] The terminal sends the acquired voice data to the voice recognition means. A general voice recognition API is used as the voice recognition means. For example, Google Cloud Speech-to-Text API is used. This API converts the voice data into text data. The input is the voice data, and the output is the text data "The Earth is flat."

[0064] Step 3:

[0065] The server receives the text data sent from the terminal and analyzes the text. The analysis uses a predefined rule set, which contains a list of true and false statements. The server determines that the statement "The Earth is flat" is false. The input is the text data "The Earth is flat" and the output is the judgment result that it is "false."

[0066] Step 4:

[0067] The server generates correct information using a generative artificial intelligence means. A general generative AI model is used as this generative artificial intelligence means. For example, OpenAI's GPT-4 model is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical." The input is the prompt sentence, and the output is the correct information, "That is incorrect. The Earth is spherical."

[0068] Step 5:

[0069] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A common speech synthesis API is used as this speech synthesis means. For example, Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical." The input is the text data of the correct information, and the output is speech data.

[0070] (Application example 1)

[0071] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] When workers perform incorrect work procedures in a factory, it can lead to reduced production efficiency and safety issues. In particular, mistakes are likely to occur when workers are new or when the work procedures are complex. There is a demand for a system that can detect such mistakes in real time and provide the correct work procedures.

[0073] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a means for a worker to speak an incorrect work procedure and to generate and speak information for correcting it, a means for recognizing the worker's statement and judging whether the work procedure is correct, and a means for generating and speaking a correct work procedure. This makes it possible, when a worker speaks an incorrect work procedure, to point out the error in real time and provide the correct work procedure.

[0074] The "voice recognition means" is a device or system that acquires the speech of a worker as voice data and converts the voice data into text data.

[0075] A "generative AI means" is a device or system that uses an artificial intelligence model to generate appropriate information based on input data.

[0076] "Workers" refers to personnel who perform work within a factory.

[0077] A "work procedure" refers to the sequence of steps or methods required to complete a particular task.

[0078] "Utterance" refers to words or sentences spoken by a worker.

[0079] "Good work procedures" refer to the precise steps or methods needed to complete a task safely and efficiently.

[0080] An "error" refers to a situation in which what a worker says does not match the correct work procedure.

[0081] "Real-time" refers to responding immediately to what workers say and providing information.

[0082] A system for implementing this invention includes a voice recognition means, a generative AI means, a means for recognizing utterances by a worker and determining whether or not the work procedure is correct, and a means for generating and uttering the correct work procedure.

[0083] Hardware and Software Used

[0084] Hardware:

[0085] Voice recognition microphone (e.g. high performance microphone)

[0086] Robot body (e.g. multi-function robot)

[0087] Speakers (e.g. high quality speakers)

[0088] software:

[0089] Speech recognition software (e.g. voice recognition engine)

[0090] Generative AI models (e.g. advanced generative AI models)

[0091] Robot control software (e.g. robot operating system)

[0092] Processing flow

[0093] 1. Voice Recognition:

[0094] A worker says, "This part goes here."

[0095] The voice recognition microphone captures what is said, and the voice recognition engine converts it into text data.

[0096] 2. Judging statements:

[0097] The converted text data is analyzed to determine whether the work procedures are correct.

[0098] For example, if a statement such as "This part should be installed here" is incorrect, a prompt is sent to the generative AI model.

[0099] 3. Producing the right information:

[0100] The generative AI model generates the correct information: "That's wrong. This part needs to be installed in a different location."

[0101] 4. Information Statement:

[0102] The correct information generated is communicated to the worker through the robot's built-in speaker.

[0103] Examples

[0104] If a worker says, "This part should be installed here," the robot will point out, "That's wrong. This part needs to be installed somewhere else."

[0105] Examples of prompt statements

[0106] User says: "This part goes here"

[0107] Generate the correct information.

[0108] With this system, when a worker makes an incorrect statement about a work procedure, the system can point out the mistake in real time and provide the correct work procedure, thereby improving work efficiency and safety in the factory.

[0109] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0110] Step 1:

[0111] Audio capture

[0112] Input: A worker says, "This part goes here."

[0113] Specific operation: A voice recognition microphone captures what the worker says.

[0114] Output: The captured audio data.

[0115] Step 2:

[0116] Speech to text conversion

[0117] Input: The captured audio data.

[0118] Specific operation: The voice recognition engine converts the voice data into text data.

[0119] Output: The converted text data (e.g. "Install this part here").

[0120] Step 3:

[0121] Speech analysis

[0122] Input: The converted text data.

[0123] Specific operation: The server analyzes the text data and determines whether the work procedure is correct.

[0124] Output: The result of a judgment as to whether the statement is correct or not (e.g., false).

[0125] Step 4:

[0126] Prompt Generation

[0127] Input: The judgement result that the statement is false.

[0128] Specific operation: The server sends a prompt to the generative AI model.

[0129] Output: A prompt to the generative AI model (e.g., "User says: 'This part should be installed here' Please generate the correct information.").

[0130] Step 5:

[0131] Producing the right information

[0132] Input: A prompt for the generative AI model.

[0133] Specific operation: The generative AI model generates the correct work steps.

[0134] Output: Generated correct work instructions (e.g. "That's wrong. This part needs to be installed somewhere else.").

[0135] Step 6:

[0136] Statement of correct information

[0137] Input: The generated correct work instructions.

[0138] Specific operation: The correct information generated is communicated to the worker through the speaker on the robot's body.

[0139] Output: The worker can hear the correct work procedure.

[0140] Example 2

[0141] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0142] In conventional educational systems, when a learner gives a wrong answer, there is a lack of a way to immediately provide correct information, so the learner may continue to hold on to the wrong knowledge. In addition, it is difficult to provide an environment in which the learner can actively learn.

[0143] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0144] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and speaking information for correcting an incorrect answer given by a learner, a means for converting voice data into text data, a means for analyzing the text data and judging the correctness of the statement, and a means for converting correct information into voice data and feeding it back to the learner. This makes it possible to provide correct information immediately when a learner gives a wrong answer, and to provide an environment in which the learner can actively learn.

[0145] A "voice recognition means" is a device or software for collecting voice data and converting it into text data.

[0146] A "generative AI means" is an artificial intelligence model that generates appropriate responses or information based on input data.

[0147] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information, and conveys it to the learner.

[0148] The "means for converting voice data into text data" refers to a device or software that converts voice data into text information using voice recognition technology.

[0149] The "means for analyzing text data and judging whether the content of a statement is correct" refers to a device or software for analyzing text data and judging whether the content is correct.

[0150] The "means for converting correct information into voice data and feeding it back to the learner" refers to a device or software for converting the generated correct information into voice data and conveying it to the learner.

[0151] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0152] This invention relates to an educational robot system, and provides an effective means for realizing a lesson in which students actively learn. Specifically, when a student gives a wrong answer, the educational robot system generates and speaks information to correct the answer, allowing the student to recognize his / her own mistake and obtain the correct information.

[0153] Hardware and Software Used

[0154] The system uses the following hardware and software:

[0155] Speech recognition means: A device or software for collecting voice data and converting it into text data. Specifically, we use voice recognition software (e.g., Google Cloud Speech-to-Text).

[0156] Generative AI means: An artificial intelligence model that generates appropriate responses or information based on input data. Specifically, we use a generative AI model (e.g., OpenAI's GPT-4).

[0157] Means for converting voice data into text data: A device or software that converts voice data into text information using voice recognition technology. Specifically, we use Google Cloud Speech-to-Text.

[0158] A means of analyzing text data and judging whether a statement is correct: A device or software that analyzes text data and judges whether the content is correct. Specifically, a generative AI model (e.g., OpenAI's GPT-4) is used.

[0159] Means of converting correct information into voice data and feeding it back to the learner: A device or software that converts the generated correct information into voice data and conveys it to the learner. Specifically, speech synthesis software (e.g. Amazon Polly) is used.

[0160] Examples

[0161] If a user says "The capital of Japan is Osaka", the following processing occurs:

[0162] 1. Voice recognition: The device collects the user's speech through a microphone. For example, the user says, "The capital of Japan is Osaka."

[0163] 2. Text conversion: The device converts the collected voice data into text data using the Google Cloud Speech-to-Text API. The converted text data becomes "The capital of Japan is Osaka."

[0164] 3. Correct / incorrect judgment: The server receives the text data sent from the terminal. Then, the server inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[0165] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[0166] 4. Information generation: The generative AI model generates the correct information: "That is incorrect. The capital of Japan is Tokyo." The server receives this generated information and sends it to the terminal.

[0167] 5. Voice feedback: The device converts the correct information received from the server into voice data using voice synthesis software (e.g. Amazon Polly). The device then communicates the converted voice data to the user through the speaker. Specifically, the device provides voice feedback saying, "That's incorrect. The capital of Japan is Tokyo."

[0168] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[0169] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0170] Step 1:

[0171] The user speaks.

[0172] The user says, "The capital of Japan is Osaka." This becomes the input to the system.

[0173] Step 2:

[0174] The terminal collects the user's utterances as voice data.

[0175] The device captures the user's speech as voice data through a microphone, and this voice data becomes the input for the next processing step.

[0176] Step 3:

[0177] The terminal converts the voice data into text data.

[0178] The device uses the Google Cloud Speech-to-Text API to convert the collected voice data into text data. For example, the generated text data is "The capital of Japan is Osaka." This text data becomes the input for the next processing step.

[0179] Step 4:

[0180] The server receives the text data and inputs a prompt sentence into the generative AI model.

[0181] The server receives the text data sent from the device and inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[0182] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[0183] This prompt sentence becomes the input to the generative AI model.

[0184] Step 5:

[0185] The generative AI model generates the correct information based on the prompt sentence.

[0186] The generative AI model analyzes the input prompt sentence and generates the correct information, "That is incorrect. The capital of Japan is Tokyo." This generated information becomes the input for the next processing step.

[0187] Step 6:

[0188] The server transmits the generated information to the terminal.

[0189] The server sends the correct information received from the generative AI model to the terminal. For example, text data such as "That is incorrect. The capital of Japan is Tokyo" is sent to the terminal.

[0190] Step 7:

[0191] The terminal converts the correct information into voice data.

[0192] The device uses Amazon Polly to convert the received text data into speech data. For example, speech data such as "That's wrong. The capital of Japan is Tokyo" is generated.

[0193] Step 8:

[0194] The terminal feeds back the voice data to the user.

[0195] The terminal transmits the generated voice data to the user through the speaker, and the user receives voice feedback saying, "That is incorrect. The capital of Japan is Tokyo."

[0196] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[0197] (Application example 2)

[0198] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0199] There is a demand for a system that can immediately instruct workers in the correct procedure when they make a mistake in a factory. With conventional methods, when a worker makes a mistake, it takes time for the worker to realize the mistake, resulting in a decrease in production efficiency. There was also the issue that education and training to correct mistakes takes a lot of time and costs money.

[0200] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0201] In this invention, the server includes a voice recognition means, a generative AI means, and a means for generating and announcing information for correcting an incorrect procedure made by a worker. This makes it possible to immediately instruct the correct procedure when a worker makes a mistake.

[0202] A "voice recognition means" is a technology that receives voice as input and converts the voice into text data.

[0203] "Generative AI means" refers to artificial intelligence technology that generates appropriate responses or information based on input data.

[0204] "Workers" refers to personnel who perform work within a factory.

[0205] A "procedure" refers to the sequence of operations or steps required to complete a particular task.

[0206] "Means for generating and uttering information" refers to technology for outputting the generated information as voice.

[0207] The system for carrying out the present invention includes a voice recognition means, a generative AI means, and a means for a worker to speak out a procedure that he or she has made a mistake in and to generate and speak information for correcting that procedure.

[0208] First, the speech recognition means requires technology that converts speech into text data. Specifically, speech recognition APIs such as Google Cloud Speech-to-Text and IBM Watson (registered trademark) Speech to Text can be used. This makes it possible to convert the speech of workers into text data in real time.

[0209] Next, as a generative AI method, a generative AI model such as OpenAI's GPT-3 (registered trademark) is used. The generative AI model generates appropriate responses and information based on the input text data. For example, if a worker says, "Assemble the parts in the order B→A→C," the generative AI model generates information such as, "That's incorrect. The correct order is A→B→C."

[0210] Finally, a technology to convert the generated text data into voice is required as a means of generating and speaking information. Specifically, speech synthesis APIs such as Google Cloud Text-to-Speech and Amazon Polly can be used. This makes it possible to output the generated information as voice and convey it to workers.

[0211] As a concrete example, if a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the means for generating and uttering information converts the information into voice and conveys it to the worker.

[0212] Examples of prompts include:

[0213] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[0214] In this way, when a worker makes a mistake in a procedure, it is possible to immediately instruct the worker to follow the correct procedure, thereby improving the efficiency and accuracy of work in the factory.

[0215] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0216] Step 1:

[0217] The server receives the speech of the worker as voice data using a voice recognition means. The voice recognition means converts the voice data into text data. The input is the voice data of the worker, and the output is text data.

[0218] Step 2:

[0219] The server passes text data to the generative AI means as input. The generative AI means generates an appropriate response based on this text data. Specifically, if the worker's statement is incorrect, it generates a response including a correct procedure. The input is text data, and the output is the generated response text.

[0220] Step 3:

[0221] The server passes the generated response text as input to the speech synthesis means, which converts the text data into speech data, with the input being the generated response text and the output being the speech data.

[0222] Step 4:

[0223] The server transmits the voice data to the worker through a speaker, allowing the worker to confirm the correct procedure by voice. The input is the voice data, and the output is the voice that the worker hears.

[0224] Specifically, when a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the speech synthesis means converts the information into voice and conveys it to the worker through a speaker.

[0225] Examples of prompts include:

[0226] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[0227] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0228] "Example 1"

[0229] One embodiment of the present invention is an educational robot system that incorporates an emotion engine. In this system, a voice recognition means recognizes a learner's utterance, and a generative AI means judges whether the utterance is incorrect. If the utterance is incorrect, the generative AI means generates correct information and speaks that information. Furthermore, this system includes an emotion engine and can recognize the learner's emotions. For example, if the learner feels confused, the generative AI means generates information according to the learner's emotion, such as encouraging words like "It's okay, let's think about it together," and speaks that information.

[0230] "Example 2"

[0231] In another embodiment, the emotion engine adjusts the response of the educational robot system according to the emotion of the learner. For example, if the emotion engine senses that the learner is angry or frustrated, it conveys the information to the generative AI means. Based on the information, the generative AI means generates words of apology or re-explanation, such as "I'm sorry, I'll explain it again," and speaks the information. This allows an appropriate response to be provided according to the learner's emotion.

[0232] The process flow of each embodiment will be described below.

[0233] "Example 1"

[0234] Step 1: The learner says something.

[0235] Step 2: The speech recognition means recognizes the learner's utterance.

[0236] Step 3: Determine whether the utterance recognized by the generative AI means is incorrect.

[0237] Step 4: If incorrect, the generative AI means generates the correct information and speaks that information.

[0238] Step 5: The emotion engine recognizes the learner's emotions.

[0239] Step 6: The generative AI means generates information according to the emotion based on the information from the emotion engine, and speaks that information.

[0240] "Example 2"

[0241] Step 1: The learner says something.

[0242] Step 2: The speech recognition means recognizes the learner's utterance.

[0243] Step 3: The emotion engine recognizes the learner's emotions.

[0244] Step 4: Based on the emotion engine's recognized emotion, the emotion engine conveys information to the generative AI means. Step 5: Based on the information from the emotion engine, the generative AI means generates information corresponding to the emotion, and speaks that information.

[0245] Example 1

[0246] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0247] Conventional educational systems can accurately recognize what students say and point out their mistakes, but they cannot respond according to the student's emotions. This makes it difficult to provide appropriate support when a student feels confused or anxious. There is also a need to improve learning outcomes by providing feedback according to the student's emotions.

[0248] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0249] In this invention, the server includes a speech recognition means, a generative AI means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and uttering information according to the emotions. This makes it possible to accurately recognize the learner's utterances, point out mistakes, and provide appropriate feedback according to the learner's emotions.

[0250] A "voice recognition means" is a device or software that captures a learner's speech as voice data and converts the voice data into text data.

[0251] "Generative AI means" refers to an artificial intelligence model that generates appropriate information based on input data, and in particular has the ability to determine whether what a learner says is correct or incorrect and generate correct information.

[0252] "A means for allowing a learner to speak an incorrect answer and generating and speaking information to correct it" refers to a device or software that, when what a learner says is incorrect, points out the mistake and generates and speaks the correct information.

[0253] "Means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions" refers to a device or software that analyzes the learner's emotions from their facial expressions, tone of voice, etc., and generates and speaks appropriate feedback or words of encouragement corresponding to those emotions.

[0254] The educational robot system of the present invention includes a voice recognition means, a generative AI means, a means for generating and speaking information to correct an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions.

[0255] Hardware and Software Used

[0256] Hardware: Microphone, speaker, camera (for emotion recognition)

[0257] Software: voice recognition software (e.g. voice recognition APIs), generative AI models (e.g. AI models), emotion recognition engines (e.g. emotion recognition software)

[0258] Program processing flow

[0259] Speech Recognition

[0260] The device uses a microphone to capture what the learner says.

[0261] The server uses voice recognition software to convert the voice data into text data.

[0262] Example: If a learner says, "The Earth is flat," speech recognition software converts this utterance into text.

[0263] Evaluation of comments

[0264] The server uses a generative AI model to evaluate whether the text data is correct.

[0265] Example: determining that the statement "The Earth is flat" is false.

[0266] Producing the right information

[0267] The server uses a generative AI model to generate correct information.

[0268] Example: Producing the correct information, "That's wrong. The Earth is spherical."

[0269] emotion recognition

[0270] The device uses a camera to capture the learner's facial expressions.

[0271] The server uses an emotion recognition engine to analyze the learner's emotions.

[0272] Example: A camera captures a learner's confused expression, and an emotion recognition engine identifies the expression as "confused."

[0273] Generating responses based on emotions

[0274] The server uses a generative AI model to generate information based on the learner's emotions.

[0275] Example: A generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[0276] Information statement

[0277] The device uses a speaker to communicate the generated information to the learner.

[0278] Example: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's figure it out together."

[0279] Examples of prompt statements

[0280] Prompt: "If the learner says, 'The Earth is flat,' generate and speak the correct information. Also generate encouraging words if the learner is confused."

[0281] In this way, the educational robot system of the present invention can not only recognize what the learner says and provide correct information, but also respond according to the learner's emotions, making it possible to provide appropriate support to the learner even when he or she feels confused or anxious, thereby improving the effectiveness of learning.

[0282] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0283] Step 1:

[0284] Capturing audio input

[0285] The device uses a microphone to capture what the learner says.

[0286] Input: Learner's spoken utterance (e.g. "The Earth is flat")

[0287] Output: The captured audio data.

[0288] What happens: A learner says, "The Earth is flat," and their voice is captured by a microphone.

[0289] Step 2:

[0290] Converting audio data to text

[0291] The server uses voice recognition software to convert the captured voice data into text data.

[0292] Input: Captured audio data

[0293] Output: Text data (e.g. "The Earth is flat")

[0294] What it does: Speech recognition software analyzes the speech data and converts it into text: "The Earth is flat."

[0295] Step 3:

[0296] Evaluation of comments

[0297] The server uses a generative AI model to evaluate whether the text data is correct.

[0298] Input: Text data (e.g. "The Earth is flat")

[0299] Output: Evaluation result (e.g., error)

[0300] How it works: A generative AI model analyzes the statement "The Earth is flat" and determines that it is false.

[0301] Step 4:

[0302] Producing the right information

[0303] The server uses a generative AI model to generate correct information.

[0304] Input: Evaluation result (e.g. mistake)

[0305] Output: Correct information (e.g. "That's not true. The Earth is a sphere.")

[0306] Specific behavior: The generative AI model generates the correct information, "That's wrong. The Earth is spherical."

[0307] Step 5:

[0308] emotion recognition

[0309] The device uses a camera to capture the learner's facial expressions.

[0310] The server uses an emotion recognition engine to analyze the learner's emotions.

[0311] Input: Learner's facial expression data

[0312] Output: Sentiment analysis result (e.g. confused)

[0313] Specific action: The camera captures the learner's confused expression, and the emotion recognition engine recognizes the expression as "confused."

[0314] Step 6:

[0315] Generating responses based on emotions

[0316] The server uses a generative AI model to generate information based on the learner's emotions.

[0317] Input: Sentiment analysis result (e.g. confused)

[0318] Output: Information based on the emotion (e.g. "It's okay, let's think about it together")

[0319] Specific behavior: The generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[0320] Step 7:

[0321] Information statement

[0322] The device uses a speaker to communicate the generated information to the learner.

[0323] Input: Correct information and emotion-specific information (e.g., "That's wrong. The Earth is a sphere." "It's okay, let's figure it out together.")

[0324] Output: Audio feedback to the learner

[0325] Specific actions: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's think about it together."

[0326] (Application example 1)

[0327] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] There is a need to point out mistakes in factory worker remarks and procedures in real time and provide correct information. Another challenge is to improve work efficiency and safety by recognizing the emotions of workers and providing appropriate encouragement and advice.

[0329] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0330] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and uttering information for correcting a procedure that the worker has made a mistake in, an emotion recognition means, and a means for generating and uttering information according to the emotion of the worker. This makes it possible to point out the worker's comments and errors in the procedure in real time, provide correct information, and take appropriate action according to the emotion of the worker.

[0331] The "voice recognition means" is a device or system that recognizes the speech of the worker as voice and converts it into text data.

[0332] "Generative AI means" refers to artificial intelligence techniques that generate appropriate information based on input data.

[0333] "Means for a worker to state a procedure that he or she made a mistake and to generate and state information to correct it" refers to a device or system that evaluates the content of the worker's statement and generates and provides the correct procedure or information if a mistake is made.

[0334] An "emotion recognition means" is a device or system that analyzes and recognizes emotions from a worker's facial expressions, tone of voice, etc.

[0335] The "means for generating and uttering information corresponding to the emotions of the worker" is a device or system that generates and provides appropriate encouragement or advice based on the emotions recognized by the emotion recognition means.

[0336] As an embodiment of the present invention, a factory work support robot system will be described as an example. This system includes a voice recognition means, a generative AI means, an emotion recognition means, and software for integrating them.

[0337] Hardware used:

[0338] Voice recognition microphone

[0339] speaker

[0340] Camera (for emotion recognition)

[0341] Factory robot body

[0342] Software used:

[0343] Speech recognition software (Google Speech-to-Text API)

[0344] Generative AI model (OpenAI GPT-4)

[0345] Emotion recognition software (Microsoft® Azure® Emotion API)

[0346] System Operation:

[0347] 1. Speech recognition: When a worker speaks, the speech is picked up by the speech recognition microphone and converted into text data using the Google Speech-to-Text API. This text data is then sent to the server.

[0348] 2. Evaluating the utterance: The server uses a generative AI model (GPT-4) to evaluate whether the worker's utterance is correct. For example, it judges whether the utterance "This part should be installed here" is correct.

[0349] 3. Generation of correct information: If the statement is incorrect, the server uses GPT-4 to generate correct information and communicates it to the worker through the speaker. For example, it provides information such as "That part should not be installed here, but next to it."

[0350] 4. Emotion Recognition: Using a camera and the Microsoft Azure Emotion API, the system analyzes the worker's facial expressions and recognizes their emotions. For example, if the worker is confused, that information is sent to the server.

[0351] 5. Emotion-based response: Based on the result of emotion recognition, the server uses GPT-4 to generate appropriate encouragement or advice and convey it to the worker through the speaker. For example, encouraging words such as "It's okay, let's check it together" are provided.

[0352] Examples:

[0353] When a worker says, "This part should be installed here," the voice recognition microphone picks up the statement and converts it into text using the Google Speech-to-Text API. The server uses GPT-4 to evaluate the statement, and if it determines that the statement is incorrect, it generates correct information such as, "That part should be installed in the next position, not here," and conveys this to the worker through the speaker. In addition, the camera and Microsoft Azure Emotion API analyze the worker's facial expressions, and if it recognizes that the worker is confused, it provides encouraging words such as, "It's okay, let's check together."

[0354] Example prompt for a generative AI model:

[0355] The worker says, "This part goes here." Evaluate whether this statement is correct and provide the correct information if it is incorrect. Also, generate appropriate words of encouragement if the worker is confused.

[0356] In this way, the factory work support robot system can evaluate the worker's comments in real time, provide correct information, and take appropriate action according to the worker's emotions.

[0357] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0358] Step 1:

[0359] Speech Recognition

[0360] Input: What the worker says

[0361] Processing: A voice recognition microphone picks up what the worker is saying and converts the voice data into text using the Google Speech-to-Text API.

[0362] Output: Text data

[0363] Specific operation: When a worker says, "This part goes here," the voice recognition microphone picks up the speech, and the Google Speech-to-Text API converts the speech into text.

[0364] Step 2:

[0365] Evaluation of comments

[0366] Input: Text data obtained in step 1

[0367] Processing: The server uses a generative AI model (GPT-4) to evaluate whether the text data is correct.

[0368] Output: Evaluation result (correct / incorrect)

[0369] Specific operation: The server inputs text data such as "This part should be installed here" into GPT-4 and evaluates whether the statement is correct.

[0370] Step 3:

[0371] Producing the right information

[0372] Input: If the evaluation result obtained in step 2 is "wrong"

[0373] Processing: The server uses a generative AI model (GPT-4) to generate the correct information.

[0374] Output: Text data with correct information

[0375] Specific operation: The server makes GPT-4 generate the correct information, such as "That part should not be installed here, but in the adjacent position."

[0376] Step 4:

[0377] Statement of correct information

[0378] Input: Text data of the correct information obtained in step 3

[0379] Processing: The server communicates the correct information to the worker via voice through a speaker.

[0380] Output: Audio information

[0381] Specific operation: The server communicates to the worker through a speaker, "That part should be installed in the next position, not here."

[0382] Step 5:

[0383] emotion recognition

[0384] Input: Facial expression data of the worker

[0385] Processing: The camera captures the worker's facial expressions and analyzes their emotions using the Microsoft Azure Emotion API.

[0386] Output: Emotion data (e.g. confusion, relief, etc.)

[0387] Specific operation: The camera captures the worker's facial expression, which is then analyzed by the Microsoft Azure Emotion API and recognized as "confusion."

[0388] Step 6:

[0389] Generating information according to emotions

[0390] Input: Emotion data obtained in step 5

[0391] Processing: The server uses a generative AI model (GPT-4) to generate appropriate information based on the emotion.

[0392] Output: Text data of information according to emotions

[0393] Specific operation: The server asks GPT-4 to generate encouraging words such as, "It's okay, let's check it together."

[0394] Step 7:

[0395] Expressing information according to emotions

[0396] Input: Text data of emotion-related information obtained in step 6

[0397] Processing: The server communicates information corresponding to the emotion to the worker through a speaker.

[0398] Output: Audio information

[0399] Specific operation: The server communicates to the worker through a speaker, "It's OK, let's check it together."

[0400] Example 2

[0401] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0402] In conventional educational systems, it is difficult to immediately provide correct information when a learner gives a wrong answer, making it difficult for the learner to recognize their own mistake. In addition, it is not possible to provide an appropriate response according to the learner's emotions, which can reduce the learner's motivation to learn. To solve these problems, a system is needed that can accurately recognize the learner's utterances, generate information to correct the mistake, and provide a response according to the learner's emotions.

[0403] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0404] In this invention, the server includes a speech recognition means, a natural language processing means, a generative AI means, an emotion recognition means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for adjusting a response according to the learner's emotion. This makes it possible to accurately recognize the learner's utterance, provide information for immediately correcting the mistake, and further provide an appropriate response according to the learner's emotion.

[0405] "Speech recognition means" refers to a technology that acquires a learner's speech as voice data and converts it into text data.

[0406] "Natural language processing means" refers to technology that analyzes acquired text data and understands the content of statements.

[0407] "Generative AI means" is an artificial intelligence technology that generates appropriate information based on what the learner says.

[0408] "Emotion recognition means" is a technology that analyzes a learner's facial expressions and tone of voice to detect the learner's emotions.

[0409] "A means for generating and uttering information to correct an incorrect answer given by a learner" is a technology that detects an incorrect answer given by a learner and generates and utters correct information.

[0410] "Means of adjusting responses according to the learner's emotions" refers to techniques for generating and uttering appropriate responses based on the learner's emotions.

[0411] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0412] The present invention relates to an educational robot system for implementing a lesson in which students actively learn. A specific embodiment of this system will be described below.

[0413] System Configuration

[0414] This educational robot system includes the following main components:

[0415] 1. Speech recognition method: Acquire the learner's speech as voice data and convert it into text data. Specifically, use voice recognition software (e.g. Google Speech-to-Text).

[0416] 2. Natural language processing means: Analyze the acquired text data and understand the content of the speech. Specifically, natural language processing software (e.g. spaCy) is used.

[0417] 3. Generative AI means: Generate appropriate information based on the learner's utterances. Specifically, we use a generative AI model (e.g., OpenAI GPT-4).

[0418] 4. Emotion recognition: Analyze the learner's facial expressions and tone of voice to detect the learner's emotions. Specifically, use emotion recognition software (e.g. Microsoft Azure Emotion API).

[0419] 5. Information generation and speech means: When a learner gives an incorrect answer, generate and speak information to correct it. Specifically, use speech synthesis software (e.g. Amazon Polly).

[0420] 6. Emotion-based response adjustment: Generate and speak appropriate responses based on the learner's emotions.

[0421] System Operation

[0422] The device picks up what the learner says through a microphone and converts it into text data using voice recognition software. The server then receives the text data sent from the device and analyzes it using natural language processing software. Based on the analysis results, the server determines whether what the learner said is correct. For example, if the learner says, "The capital of Japan is Osaka," the server detects that this is incorrect.

[0423] The server uses the generative AI model to generate the correct information. Specifically, it generates the information, "That is incorrect. The capital of Japan is Tokyo." Next, the device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions. The detected emotion data is sent to the server.

[0424] The server uses an emotion engine to generate a response based on the learner's emotions. For example, if the learner expresses anger or dissatisfaction, the server uses a generative AI model to generate an apology or re-explanation such as "I'm sorry, I'll explain it again." Finally, the device converts the generated information into speech using speech synthesis software and speaks it to the learner through the speaker.

[0425] Examples

[0426] For example, if a learner says "The capital of Japan is Osaka," the system will operate as follows:

[0427] 1. The device picks up speech through the microphone and uses Google Speech-to-Text to convert it into text data such as "The capital of Japan is Osaka."

[0428] 2. The server receives the text data and analyzes it using spaCy.

[0429] 3. Based on the analysis results, the server detects that the statement "The capital of Japan is Osaka" is incorrect.

[0430] 4. The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[0431] 5. The device analyzes the learner's facial expressions and tone of voice using the Microsoft Azure Emotion API to detect when the learner is expressing dissatisfaction.

[0432] 6. The server uses an emotion engine to generate an apology such as "I'm sorry, I'll explain it again."

[0433] 7. The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[0434] Examples of prompt statements

[0435] "If a learner says, 'The capital of Japan is Osaka,' please generate the correct information."

[0436] "If the learner expresses anger or frustration, generate appropriate apologies or restatements."

[0437] The above is an embodiment of the invention.

[0438] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0439] Step 1:

[0440] The user makes a statement as a learner.

[0441] Input: Learner's spoken words

[0442] Output: Audio data

[0443] Specific action: The learner says, "The capital of Japan is Osaka."

[0444] Step 2:

[0445] The terminal captures the learner's voice data through a microphone and converts it into text data using voice recognition software.

[0446] Input: Audio data

[0447] Output: Text data

[0448] Specific operation: The device uses Google Speech-to-Text to generate text data such as "The capital of Japan is Osaka."

[0449] Step 3:

[0450] The server receives the text data sent from the terminal and analyzes it using natural language processing software.

[0451] Input: Text data

[0452] Output: Analysis results

[0453] Specific operation: The server uses spaCy to analyze the text data and understand the content of the statement, "The capital of Japan is Osaka."

[0454] Step 4:

[0455] The server uses the analysis results to determine whether the learner's statement is correct.

[0456] Input: Analysis results

[0457] Output: True or False

[0458] Specific operation: The server detects that the statement "The capital of Japan is Osaka" is false.

[0459] Step 5:

[0460] The server generates the correct information using a generative AI model.

[0461] Input: Correct / Incorrect

[0462] Output: Correct information

[0463] Specific operation: The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[0464] Step 6:

[0465] The device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions.

[0466] Input: Learner's facial expressions and tone of voice

[0467] Output: Emotion data

[0468] Specific behavior: The device uses the Microsoft Azure Emotion API to detect when a learner is expressing dissatisfaction.

[0469] Step 7:

[0470] The terminal transmits the detected emotion data to the server.

[0471] Input: Emotion data

[0472] Output: Emotion data (sent to server)

[0473] Specific operation: The terminal transmits emotion data to the server.

[0474] Step 8:

[0475] The server uses an emotion engine to generate a response according to the learner's emotion.

[0476] Input: Emotion data

[0477] Output: Appropriate response

[0478] What it does: The server uses a generative AI model to generate an apology saying, "I'm sorry, I'll explain it again."

[0479] Step 9:

[0480] The terminal uses speech synthesis software to convert the generated information into speech, which is then spoken to the learner through a speaker.

[0481] Input: appropriate response

[0482] Output: Audio data

[0483] Specific operation: The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[0484] (Application example 2)

[0485] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0486] Conventional educational robot systems have the function of providing correct information when a learner gives an incorrect answer, but they are unable to show appropriate responses according to the learner's emotions, making it difficult to improve the learner's motivation and level of understanding.In addition, when it comes to work guidance in factories, there is a lack of a function to teach the correct procedure when a worker performs an incorrect work procedure, making it difficult to improve work efficiency and quality.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0488] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and speaking information for correcting an incorrect answer given by a learner, an emotion recognition means, a means for adjusting a response according to the emotion of the learner, and a means for instructing a work procedure. This makes it possible to provide correct information when a learner gives a wrong answer, and to show an appropriate response according to the emotion of the learner. In addition, in the case of work guidance in a factory, if a worker performs an incorrect work procedure, the correct procedure can be taught, which is expected to improve work efficiency and quality.

[0489] "Speech recognition means" refers to technology that analyzes speech and converts what is being said into text data.

[0490] "Generative AI means" refers to artificial intelligence technology that generates appropriate information based on input data.

[0491] "A means for allowing a learner to give an incorrect answer and for generating and giving information to correct it" refers to technology that has the function of detecting incorrect answers given by a learner and providing the correct information.

[0492] "Emotion recognition means" is a technology that analyzes and recognizes emotions from learners' facial expressions and voices.

[0493] A "means for tailoring responses according to learner emotions" is a technique for generating and providing appropriate responses based on recognized emotions.

[0494] The "means for instructing work procedures" refers to technology that has the function of teaching correct procedures when a worker performs an incorrect work procedure.

[0495] A system for implementing this invention includes a voice recognition means, a generative AI means, a means for generating and speaking information to correct an incorrect answer provided by a learner, an emotion recognition means, a means for adjusting a response in accordance with the learner's emotions, and a means for instructing the learner on work procedures.

[0496] Hardware and Software Used

[0497] Hardware: Task guidance robot, microphone, camera

[0498] Software: OpenAI API, Emotion Recognition API

[0499] System program processing

[0500] Voice Recognition Means

[0501] The server collects the learner's utterances through a microphone and converts them into text data using a voice recognition means. The voice recognition means accurately analyzes the contents of the learner's utterances and transmits them in text form to the server.

[0502] Generative AI means

[0503] The server uses the generative AI means to determine whether the learner's utterance is correct. If the utterance is incorrect, the generative AI means generates correct information and creates text data to utter that information.

[0504] emotion recognition means

[0505] The server collects the learner's facial expressions and voices through a camera and microphone, and analyzes the learner's emotions using an emotion recognition means. This emotion recognition means judges whether the learner is showing anger or dissatisfaction, and transmits the result to the server.

[0506] A means of adjusting responses according to learners' emotions

[0507] The server generates an appropriate response according to the learner's emotions based on the data from the emotion recognition means. For example, if the learner shows anger or dissatisfaction, the generative AI means generates an apology or re-explanation such as "I'm sorry, I'll explain it again."

[0508] A means of instructing work procedures

[0509] The server generates information to teach correct work procedures when a worker performs an incorrect work procedure. The generative AI means detects incorrect procedures and creates text data to explain the correct procedures.

[0510] Examples

[0511] For example, if a worker says, "Put this part here," but it should actually be installed in a different place, the server will tell them, "That's wrong. The correct procedure is to install this part here." Also, if a worker is stressed, the server will respond to the worker's emotions by saying, "I'm sorry, let me explain it again."

[0512] Examples of prompt statements

[0513] If a user says "install this part here", what is the correct procedure?

[0514] If the user is stressed, generate an appropriate response.

[0515] In this way, a system is realized that can provide correct information when a learner or worker utters incorrect information, and can also show an appropriate response according to their emotions.

[0516] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0517] Step 1:

[0518] The server collects the user's speech through a microphone. The input is the user's voice, and the output is voice data. This voice data is sent to a voice recognition means.

[0519] Step 2:

[0520] The server converts the voice data into text data using a voice recognition means. The input is voice data, and the output is text data. This text data is sent to the generative AI means.

[0521] Step 3:

[0522] The server uses a generative AI means to determine whether the text data is correct. The input is the text data, and the output is the result of the judgment. If the statement is incorrect, the generative AI means generates the correct information.

[0523] Step 4:

[0524] The server creates text data for uttering the correct information from the generative AI means. The input is the result of the judgment of correctness or incorrectness and the correct information, and the output is the text data for utterance. This text data is sent to the speech synthesis means.

[0525] Step 5:

[0526] The server collects the user's facial expressions and voice through a camera and microphone, and analyzes the user's emotions using an emotion recognition means. The input is the user's facial expression data and voice data, and the output is emotion data. This emotion data is sent to the generative AI means.

[0527] Step 6:

[0528] The server generates an appropriate response according to the user's emotion based on the emotion data from the emotion recognition means. The input is the emotion data, and the output is text data of the response according to the emotion. This text data is sent to the voice synthesis means.

[0529] Step 7:

[0530] The server generates information to teach the correct work procedure when the worker performs an incorrect work procedure. The input is text data of the incorrect work procedure, and the output is text data of the correct work procedure. This text data is sent to the voice synthesis means.

[0531] Step 8:

[0532] The server uses a speech synthesis means to convert the generated text data into speech and speaks it to the user. The input is the text data to be spoken, and the output is speech data. This speech data is provided to the user through a speaker.

[0533] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0534] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0535] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0536] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0537] [Second embodiment]

[0538] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0539] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0540] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. In addition,

[0541] The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a Wide Area Network (WAN) and / or a Local Area Network (LAN).

[0542] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0543] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0544] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0545] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0546] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0547] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0548] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0549] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0550] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0551] "Example 1"

[0552] The educational robot system of the present invention is equipped with a voice recognition means and a generative AI means.

[0553] The recognition means recognizes what the learner says and determines whether the statement is incorrect. The generative AI means generates correct information for the learner's incorrect statement and speaks that information. Specifically, if the learner says, "The Earth is flat," the speech recognition means recognizes this statement, and the generative AI means generates and speaks the information, "That's incorrect. The Earth is spherical."

[0554] "Example 2"

[0555] The educational robot system of the present invention provides an effective means for implementing lesson development in which learners actively learn. Specifically, when a learner gives a wrong answer, the educational robot system generates and speaks information to correct it. This allows the learner to recognize his or her own mistake and obtain the correct information. For example, if a learner says, "The capital of Japan is Osaka," the educational robot system generates and speaks the information, "That is incorrect. The capital of Japan is Tokyo."

[0556] The process flow of each embodiment will be described below.

[0557] "Example 1"

[0558] Step 1: The learner makes an utterance, for example, "The Earth is flat."

[0559] Step 2: The speech recognition means recognizes the learner's utterance. In this example, it recognizes the utterance "The Earth is flat."

[0560] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information "That is incorrect. The Earth is spherical." Step 4: The generative AI means speaks out the generated information. In this example, it speaks out "That is incorrect. The Earth is spherical."

[0561] "Example 2"

[0562] Step 1: A learner makes an utterance. For example, the learner may say, "The capital of Japan is Osaka." Step 2: A speech recognition means recognizes the learner's utterance. In this example, the speech recognition means recognizes the utterance, "The capital of Japan is Osaka."

[0563] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information, "That is incorrect. The capital of Japan is Tokyo."

[0564] Step 4: The generative AI means speaks the generated information. In this example, it speaks, "That is incorrect. The capital of Japan is Tokyo."

[0565] Example 1

[0566] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0567] In conventional educational systems, it was difficult to determine in real time whether what a learner said was correct and to immediately provide the correct information. In addition, if a learner said incorrect information, there was a lack of means to provide appropriate feedback to correct the mistake. This resulted in problems such as learners not deepening their understanding and preventing effective learning.

[0568] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0569] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, a means for a learner to speak an incorrect answer and to generate and speak information for correcting the answer, a means for converting speech to text, a means for analyzing the text and determining the error, a means for generating a prompt sentence for generating correct information, and a means for converting the generated correct information into speech. This makes it possible to recognize a learner's speech in real time, immediately determine an error, and provide correct information.

[0570] A "voice recognition means" is a device or software for converting voice data into text data.

[0571] A "generative artificial intelligence means" is an artificial intelligence technique for generating new information based on input data.

[0572] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information in response to it, and provides feedback in the form of voice.

[0573] A "speech-to-text conversion means" is a technology for analyzing speech data and converting it into corresponding text data.

[0574] A "means for analyzing text and determining errors" is an algorithm or set of rules for analyzing text data and determining whether its content is correct.

[0575] "Means for generating a prompt sentence for generating correct information" refers to a technology for creating an instruction sentence for causing a generative artificial intelligence to generate correct information.

[0576] The "means for converting the generated correct information into voice" refers to a device or software for converting text data into voice data.

[0577] The present invention provides a technology for an educational robot system that recognizes a learner's utterances in real time, immediately detects mistakes, and provides correct information. A specific embodiment of this system will be described below.

[0578] First, the user speaks to the educational robot. For example, they say, "The Earth is flat." The device captures this speech with a microphone and saves it as voice data.

[0579] Next, the terminal transmits the acquired voice data to a voice recognition means. A general voice recognition API can be used as this voice recognition means. For example, Google Cloud Speech-to-Text API is used. This API converts the voice data into text data. The converted text data becomes "The Earth is flat."

[0580] The server receives the text data sent from the device and analyzes the text using a predefined set of rules that includes a list of true and false statements. The server determines that the statement "The Earth is flat" is false.

[0581] Next, the server uses a generative artificial intelligence means to generate correct information. A general generative AI model can be used as this generative artificial intelligence means. For example, OpenAI's GPT-4 model is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical."

[0582] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A general speech synthesis API can be used as the speech synthesis means. For example, the Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical."

[0583] In this way, the educational robot system can recognize what the learner says in real time, immediately identify mistakes, and provide correct information, which will help the learner deepen their understanding and learn more effectively.

[0584] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0585] Step 1:

[0586] The user speaks to the educational robot. For example, "The Earth is flat." The device captures this speech with a microphone and obtains it as voice data. The input is the user's voice, and the output is the captured voice data.

[0587] Step 2:

[0588] The terminal sends the acquired voice data to the voice recognition means. A general voice recognition API is used as the voice recognition means. For example, Google Cloud Speech-to-Text API is used. This API converts the voice data into text data. The input is the voice data, and the output is the text data "The Earth is flat."

[0589] Step 3:

[0590] The server receives the text data sent from the terminal and analyzes the text. The analysis uses a predefined rule set, which contains a list of true and false statements. The server determines that the statement "The Earth is flat" is false. The input is the text data "The Earth is flat" and the output is the judgment result that it is "false."

[0591] Step 4:

[0592] The server generates correct information using a generative artificial intelligence means. A general generative AI model is used as this generative artificial intelligence means. For example, OpenAI's GPT-4 model is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical." The input is the prompt sentence, and the output is the correct information, "That is incorrect. The Earth is spherical."

[0593] Step 5:

[0594] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A common speech synthesis API is used as this speech synthesis means. For example, Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical." The input is the text data of the correct information, and the output is speech data.

[0595] (Application example 1)

[0596] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0597] When workers perform incorrect work procedures in a factory, it can lead to reduced production efficiency and safety issues. In particular, mistakes are likely to occur when workers are new or when the work procedures are complex. There is a demand for a system that can detect such mistakes in real time and provide the correct work procedures.

[0598] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a means for a worker to speak an incorrect work procedure and to generate and speak information for correcting it, a means for recognizing the worker's statement and judging whether the work procedure is correct, and a means for generating and speaking a correct work procedure. This makes it possible, when a worker speaks an incorrect work procedure, to point out the error in real time and provide the correct work procedure.

[0599] The "voice recognition means" is a device or system that acquires the speech of a worker as voice data and converts the voice data into text data.

[0600] A "generative AI means" is a device or system that uses an artificial intelligence model to generate appropriate information based on input data.

[0601] "Workers" refers to personnel who perform work within a factory.

[0602] A "work procedure" refers to the sequence of steps or methods required to complete a particular task.

[0603] "Utterance" refers to words or sentences spoken by a worker.

[0604] "Proper work procedures" refer to the precise steps or methods needed to complete a task safely and efficiently.

[0605] An "error" refers to a situation in which what a worker says does not match the correct work procedure.

[0606] "Real-time" refers to responding immediately to what workers say and providing information.

[0607] A system for implementing this invention includes a voice recognition means, a generative AI means, a means for recognizing utterances by a worker and determining whether or not the work procedure is correct, and a means for generating and uttering the correct work procedure.

[0608] Hardware and Software Used

[0609] Hardware:

[0610] Voice recognition microphone (e.g. high performance microphone)

[0611] Robot body (e.g. multi-function robot)

[0612] Speakers (e.g. high quality speakers)

[0613] software:

[0614] Speech recognition software (e.g. voice recognition engine)

[0615] Generative AI models (e.g. advanced generative AI models)

[0616] Robot control software (e.g. robot operating system)

[0617] Processing flow

[0618] 1. Voice Recognition:

[0619] A worker says, "This part goes here."

[0620] The voice recognition microphone captures what is said, and the voice recognition engine converts it into text data.

[0621] 2. Judging statements:

[0622] The converted text data is analyzed to determine whether the work procedures are correct.

[0623] For example, if a statement such as "This part should be installed here" is incorrect, a prompt is sent to the generative AI model.

[0624] 3. Producing the right information:

[0625] The generative AI model generates the correct information: "That's wrong. This part needs to be installed in a different location."

[0626] 4. Information Statement:

[0627] The correct information generated is communicated to the worker through the robot's speaker.

[0628] Examples

[0629] If a worker says, "This part should be installed here," the robot will point out, "That's wrong. This part needs to be installed somewhere else."

[0630] Examples of prompt statements

[0631] User says: "This part goes here"

[0632] Generate the correct information.

[0633] With this system, when a worker makes an incorrect statement about a work procedure, the system can point out the mistake in real time and provide the correct work procedure, thereby improving work efficiency and safety in the factory.

[0634] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0635] Step 1:

[0636] Audio capture

[0637] Input: A worker says, "This part goes here."

[0638] Specific operation: A voice recognition microphone captures what the worker says.

[0639] Output: The captured audio data.

[0640] Step 2:

[0641] Speech to text conversion

[0642] Input: The captured audio data.

[0643] Specific operation: The voice recognition engine converts the voice data into text data.

[0644] Output: The converted text data (e.g. "Install this part here").

[0645] Step 3:

[0646] Speech analysis

[0647] Input: The converted text data.

[0648] Specific operation: The server analyzes the text data and determines whether the work procedure is correct.

[0649] Output: The result of a judgment as to whether the statement is correct or not (e.g., false).

[0650] Step 4:

[0651] Prompt Generation

[0652] Input: The judgement result that the statement is false.

[0653] Specific operation: The server sends a prompt to the generative AI model.

[0654] Output: A prompt to the generative AI model (e.g., "User says: 'This part should be installed here' Please generate the correct information.").

[0655] Step 5:

[0656] Producing the right information

[0657] Input: A prompt for the generative AI model.

[0658] Specific operation: The generative AI model generates the correct work steps.

[0659] Output: Generated correct work instructions (e.g. "That's wrong. This part needs to be installed somewhere else.").

[0660] Step 6:

[0661] Statement of correct information

[0662] Input: The generated correct work instructions.

[0663] Specific operation: The correct information generated is communicated to the worker through the speaker on the robot's body.

[0664] Output: The worker can hear the correct work procedure.

[0665] Example 2

[0666] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0667] In conventional educational systems, when a learner gives a wrong answer, there is a lack of a way to immediately provide correct information, so the learner may continue to hold on to the wrong knowledge. In addition, it is difficult to provide an environment in which the learner can actively learn.

[0668] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0669] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and speaking information for correcting an incorrect answer given by a learner, a means for converting voice data into text data, a means for analyzing the text data and judging the correctness of the statement, and a means for converting correct information into voice data and feeding it back to the learner. This makes it possible to provide correct information immediately when a learner gives a wrong answer, and to provide an environment in which the learner can actively learn.

[0670] A "voice recognition means" is a device or software for collecting voice data and converting it into text data.

[0671] A "generative AI means" is an artificial intelligence model that generates appropriate responses or information based on input data.

[0672] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information, and conveys it to the learner.

[0673] The "means for converting voice data into text data" refers to a device or software that converts voice data into text information using voice recognition technology.

[0674] The "means for analyzing text data and judging whether the content of a statement is correct" refers to a device or software for analyzing text data and judging whether the content is correct.

[0675] The "means for converting correct information into voice data and feeding it back to the learner" refers to a device or software for converting the generated correct information into voice data and conveying it to the learner.

[0676] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0677] This invention relates to an educational robot system, and provides an effective means for realizing a lesson in which students actively learn. Specifically, when a student gives a wrong answer, the educational robot system generates and speaks information to correct the answer, allowing the student to recognize his / her own mistake and obtain the correct information.

[0678] Hardware and Software Used

[0679] The system uses the following hardware and software:

[0680] Speech recognition means: A device or software for collecting voice data and converting it into text data. Specifically, we use voice recognition software (e.g., Google Cloud Speech-to-Text).

[0681] Generative AI means: An artificial intelligence model that generates appropriate responses or information based on input data. Specifically, we use a generative AI model (e.g., OpenAI's GPT-4).

[0682] Means for converting voice data into text data: A device or software that converts voice data into text information using voice recognition technology. Specifically, we use Google Cloud Speech-to-Text.

[0683] A means of analyzing text data and judging whether a statement is correct: A device or software that analyzes text data and judges whether the content is correct. Specifically, a generative AI model (e.g., OpenAI's GPT-4) is used.

[0684] Means of converting correct information into voice data and feeding it back to the learner: A device or software that converts the generated correct information into voice data and conveys it to the learner. Specifically, speech synthesis software (e.g. Amazon Polly) is used.

[0685] Examples

[0686] If a user says "The capital of Japan is Osaka", the following processing occurs:

[0687] 1. Voice recognition: The device collects the user's speech through a microphone. For example, the user says, "The capital of Japan is Osaka."

[0688] 2. Text conversion: The device converts the collected voice data into text data using the Google Cloud Speech-to-Text API. The converted text data becomes "The capital of Japan is Osaka."

[0689] 3. Correct / incorrect judgment: The server receives the text data sent from the terminal. Then, the server inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[0690] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[0691] 4. Information generation: The generative AI model generates the correct information: "That is incorrect. The capital of Japan is Tokyo." The server receives this generated information and sends it to the terminal.

[0692] 5. Voice feedback: The device converts the correct information received from the server into voice data using voice synthesis software (e.g. Amazon Polly). The device then communicates the converted voice data to the user through the speaker. Specifically, the device provides voice feedback saying, "That's incorrect. The capital of Japan is Tokyo."

[0693] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[0694] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0695] Step 1:

[0696] The user speaks.

[0697] The user says, "The capital of Japan is Osaka." This becomes the input to the system.

[0698] Step 2:

[0699] The terminal collects the user's utterances as voice data.

[0700] The device captures the user's speech as voice data through a microphone, and this voice data becomes the input for the next processing step.

[0701] Step 3:

[0702] The terminal converts the voice data into text data.

[0703] The device uses the Google Cloud Speech-to-Text API to convert the collected voice data into text data. For example, the generated text data is "The capital of Japan is Osaka." This text data becomes the input for the next processing step.

[0704] Step 4:

[0705] The server receives the text data and inputs a prompt sentence into the generative AI model.

[0706] The server receives the text data sent from the device and inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[0707] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[0708] This prompt sentence becomes the input to the generative AI model.

[0709] Step 5:

[0710] The generative AI model generates the correct information based on the prompt sentence.

[0711] The generative AI model analyzes the input prompt sentence and generates the correct information, "That is incorrect. The capital of Japan is Tokyo." This generated information becomes the input for the next processing step.

[0712] Step 6:

[0713] The server transmits the generated information to the terminal.

[0714] The server sends the correct information received from the generative AI model to the terminal. For example, text data such as "That is incorrect. The capital of Japan is Tokyo" is sent to the terminal.

[0715] Step 7:

[0716] The terminal converts the correct information into voice data.

[0717] The device uses Amazon Polly to convert the received text data into speech data. For example, speech data such as "That's wrong. The capital of Japan is Tokyo" is generated.

[0718] Step 8:

[0719] The terminal feeds back the voice data to the user.

[0720] The terminal transmits the generated voice data to the user through the speaker, and the user receives voice feedback saying, "That is incorrect. The capital of Japan is Tokyo."

[0721] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[0722] (Application example 2)

[0723] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0724] There is a demand for a system that can immediately instruct workers in the correct procedure when they make a mistake in a factory. With conventional methods, when a worker makes a mistake, it takes time for the worker to realize the mistake, resulting in a decrease in production efficiency. There was also the issue that education and training to correct mistakes takes a lot of time and costs money.

[0725] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0726] In this invention, the server includes a voice recognition means, a generative AI means, and a means for generating and announcing information for correcting an incorrect procedure made by a worker. This makes it possible to immediately instruct the correct procedure when a worker makes a mistake.

[0727] A "voice recognition means" is a technology that receives voice as input and converts the voice into text data.

[0728] "Generative AI means" refers to artificial intelligence technology that generates appropriate responses or information based on input data.

[0729] "Workers" refers to personnel who perform work within a factory.

[0730] A "procedure" refers to the sequence of operations or steps required to complete a particular task.

[0731] "Means for generating and uttering information" refers to technology for outputting the generated information as voice.

[0732] The system for carrying out the present invention includes a voice recognition means, a generative AI means, and a means for a worker to speak out a procedure that he or she has made a mistake in and to generate and speak information for correcting that procedure.

[0733] First, the speech recognition method requires technology that converts speech into text data. Specifically, speech recognition APIs such as Google Cloud Speech-to-Text and IBM Watson Speech to Text can be used. This makes it possible to convert the speech of workers into text data in real time.

[0734] Next, generative AI models such as OpenAI's GPT-3 are used as generative AI methods. Generative AI models generate appropriate responses and information based on input text data. For example, if a worker says, "Assemble the parts in the order B→A→C," the generative AI model generates information such as, "That's incorrect. The correct order is A→B→C."

[0735] Finally, a technology to convert the generated text data into voice is required as a means of generating and speaking information. Specifically, speech synthesis APIs such as Google Cloud Text-to-Speech and Amazon Polly can be used. This makes it possible to output the generated information as voice and convey it to workers.

[0736] As a concrete example, if a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the means for generating and uttering information converts the information into voice and conveys it to the worker.

[0737] Examples of prompts include:

[0738] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[0739] In this way, when a worker makes a mistake in a procedure, it is possible to immediately instruct the worker to follow the correct procedure, thereby improving the efficiency and accuracy of work in the factory.

[0740] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0741] Step 1:

[0742] The server receives the speech of the worker as voice data using a voice recognition means. The voice recognition means converts the voice data into text data. The input is the voice data of the worker, and the output is text data.

[0743] Step 2:

[0744] The server passes text data to the generative AI means as input. The generative AI means generates an appropriate response based on this text data. Specifically, if the worker's statement is incorrect, it generates a response including a correct procedure. The input is text data, and the output is the generated response text.

[0745] Step 3:

[0746] The server passes the generated response text as input to the speech synthesis means, which converts the text data into speech data, with the input being the generated response text and the output being the speech data.

[0747] Step 4:

[0748] The server transmits the voice data to the worker through a speaker, allowing the worker to confirm the correct procedure by voice. The input is the voice data, and the output is the voice that the worker hears.

[0749] Specifically, when a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the speech synthesis means converts the information into voice and conveys it to the worker through a speaker.

[0750] Examples of prompts include:

[0751] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[0752] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0753] "Example 1"

[0754] One embodiment of the present invention is an educational robot system that incorporates an emotion engine. In this system, a voice recognition means recognizes a learner's utterance, and a generative AI means judges whether the utterance is incorrect. If the utterance is incorrect, the generative AI means generates correct information and speaks that information. Furthermore, this system includes an emotion engine and can recognize the learner's emotions. For example, if the learner feels confused, the generative AI means generates information according to the learner's emotion, such as encouraging words like "It's okay, let's think about it together," and speaks that information.

[0755] "Example 2"

[0756] In another embodiment, the emotion engine adjusts the response of the educational robot system according to the emotion of the learner. For example, if the emotion engine senses that the learner is angry or frustrated, it conveys the information to the generative AI means. Based on the information, the generative AI means generates words of apology or re-explanation, such as "I'm sorry, I'll explain it again," and speaks the information. This allows an appropriate response to be provided according to the learner's emotion.

[0757] The process flow of each embodiment will be described below.

[0758] "Example 1"

[0759] Step 1: The learner says something.

[0760] Step 2: The speech recognition means recognizes the learner's utterance.

[0761] Step 3: Determine whether the utterance recognized by the generative AI means is incorrect.

[0762] Step 4: If incorrect, the generative AI means generates the correct information and speaks that information.

[0763] Step 5: The emotion engine recognizes the learner's emotions.

[0764] Step 6: The generative AI means generates information according to the emotion based on the information from the emotion engine, and speaks that information.

[0765] "Example 2"

[0766] Step 1: The learner says something.

[0767] Step 2: The speech recognition means recognizes the learner's utterance.

[0768] Step 3: The emotion engine recognizes the learner's emotions.

[0769] Step 4: Based on the emotion engine's recognized emotion, the emotion engine conveys information to the generative AI means. Step 5: Based on the information from the emotion engine, the generative AI means generates information corresponding to the emotion, and speaks that information.

[0770] Example 1

[0771] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0772] Conventional educational systems can accurately recognize what students say and point out their mistakes, but they cannot respond according to the student's emotions. This makes it difficult to provide appropriate support when a student feels confused or anxious. There is also a need to improve learning outcomes by providing feedback according to the student's emotions.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0774] In this invention, the server includes a speech recognition means, a generative AI means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and uttering information according to the emotions. This makes it possible to accurately recognize the learner's utterances, point out mistakes, and provide appropriate feedback according to the learner's emotions.

[0775] A "voice recognition means" is a device or software that captures a learner's speech as voice data and converts the voice data into text data.

[0776] "Generative AI means" refers to an artificial intelligence model that generates appropriate information based on input data, and in particular has the ability to determine whether what a learner says is correct or incorrect and generate correct information.

[0777] "A means for allowing a learner to speak an incorrect answer and for generating and speaking information to correct it" refers to a device or software that, when what a learner says is incorrect, points out the mistake and generates and speaks the correct information.

[0778] "Means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions" refers to a device or software that analyzes the learner's emotions from their facial expressions, tone of voice, etc., and generates and speaks appropriate feedback or words of encouragement corresponding to those emotions.

[0779] The educational robot system of the present invention includes a voice recognition means, a generative AI means, a means for generating and speaking information to correct an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions.

[0780] Hardware and Software Used

[0781] Hardware: Microphone, speaker, camera (for emotion recognition)

[0782] Software: voice recognition software (e.g. voice recognition APIs), generative AI models (e.g. AI models), emotion recognition engines (e.g. emotion recognition software)

[0783] Program processing flow

[0784] Speech Recognition

[0785] The device uses a microphone to capture what the learner says.

[0786] The server uses voice recognition software to convert the voice data into text data.

[0787] Example: If a learner says, "The Earth is flat," speech recognition software converts this utterance into text.

[0788] Evaluation of comments

[0789] The server uses a generative AI model to evaluate whether the text data is correct.

[0790] Example: determining that the statement "The Earth is flat" is false.

[0791] Producing the right information

[0792] The server uses a generative AI model to generate correct information.

[0793] Example: Producing the correct information, "That's wrong. The Earth is spherical."

[0794] emotion recognition

[0795] The device uses a camera to capture the learner's facial expressions.

[0796] The server uses an emotion recognition engine to analyze the learner's emotions.

[0797] Example: A camera captures a learner's confused expression, and an emotion recognition engine identifies the expression as "confused."

[0798] Generating responses based on emotions

[0799] The server uses a generative AI model to generate information based on the learner's emotions.

[0800] Example: A generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[0801] Information statement

[0802] The device uses a speaker to communicate the generated information to the learner.

[0803] Example: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's figure it out together."

[0804] Examples of prompt statements

[0805] Prompt: "If the learner says, 'The Earth is flat,' generate and speak the correct information. Also generate encouraging words if the learner is confused."

[0806] In this way, the educational robot system of the present invention can not only recognize what the learner says and provide correct information, but also respond according to the learner's emotions, making it possible to provide appropriate support to the learner even when he or she feels confused or anxious, thereby improving the effectiveness of learning.

[0807] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] Capturing audio input

[0810] The device uses a microphone to capture what the learner says.

[0811] Input: Learner's spoken utterance (e.g. "The Earth is flat")

[0812] Output: The captured audio data.

[0813] What happens: A learner says, "The Earth is flat," and their voice is captured by a microphone.

[0814] Step 2:

[0815] Converting audio data to text

[0816] The server uses voice recognition software to convert the captured voice data into text data.

[0817] Input: Captured audio data

[0818] Output: Text data (e.g. "The Earth is flat")

[0819] What it does: Speech recognition software analyzes the speech data and converts it into text: "The Earth is flat."

[0820] Step 3:

[0821] Evaluation of comments

[0822] The server uses a generative AI model to evaluate whether the text data is correct.

[0823] Input: Text data (e.g. "The Earth is flat")

[0824] Output: Evaluation result (e.g., error)

[0825] How it works: A generative AI model analyzes the statement "The Earth is flat" and determines that it is false.

[0826] Step 4:

[0827] Producing the right information

[0828] The server uses a generative AI model to generate correct information.

[0829] Input: Evaluation result (e.g. mistake)

[0830] Output: Correct information (e.g. "That's not true. The Earth is a sphere.")

[0831] Specific behavior: The generative AI model generates the correct information, "That's wrong. The Earth is spherical."

[0832] Step 5:

[0833] emotion recognition

[0834] The device uses a camera to capture the learner's facial expressions.

[0835] The server uses an emotion recognition engine to analyze the learner's emotions.

[0836] Input: Learner's facial expression data

[0837] Output: Sentiment analysis result (e.g. confused)

[0838] Specific action: The camera captures the learner's confused expression, and the emotion recognition engine recognizes the expression as "confused."

[0839] Step 6:

[0840] Generating responses based on emotions

[0841] The server uses a generative AI model to generate information based on the learner's emotions.

[0842] Input: Sentiment analysis result (e.g. confused)

[0843] Output: Information based on the emotion (e.g. "It's okay, let's think about it together")

[0844] Specific behavior: The generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[0845] Step 7:

[0846] Information statement

[0847] The device uses a speaker to communicate the generated information to the learner.

[0848] Input: Correct information and emotion-specific information (e.g., "That's wrong. The Earth is a sphere." "It's okay, let's figure it out together.")

[0849] Output: Audio feedback to the learner

[0850] Specific actions: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's think about it together."

[0851] (Application example 1)

[0852] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0853] There is a need to point out mistakes in factory worker remarks and procedures in real time and provide correct information. Another challenge is to improve work efficiency and safety by recognizing the emotions of workers and providing appropriate encouragement and advice.

[0854] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0855] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and uttering information for correcting a procedure that the worker has made a mistake in, an emotion recognition means, and a means for generating and uttering information according to the emotion of the worker. This makes it possible to point out the worker's comments and errors in the procedure in real time, provide correct information, and take appropriate action according to the emotion of the worker.

[0856] The "voice recognition means" is a device or system that recognizes the speech of the worker as voice and converts it into text data.

[0857] "Generative AI means" refers to artificial intelligence techniques that generate appropriate information based on input data.

[0858] "Means for a worker to state a procedure that he or she made a mistake and to generate and state information to correct it" refers to a device or system that evaluates the content of the worker's statement and generates and provides the correct procedure or information if a mistake is made.

[0859] An "emotion recognition means" is a device or system that analyzes and recognizes emotions from a worker's facial expressions, tone of voice, etc.

[0860] The "means for generating and uttering information corresponding to the emotions of the worker" is a device or system that generates and provides appropriate encouragement or advice based on the emotions recognized by the emotion recognition means.

[0861] As an embodiment of the present invention, a factory work support robot system will be described as an example. This system includes a voice recognition means, a generative AI means, an emotion recognition means, and software for integrating them.

[0862] Hardware used:

[0863] Voice recognition microphone

[0864] speaker

[0865] Camera (for emotion recognition)

[0866] Factory robot body

[0867] Software used:

[0868] Speech recognition software (Google Speech-to-Text API)

[0869] Generative AI model (OpenAI GPT-4)

[0870] Emotion recognition software (Microsoft Azure Emotion API)

[0871] System Operation:

[0872] 1. Speech recognition: When a worker speaks, the speech is picked up by the speech recognition microphone and converted into text data using the Google Speech-to-Text API. This text data is then sent to the server.

[0873] 2. Evaluating the utterance: The server uses a generative AI model (GPT-4) to evaluate whether the worker's utterance is correct. For example, it judges whether the utterance "This part should be installed here" is correct.

[0874] 3. Generation of correct information: If the statement is incorrect, the server uses GPT-4 to generate correct information and communicates it to the worker through the speaker. For example, it provides information such as "That part should not be installed here, but next to it."

[0875] 4. Emotion Recognition: Using a camera and the Microsoft Azure Emotion API, the system analyzes the worker's facial expressions and recognizes their emotions. For example, if the worker is confused, that information is sent to the server.

[0876] 5. Emotion-based response: Based on the result of emotion recognition, the server uses GPT-4 to generate appropriate encouragement or advice and convey it to the worker through the speaker. For example, encouraging words such as "It's okay, let's check it together" are provided.

[0877] Examples:

[0878] When a worker says, "This part should be installed here," the voice recognition microphone picks up the statement and converts it into text using the Google Speech-to-Text API. The server uses GPT-4 to evaluate the statement, and if it determines that the statement is incorrect, it generates correct information such as, "That part should be installed in the next position, not here," and conveys this to the worker through the speaker. In addition, the camera and Microsoft Azure Emotion API analyze the worker's facial expressions, and if it recognizes that the worker is confused, it provides encouraging words such as, "It's okay, let's check together."

[0879] Example prompt for a generative AI model:

[0880] The worker says, "This part goes here." Evaluate whether this statement is correct and provide the correct information if it is incorrect. Also, generate appropriate words of encouragement if the worker is confused.

[0881] In this way, the factory work support robot system can evaluate the worker's comments in real time, provide correct information, and take appropriate action according to the worker's emotions.

[0882] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0883] Step 1:

[0884] Speech Recognition

[0885] Input: What the worker says

[0886] Processing: A voice recognition microphone picks up what the worker is saying and converts the voice data into text using the Google Speech-to-Text API.

[0887] Output: Text data

[0888] Specific operation: When a worker says, "This part goes here," the voice recognition microphone picks up the speech, and the Google Speech-to-Text API converts the speech into text.

[0889] Step 2:

[0890] Evaluation of comments

[0891] Input: Text data obtained in step 1

[0892] Processing: The server uses a generative AI model (GPT-4) to evaluate whether the text data is correct.

[0893] Output: Evaluation result (correct / incorrect)

[0894] Specific operation: The server inputs text data such as "This part should be installed here" into GPT-4 and evaluates whether the statement is correct.

[0895] Step 3:

[0896] Producing the right information

[0897] Input: If the evaluation result obtained in step 2 is "wrong"

[0898] Processing: The server uses a generative AI model (GPT-4) to generate the correct information.

[0899] Output: Text data with correct information

[0900] Specific operation: The server makes GPT-4 generate the correct information, such as "That part should not be installed here, but in the adjacent position."

[0901] Step 4:

[0902] Statement of correct information

[0903] Input: Text data of the correct information obtained in step 3

[0904] Processing: The server communicates the correct information to the worker via voice through a speaker.

[0905] Output: Audio information

[0906] Specific operation: The server communicates to the worker through a speaker, "That part should be installed in the next position, not here."

[0907] Step 5:

[0908] emotion recognition

[0909] Input: Facial expression data of the worker

[0910] Processing: The camera captures the worker's facial expressions and analyzes their emotions using the Microsoft Azure Emotion API.

[0911] Output: Emotion data (e.g. confusion, relief, etc.)

[0912] Specific operation: The camera captures the worker's facial expression, which is then analyzed by the Microsoft Azure Emotion API and recognized as "confusion."

[0913] Step 6:

[0914] Generating information according to emotions

[0915] Input: Emotion data obtained in step 5

[0916] Processing: The server uses a generative AI model (GPT-4) to generate appropriate information based on the emotion.

[0917] Output: Text data of information according to emotions

[0918] Specific operation: The server asks GPT-4 to generate encouraging words such as, "It's okay, let's check it together."

[0919] Step 7:

[0920] Expressing information according to emotions

[0921] Input: Text data of emotion-related information obtained in step 6

[0922] Processing: The server communicates information corresponding to the emotion to the worker through a speaker.

[0923] Output: Audio information

[0924] Specific operation: The server communicates to the worker through a speaker, "It's OK, let's check it together."

[0925] Example 2

[0926] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0927] In conventional educational systems, it is difficult to immediately provide correct information when a learner gives a wrong answer, making it difficult for the learner to recognize their own mistake. In addition, it is not possible to provide an appropriate response according to the learner's emotions, which can reduce the learner's motivation to learn. To solve these problems, a system is needed that can accurately recognize the learner's utterances, generate information to correct the mistake, and provide a response according to the learner's emotions.

[0928] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0929] In this invention, the server includes a speech recognition means, a natural language processing means, a generative AI means, an emotion recognition means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for adjusting a response according to the learner's emotion. This makes it possible to accurately recognize the learner's utterance, provide information for immediately correcting the mistake, and further provide an appropriate response according to the learner's emotion.

[0930] "Speech recognition means" refers to a technology that acquires a learner's speech as voice data and converts it into text data.

[0931] "Natural language processing means" refers to technology that analyzes acquired text data and understands the content of statements.

[0932] "Generative AI means" is an artificial intelligence technology that generates appropriate information based on what the learner says.

[0933] "Emotion recognition means" is a technology that analyzes a learner's facial expressions and tone of voice to detect the learner's emotions.

[0934] "A means for generating and uttering information to correct an incorrect answer given by a learner" is a technology that detects an incorrect answer given by a learner and generates and utters correct information.

[0935] "Means of adjusting responses according to the learner's emotions" refers to techniques for generating and uttering appropriate responses based on the learner's emotions.

[0936] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0937] The present invention relates to an educational robot system for implementing a lesson in which students actively learn. A specific embodiment of this system will be described below.

[0938] System Configuration

[0939] This educational robot system includes the following main components:

[0940] 1. Speech recognition method: Acquire the learner's speech as voice data and convert it into text data. Specifically, use voice recognition software (e.g. Google Speech-to-Text).

[0941] 2. Natural language processing means: Analyze the acquired text data and understand the content of the speech. Specifically, natural language processing software (e.g. spaCy) is used.

[0942] 3. Generative AI means: Generate appropriate information based on what the learner says. Specifically, we use a generative AI model (e.g., OpenAI GPT-4).

[0943] 4. Emotion recognition: Analyze the learner's facial expressions and tone of voice to detect the learner's emotions. Specifically, use emotion recognition software (e.g. Microsoft Azure Emotion API).

[0944] 5. Information generation and speech means: When a learner gives an incorrect answer, generate and speak information to correct it. Specifically, use speech synthesis software (e.g. Amazon Polly).

[0945] 6. Emotion-based response adjustment: Generate and speak appropriate responses based on the learner's emotions.

[0946] System Operation

[0947] The device picks up what the learner says through a microphone and converts it into text data using voice recognition software. The server then receives the text data sent from the device and analyzes it using natural language processing software. Based on the analysis results, the server determines whether what the learner said is correct. For example, if the learner says, "The capital of Japan is Osaka," the server detects that this is incorrect.

[0948] The server uses the generative AI model to generate the correct information. Specifically, it generates the information, "That is incorrect. The capital of Japan is Tokyo." Next, the device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions. The detected emotion data is sent to the server.

[0949] The server uses an emotion engine to generate a response based on the learner's emotions. For example, if the learner expresses anger or dissatisfaction, the server uses a generative AI model to generate an apology or re-explanation such as "I'm sorry, I'll explain it again." Finally, the device converts the generated information into speech using speech synthesis software and speaks it to the learner through the speaker.

[0950] Examples

[0951] For example, if a learner says "The capital of Japan is Osaka," the system will operate as follows:

[0952] 1. The device picks up speech through the microphone and uses Google Speech-to-Text to convert it into text data such as "The capital of Japan is Osaka."

[0953] 2. The server receives the text data and analyzes it using spaCy.

[0954] 3. Based on the analysis results, the server detects that the statement "The capital of Japan is Osaka" is incorrect.

[0955] 4. The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[0956] 5. The device analyzes the learner's facial expressions and tone of voice using the Microsoft Azure Emotion API to detect when the learner is expressing dissatisfaction.

[0957] 6. The server uses an emotion engine to generate an apology such as "I'm sorry, I'll explain it again."

[0958] 7. The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[0959] Examples of prompt statements

[0960] "If a learner says, 'The capital of Japan is Osaka,' please generate the correct information."

[0961] "If the learner expresses anger or frustration, generate appropriate apologies or restatements."

[0962] The above is an embodiment of the invention.

[0963] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0964] Step 1:

[0965] The user makes a statement as a learner.

[0966] Input: Learner's spoken words

[0967] Output: Audio data

[0968] Specific action: The learner says, "The capital of Japan is Osaka."

[0969] Step 2:

[0970] The terminal captures the learner's voice data through a microphone and converts it into text data using voice recognition software.

[0971] Input: Audio data

[0972] Output: Text data

[0973] Specific operation: The device uses Google Speech-to-Text to generate text data such as "The capital of Japan is Osaka."

[0974] Step 3:

[0975] The server receives the text data sent from the terminal and analyzes it using natural language processing software.

[0976] Input: Text data

[0977] Output: Analysis results

[0978] Specific operation: The server uses spaCy to analyze the text data and understand the content of the statement, "The capital of Japan is Osaka."

[0979] Step 4:

[0980] The server uses the analysis results to determine whether the learner's statement is correct.

[0981] Input: Analysis results

[0982] Output: True or False

[0983] Specific operation: The server detects that the statement "The capital of Japan is Osaka" is false.

[0984] Step 5:

[0985] The server generates the correct information using a generative AI model.

[0986] Input: Correct / Incorrect

[0987] Output: Correct information

[0988] Specific operation: The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[0989] Step 6:

[0990] The device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions.

[0991] Input: Learner's facial expressions and tone of voice

[0992] Output: Emotion data

[0993] Specific behavior: The device uses the Microsoft Azure Emotion API to detect when a learner is expressing dissatisfaction.

[0994] Step 7:

[0995] The terminal transmits the detected emotion data to the server.

[0996] Input: Emotion data

[0997] Output: Emotion data (sent to server)

[0998] Specific operation: The terminal transmits emotion data to the server.

[0999] Step 8:

[1000] The server uses an emotion engine to generate a response according to the learner's emotion.

[1001] Input: Emotion data

[1002] Output: Appropriate response

[1003] What it does: The server uses a generative AI model to generate an apology saying, "I'm sorry, I'll explain it again."

[1004] Step 9:

[1005] The terminal uses speech synthesis software to convert the generated information into speech, which is then spoken to the learner through a speaker.

[1006] Input: appropriate response

[1007] Output: Audio data

[1008] Specific operation: The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[1009] (Application example 2)

[1010] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1011] Conventional educational robot systems have the function of providing correct information when a learner gives an incorrect answer, but they are unable to show appropriate responses according to the learner's emotions, making it difficult to improve the learner's motivation and level of understanding.In addition, when it comes to work guidance in factories, there is a lack of a function to teach the correct procedure when a worker performs an incorrect work procedure, making it difficult to improve work efficiency and quality.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1013] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and speaking information for correcting an incorrect answer given by a learner, an emotion recognition means, a means for adjusting a response according to the emotion of the learner, and a means for instructing a work procedure. This makes it possible to provide correct information when a learner gives a wrong answer, and to show an appropriate response according to the emotion of the learner. Also, in the case of work guidance in a factory, if a worker performs an incorrect work procedure, the correct procedure can be taught, which is expected to improve work efficiency and quality.

[1014] "Speech recognition means" refers to technology that analyzes speech and converts what is being said into text data.

[1015] "Generative AI means" refers to artificial intelligence technology that generates appropriate information based on input data.

[1016] "A means for allowing a learner to give an incorrect answer and for generating and giving information to correct it" refers to technology that has the function of detecting incorrect answers given by a learner and providing the correct information.

[1017] "Emotion recognition means" is a technology that analyzes and recognizes emotions from learners' facial expressions and voices.

[1018] A "means for tailoring responses according to learner emotions" is a technique for generating and providing appropriate responses based on recognized emotions.

[1019] The "means for instructing work procedures" refers to technology that has the function of teaching correct procedures when a worker performs an incorrect work procedure.

[1020] A system for implementing this invention includes a voice recognition means, a generative AI means, a means for generating and speaking information to correct an incorrect answer provided by a learner, an emotion recognition means, a means for adjusting a response in accordance with the learner's emotions, and a means for instructing the learner on work procedures.

[1021] Hardware and Software Used

[1022] Hardware: Task guidance robot, microphone, camera

[1023] Software: OpenAI API, Emotion Recognition API

[1024] System program processing

[1025] Voice Recognition Method

[1026] The server collects the learner's utterances through a microphone and converts them into text data using a voice recognition means. The voice recognition means accurately analyzes the contents of the learner's utterances and transmits them in text form to the server.

[1027] Generative AI means

[1028] The server uses the generative AI means to determine whether the learner's utterance is correct. If the utterance is incorrect, the generative AI means generates correct information and creates text data to utter that information.

[1029] emotion recognition means

[1030] The server collects the learner's facial expressions and voices through a camera and microphone, and analyzes the learner's emotions using an emotion recognition means. This emotion recognition means judges whether the learner is showing anger or dissatisfaction, and transmits the result to the server.

[1031] A means of adjusting responses according to learners' emotions

[1032] The server generates an appropriate response according to the learner's emotions based on the data from the emotion recognition means. For example, if the learner shows anger or dissatisfaction, the generative AI means generates an apology or re-explanation such as "I'm sorry, I'll explain it again."

[1033] A means of instructing work procedures

[1034] The server generates information to teach correct work procedures when a worker performs an incorrect work procedure. The generative AI means detects incorrect procedures and creates text data to explain the correct procedures.

[1035] Examples

[1036] For example, if a worker says, "Put this part here," but it should actually be installed in a different place, the server will tell them, "That's wrong. The correct procedure is to install this part here." Also, if a worker is stressed, the server will respond to the worker's emotions by saying, "I'm sorry, let me explain it again."

[1037] Examples of prompt statements

[1038] If a user says "install this part here", what is the correct procedure?

[1039] If the user is stressed, generate an appropriate response.

[1040] In this way, a system is realized that can provide correct information when a learner or worker utters incorrect information, and can also show an appropriate response according to their emotions.

[1041] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1042] Step 1:

[1043] The server collects the user's speech through a microphone. The input is the user's voice, and the output is voice data. This voice data is sent to a voice recognition means.

[1044] Step 2:

[1045] The server converts the voice data into text data using a voice recognition means. The input is voice data, and the output is text data. This text data is sent to the generative AI means.

[1046] Step 3:

[1047] The server uses a generative AI means to determine whether the text data is correct. The input is the text data, and the output is the result of the judgment. If the statement is incorrect, the generative AI means generates the correct information.

[1048] Step 4:

[1049] The server creates text data for uttering the correct information from the generative AI means. The input is the result of the judgment of correctness or incorrectness and the correct information, and the output is the text data for utterance. This text data is sent to the speech synthesis means.

[1050] Step 5:

[1051] The server collects the user's facial expressions and voice through a camera and microphone, and analyzes the user's emotions using an emotion recognition means. The input is the user's facial expression data and voice data, and the output is emotion data. This emotion data is sent to the generative AI means.

[1052] Step 6:

[1053] The server generates an appropriate response according to the user's emotion based on the emotion data from the emotion recognition means. The input is the emotion data, and the output is text data of the response according to the emotion. This text data is sent to the voice synthesis means.

[1054] Step 7:

[1055] The server generates information to teach the correct work procedure when the worker performs an incorrect work procedure. The input is text data of the incorrect work procedure, and the output is text data of the correct work procedure. This text data is sent to the voice synthesis means.

[1056] Step 8:

[1057] The server uses a speech synthesis means to convert the generated text data into speech and speaks it to the user. The input is the text data to be spoken, and the output is speech data. This speech data is provided to the user through a speaker.

[1058] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1059] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1060] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1061] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1062] [Third embodiment]

[1063] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1064] As shown in Fig. 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1065] .

[1066] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1067] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1068] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1069] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1070] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1071] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1072] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1073] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1074] In the headset type terminal 314, the reception and output process is performed by the processor 46. The storage 50 stores a reception and output program 60. The processor 46 reads the reception and output program 60 from the storage 50 and executes the reception and output process according to the reception and output program 60.

[1075] is executed on the RAM 48. The reception and output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception and output program 60 executed on the RAM 48.

[1076] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1077] "Example 1"

[1078] The educational robot system of the present invention comprises a voice recognition means and a generative AI means. The voice recognition means recognizes the learner's utterance and judges whether the utterance is incorrect. The generative AI means generates correct information for the learner's incorrect utterance and speaks that information. Specifically, if the learner says "The Earth is flat," the voice recognition means recognizes this utterance, and the generative AI means generates and speaks the information "That is incorrect. The Earth is spherical."

[1079] "Example 2"

[1080] The educational robot system of the present invention provides an effective means for implementing lesson development in which learners actively learn. Specifically, when a learner gives a wrong answer, the educational robot system generates and speaks information to correct it. This allows the learner to recognize his or her own mistake and obtain the correct information. For example, if a learner says, "The capital of Japan is Osaka," the educational robot system generates and speaks the information, "That is incorrect. The capital of Japan is Tokyo."

[1081] The process flow of each embodiment will be described below.

[1082] "Example 1"

[1083] Step 1: The learner makes an utterance, for example, "The Earth is flat."

[1084] Step 2: The speech recognition means recognizes the learner's utterance. In this example, it recognizes the utterance "The Earth is flat."

[1085] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information "That is incorrect. The Earth is spherical." Step 4: The generative AI means speaks out the generated information. In this example, it speaks out "That is incorrect. The Earth is spherical."

[1086] "Example 2"

[1087] Step 1: A learner makes an utterance. For example, the learner may say, "The capital of Japan is Osaka." Step 2: A speech recognition means recognizes the learner's utterance. In this example, the speech recognition means recognizes the utterance, "The capital of Japan is Osaka."

[1088] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information, "That is incorrect. The capital of Japan is Tokyo."

[1089] Step 4: The generative AI means speaks the generated information. In this example, it speaks, "That is incorrect. The capital of Japan is Tokyo."

[1090] Example 1

[1091] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1092] In conventional educational systems, it was difficult to determine in real time whether what a learner said was correct and to immediately provide the correct information. In addition, if a learner said incorrect information, there was a lack of means to provide appropriate feedback to correct the mistake. This resulted in problems such as learners not deepening their understanding and preventing effective learning.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1094] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, a means for a learner to speak an incorrect answer and to generate and speak information for correcting the answer, a means for converting speech to text, a means for analyzing the text and determining the error, a means for generating a prompt sentence for generating correct information, and a means for converting the generated correct information into speech. This makes it possible to recognize a learner's speech in real time, immediately determine an error, and provide correct information.

[1095] A "voice recognition means" is a device or software for converting voice data into text data.

[1096] A "generative artificial intelligence means" is an artificial intelligence technique for generating new information based on input data.

[1097] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information in response to it, and provides feedback in the form of voice.

[1098] A "speech-to-text conversion means" is a technology for analyzing speech data and converting it into corresponding text data.

[1099] A "means for analyzing text and determining errors" is an algorithm or set of rules for analyzing text data and determining whether its content is correct.

[1100] "Means for generating a prompt sentence for generating correct information" refers to a technology for creating an instruction sentence for causing a generative artificial intelligence to generate correct information.

[1101] The "means for converting the generated correct information into voice" refers to a device or software for converting text data into voice data.

[1102] The present invention provides a technology for an educational robot system that recognizes a learner's utterances in real time, immediately detects mistakes, and provides correct information. A specific embodiment of this system will be described below.

[1103] First, the user speaks to the educational robot. For example, they say, "The Earth is flat." The device captures this speech with a microphone and saves it as voice data.

[1104] Next, the terminal transmits the acquired voice data to a voice recognition means. A general voice recognition API can be used as this voice recognition means. For example, Google Cloud Speech-to-Text API is used. This API converts the voice data into text data. The converted text data becomes "The Earth is flat."

[1105] The server receives the text data sent from the terminal and analyzes the text using a predefined rule set that contains a list of true and false statements. The server determines that the statement "The Earth is flat" is false.

[1106] Next, the server uses a generative artificial intelligence means to generate correct information. A general generative AI model can be used as this generative artificial intelligence means. For example, OpenAI's GPT-4 model is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical."

[1107] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A general speech synthesis API can be used as the speech synthesis means. For example, the Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical."

[1108] In this way, the educational robot system can recognize what the learner says in real time, immediately identify mistakes, and provide correct information, which will help the learner deepen their understanding and learn more effectively.

[1109] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1110] Step 1:

[1111] The user speaks to the educational robot. For example, "The Earth is flat." The device captures this speech with a microphone and obtains it as voice data. The input is the user's voice, and the output is the captured voice data.

[1112] Step 2:

[1113] The terminal sends the acquired voice data to the voice recognition means. A general voice recognition API is used as the voice recognition means. For example, Google Cloud Speech-to-Text API is used. This API converts the voice data into text data. The input is the voice data, and the output is the text data "The Earth is flat."

[1114] Step 3:

[1115] The server receives the text data sent from the terminal and analyzes the text. The analysis uses a predefined rule set, which contains a list of true and false statements. The server determines that the statement "The Earth is flat" is false. The input is the text data "The Earth is flat" and the output is the judgment result that it is "false."

[1116] Step 4:

[1117] The server generates correct information using a generative artificial intelligence means. A general generative AI model is used as this generative artificial intelligence means. For example, OpenAI's GPT-4 model is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical." The input is the prompt sentence, and the output is the correct information, "That is incorrect. The Earth is spherical."

[1118] Step 5:

[1119] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A common speech synthesis API is used as this speech synthesis means. For example, Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical." The input is the text data of the correct information, and the output is speech data.

[1120] (Application example 1)

[1121] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1122] When workers perform incorrect work procedures in a factory, it can lead to reduced production efficiency and safety issues. In particular, mistakes are likely to occur when workers are new or when the work procedures are complex. There is a demand for a system that can detect such mistakes in real time and provide the correct work procedures.

[1123] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a means for a worker to speak an incorrect work procedure and to generate and speak information for correcting it, a means for recognizing the worker's statement and judging whether the work procedure is correct, and a means for generating and speaking a correct work procedure. This makes it possible, when a worker speaks an incorrect work procedure, to point out the error in real time and provide the correct work procedure.

[1124] The "voice recognition means" is a device or system that acquires the speech of a worker as voice data and converts the voice data into text data.

[1125] A "generative AI means" is a device or system that uses an artificial intelligence model to generate appropriate information based on input data.

[1126] "Workers" refers to personnel who perform work within a factory.

[1127] A "work procedure" refers to the sequence of steps or methods required to complete a particular task.

[1128] "Utterance" refers to words or sentences spoken by a worker.

[1129] "Proper work procedures" refer to the precise steps or methods needed to complete a task safely and efficiently.

[1130] An "error" refers to a situation in which what a worker says does not match the correct work procedure.

[1131] "Real-time" refers to responding immediately to what workers say and providing information.

[1132] A system for implementing this invention includes a voice recognition means, a generative AI means, a means for recognizing utterances by a worker and determining whether or not the work procedure is correct, and a means for generating and uttering the correct work procedure.

[1133] Hardware and Software Used

[1134] Hardware:

[1135] Voice recognition microphone (e.g. high performance microphone)

[1136] Robot body (e.g. multi-function robot)

[1137] Speakers (e.g. high quality speakers)

[1138] software:

[1139] Speech recognition software (e.g. voice recognition engine)

[1140] Generative AI models (e.g. advanced generative AI models)

[1141] Robot control software (e.g. robot operating system)

[1142] Processing flow

[1143] 1. Voice Recognition:

[1144] A worker says, "This part goes here."

[1145] The voice recognition microphone captures what is said, and the voice recognition engine converts it into text data.

[1146] 2. Judging statements:

[1147] The converted text data is analyzed to determine whether the work procedures are correct.

[1148] For example, if a statement such as "This part should be installed here" is incorrect, a prompt is sent to the generative AI model.

[1149] 3. Producing the right information:

[1150] The generative AI model generates the correct information: "That's wrong. This part needs to be installed in a different location."

[1151] 4. Information Statement:

[1152] The correct information generated is communicated to the worker through the robot's built-in speaker.

[1153] Examples

[1154] If a worker says, "This part should be installed here," the robot will point out, "That's wrong. This part needs to be installed somewhere else."

[1155] Examples of prompt statements

[1156] User says: "This part goes here"

[1157] Generate the correct information.

[1158] With this system, when a worker makes an incorrect statement about a work procedure, the system can point out the mistake in real time and provide the correct work procedure, thereby improving work efficiency and safety in the factory.

[1159] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1160] Step 1:

[1161] Audio capture

[1162] Input: A worker says, "This part goes here."

[1163] Specific operation: A voice recognition microphone captures what the worker says.

[1164] Output: The captured audio data.

[1165] Step 2:

[1166] Speech to text conversion

[1167] Input: The captured audio data.

[1168] Specific operation: The voice recognition engine converts the voice data into text data.

[1169] Output: The converted text data (e.g. "Install this part here").

[1170] Step 3:

[1171] Speech analysis

[1172] Input: The converted text data.

[1173] Specific operation: The server analyzes the text data and determines whether the work procedure is correct.

[1174] Output: The result of a judgment as to whether the statement is correct or not (e.g., false).

[1175] Step 4:

[1176] Prompt Generation

[1177] Input: The judgement result that the statement is false.

[1178] Specific operation: The server sends a prompt to the generative AI model.

[1179] Output: A prompt to the generative AI model (e.g., "User says: 'This part should be installed here' Please generate the correct information.").

[1180] Step 5:

[1181] Producing the right information

[1182] Input: A prompt for the generative AI model.

[1183] Specific operation: The generative AI model generates the correct work steps.

[1184] Output: Generated correct work instructions (e.g. "That's wrong. This part needs to be installed somewhere else.").

[1185] Step 6:

[1186] Statement of correct information

[1187] Input: The generated correct work instructions.

[1188] Specific operation: The correct information generated is communicated to the worker through the speaker on the robot's body.

[1189] Output: The worker can hear the correct work procedure.

[1190] Example 2

[1191] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1192] In conventional educational systems, when a learner gives a wrong answer, there is a lack of a way to immediately provide correct information, so the learner may continue to hold on to the wrong knowledge. In addition, it is difficult to provide an environment in which the learner can actively learn.

[1193] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1194] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and speaking information for correcting an incorrect answer given by a learner, a means for converting voice data into text data, a means for analyzing the text data and judging the correctness of the statement, and a means for converting correct information into voice data and feeding it back to the learner. This makes it possible to provide correct information immediately when a learner gives a wrong answer, and to provide an environment in which the learner can actively learn.

[1195] A "voice recognition means" is a device or software for collecting voice data and converting it into text data.

[1196] A "generative AI means" is an artificial intelligence model that generates appropriate responses or information based on input data.

[1197] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information, and conveys it to the learner.

[1198] The "means for converting voice data into text data" refers to a device or software that converts voice data into text information using voice recognition technology.

[1199] The "means for analyzing text data and judging whether the content of a statement is correct" refers to a device or software for analyzing text data and judging whether the content is correct.

[1200] The "means for converting correct information into voice data and feeding it back to the learner" refers to a device or software for converting the generated correct information into voice data and conveying it to the learner.

[1201] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1202] This invention relates to an educational robot system, and provides an effective means for realizing a lesson in which students actively learn. Specifically, when a student gives a wrong answer, the educational robot system generates and speaks information to correct the answer, allowing the student to recognize his / her own mistake and obtain the correct information.

[1203] Hardware and Software Used

[1204] The system uses the following hardware and software:

[1205] Speech recognition means: A device or software for collecting voice data and converting it into text data. Specifically, we use voice recognition software (e.g., Google Cloud Speech-to-Text).

[1206] Generative AI means: An artificial intelligence model that generates appropriate responses or information based on input data. Specifically, we use a generative AI model (e.g., OpenAI's GPT-4).

[1207] Means for converting voice data into text data: A device or software that converts voice data into text information using voice recognition technology. Specifically, we use Google Cloud Speech-to-Text.

[1208] A means of analyzing text data and judging whether a statement is correct: A device or software that analyzes text data and judges whether the content is correct. Specifically, a generative AI model (e.g., OpenAI's GPT-4) is used.

[1209] Means of converting correct information into voice data and feeding it back to the learner: A device or software that converts the generated correct information into voice data and conveys it to the learner. Specifically, speech synthesis software (e.g. Amazon Polly) is used.

[1210] Examples

[1211] If a user says "The capital of Japan is Osaka", the following processing occurs:

[1212] 1. Voice recognition: The device collects the user's speech through a microphone. For example, the user says, "The capital of Japan is Osaka."

[1213] 2. Text conversion: The device converts the collected voice data into text data using the Google Cloud Speech-to-Text API. The converted text data becomes "The capital of Japan is Osaka."

[1214] 3. Correct / incorrect judgment: The server receives the text data sent from the terminal. Then, the server inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[1215] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[1216] 4. Information generation: The generative AI model generates the correct information: "That is incorrect. The capital of Japan is Tokyo." The server receives this generated information and sends it to the terminal.

[1217] 5. Voice feedback: The device converts the correct information received from the server into voice data using voice synthesis software (e.g. Amazon Polly). The device then communicates the converted voice data to the user through the speaker. Specifically, the device provides voice feedback saying, "That's incorrect. The capital of Japan is Tokyo."

[1218] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[1219] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1220] Step 1:

[1221] The user speaks.

[1222] The user says, "The capital of Japan is Osaka." This becomes the input to the system.

[1223] Step 2:

[1224] The terminal collects the user's utterances as voice data.

[1225] The device captures the user's speech as voice data through a microphone, and this voice data becomes the input for the next processing step.

[1226] Step 3:

[1227] The terminal converts the voice data into text data.

[1228] The device uses the Google Cloud Speech-to-Text API to convert the collected voice data into text data. For example, the generated text data is "The capital of Japan is Osaka." This text data becomes the input for the next processing step.

[1229] Step 4:

[1230] The server receives the text data and inputs a prompt sentence into the generative AI model.

[1231] The server receives the text data sent from the device and inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[1232] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[1233] This prompt sentence becomes the input to the generative AI model.

[1234] Step 5:

[1235] The generative AI model generates the correct information based on the prompt sentence.

[1236] The generative AI model analyzes the input prompt sentence and generates the correct information, "That is incorrect. The capital of Japan is Tokyo." This generated information becomes the input for the next processing step.

[1237] Step 6:

[1238] The server transmits the generated information to the terminal.

[1239] The server sends the correct information received from the generative AI model to the terminal. For example, text data such as "That is incorrect. The capital of Japan is Tokyo" is sent to the terminal.

[1240] Step 7:

[1241] The terminal converts the correct information into voice data.

[1242] The device uses Amazon Polly to convert the received text data into speech data. For example, speech data such as "That's wrong. The capital of Japan is Tokyo" is generated.

[1243] Step 8:

[1244] The terminal feeds back the voice data to the user.

[1245] The terminal transmits the generated voice data to the user through the speaker, and the user receives voice feedback saying, "That is incorrect. The capital of Japan is Tokyo."

[1246] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[1247] (Application example 2)

[1248] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1249] There is a demand for a system that can immediately instruct workers in the correct procedure when they make a mistake in a factory. With conventional methods, when a worker makes a mistake, it takes time for the worker to realize the mistake, resulting in a decrease in production efficiency. There was also the issue that education and training to correct mistakes takes a lot of time and costs money.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1251] In this invention, the server includes a voice recognition means, a generative AI means, and a means for generating and issuing information for correcting an incorrect procedure when the worker issues the incorrect procedure. This makes it possible to immediately instruct the correct procedure when the worker issues the incorrect procedure.

[1252] A "voice recognition means" is a technology that receives voice as input and converts the voice into text data.

[1253] "Generative AI means" refers to artificial intelligence technology that generates appropriate responses or information based on input data.

[1254] "Workers" refers to personnel who perform work within a factory.

[1255] A "procedure" refers to the sequence of operations or steps required to complete a particular task.

[1256] "Means for generating and uttering information" refers to technology for outputting the generated information as voice.

[1257] The system for carrying out the present invention includes a voice recognition means, a generative AI means, and a means for a worker to speak out a procedure that he or she has made a mistake in and to generate and speak information for correcting that procedure.

[1258] First, the speech recognition method requires technology that converts speech into text data. Specifically, speech recognition APIs such as Google Cloud Speech-to-Text and IBM Watson Speech to Text can be used. This makes it possible to convert the speech of workers into text data in real time.

[1259] Next, generative AI models such as OpenAI's GPT-3 are used as generative AI methods. Generative AI models generate appropriate responses and information based on input text data. For example, if a worker says, "Assemble the parts in the order B→A→C," the generative AI model generates information such as, "That's incorrect. The correct order is A→B→C."

[1260] Finally, a technology to convert the generated text data into voice is required as a means of generating and speaking information. Specifically, speech synthesis APIs such as Google Cloud Text-to-Speech and Amazon Polly can be used. This makes it possible to output the generated information as voice and convey it to workers.

[1261] As a concrete example, if a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the means for generating and uttering information converts the information into voice and conveys it to the worker.

[1262] Examples of prompts include:

[1263] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[1264] In this way, when a worker makes a mistake in a procedure, it is possible to immediately instruct the worker to follow the correct procedure, thereby improving the efficiency and accuracy of work in the factory.

[1265] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1266] Step 1:

[1267] The server receives the speech of the worker as voice data using a voice recognition means. The voice recognition means converts the voice data into text data. The input is the voice data of the worker, and the output is text data.

[1268] Step 2:

[1269] The server passes text data to the generative AI means as input. The generative AI means generates an appropriate response based on this text data. Specifically, if the worker's statement is incorrect, it generates a response including a correct procedure. The input is text data, and the output is the generated response text.

[1270] Step 3:

[1271] The server passes the generated response text as input to the speech synthesis means, which converts the text data into speech data, with the input being the generated response text and the output being the speech data.

[1272] Step 4:

[1273] The server transmits the voice data to the worker through a speaker, allowing the worker to confirm the correct procedure by voice. The input is the voice data, and the output is the voice that the worker hears.

[1274] Specifically, when a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the speech synthesis means converts the information into voice and conveys it to the worker through a speaker.

[1275] Examples of prompts include:

[1276] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[1277] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1278] "Example 1"

[1279] One embodiment of the present invention is an educational robot system that incorporates an emotion engine. In this system, a voice recognition means recognizes a learner's utterance, and a generative AI means judges whether the utterance is incorrect. If the utterance is incorrect, the generative AI means generates correct information and speaks that information. Furthermore, this system includes an emotion engine and can recognize the learner's emotions. For example, if the learner feels confused, the generative AI means generates information according to the learner's emotion, such as encouraging words like "It's okay, let's think about it together," and speaks that information.

[1280] "Example 2"

[1281] In another embodiment, the emotion engine adjusts the response of the educational robot system according to the emotion of the learner. For example, if the emotion engine senses that the learner is angry or frustrated, it conveys the information to the generative AI means. Based on the information, the generative AI means generates words of apology or re-explanation, such as "I'm sorry, I'll explain it again," and speaks the information. This allows an appropriate response to be provided according to the learner's emotion.

[1282] The process flow of each embodiment will be described below.

[1283] "Example 1"

[1284] Step 1: The learner says something.

[1285] Step 2: The speech recognition means recognizes the learner's utterance.

[1286] Step 3: Determine whether the utterance recognized by the generative AI means is incorrect.

[1287] Step 4: If incorrect, the generative AI means generates the correct information and speaks that information.

[1288] Step 5: The emotion engine recognizes the learner's emotions.

[1289] Step 6: The generative AI means generates information according to the emotion based on the information from the emotion engine, and speaks that information.

[1290] "Example 2"

[1291] Step 1: The learner says something.

[1292] Step 2: The speech recognition means recognizes the learner's utterance.

[1293] Step 3: The emotion engine recognizes the learner's emotions.

[1294] Step 4: Based on the emotion engine's recognized emotion, the emotion engine conveys information to the generative AI means. Step 5: Based on the information from the emotion engine, the generative AI means generates information corresponding to the emotion, and speaks that information.

[1295] Example 1

[1296] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1297] Conventional educational systems can accurately recognize what students say and point out their mistakes, but they cannot respond according to the student's emotions. This makes it difficult to provide appropriate support when a student feels confused or anxious. There is also a need to improve learning outcomes by providing feedback according to the student's emotions.

[1298] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1299] In this invention, the server includes a speech recognition means, a generative AI means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and uttering information according to the emotions. This makes it possible to accurately recognize the learner's utterances, point out mistakes, and provide appropriate feedback according to the learner's emotions.

[1300] A "voice recognition means" is a device or software that captures a learner's speech as voice data and converts the voice data into text data.

[1301] "Generative AI means" refers to an artificial intelligence model that generates appropriate information based on input data, and in particular has the ability to determine whether what a learner says is correct or incorrect and generate correct information.

[1302] "A means for allowing a learner to speak an incorrect answer and for generating and speaking information to correct it" refers to a device or software that, when what a learner says is incorrect, points out the mistake and generates and speaks the correct information.

[1303] "Means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions" refers to a device or software that analyzes the learner's emotions from their facial expressions, tone of voice, etc., and generates and speaks appropriate feedback or words of encouragement corresponding to those emotions.

[1304] The educational robot system of the present invention includes a voice recognition means, a generative AI means, a means for generating and speaking information to correct an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions.

[1305] Hardware and Software Used

[1306] Hardware: Microphone, speaker, camera (for emotion recognition)

[1307] Software: voice recognition software (e.g. voice recognition APIs), generative AI models (e.g. AI models), emotion recognition engines (e.g. emotion recognition software)

[1308] Program processing flow

[1309] Speech Recognition

[1310] The device uses a microphone to capture what the learner says.

[1311] The server uses voice recognition software to convert the voice data into text data.

[1312] Example: If a learner says, "The Earth is flat," speech recognition software converts this utterance into text.

[1313] Evaluation of comments

[1314] The server uses a generative AI model to evaluate whether the text data is correct.

[1315] Example: determining that the statement "The Earth is flat" is false.

[1316] Producing the right information

[1317] The server uses a generative AI model to generate correct information.

[1318] Example: Producing the correct information, "That's wrong. The Earth is spherical."

[1319] emotion recognition

[1320] The device uses a camera to capture the learner's facial expressions.

[1321] The server uses an emotion recognition engine to analyze the learner's emotions.

[1322] Example: A camera captures a learner's confused expression, and an emotion recognition engine identifies the expression as "confused."

[1323] Generating responses based on emotions

[1324] The server uses a generative AI model to generate information based on the learner's emotions.

[1325] Example: A generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[1326] Information statement

[1327] The device uses a speaker to communicate the generated information to the learner.

[1328] Example: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's figure it out together."

[1329] Examples of prompt statements

[1330] Prompt: "If the learner says, 'The Earth is flat,' generate and speak the correct information. Also generate encouraging words if the learner is confused."

[1331] In this way, the educational robot system of the present invention can not only recognize what the learner says and provide correct information, but also respond according to the learner's emotions, making it possible to provide appropriate support to the learner even when he or she feels confused or anxious, thereby improving the effectiveness of learning.

[1332] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1333] Step 1:

[1334] Capturing audio input

[1335] The device uses a microphone to capture what the learner says.

[1336] Input: Learner's spoken utterance (e.g. "The Earth is flat")

[1337] Output: The captured audio data.

[1338] What happens: A learner says, "The Earth is flat," and their voice is captured by a microphone.

[1339] Step 2:

[1340] Converting audio data to text

[1341] The server uses voice recognition software to convert the captured voice data into text data.

[1342] Input: Captured audio data

[1343] Output: Text data (e.g. "The Earth is flat")

[1344] What it does: Speech recognition software analyzes the speech data and converts it into text: "The Earth is flat."

[1345] Step 3:

[1346] Evaluation of comments

[1347] The server uses a generative AI model to evaluate whether the text data is correct.

[1348] Input: Text data (e.g. "The Earth is flat")

[1349] Output: Evaluation result (e.g., error)

[1350] How it works: A generative AI model analyzes the statement "The Earth is flat" and determines that it is false.

[1351] Step 4:

[1352] Producing the right information

[1353] The server uses a generative AI model to generate correct information.

[1354] Input: Evaluation result (e.g. mistake)

[1355] Output: Correct information (e.g. "That's wrong. The Earth is a sphere.")

[1356] Specific behavior: The generative AI model generates the correct information, "That's wrong. The Earth is spherical."

[1357] Step 5:

[1358] emotion recognition

[1359] The device uses a camera to capture the learner's facial expressions.

[1360] The server uses an emotion recognition engine to analyze the learner's emotions.

[1361] Input: Learner's facial expression data

[1362] Output: Sentiment analysis result (e.g. confused)

[1363] Specific action: The camera captures the learner's confused expression, and the emotion recognition engine recognizes the expression as "confused."

[1364] Step 6:

[1365] Generating responses based on emotions

[1366] The server uses a generative AI model to generate information based on the learner's emotions.

[1367] Input: Sentiment analysis result (e.g. confused)

[1368] Output: Information based on the emotion (e.g. "It's okay, let's think about it together")

[1369] Specific behavior: The generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[1370] Step 7:

[1371] Information statement

[1372] The device uses a speaker to communicate the generated information to the learner.

[1373] Input: Correct information and emotion-specific information (e.g., "That's wrong. The Earth is a sphere." "It's okay, let's figure it out together.")

[1374] Output: Audio feedback to the learner

[1375] Specific actions: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's think about it together."

[1376] (Application example 1)

[1377] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1378] There is a need to point out mistakes in factory worker remarks and procedures in real time and provide correct information. Another challenge is to improve work efficiency and safety by recognizing the emotions of workers and providing appropriate encouragement and advice.

[1379] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1380] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and uttering information for correcting a procedure that the worker has made a mistake in, an emotion recognition means, and a means for generating and uttering information according to the emotion of the worker. This makes it possible to point out the worker's comments and errors in the procedure in real time, provide correct information, and take appropriate action according to the emotion of the worker.

[1381] The "voice recognition means" is a device or system that recognizes the speech of the worker as voice and converts it into text data.

[1382] "Generative AI means" refers to artificial intelligence techniques that generate appropriate information based on input data.

[1383] "Means for a worker to state a procedure that he or she made a mistake and to generate and state information to correct it" refers to a device or system that evaluates the content of the worker's statement and generates and provides the correct procedure or information if a mistake is made.

[1384] An "emotion recognition means" is a device or system that analyzes and recognizes emotions from a worker's facial expressions, tone of voice, etc.

[1385] The "means for generating and uttering information corresponding to the emotions of the worker" is a device or system that generates and provides appropriate encouragement or advice based on the emotions recognized by the emotion recognition means.

[1386] As an embodiment of the present invention, a factory work support robot system will be described as an example. This system includes a voice recognition means, a generative AI means, an emotion recognition means, and software for integrating them.

[1387] Hardware used:

[1388] Voice recognition microphone

[1389] speaker

[1390] Camera (for emotion recognition)

[1391] Factory robot body

[1392] Software used:

[1393] Speech recognition software (Google Speech-to-Text API)

[1394] Generative AI model (OpenAI GPT-4)

[1395] Emotion recognition software (Microsoft Azure Emotion API)

[1396] System Operation:

[1397] 1. Speech recognition: When a worker speaks, the speech is picked up by the speech recognition microphone and converted into text data using the Google Speech-to-Text API. This text data is then sent to the server.

[1398] 2. Evaluating the utterance: The server uses a generative AI model (GPT-4) to evaluate whether the worker's utterance is correct. For example, it judges whether the utterance "This part should be installed here" is correct.

[1399] 3. Generation of correct information: If the statement is incorrect, the server uses GPT-4 to generate correct information and communicates it to the worker through the speaker. For example, it provides information such as "That part should not be installed here, but next to it."

[1400] 4. Emotion Recognition: Using a camera and the Microsoft Azure Emotion API, the system analyzes the worker's facial expressions and recognizes their emotions. For example, if the worker is confused, that information is sent to the server.

[1401] 5. Emotion-based response: Based on the result of emotion recognition, the server uses GPT-4 to generate appropriate encouragement or advice and convey it to the worker through the speaker. For example, encouraging words such as "It's okay, let's check it together" are provided.

[1402] Examples:

[1403] When a worker says, "This part should be installed here," the voice recognition microphone picks up the statement and converts it into text using the Google Speech-to-Text API. The server uses GPT-4 to evaluate the statement, and if it determines that the statement is incorrect, it generates correct information such as, "That part should be installed in the next position, not here," and conveys this to the worker through the speaker. In addition, the camera and Microsoft Azure Emotion API analyze the worker's facial expressions, and if it recognizes that the worker is confused, it provides encouraging words such as, "It's okay, let's check together."

[1404] Example prompt for a generative AI model:

[1405] The worker says, "This part goes here." Evaluate whether this statement is correct and provide the correct information if it is incorrect. Also, generate appropriate words of encouragement if the worker is confused.

[1406] In this way, the factory work support robot system can evaluate the worker's comments in real time, provide correct information, and take appropriate action according to the worker's emotions.

[1407] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1408] Step 1:

[1409] Speech Recognition

[1410] Input: What the worker says

[1411] Processing: A voice recognition microphone picks up what the worker is saying and converts the voice data into text using the Google Speech-to-Text API.

[1412] Output: Text data

[1413] Specific operation: When a worker says, "This part goes here," the voice recognition microphone picks up the speech, and the Google Speech-to-Text API converts the speech into text.

[1414] Step 2:

[1415] Evaluation of comments

[1416] Input: Text data obtained in step 1

[1417] Processing: The server uses a generative AI model (GPT-4) to evaluate whether the text data is correct.

[1418] Output: Evaluation result (correct / incorrect)

[1419] Specific operation: The server inputs text data such as "This part should be installed here" into GPT-4 and evaluates whether the statement is correct.

[1420] Step 3:

[1421] Producing the right information

[1422] Input: If the evaluation result obtained in step 2 is "wrong"

[1423] Processing: The server uses a generative AI model (GPT-4) to generate the correct information.

[1424] Output: Text data with correct information

[1425] Specific operation: The server makes GPT-4 generate the correct information, such as "That part should not be installed here, but in the adjacent position."

[1426] Step 4:

[1427] Statement of correct information

[1428] Input: Text data of the correct information obtained in step 3

[1429] Processing: The server communicates the correct information to the worker via voice through a speaker.

[1430] Output: Audio information

[1431] Specific operation: The server communicates to the worker through a speaker, "That part should be installed in the next position, not here."

[1432] Step 5:

[1433] emotion recognition

[1434] Input: Facial expression data of the worker

[1435] Processing: The camera captures the worker's facial expressions and analyzes their emotions using the Microsoft Azure Emotion API.

[1436] Output: Emotion data (e.g. confusion, relief, etc.)

[1437] Specific operation: The camera captures the worker's facial expression, which is then analyzed by the Microsoft Azure Emotion API and recognized as "confusion."

[1438] Step 6:

[1439] Generating information according to emotions

[1440] Input: Emotion data obtained in step 5

[1441] Processing: The server uses a generative AI model (GPT-4) to generate appropriate information based on the emotion.

[1442] Output: Text data of information according to emotions

[1443] Specific operation: The server asks GPT-4 to generate encouraging words such as, "It's okay, let's check it together."

[1444] Step 7:

[1445] Expressing information according to emotions

[1446] Input: Text data of emotion-related information obtained in step 6

[1447] Processing: The server communicates information corresponding to the emotion to the worker through a speaker.

[1448] Output: Audio information

[1449] Specific operation: The server communicates to the worker through a speaker, "It's OK, let's check it together."

[1450] Example 2

[1451] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1452] In conventional educational systems, it is difficult to immediately provide correct information when a learner gives a wrong answer, making it difficult for the learner to recognize their own mistake. In addition, it is not possible to provide an appropriate response according to the learner's emotions, which can reduce the learner's motivation to learn. To solve these problems, a system is needed that can accurately recognize the learner's utterances, generate information to correct the mistake, and provide a response according to the learner's emotions.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1454] In this invention, the server includes a speech recognition means, a natural language processing means, a generative AI means, an emotion recognition means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for adjusting a response according to the learner's emotion. This makes it possible to accurately recognize the learner's utterance, provide information for immediately correcting the mistake, and further provide an appropriate response according to the learner's emotion.

[1455] "Speech recognition means" refers to a technology that acquires a learner's speech as voice data and converts it into text data.

[1456] "Natural language processing means" refers to technology that analyzes acquired text data and understands the content of statements.

[1457] "Generative AI means" is an artificial intelligence technology that generates appropriate information based on what the learner says.

[1458] "Emotion recognition means" is a technology that analyzes a learner's facial expressions and tone of voice to detect the learner's emotions.

[1459] "A means for generating and uttering information to correct an incorrect answer provided by a learner" is a technology that detects an incorrect answer provided by a learner and generates and utters correct information.

[1460] "Means of adjusting responses according to the learner's emotions" refers to techniques for generating and uttering appropriate responses based on the learner's emotions.

[1461] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1462] The present invention relates to an educational robot system for implementing a lesson in which students actively learn. A specific embodiment of this system will be described below.

[1463] System Configuration

[1464] This educational robot system includes the following main components:

[1465] 1. Speech recognition method: Acquire the learner's speech as voice data and convert it into text data. Specifically, use voice recognition software (e.g. Google Speech-to-Text).

[1466] 2. Natural language processing means: Analyze the acquired text data and understand the content of the speech. Specifically, natural language processing software (e.g. spaCy) is used.

[1467] 3. Generative AI means: Generate appropriate information based on what the learner says. Specifically, we use a generative AI model (e.g., OpenAI GPT-4).

[1468] 4. Emotion recognition: Analyze the learner's facial expressions and tone of voice to detect the learner's emotions. Specifically, use emotion recognition software (e.g. Microsoft Azure Emotion API).

[1469] 5. Information generation and speech means: When a learner gives an incorrect answer, generate and speak information to correct it. Specifically, use speech synthesis software (e.g. Amazon Polly).

[1470] 6. Emotion-based response adjustment: Generate and speak appropriate responses based on the learner's emotions.

[1471] System Operation

[1472] The device picks up what the learner says through a microphone and converts it into text data using voice recognition software. The server then receives the text data sent from the device and analyzes it using natural language processing software. Based on the analysis results, the server determines whether what the learner said is correct. For example, if the learner says, "The capital of Japan is Osaka," the server detects that this is incorrect.

[1473] The server uses the generative AI model to generate the correct information. Specifically, it generates the information, "That is incorrect. The capital of Japan is Tokyo." Next, the device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions. The detected emotion data is sent to the server.

[1474] The server uses an emotion engine to generate a response based on the learner's emotions. For example, if the learner expresses anger or dissatisfaction, the server uses a generative AI model to generate an apology or re-explanation such as "I'm sorry, I'll explain it again." Finally, the device converts the generated information into speech using speech synthesis software and speaks it to the learner through the speaker.

[1475] Examples

[1476] For example, if a learner says "The capital of Japan is Osaka," the system will operate as follows:

[1477] 1. The device picks up speech through the microphone and uses Google Speech-to-Text to convert it into text data such as "The capital of Japan is Osaka."

[1478] 2. The server receives the text data and analyzes it using spaCy.

[1479] 3. Based on the analysis results, the server detects that the statement "The capital of Japan is Osaka" is incorrect.

[1480] 4. The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[1481] 5. The device analyzes the learner's facial expressions and tone of voice using the Microsoft Azure Emotion API to detect when the learner is expressing dissatisfaction.

[1482] 6. The server uses an emotion engine to generate an apology such as "I'm sorry, I'll explain it again."

[1483] 7. The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[1484] Examples of prompt statements

[1485] "If a learner says, 'The capital of Japan is Osaka,' please generate the correct information."

[1486] "If the learner expresses anger or frustration, generate appropriate apologies or restatements."

[1487] The above is an embodiment of the invention.

[1488] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1489] Step 1:

[1490] The user makes a statement as a learner.

[1491] Input: Learner's spoken words

[1492] Output: Audio data

[1493] Specific action: The learner says, "The capital of Japan is Osaka."

[1494] Step 2:

[1495] The terminal captures the learner's voice data through a microphone and converts it into text data using voice recognition software.

[1496] Input: Audio data

[1497] Output: Text data

[1498] Specific operation: The device uses Google Speech-to-Text to generate text data such as "The capital of Japan is Osaka."

[1499] Step 3:

[1500] The server receives the text data sent from the terminal and analyzes it using natural language processing software.

[1501] Input: Text data

[1502] Output: Analysis results

[1503] Specific operation: The server uses spaCy to analyze the text data and understand the content of the statement, "The capital of Japan is Osaka."

[1504] Step 4:

[1505] The server uses the analysis results to determine whether the learner's statement is correct.

[1506] Input: Analysis results

[1507] Output: True or False

[1508] Specific operation: The server detects that the statement "The capital of Japan is Osaka" is false.

[1509] Step 5:

[1510] The server generates the correct information using a generative AI model.

[1511] Input: Correct / Incorrect

[1512] Output: Correct information

[1513] Specific operation: The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[1514] Step 6:

[1515] The device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions.

[1516] Input: Learner's facial expressions and tone of voice

[1517] Output: Emotion data

[1518] Specific behavior: The device uses the Microsoft Azure Emotion API to detect when a learner is expressing dissatisfaction.

[1519] Step 7:

[1520] The terminal transmits the detected emotion data to the server.

[1521] Input: Emotion data

[1522] Output: Emotion data (sent to server)

[1523] Specific operation: The terminal transmits emotion data to the server.

[1524] Step 8:

[1525] The server uses an emotion engine to generate a response according to the learner's emotion.

[1526] Input: Emotion data

[1527] Output: Appropriate response

[1528] What it does: The server uses a generative AI model to generate an apology saying, "I'm sorry, I'll explain it again."

[1529] Step 9:

[1530] The terminal uses speech synthesis software to convert the generated information into speech, which is then spoken to the learner through a speaker.

[1531] Input: appropriate response

[1532] Output: Audio data

[1533] Specific operation: The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[1534] (Application example 2)

[1535] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1536] Conventional educational robot systems have the function of providing correct information when a learner gives an incorrect answer, but they are unable to show appropriate responses according to the learner's emotions, making it difficult to improve the learner's motivation and level of understanding.In addition, when it comes to work guidance in factories, there is a lack of a function to teach the correct procedure when a worker performs an incorrect work procedure, making it difficult to improve work efficiency and quality.

[1537] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1538] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and speaking information for correcting an incorrect answer given by a learner, an emotion recognition means, a means for adjusting a response according to the emotion of the learner, and a means for instructing a work procedure. This makes it possible to provide correct information when a learner gives a wrong answer, and to show an appropriate response according to the emotion of the learner. In addition, in the case of work guidance in a factory, if a worker performs an incorrect work procedure, the correct procedure can be taught, which is expected to improve work efficiency and quality.

[1539] "Speech recognition means" refers to technology that analyzes speech and converts what is being said into text data.

[1540] "Generative AI means" refers to artificial intelligence technology that generates appropriate information based on input data.

[1541] "A means for allowing a learner to give an incorrect answer and for generating and giving information to correct it" refers to technology that has the function of detecting incorrect answers given by a learner and providing the correct information.

[1542] "Emotion recognition means" is a technology that analyzes and recognizes emotions from learners' facial expressions and voices.

[1543] A "means for tailoring responses according to learner emotions" is a technique for generating and providing appropriate responses based on recognized emotions.

[1544] The "means for instructing work procedures" refers to technology that has the function of teaching correct procedures when a worker performs an incorrect work procedure.

[1545] A system for implementing this invention includes a voice recognition means, a generative AI means, a means for generating and speaking information to correct an incorrect answer provided by a learner, an emotion recognition means, a means for adjusting a response in accordance with the learner's emotions, and a means for instructing the learner on work procedures.

[1546] Hardware and Software Used

[1547] Hardware: Task guidance robot, microphone, camera

[1548] Software: OpenAI API, Emotion Recognition API

[1549] System program processing

[1550] Voice Recognition Means

[1551] The server collects the learner's utterances through a microphone and converts them into text data using a voice recognition means. The voice recognition means accurately analyzes the contents of the learner's utterances and transmits them in text form to the server.

[1552] Generative AI means

[1553] The server uses the generative AI means to determine whether the learner's utterance is correct. If the utterance is incorrect, the generative AI means generates correct information and creates text data to utter that information.

[1554] emotion recognition means

[1555] The server collects the learner's facial expressions and voices through a camera and microphone, and analyzes the learner's emotions using an emotion recognition means. This emotion recognition means judges whether the learner is showing anger or dissatisfaction, and transmits the result to the server.

[1556] A means of adjusting responses according to learners' emotions

[1557] The server generates an appropriate response according to the learner's emotions based on the data from the emotion recognition means. For example, if the learner shows anger or dissatisfaction, the generative AI means generates an apology or re-explanation such as "I'm sorry, I'll explain it again."

[1558] A means of instructing work procedures

[1559] The server generates information to teach correct work procedures when a worker performs an incorrect work procedure. The generative AI means detects incorrect procedures and creates text data to explain the correct procedures.

[1560] Examples

[1561] For example, if a worker says, "Put this part here," but it should actually be installed in a different place, the server will tell them, "That's wrong. The correct procedure is to install this part here." Also, if a worker is stressed, the server will respond to the worker's emotions by saying, "I'm sorry, let me explain it again."

[1562] Examples of prompt statements

[1563] If a user says "install this part here", what is the correct procedure?

[1564] If the user is stressed, generate an appropriate response.

[1565] In this way, a system is realized that can provide correct information when a learner or worker utters incorrect information, and can also show an appropriate response according to their emotions.

[1566] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1567] Step 1:

[1568] The server collects the user's speech through a microphone. The input is the user's voice, and the output is voice data. This voice data is sent to a voice recognition means.

[1569] Step 2:

[1570] The server converts the voice data into text data using a voice recognition means. The input is voice data, and the output is text data. This text data is sent to the generative AI means.

[1571] Step 3:

[1572] The server uses a generative AI means to determine whether the text data is correct. The input is the text data, and the output is the result of the judgment. If the statement is incorrect, the generative AI means generates the correct information.

[1573] Step 4:

[1574] The server creates text data for uttering the correct information from the generative AI means. The input is the result of the judgment of correctness or incorrectness and the correct information, and the output is the text data for utterance. This text data is sent to the speech synthesis means.

[1575] Step 5:

[1576] The server collects the user's facial expressions and voice through a camera and microphone, and analyzes the user's emotions using an emotion recognition means. The input is the user's facial expression data and voice data, and the output is emotion data. This emotion data is sent to the generative AI means.

[1577] Step 6:

[1578] The server generates an appropriate response according to the user's emotion based on the emotion data from the emotion recognition means. The input is the emotion data, and the output is text data of the response according to the emotion. This text data is sent to the voice synthesis means.

[1579] Step 7:

[1580] The server generates information to teach the correct work procedure when the worker performs an incorrect work procedure. The input is text data of the incorrect work procedure, and the output is text data of the correct work procedure. This text data is sent to the voice synthesis means.

[1581] Step 8:

[1582] The server uses a speech synthesis means to convert the generated text data into speech and speaks it to the user. The input is the text data to be spoken, and the output is speech data. This speech data is provided to the user through a speaker.

[1583] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1584] The data generation model 58 is a so-called generative AI (Artificial Intelligence). One example of a data generation model58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1585] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1586] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1587] [Fourth embodiment]

[1588] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1589] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1590] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1591] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1592] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1593] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1594] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1595] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1596] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1597] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program.

[1598] The processor 28 executes the specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1599] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1600] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1601] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1602] "Example 1"

[1603] The educational robot system of the present invention comprises a voice recognition means and a generative AI means. The voice recognition means recognizes the learner's utterance and judges whether the utterance is incorrect. The generative AI means generates correct information for the learner's incorrect utterance and speaks that information. Specifically, if the learner says "The Earth is flat," the voice recognition means recognizes this utterance, and the generative AI means generates and speaks the information "That is incorrect. The Earth is spherical."

[1604] "Example 2"

[1605] The educational robot system of the present invention provides an effective means for implementing lesson development in which learners actively learn. Specifically, when a learner gives a wrong answer, the educational robot system generates and speaks information to correct it. This allows the learner to recognize his or her own mistake and obtain the correct information. For example, if a learner says, "The capital of Japan is Osaka," the educational robot system generates and speaks the information, "That is incorrect. The capital of Japan is Tokyo."

[1606] The process flow of each embodiment will be described below.

[1607] "Example 1"

[1608] Step 1: The learner makes an utterance, for example, "The Earth is flat."

[1609] Step 2: The speech recognition means recognizes the learner's utterance. In this example, it recognizes the utterance "The Earth is flat."

[1610] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information "That is incorrect. The Earth is spherical." Step 4: The generative AI means speaks out the generated information. In this example, it speaks out "That is incorrect. The Earth is spherical."

[1611] "Example 2"

[1612] Step 1: A learner makes an utterance. For example, the learner may say, "The capital of Japan is Osaka." Step 2: A speech recognition means recognizes the learner's utterance. In this example, the speech recognition means recognizes the utterance, "The capital of Japan is Osaka."

[1613] Step 3: The generative AI means determines that the learner's statement is incorrect and generates the correct information. In this example, it generates the information, "That is incorrect. The capital of Japan is Tokyo."

[1614] Step 4: The generative AI means speaks the generated information. In this example, it speaks, "That is incorrect. The capital of Japan is Tokyo."

[1615] Example 1

[1616] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1617] In conventional educational systems, it was difficult to determine in real time whether what a learner said was correct and to immediately provide the correct information. In addition, if a learner said incorrect information, there was a lack of means to provide appropriate feedback to correct the mistake. This resulted in problems such as learners not deepening their understanding and preventing effective learning.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1619] In this invention, the server includes a speech recognition means, a generative artificial intelligence means, a means for a learner to speak an incorrect answer and to generate and speak information for correcting the answer, a means for converting speech to text, a means for analyzing the text and determining the error, a means for generating a prompt sentence for generating correct information, and a means for converting the generated correct information into speech. This makes it possible to recognize a learner's speech in real time, immediately determine an error, and provide correct information.

[1620] A "voice recognition means" is a device or software for converting voice data into text data.

[1621] A "generative artificial intelligence means" is an artificial intelligence technique for generating new information based on input data.

[1622] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information in response to it, and provides feedback in the form of voice.

[1623] A "speech-to-text conversion means" is a technology for analyzing speech data and converting it into corresponding text data.

[1624] A "means for analyzing text and determining errors" is an algorithm or set of rules for analyzing text data and determining whether its content is correct.

[1625] "Means for generating a prompt sentence for generating correct information" refers to a technology for creating an instruction sentence for causing a generative artificial intelligence to generate correct information.

[1626] The "means for converting the generated correct information into voice" refers to a device or software for converting text data into voice data.

[1627] The present invention provides a technology for an educational robot system that recognizes a learner's utterances in real time, immediately detects mistakes, and provides correct information. A specific embodiment of this system will be described below.

[1628] First, the user speaks to the educational robot. For example, they say, "The Earth is flat." The device captures this speech with a microphone and saves it as voice data.

[1629] Next, the terminal transmits the acquired voice data to a voice recognition means. A general voice recognition API can be used as this voice recognition means. For example, Google Cloud Speech-to-Text API is used. This API converts the voice data into text data. The converted text data becomes "The Earth is flat."

[1630] The server receives the text data sent from the terminal and analyzes the text using a predefined rule set that contains a list of true and false statements. The server determines that the statement "The Earth is flat" is false.

[1631] Next, the server uses a generative artificial intelligence means to generate correct information. A general generative AI model can be used as this generative artificial intelligence means. For example, OpenAI's GPT-4 model is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical."

[1632] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A general speech synthesis API can be used as the speech synthesis means. For example, the Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical."

[1633] In this way, the educational robot system can recognize what the learner says in real time, immediately identify mistakes, and provide correct information, which will help the learner deepen their understanding and learn more effectively.

[1634] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1635] Step 1:

[1636] The user speaks to the educational robot. For example, "The Earth is flat." The device captures this speech with a microphone and obtains it as voice data. The input is the user's voice, and the output is the captured voice data.

[1637] Step 2:

[1638] The terminal sends the acquired voice data to the voice recognition means. A general voice recognition API is used as the voice recognition means. For example, Google Cloud Speech-to-Text API is used. This API converts the voice data into text data. The input is the voice data, and the output is the text data "The Earth is flat."

[1639] Step 3:

[1640] The server receives the text data sent from the terminal and analyzes the text. The analysis uses a predefined rule set, which contains a list of true and false statements. The server determines that the statement "The Earth is flat" is false. The input is the text data "The Earth is flat" and the output is the judgment result that it is "false."

[1641] Step 4:

[1642] The server generates correct information using a generative artificial intelligence means. A general generative AI model is used as this generative artificial intelligence means. For example, OpenAI's GPT-4 model is used. The server sends a prompt sentence to the GPT-4 model to generate correct information for the incorrect statement. An example of a prompt sentence is "The learner said, 'The Earth is flat.' Please provide the correct information." The GPT-4 model generates the correct information, "That is incorrect. The Earth is spherical." The input is the prompt sentence, and the output is the correct information, "That is incorrect. The Earth is spherical."

[1643] Step 5:

[1644] The generated correct information is returned to the server as text data. The server uses a speech synthesis means to convert this text data into speech. A common speech synthesis API is used as this speech synthesis means. For example, Google Cloud Text-to-Speech API is used. The generated speech data is sent to the terminal, and the terminal provides feedback to the user through the speaker. The user receives the feedback, "That's wrong. The Earth is spherical." The input is the text data of the correct information, and the output is speech data.

[1645] (Application example 1)

[1646] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1647] When workers perform incorrect work procedures in a factory, it can lead to reduced production efficiency and safety issues. In particular, mistakes are likely to occur when workers are new or when the work procedures are complex. There is a demand for a system that can detect such mistakes in real time and provide the correct work procedures.

[1648] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a voice recognition means, a generative AI means, a means for a worker to speak an incorrect work procedure and to generate and speak information for correcting it, a means for recognizing the worker's statement and judging whether the work procedure is correct, and a means for generating and speaking a correct work procedure. This makes it possible, when a worker speaks an incorrect work procedure, to point out the error in real time and provide the correct work procedure.

[1649] The "voice recognition means" is a device or system that acquires the speech of a worker as voice data and converts the voice data into text data.

[1650] A "generative AI means" is a device or system that uses an artificial intelligence model to generate appropriate information based on input data.

[1651] "Workers" refers to personnel who perform work within a factory.

[1652] A "work procedure" refers to the sequence of steps or methods required to complete a particular task.

[1653] "Utterance" refers to words or sentences spoken by a worker.

[1654] "Good work procedures" refer to the precise steps or methods needed to complete a task safely and efficiently.

[1655] An "error" refers to a situation in which what a worker says does not match the correct work procedure.

[1656] "Real-time" refers to responding immediately to what workers say and providing information.

[1657] A system for implementing this invention includes a voice recognition means, a generative AI means, a means for recognizing utterances by a worker and determining whether or not the work procedure is correct, and a means for generating and uttering the correct work procedure.

[1658] Hardware and Software Used

[1659] Hardware:

[1660] Voice recognition microphone (e.g. high performance microphone)

[1661] Robot body (e.g. multi-function robot)

[1662] Speakers (e.g. high quality speakers)

[1663] software:

[1664] Speech recognition software (e.g. voice recognition engine)

[1665] Generative AI models (e.g. advanced generative AI models)

[1666] Robot control software (e.g. robot operating system)

[1667] Processing flow

[1668] 1. Voice Recognition:

[1669] A worker says, "This part goes here."

[1670] The voice recognition microphone captures what is said, and the voice recognition engine converts it into text data.

[1671] 2. Judging statements:

[1672] The converted text data is analyzed to determine whether the work procedures are correct.

[1673] For example, if a statement such as "This part should be installed here" is incorrect, a prompt is sent to the generative AI model.

[1674] 3. Producing the right information:

[1675] The generative AI model generates the correct information: "That's wrong. This part needs to be installed in a different location."

[1676] 4. Information Statement:

[1677] The correct information generated is communicated to the worker through the robot's built-in speaker.

[1678] Examples

[1679] If a worker says, "This part should be installed here," the robot will point out, "That's wrong. This part needs to be installed somewhere else."

[1680] Examples of prompt statements

[1681] User says: "This part goes here"

[1682] Generate the correct information.

[1683] With this system, when a worker makes an incorrect statement about a work procedure, the system can point out the mistake in real time and provide the correct work procedure, thereby improving work efficiency and safety in the factory.

[1684] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1685] Step 1:

[1686] Audio capture

[1687] Input: A worker says, "This part goes here."

[1688] Specific operation: A voice recognition microphone captures what the worker says.

[1689] Output: The captured audio data.

[1690] Step 2:

[1691] Speech to text conversion

[1692] Input: The captured audio data.

[1693] Specific operation: The voice recognition engine converts the voice data into text data.

[1694] Output: The converted text data (e.g. "Install this part here").

[1695] Step 3:

[1696] Speech analysis

[1697] Input: The converted text data.

[1698] Specific operation: The server analyzes the text data and determines whether the work procedure is correct.

[1699] Output: The result of a judgment as to whether the statement is correct or not (e.g., false).

[1700] Step 4:

[1701] Prompt Generation

[1702] Input: The judgement result that the statement is false.

[1703] Specific operation: The server sends a prompt to the generative AI model.

[1704] Output: A prompt to the generative AI model (e.g., "User says: 'This part should be installed here' Please generate the correct information.").

[1705] Step 5:

[1706] Producing the right information

[1707] Input: A prompt for the generative AI model.

[1708] Specific operation: The generative AI model generates the correct work steps.

[1709] Output: Generated correct work instructions (e.g. "That's wrong. This part needs to be installed somewhere else.").

[1710] Step 6:

[1711] Statement of correct information

[1712] Input: The generated correct work instructions.

[1713] Specific operation: The correct information generated is communicated to the worker through the speaker on the robot's body.

[1714] Output: The worker can hear the correct work procedure.

[1715] Example 2

[1716] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1717] In conventional educational systems, when a learner gives a wrong answer, there is a lack of a way to immediately provide correct information, so the learner may continue to hold on to the wrong knowledge. In addition, it is difficult to provide an environment in which the learner can actively learn.

[1718] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1719] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and speaking information for correcting an incorrect answer given by a learner, a means for converting voice data into text data, a means for analyzing the text data and judging the correctness of the statement, and a means for converting correct information into voice data and feeding it back to the learner. This makes it possible to provide correct information immediately when a learner gives a wrong answer, and to provide an environment in which the learner can actively learn.

[1720] A "voice recognition means" is a device or software for collecting voice data and converting it into text data.

[1721] A "generative AI means" is an artificial intelligence model that generates appropriate responses or information based on input data.

[1722] "Means for generating and uttering information to correct an incorrect answer by a learner" refers to a device or software that detects an incorrect answer by a learner, generates correct information, and conveys it to the learner.

[1723] The "means for converting voice data into text data" refers to a device or software that converts voice data into text information using voice recognition technology.

[1724] The "means for analyzing text data and judging whether the content of a statement is correct" refers to a device or software for analyzing text data and judging whether the content is correct.

[1725] The "means for converting correct information into voice data and feeding it back to the learner" refers to a device or software for converting the generated correct information into voice data and conveying it to the learner.

[1726] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1727] This invention relates to an educational robot system, and provides an effective means for realizing a lesson in which students actively learn. Specifically, when a student gives a wrong answer, the educational robot system generates and speaks information to correct the answer, allowing the student to recognize his / her own mistake and obtain the correct information.

[1728] Hardware and Software Used

[1729] The system uses the following hardware and software:

[1730] Speech recognition means: A device or software for collecting voice data and converting it into text data. Specifically, we use voice recognition software (e.g., Google Cloud Speech-to-Text).

[1731] Generative AI means: An artificial intelligence model that generates appropriate responses or information based on input data. Specifically, we use a generative AI model (e.g., OpenAI's GPT-4).

[1732] Means for converting voice data into text data: A device or software that converts voice data into text information using voice recognition technology. Specifically, we use Google Cloud Speech-to-Text.

[1733] A means of analyzing text data and judging whether a statement is correct: A device or software that analyzes text data and judges whether the content is correct. Specifically, a generative AI model (e.g., OpenAI's GPT-4) is used.

[1734] Means of converting correct information into voice data and feeding it back to the learner: A device or software that converts the generated correct information into voice data and conveys it to the learner. Specifically, speech synthesis software (e.g. Amazon Polly) is used.

[1735] Examples

[1736] If a user says "The capital of Japan is Osaka", the following processing occurs:

[1737] 1. Voice recognition: The device collects the user's speech through a microphone. For example, the user says, "The capital of Japan is Osaka."

[1738] 2. Text conversion: The device converts the collected voice data into text data using the Google Cloud Speech-to-Text API. The converted text data becomes "The capital of Japan is Osaka."

[1739] 3. Correct / incorrect judgment: The server receives the text data sent from the terminal. Then, the server inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[1740] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[1741] 4. Information generation: The generative AI model generates the correct information: "That is incorrect. The capital of Japan is Tokyo." The server receives this generated information and sends it to the terminal.

[1742] 5. Voice feedback: The device converts the correct information received from the server into voice data using voice synthesis software (e.g. Amazon Polly). The device then communicates the converted voice data to the user through the speaker. Specifically, the device provides voice feedback saying, "That's incorrect. The capital of Japan is Tokyo."

[1743] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[1744] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1745] Step 1:

[1746] The user speaks.

[1747] The user says, "The capital of Japan is Osaka." This becomes the input to the system.

[1748] Step 2:

[1749] The terminal collects the user's utterances as voice data.

[1750] The device captures the user's speech as voice data through a microphone, and this voice data becomes the input for the next processing step.

[1751] Step 3:

[1752] The terminal converts the voice data into text data.

[1753] The device uses the Google Cloud Speech-to-Text API to convert the collected voice data into text data. For example, the generated text data is "The capital of Japan is Osaka." This text data becomes the input for the next processing step.

[1754] Step 4:

[1755] The server receives the text data and inputs a prompt sentence into the generative AI model.

[1756] The server receives the text data sent from the device and inputs the following prompt sentence into the generative AI model (e.g., OpenAI's GPT-4):

[1757] A user said, "The capital of Japan is Osaka." Is this correct? If not, please provide the correct information.

[1758] This prompt sentence becomes the input to the generative AI model.

[1759] Step 5:

[1760] The generative AI model generates the correct information based on the prompt sentence.

[1761] The generative AI model analyzes the input prompt sentence and generates the correct information, "That is incorrect. The capital of Japan is Tokyo." This generated information becomes the input for the next processing step.

[1762] Step 6:

[1763] The server transmits the generated information to the terminal.

[1764] The server sends the correct information received from the generative AI model to the terminal. For example, text data such as "That is incorrect. The capital of Japan is Tokyo" is sent to the terminal.

[1765] Step 7:

[1766] The terminal converts the correct information into voice data.

[1767] The device uses Amazon Polly to convert the received text data into speech data. For example, speech data such as "That's wrong. The capital of Japan is Tokyo" is generated.

[1768] Step 8:

[1769] The terminal feeds back the voice data to the user.

[1770] The terminal transmits the generated voice data to the user through the speaker, and the user receives voice feedback saying, "That is incorrect. The capital of Japan is Tokyo."

[1771] In this way, the educational robot system can deepen the learner's understanding by providing correct information when the learner gives an incorrect answer.

[1772] (Application example 2)

[1773] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1774] There is a demand for a system that can immediately instruct workers in the correct procedure when they make a mistake in a factory. With conventional methods, when a worker makes a mistake, it takes time for the worker to realize the mistake, resulting in a decrease in production efficiency. There was also the issue that education and training to correct mistakes takes a lot of time and costs money.

[1775] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1776] In this invention, the server includes a voice recognition means, a generative AI means, and a means for generating and announcing information for correcting an incorrect procedure made by a worker. This makes it possible to immediately instruct the correct procedure when a worker makes a mistake.

[1777] A "voice recognition means" is a technology that receives voice as input and converts the voice into text data.

[1778] "Generative AI means" refers to artificial intelligence technology that generates appropriate responses or information based on input data.

[1779] "Workers" refers to personnel who perform work within a factory.

[1780] A "procedure" refers to the sequence of operations or steps required to complete a particular task.

[1781] "Means for generating and uttering information" refers to technology for outputting the generated information as voice.

[1782] The system for carrying out the present invention includes a voice recognition means, a generative AI means, and a means for a worker to speak out a procedure that he or she has made a mistake in and to generate and speak information for correcting that procedure.

[1783] First, the speech recognition method requires technology that converts speech into text data. Specifically, speech recognition APIs such as Google Cloud Speech-to-Text and IBM Watson Speech to Text can be used. This makes it possible to convert the speech of workers into text data in real time.

[1784] Next, generative AI models such as OpenAI's GPT-3 are used as generative AI methods. Generative AI models generate appropriate responses and information based on input text data. For example, if a worker says, "Assemble the parts in the order B→A→C," the generative AI model generates information such as, "That's incorrect. The correct order is A→B→C."

[1785] Finally, a technology to convert the generated text data into voice is required as a means of generating and speaking information. Specifically, speech synthesis APIs such as Google Cloud Text-to-Speech and Amazon Polly can be used. This makes it possible to output the generated information as voice and convey it to workers.

[1786] As a concrete example, if a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the means for generating and uttering information converts the information into voice and conveys it to the worker.

[1787] Examples of prompts include:

[1788] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[1789] In this way, when a worker makes a mistake in a procedure, it is possible to immediately instruct the worker to follow the correct procedure, thereby improving the efficiency and accuracy of work in the factory.

[1790] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1791] Step 1:

[1792] The server receives the speech of the worker as voice data using a voice recognition means. The voice recognition means converts the voice data into text data. The input is the voice data of the worker, and the output is text data.

[1793] Step 2:

[1794] The server passes text data to the generative AI means as input. The generative AI means generates an appropriate response based on this text data. Specifically, if the worker's statement is incorrect, it generates a response including a correct procedure. The input is text data, and the output is the generated response text.

[1795] Step 3:

[1796] The server passes the generated response text as input to the speech synthesis means, which converts the text data into speech data, with the input being the generated response text and the output being the speech data.

[1797] Step 4:

[1798] The server transmits the voice data to the worker through a speaker, allowing the worker to confirm the correct procedure by voice. The input is the voice data, and the output is the voice that the worker hears.

[1799] Specifically, when a worker says, "Assemble the parts in the order B→A→C," the speech recognition means converts the utterance into text data, and the generative AI means generates information saying, "That's wrong. The correct order is A→B→C." Next, the speech synthesis means converts the information into voice and conveys it to the worker through a speaker.

[1800] Examples of prompts include:

[1801] A worker says: 'Assemble the parts in the order B→A→C'. Please tell me the correct procedure.

[1802] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1803] "Example 1"

[1804] One embodiment of the present invention is an educational robot system that incorporates an emotion engine. In this system, a voice recognition means recognizes a learner's utterance, and a generative AI means judges whether the utterance is incorrect. If the utterance is incorrect, the generative AI means generates correct information and speaks that information. Furthermore, this system includes an emotion engine and can recognize the learner's emotions. For example, if the learner feels confused, the generative AI means generates information according to the learner's emotion, such as encouraging words like "It's okay, let's think about it together," and speaks that information.

[1805] "Example 2"

[1806] In another embodiment, the emotion engine adjusts the response of the educational robot system according to the emotion of the learner. For example, if the emotion engine senses that the learner is angry or frustrated, it conveys the information to the generative AI means. Based on the information, the generative AI means generates words of apology or re-explanation, such as "I'm sorry, I'll explain it again," and speaks the information. This allows an appropriate response to be provided according to the learner's emotion.

[1807] The process flow of each embodiment will be described below.

[1808] "Example 1"

[1809] Step 1: The learner says something.

[1810] Step 2: The speech recognition means recognizes the learner's utterance.

[1811] Step 3: Determine whether the utterance recognized by the generative AI means is incorrect.

[1812] Step 4: If incorrect, the generative AI means generates the correct information and speaks that information.

[1813] Step 5: The emotion engine recognizes the learner's emotions.

[1814] Step 6: The generative AI means generates information according to the emotion based on the information from the emotion engine, and speaks that information.

[1815] "Example 2"

[1816] Step 1: The learner says something.

[1817] Step 2: The speech recognition means recognizes the learner's utterance.

[1818] Step 3: The emotion engine recognizes the learner's emotions.

[1819] Step 4: Based on the emotion engine's recognized emotion, the emotion engine conveys information to the generative AI means. Step 5: Based on the information from the emotion engine, the generative AI means generates information corresponding to the emotion, and speaks that information.

[1820] Example 1

[1821] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1822] Conventional educational systems can accurately recognize what students say and point out their mistakes, but they cannot respond according to the student's emotions. This makes it difficult to provide appropriate support when a student feels confused or anxious. There is also a need to improve learning outcomes by providing feedback according to the student's emotions.

[1823] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1824] In this invention, the server includes a speech recognition means, a generative AI means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and uttering information according to the emotions. This makes it possible to accurately recognize the learner's utterances, point out mistakes, and provide appropriate feedback according to the learner's emotions.

[1825] A "voice recognition means" is a device or software that captures a learner's speech as voice data and converts the voice data into text data.

[1826] "Generative AI means" refers to an artificial intelligence model that generates appropriate information based on input data, and in particular has the ability to determine whether what a learner says is correct or incorrect and generate correct information.

[1827] "A means for allowing a learner to speak an incorrect answer and generating and speaking information to correct it" refers to a device or software that, when what a learner says is incorrect, points out the mistake and generates and speaks the correct information.

[1828] "Means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions" refers to a device or software that analyzes the learner's emotions from their facial expressions, tone of voice, etc., and generates and speaks appropriate feedback or words of encouragement corresponding to those emotions.

[1829] The educational robot system of the present invention includes a voice recognition means, a generative AI means, a means for generating and speaking information to correct an incorrect answer given by a learner, and a means for recognizing the learner's emotions and generating and speaking information corresponding to those emotions.

[1830] Hardware and Software Used

[1831] Hardware: Microphone, speaker, camera (for emotion recognition)

[1832] Software: voice recognition software (e.g. voice recognition APIs), generative AI models (e.g. AI models), emotion recognition engines (e.g. emotion recognition software)

[1833] Program processing flow

[1834] Speech Recognition

[1835] The device uses a microphone to capture what the learner says.

[1836] The server uses voice recognition software to convert the voice data into text data.

[1837] Example: If a learner says, "The Earth is flat," speech recognition software converts this utterance into text.

[1838] Evaluation of comments

[1839] The server uses a generative AI model to evaluate whether the text data is correct.

[1840] Example: determining that the statement "The Earth is flat" is false.

[1841] Producing the right information

[1842] The server uses a generative AI model to generate correct information.

[1843] Example: Producing the correct information, "That's wrong. The Earth is spherical."

[1844] emotion recognition

[1845] The device uses a camera to capture the learner's facial expressions.

[1846] The server uses an emotion recognition engine to analyze the learner's emotions.

[1847] Example: A camera captures a learner's confused expression, and an emotion recognition engine identifies the expression as "confused."

[1848] Generating responses based on emotions

[1849] The server uses a generative AI model to generate information based on the learner's emotions.

[1850] Example: A generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[1851] Information statement

[1852] The device uses a speaker to communicate the generated information to the learner.

[1853] Example: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's figure it out together."

[1854] Examples of prompt statements

[1855] Prompt: "If the learner says, 'The Earth is flat,' generate and speak the correct information. Also generate encouraging words if the learner is confused."

[1856] In this way, the educational robot system of the present invention can not only recognize what the learner says and provide correct information, but also respond according to the learner's emotions, making it possible to provide appropriate support to the learner even when he or she feels confused or anxious, thereby improving the effectiveness of learning.

[1857] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1858] Step 1:

[1859] Capturing audio input

[1860] The device uses a microphone to capture what the learner says.

[1861] Input: Learner's spoken utterance (e.g. "The Earth is flat")

[1862] Output: The captured audio data.

[1863] What happens: A learner says, "The Earth is flat," and their voice is captured by a microphone.

[1864] Step 2:

[1865] Converting audio data to text

[1866] The server uses voice recognition software to convert the captured voice data into text data.

[1867] Input: Captured audio data

[1868] Output: Text data (e.g. "The Earth is flat")

[1869] What it does: Speech recognition software analyzes the speech data and converts it into text: "The Earth is flat."

[1870] Step 3:

[1871] Evaluation of comments

[1872] The server uses a generative AI model to evaluate whether the text data is correct.

[1873] Input: Text data (e.g. "The Earth is flat")

[1874] Output: Evaluation result (e.g., error)

[1875] How it works: A generative AI model analyzes the statement "The Earth is flat" and determines that it is false.

[1876] Step 4:

[1877] Producing the right information

[1878] The server uses a generative AI model to generate correct information.

[1879] Input: Evaluation result (e.g. mistake)

[1880] Output: Correct information (e.g. "That's not true. The Earth is a sphere.")

[1881] Specific behavior: The generative AI model generates the correct information, "That's wrong. The Earth is spherical."

[1882] Step 5:

[1883] emotion recognition

[1884] The device uses a camera to capture the learner's facial expressions.

[1885] The server uses an emotion recognition engine to analyze the learner's emotions.

[1886] Input: Learner's facial expression data

[1887] Output: Sentiment analysis result (e.g. confused)

[1888] Specific action: The camera captures the learner's confused expression, and the emotion recognition engine recognizes the expression as "confused."

[1889] Step 6:

[1890] Generating responses based on emotions

[1891] The server uses a generative AI model to generate information based on the learner's emotions.

[1892] Input: Sentiment analysis result (e.g. confused)

[1893] Output: Information based on the emotion (e.g. "It's okay, let's think about it together")

[1894] Specific behavior: The generative AI model generates encouraging words such as, "It's okay, let's think about it together."

[1895] Step 7:

[1896] Information statement

[1897] The device uses a speaker to communicate the generated information to the learner.

[1898] Input: Correct information and emotion-specific information (e.g., "That's wrong. The Earth is a sphere." "It's okay, let's figure it out together.")

[1899] Output: Audio feedback to the learner

[1900] Specific actions: The speaker says, "That's wrong. The Earth is a sphere," followed by an encouraging remark, "It's okay, let's think about it together."

[1901] (Application example 1)

[1902] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1903] There is a need to point out mistakes in factory worker remarks and procedures in real time and provide correct information. Another challenge is to improve work efficiency and safety by recognizing the emotions of workers and providing appropriate encouragement and advice.

[1904] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1905] In this invention, the server includes a voice recognition means, a generative AI means, a means for generating and uttering information for correcting a procedure that the worker has made a mistake in, an emotion recognition means, and a means for generating and uttering information according to the emotion of the worker. This makes it possible to point out the worker's comments and errors in the procedure in real time, provide correct information, and take appropriate action according to the emotion of the worker.

[1906] The "voice recognition means" is a device or system that recognizes the speech of the worker as voice and converts it into text data.

[1907] "Generative AI means" refers to artificial intelligence techniques that generate appropriate information based on input data.

[1908] "Means for a worker to state a procedure that he or she made a mistake and to generate and state information to correct it" refers to a device or system that evaluates the content of the worker's statement and generates and provides the correct procedure or information if a mistake is made.

[1909] An "emotion recognition means" is a device or system that analyzes and recognizes emotions from a worker's facial expressions, tone of voice, etc.

[1910] The "means for generating and uttering information corresponding to the emotions of the worker" is a device or system that generates and provides appropriate encouragement or advice based on the emotions recognized by the emotion recognition means.

[1911] As an embodiment of the present invention, a factory work support robot system will be described as an example. This system includes a voice recognition means, a generative AI means, an emotion recognition means, and software for integrating them.

[1912] Hardware used:

[1913] Voice recognition microphone

[1914] speaker

[1915] Camera (for emotion recognition)

[1916] Factory robot body

[1917] Software used:

[1918] Speech recognition software (Google Speech-to-Text API)

[1919] Generative AI model (OpenAI GPT-4)

[1920] Emotion recognition software (Microsoft Azure Emotion API)

[1921] System Operation:

[1922] 1. Speech recognition: When a worker speaks, the speech is picked up by the speech recognition microphone and converted into text data using the Google Speech-to-Text API. This text data is then sent to the server.

[1923] 2. Evaluating the utterance: The server uses a generative AI model (GPT-4) to evaluate whether the worker's utterance is correct. For example, it judges whether the utterance "This part should be installed here" is correct.

[1924] 3. Generation of correct information: If the statement is incorrect, the server uses GPT-4 to generate correct information and communicates it to the worker through the speaker. For example, it provides information such as "That part should not be installed here, but next to it."

[1925] 4. Emotion Recognition: Using a camera and the Microsoft Azure Emotion API, the system analyzes the worker's facial expressions and recognizes their emotions. For example, if the worker is confused, that information is sent to the server.

[1926] 5. Emotion-based response: Based on the result of emotion recognition, the server uses GPT-4 to generate appropriate encouragement or advice and convey it to the worker through the speaker. For example, encouraging words such as "It's okay, let's check it together" are provided.

[1927] Examples:

[1928] When a worker says, "This part should be installed here," the voice recognition microphone picks up the statement and converts it into text using the Google Speech-to-Text API. The server uses GPT-4 to evaluate the statement, and if it determines that the statement is incorrect, it generates correct information such as, "That part should be installed in the next position, not here," and conveys this to the worker through the speaker. In addition, the camera and Microsoft Azure Emotion API analyze the worker's facial expressions, and if it recognizes that the worker is confused, it provides encouraging words such as, "It's okay, let's check together."

[1929] Example prompt for a generative AI model:

[1930] The worker says, "This part goes here." Evaluate whether this statement is correct and provide the correct information if it is incorrect. Also, generate appropriate words of encouragement if the worker is confused.

[1931] In this way, the factory work support robot system can evaluate the worker's comments in real time, provide correct information, and take appropriate action according to the worker's emotions.

[1932] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1933] Step 1:

[1934] Speech Recognition

[1935] Input: What the worker says

[1936] Processing: A voice recognition microphone picks up what the worker is saying and converts the voice data into text using the Google Speech-to-Text API.

[1937] Output: Text data

[1938] Specific operation: When a worker says, "This part goes here," the voice recognition microphone picks up the speech, and the Google Speech-to-Text API converts the speech into text.

[1939] Step 2:

[1940] Evaluation of comments

[1941] Input: Text data obtained in step 1

[1942] Processing: The server uses a generative AI model (GPT-4) to evaluate whether the text data is correct.

[1943] Output: Evaluation result (correct / incorrect)

[1944] Specific operation: The server inputs text data such as "This part should be installed here" into GPT-4 and evaluates whether the statement is correct.

[1945] Step 3:

[1946] Producing the right information

[1947] Input: If the evaluation result obtained in step 2 is "wrong"

[1948] Processing: The server uses a generative AI model (GPT-4) to generate the correct information.

[1949] Output: Text data with correct information

[1950] Specific operation: The server makes GPT-4 generate the correct information, such as "That part should not be installed here, but in the adjacent position."

[1951] Step 4:

[1952] Statement of correct information

[1953] Input: Text data of the correct information obtained in step 3

[1954] Processing: The server communicates the correct information to the worker via voice through a speaker.

[1955] Output: Audio information

[1956] Specific operation: The server communicates to the worker through a speaker, "That part should be installed in the next position, not here."

[1957] Step 5:

[1958] emotion recognition

[1959] Input: Facial expression data of the worker

[1960] Processing: The camera captures the worker's facial expressions and analyzes their emotions using the Microsoft Azure Emotion API.

[1961] Output: Emotion data (e.g. confusion, relief, etc.)

[1962] Specific operation: The camera captures the worker's facial expression, which is then analyzed by the Microsoft Azure Emotion API and recognized as "confusion."

[1963] Step 6:

[1964] Generating information according to emotions

[1965] Input: Emotion data obtained in step 5

[1966] Processing: The server uses a generative AI model (GPT-4) to generate appropriate information based on the emotion.

[1967] Output: Text data of information according to emotions

[1968] Specific operation: The server asks GPT-4 to generate encouraging words such as, "It's okay, let's check it together."

[1969] Step 7:

[1970] Expressing information according to emotions

[1971] Input: Text data of emotion-related information obtained in step 6

[1972] Processing: The server communicates information corresponding to the emotion to the worker through a speaker.

[1973] Output: Audio information

[1974] Specific operation: The server communicates to the worker through a speaker, "It's OK, let's check it together."

[1975] Example 2

[1976] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1977] In conventional educational systems, it is difficult to immediately provide correct information when a learner gives a wrong answer, making it difficult for the learner to recognize their own mistake. In addition, it is not possible to provide an appropriate response according to the learner's emotions, which can reduce the learner's motivation to learn. To solve these problems, a system is needed that can accurately recognize the learner's utterances, generate information to correct the mistake, and provide a response according to the learner's emotions.

[1978] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1979] In this invention, the server includes a speech recognition means, a natural language processing means, a generative AI means, an emotion recognition means, a means for generating and uttering information for correcting an incorrect answer given by a learner, and a means for adjusting a response according to the learner's emotion. This makes it possible to accurately recognize the learner's utterance, provide information for immediately correcting the mistake, and further provide an appropriate response according to the learner's emotion.

[1980] "Speech recognition means" refers to a technology that acquires a learner's speech as voice data and converts it into text data.

[1981] "Natural language processing means" refers to technology that analyzes acquired text data and understands the content of statements.

[1982] "Generative AI means" is an artificial intelligence technology that generates appropriate information based on what the learner says.

[1983] "Emotion recognition means" is a technology that analyzes a learner's facial expressions and tone of voice to detect the learner's emotions.

[1984] "A means for generating and uttering information to correct an incorrect answer provided by a learner" is a technology that detects an incorrect answer provided by a learner and generates and utters correct information.

[1985] "Means of adjusting responses according to the learner's emotions" refers to techniques for generating and uttering appropriate responses based on the learner's emotions.

[1986] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1987] The present invention relates to an educational robot system for implementing a lesson in which students actively learn. A specific embodiment of this system will be described below.

[1988] System Configuration

[1989] This educational robot system includes the following main components:

[1990] 1. Speech recognition method: Acquire the learner's speech as voice data and convert it into text data. Specifically, use voice recognition software (e.g. Google Speech-to-Text).

[1991] 2. Natural language processing means: Analyze the acquired text data and understand the content of the speech. Specifically, natural language processing software (e.g. spaCy) is used.

[1992] 3. Generative AI means: Generate appropriate information based on the learner's utterances. Specifically, we use a generative AI model (e.g., OpenAI GPT-4).

[1993] 4. Emotion recognition: Analyze the learner's facial expressions and tone of voice to detect the learner's emotions. Specifically, use emotion recognition software (e.g. Microsoft Azure Emotion API).

[1994] 5. Information generation and speech means: When a learner gives an incorrect answer, generate and speak information to correct it. Specifically, use speech synthesis software (e.g. Amazon Polly).

[1995] 6. Emotion-based response adjustment: Generate and speak appropriate responses based on the learner's emotions.

[1996] System Operation

[1997] The device picks up what the learner says through a microphone and converts it into text data using voice recognition software. The server then receives the text data sent from the device and analyzes it using natural language processing software. Based on the analysis results, the server determines whether what the learner said is correct. For example, if the learner says, "The capital of Japan is Osaka," the server detects that this is incorrect.

[1998] The server uses the generative AI model to generate the correct information. Specifically, it generates the information, "That is incorrect. The capital of Japan is Tokyo." Next, the device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions. The detected emotion data is sent to the server.

[1999] The server uses an emotion engine to generate a response based on the learner's emotions. For example, if the learner expresses anger or dissatisfaction, the server uses a generative AI model to generate an apology or re-explanation such as "I'm sorry, I'll explain it again." Finally, the device converts the generated information into speech using speech synthesis software and speaks it to the learner through the speaker.

[2000] Examples

[2001] For example, if a learner says "The capital of Japan is Osaka," the system will operate as follows:

[2002] 1. The device picks up speech through the microphone and uses Google Speech-to-Text to convert it into text data such as "The capital of Japan is Osaka."

[2003] 2. The server receives the text data and analyzes it using spaCy.

[2004] 3. Based on the analysis results, the server detects that the statement "The capital of Japan is Osaka" is incorrect.

[2005] 4. The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[2006] 5. The device analyzes the learner's facial expressions and tone of voice using the Microsoft Azure Emotion API to detect when the learner is expressing dissatisfaction.

[2007] 6. The server uses an emotion engine to generate an apology such as "I'm sorry, I'll explain it again."

[2008] 7. The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[2009] Examples of prompt statements

[2010] "If a learner says, 'The capital of Japan is Osaka,' please generate the correct information."

[2011] "If the learner expresses anger or frustration, generate appropriate apologies or restatements."

[2012] The above is an embodiment of the invention.

[2013] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2014] Step 1:

[2015] The user makes a statement as a learner.

[2016] Input: Learner's spoken words

[2017] Output: Audio data

[2018] Specific action: The learner says, "The capital of Japan is Osaka."

[2019] Step 2:

[2020] The terminal captures the learner's voice data through a microphone and converts it into text data using voice recognition software.

[2021] Input: Audio data

[2022] Output: Text data

[2023] Specific operation: The device uses Google Speech-to-Text to generate text data such as "The capital of Japan is Osaka."

[2024] Step 3:

[2025] The server receives the text data sent from the terminal and analyzes it using natural language processing software.

[2026] Input: Text data

[2027] Output: Analysis results

[2028] Specific operation: The server uses spaCy to analyze the text data and understand the content of the statement, "The capital of Japan is Osaka."

[2029] Step 4:

[2030] The server uses the analysis results to determine whether the learner's statement is correct.

[2031] Input: Analysis results

[2032] Output: True or False

[2033] Specific operation: The server detects that the statement "The capital of Japan is Osaka" is false.

[2034] Step 5:

[2035] The server generates the correct information using a generative AI model.

[2036] Input: Correct / Incorrect

[2037] Output: Correct information

[2038] Specific operation: The server uses OpenAI GPT-4 to generate the correct information: "That is incorrect. The capital of Japan is Tokyo."

[2039] Step 6:

[2040] The device uses emotion recognition software to analyze the learner's facial expressions and tone of voice to detect the learner's emotions.

[2041] Input: Learner's facial expressions and tone of voice

[2042] Output: Emotion data

[2043] Specific behavior: The device uses the Microsoft Azure Emotion API to detect when a learner is expressing dissatisfaction.

[2044] Step 7:

[2045] The terminal transmits the detected emotion data to the server.

[2046] Input: Emotion data

[2047] Output: Emotion data (sent to server)

[2048] Specific operation: The terminal transmits emotion data to the server.

[2049] Step 8:

[2050] The server uses an emotion engine to generate a response according to the learner's emotion.

[2051] Input: Emotion data

[2052] Output: Appropriate response

[2053] What it does: The server uses a generative AI model to generate an apology saying, "I'm sorry, I'll explain it again."

[2054] Step 9:

[2055] The terminal uses speech synthesis software to convert the generated information into speech, which is then spoken to the learner through a speaker.

[2056] Input: appropriate response

[2057] Output: Audio data

[2058] Specific operation: The device uses Amazon Polly to convert the generated information into speech and speaks it to the learner through the speaker.

[2059] (Application example 2)

[2060] Next, a description will be given of Application Example 2 of Form Example 2. In the following description,...

Claims

1. A means for converting speech into text; A means for analyzing the converted text and determining whether information contained in the speech is correct or incorrect; A means for generating a prompt sentence for generating correct information when an error is included in the information included in the speech voice, and inputting the prompt sentence to a generation system AI; A means for outputting correct information generated by the generation system AI; A system including:

2. A means for converting speech into text; A means for analyzing the converted text and determining whether the information regarding the work procedure contained in the spoken voice is correct or not; a means for generating a prompt sentence for generating information showing a correct work procedure when the information on the work procedure contains an error, and inputting the prompt sentence to a generation system AI; A means for outputting information indicating a correct work procedure generated by the generation system AI; A system including:

3. The correct information generated by the generation system AI is converted into voice data and fed back to the speaker. A system according to claim 1 or claim 2.

Images