System

Abstract

To provide a system.SOLUTION: A system includes: means for a user to input a question; means for a terminal to recognize the question; means for the terminal to transmit the question to a server; means for the server to analyze the question; means for the server to generate an answer by using a generative AI model; means for the server to transmit the answer to the terminal; means for the terminal to present the answer to the user; a robot equipped with artificial intelligence; means for the robot to provide additional knowledge about exam study content; means for the robot to discuss with the user about a theme the user is interested in; and means for the robot to put forth its own opinion in addition to its answer.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, young people who use social media and elderly people with early-stage dementia are facing the challenges of declining thinking and academic ability, as well as accelerating dementia. [Means for solving the problem]

[0005] As a means to solve this problem, a robot equipped with artificial intelligence is provided. This robot provides further knowledge about the contents of exam preparation, etc., and engages in conversation and discussions on topics of interest. In addition, by adding its own opinions to the user's answers, the robot contributes to improving the user's thinking and academic abilities, and slowing the progression of dementia. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] As one embodiment of the present invention, a robot equipped with artificial intelligence is provided. This robot has a function of providing further knowledge about the contents of exam preparation, etc. For example, when a user is working on a mathematical problem, the robot can provide related mathematical theories and solutions.

[0029] "Example 2"

[0030] The robot also has the ability to engage in conversation and discussion on topics of interest to the user. For example, if the user brings up the topic of advances in science and technology, the robot can engage in a deeper discussion about the latest trends in science and technology and their impact.

[0031] "Example 3"

[0032] In addition, the robot has the ability to respond with its own opinions in addition to the user's own answers. This allows the user to deepen their own opinions. For example, if a user expresses an opinion about an environmental issue, the robot can provide additional information or perspectives on that opinion.

[0033] The process flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The user presents the robot with learning content (e.g. a math problem).

[0036] Step 2: The robot acquires knowledge about the relevant learning content from artificial intelligence.

[0037] Step 3: The robot provides the user with the knowledge it has acquired, providing concrete explanations of the relevant mathematical theories and solutions.

[0038] "Example 2"

[0039] Step 1: The user presents the robot with a topic of interest (e.g., advances in science and technology).

[0040] Step 2: The robot gets information about the relevant topic from artificial intelligence.

[0041] Step 3: Based on the information acquired by the robot, the robot will hold a discussion with the user, providing specific details about the latest trends in science and technology and their impact.

[0042] "Example 3"

[0043] Step 1: The user presents the robot with their opinion (e.g., their opinion on environmental issues).

[0044] Step 2: The robot obtains additional information and perspectives on the subject from artificial intelligence.

[0045] Step 3: The robot provides the user with additional information and perspectives that it has obtained, with specific explanations of the additional information and perspectives in response to the user's opinion.

[0046] Example 1

[0047] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0048] Conventional learning support systems have the problem that it is difficult to get an appropriate and prompt answer when a user asks a question about exam preparation or a specific topic. There are also only a limited number of systems that allow users to have in-depth discussions about topics that interest them. Furthermore, there is a lack of functionality for robots to have their own opinions and provide additional information to users.

[0049] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0050] In this invention, the server includes a means for a user to input a question, a means for the terminal to recognize the question, a means for the terminal to send the question to the server, a means for the server to analyze the question, a means for the server to generate an answer using a generative AI model, a means for the server to send the answer to the terminal, and a means for the terminal to present the answer to the user. This allows the user to get an appropriate and prompt answer when asking a question about studying for an exam or a specific topic. In addition, the robot can deeply discuss a topic that interests the user and provide additional information with its own opinion.

[0051] "Means for user input of question" refers to an interface through which a user can input a question by voice or text.

[0052] "Means by which the device recognizes the question" refers to speech recognition software that the device uses to convert the user's speech into text, or the device's ability to directly obtain text input.

[0053] "Means for the terminal to send a question to the server" refers to a communication function for sending text data recognized by the terminal to the server.

[0054] "Means for the server to analyze the question" refers to natural language processing technology used by the server to analyze the text data it receives and understand the intent of the question.

[0055] "Means for the server to generate an answer using a generative AI model" refers to the function of the server using a generative AI model to generate an optimal answer to a question.

[0056] "Means for the server to transmit a response to the terminal" refers to a communication function for transmitting a response generated by the server to the terminal.

[0057] "Means for the terminal to present the answer to the user" refers to an output function for presenting the answer received by the terminal to the user in the form of voice or text.

[0058] A "robot equipped with artificial intelligence" refers to a robot that has built-in artificial intelligence technology and is capable of interacting with users and providing them with information.

[0059] "Means for providing further knowledge about the contents of examination study, etc." refers to a function that provides related knowledge and solutions when a user asks a question related to examination study.

[0060] "Means of talking and discussing topics of interest" refers to the function of the robot to converse and discuss topics that interest the user.

[0061] "A means for the robot to respond with its own opinion in addition to its own answer" refers to a function in which the robot provides its own opinion or additional information in addition to the answer to the user's question.

[0062] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0063] This invention is a system that uses a robot equipped with artificial intelligence to provide appropriate and prompt answers to questions from users about studying for exams or specific topics. A specific embodiment of this system will be described below.

[0064] Hardware and Software Used

[0065] Hardware: High performance processor (e.g. Intel Core i7), camera, microphone, speakers

[0066] Software: Generative AI models (e.g., GPT-4®), speech recognition software (e.g., Google® Speech-to-Text), natural language processing libraries (e.g., spaCy)

[0067] System Configuration

[0068] 1. How the user enters their question:

[0069] The user inputs a question to the robot by voice or text. For example, the user may ask, "Teach me how to solve a quadratic equation."

[0070] 2. How the device recognizes the question:

[0071] The device uses a built-in microphone and speech recognition software to convert the user's speech into text, or, in the case of text input, to capture the text as is.

[0072] 3. How the device sends a query to the server:

[0073] The terminal transmits the recognized text data to the server. At this time, the data is encrypted before being transmitted.

[0074] 4. How the server parses the query:

[0075] The server analyzes the received text data and uses a natural language processing library to understand the intent of the question.

[0076] 5. How the server uses the generative AI model to generate an answer:

[0077] The server uses the generative AI model to generate optimal answers to questions, such as "How do I solve a quadratic equation?", generating a solution using a formula.

[0078] 6. The server sends the response to the terminal by:

[0079] The server then sends the generated response to the terminal. The data is also encrypted when sent.

[0080] 7. The means by which the terminal presents the answer to the user:

[0081] The device will then present the answer to the user either by voice or text, in which case the answer will be read out loud using a speaker.

[0082] Examples

[0083] For example, if a user asks the robot, "Teach me how to solve a quadratic equation," the system operates as follows.

[0084] User: "How do I solve a quadratic equation?"

[0085] Device: Capture audio through the microphone and convert it to text using Google Speech-to-Text: "Can you help me solve a quadratic equation?"

[0086] Terminal: Sends text data to the server.

[0087] Server: Use spaCy to analyze the text and understand the intent of the question.

[0088] Server: Uses GPT-4 to generate the answer, "To solve quadratic equations, use the formula. The formula is x = (-b ± √(b^2 - 4ac)) / 2a."

[0089] Server: Generates the answer and sends it to the device.

[0090] Device: Using the speaker, read to the user, "To solve quadratic equations, use the formula. The formula is x = (-b ± √(b^2 - 4ac)) / 2a."

[0091] Examples of prompt statements

[0092] "When a user asks how to solve a quadratic equation, explain how to solve it using the formula."

[0093] In this way, the robot supports the user's learning.

[0094] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0095] Step 1:

[0096] The user enters a question.

[0097] The user inputs a question to the robot by voice or text. For example, the user may ask, "Please tell me how to solve a quadratic equation." The input is voice or text data.

[0098] Step 2:

[0099] The device recognizes the question.

[0100] The terminal uses a built-in microphone and voice recognition software to convert the user's speech into text. In the case of text input, the terminal obtains the text as is. The input is voice data or text data, and the output is text data.

[0101] Step 3:

[0102] The terminal sends a question to the server.

[0103] The terminal transmits the recognized text data to the server. At this time, the data is encrypted before being transmitted. The input is text data, and the output is encrypted text data.

[0104] Step 4:

[0105] The server parses the question.

[0106] The server analyzes the received text data. It uses a natural language processing library to understand the intent of the question. The input is the encrypted text data, and the output is the analyzed intent of the question.

[0107] Step 5:

[0108] The server generates the answer using a generative AI model.

[0109] The server uses the generative AI model to generate the best answer to the question. For example, for the question "How do I solve a quadratic equation?", it generates a solution using the solution formula. The input is the parsed question intent, and the output is the generated answer.

[0110] Step 6:

[0111] The server sends the response to the terminal.

[0112] The server sends the generated answer to the terminal. The data is also encrypted when sent. The input is the generated answer, and the output is the encrypted answer.

[0113] Step 7:

[0114] The terminal presents the answer to the user.

[0115] The terminal presents the received answer to the user either by voice or text, in the latter case by reading the answer aloud using a speaker. The input is the encrypted answer, and the output is the presented answer by voice or text.

[0116] (Application example 1)

[0117] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0118] Conventional educational support robots were limited to providing knowledge on exam preparation or specific topics, and had the problem of not being able to support the education and training of workers in factories. In addition, since they could not provide real-time knowledge using generative AI models, it was difficult to immediately provide the latest information or specific operating procedures.

[0119] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0120] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to provide knowledge about new machine operation methods and manufacturing processes to workers in a factory, and a means for the robot to obtain and provide related knowledge based on a prompt sentence using a generative AI model. This makes it possible to provide the latest knowledge and specific operating procedures in real time during worker education and training in a factory.

[0121] A "robot equipped with artificial intelligence" is an autonomous mechanical device that incorporates artificial intelligence technologies such as machine learning and natural language processing, and is capable of interacting with users and providing them with knowledge.

[0122] "Means of providing further knowledge about the content of exam preparation, etc." is a function that provides additional information or explanations related to the study content the user is working on.

[0123] "A means to talk to and discuss topics of interest" is a function that enables users to engage in dialogue, exchange opinions, and debate on topics that interest them.

[0124] "A means for the robot to respond with its own opinion by adding it to the user's answer" is a function that enables the robot to respond to the user's question by adding its own knowledge and opinions.

[0125] "A means of providing workers in a factory with knowledge about how to operate new machines and manufacturing processes" is a function that provides workers in a factory with information about how to use new machines and manufacturing processes.

[0126] "Means for obtaining and providing relevant knowledge based on a prompt sentence using a generative AI model" refers to a function that uses a generative AI model to generate relevant information based on a prompt sentence entered by a user and provides it to the user.

[0127] The system for implementing this invention is centered around a robot equipped with artificial intelligence. The robot has the function of providing knowledge in various scenarios, such as studying for exams or educating workers in a factory.

[0128] System Configuration

[0129] The system consists of the following main components:

[0130] 1. Robots equipped with artificial intelligence: Equipped with machine learning and natural language processing technology, they can converse with users and provide them with knowledge.

[0131] 2. Generative AI model: Generates relevant knowledge based on the prompt sentence entered by the user.

[0132] 3. Database: Stores information about training content and operating procedures within the factory.

[0133] Hardware and Software Used

[0134] Hardware: Robot body, server, user device (smartphone, tablet, etc.)

[0135] Software: Python, OpenAI® API, Database Management System

[0136] Data processing and calculation

[0137] The server sends the prompt sentence entered by the user to the generative AI model to obtain related knowledge, which is then provided to the user through the robot.

[0138] Examples

[0139] For example, if a worker in a factory wants to learn how to operate a new welding machine, the system works as follows.

[0140] 1. User input: A worker says to the robot, "Please teach me the basic principles of welding."

[0141] 2. Prompt generation: The server sends this input as a prompt to the generative AI model.

[0142] Example prompt: "Please provide a detailed explanation of the basic principles of welding."

[0143] 3. Knowledge Acquisition: The generative AI model generates relevant knowledge based on the prompt sentence and returns it to the server.

[0144] 4. Knowledge provision: The server transmits the acquired knowledge to the robot, and the robot provides the knowledge to the worker via voice or display.

[0145] In this way, the system is able to provide up-to-date knowledge and specific operating procedures in real time.

[0146] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0147] Step 1:

[0148] The user inputs a question to the robot. For example, a worker might say, "Please tell me the basic principles of welding." This input is converted into text data through a voice recognition system.

[0149] Step 2:

[0150] The terminal generates a prompt sentence based on the user's input, for example, "Please provide a detailed explanation of the basic principles of welding." This prompt sentence is then sent to the generative AI model.

[0151] Step 3:

[0152] The server sends a prompt sentence to the generative AI model, which generates relevant knowledge based on the prompt sentence. This knowledge is returned to the server as text data.

[0153] Step 4:

[0154] The server receives the generated knowledge and matches it with related information stored in a database, retrieving additional information from the database if necessary.

[0155] Step 5:

[0156] The server transmits the final knowledge data to the robot, which converts the data into a format suitable for providing the data to the user, such as voice data or text data for display.

[0157] Step 6:

[0158] The robot provides the knowledge data received from the server to the user, either by explaining it aloud using a speech synthesis system or by displaying text on a display.

[0159] Step 7:

[0160] The user receives the knowledge provided by the robot and uses it to learn or work, and if necessary, can ask further questions to gain additional knowledge.

[0161] Example 2

[0162] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0163] Conventional robot systems have the problem that it is difficult to have in-depth discussions on topics that interest the user. In addition, there is a lack of technology to accurately recognize the user's voice input and generate appropriate responses. Furthermore, there is a need for a system that can speak the generated responses in a natural voice.

[0164] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting a user's voice input into text data using voice recognition software, a means for inputting a prompt sentence into a generative AI model and generating an appropriate response, and a means for converting the generated text response into voice data using voice synthesis software. This enables users to have in-depth discussions on topics that interest them.

[0165] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[0166] A "robot" is a mechanical device equipped with artificial intelligence and capable of communicating with humans through physical movements and dialogue.

[0167] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass a particular exam.

[0168] A "knowledge delivery medium" is a method or technique for conveying information or data to a user.

[0169] "Means of discussion" are methods and technologies for exchanging opinions and sharing information through dialogue with users.

[0170] "Speech recognition software" is a program for converting voice data into text data.

[0171] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses based on input text data.

[0172] A "prompt sentence" is a piece of text data input into a generative AI model and is an instruction sentence for generating a response.

[0173] "Speech synthesis software" is a program for converting text data into voice data.

[0174] "Audio data" refers to data that represents audio information in digital form.

[0175] The "means for uttering" refers to a method or technology for playing back the generated voice data through a speaker and conveying information to the user by voice.

[0176] This invention is a system for allowing a user to discuss with a robot a topic that the user is interested in. This system uses a robot equipped with artificial intelligence to carry out a series of processes, including recognizing the user's voice input, generating an appropriate response, and speaking the response aloud.

[0177] Hardware and Software Used

[0178] Hardware: The robot itself (e.g. a typical interactive robot)

[0179] Software: speech recognition software (e.g., a general speech recognition program), generative AI models (e.g., a general generative AI algorithm), voice synthesis software (e.g., a general voice synthesis program)

[0180] System Operation

[0181] 1. User voice input:

[0182] The user talks to the robot, for example, saying, "Tell me about recent advances in science and technology."

[0183] The robot uses a built-in microphone to capture the user's voice.

[0184] 2. Speech to text conversion:

[0185] The terminal (robot) sends the captured voice data to the voice recognition software.

[0186] The server uses voice recognition software to convert the voice data into text data.

[0187] The terminal receives the converted text data.

[0188] 3. Use of generative AI models:

[0189] The device inputs the received text data into the generative AI model.

[0190] The server sends prompts to the generative AI model, which then generates an appropriate response.

[0191] 4. Generate the response:

[0192] The server receives the response from the generative AI model.

[0193] The terminal receives the generated text response from the server.

[0194] 5. Response transcription:

[0195] The terminal sends the received text response to the speech synthesis software.

[0196] The server uses speech synthesis software to convert the text data into voice data.

[0197] The terminal receives the converted voice data.

[0198] 6. Robot response utterance:

[0199] The terminal transmits the received voice data to the robot's speaker.

[0200] The robot speaks the generated voice to the user.

[0201] Examples

[0202] User: "Tell me about recent advances in science and technology."

[0203] Robot: "A recent advance in science and technology is the development of quantum computing. Quantum computers have the potential to rapidly solve problems that cannot be solved by conventional computers."

[0204] Examples of prompt statements

[0205] Please tell me about recent advances in science and technology.

[0206] "Let's discuss the latest developments in AI technology."

[0207] "Tell me about the future of robotics."

[0208] In this way, users can deepen discussions with the robot about topics of interest.

[0209] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0210] Step 1:

[0211] A user speaks to the robot, for example, "Tell me about recent advances in science and technology." The robot uses a built-in microphone to capture the user's voice. The input is the user's voice, and the output is the voice data.

[0212] Step 2:

[0213] The terminal (robot) sends the captured voice data to the voice recognition software. The server uses the voice recognition software to convert the voice data into text data. The input is voice data, and the output is text data. In concrete terms, the voice recognition software analyzes the voice waveform and generates the corresponding text.

[0214] Step 3:

[0215] The terminal receives the converted text data and inputs it into the generative AI model. The server sends a prompt sentence to the generative AI model and generates an appropriate response. The input is the text data, and the output is the generated response text. In concrete terms, the generative AI model analyzes the prompt sentence and generates a response based on related information.

[0216] Step 4:

[0217] The server receives the response from the generative AI model and sends it to the terminal. The terminal receives the generated text response from the server. The input is the generated response text, and the output is the same response text. In specific operations, the server receives the response from the generative AI model and forwards it to the terminal.

[0218] Step 5:

[0219] The terminal sends the received text response to the speech synthesis software. The server uses the speech synthesis software to convert the text data into speech data. The input is text data and the output is speech data. In concrete terms, the speech synthesis software analyzes the text and generates a corresponding speech waveform.

[0220] Step 6:

[0221] The terminal receives the converted voice data and transmits it to the robot's speaker. The robot speaks the generated voice to the user. The input is the voice data, and the output is a voice response to the user. In concrete terms, the robot's speaker plays the voice data and conveys information to the user.

[0222] (Application example 2)

[0223] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0224] Conventional robots are limited to specific tasks and simple dialogue, making it difficult to improve work efficiency through real-time communication and discussion with workers. In addition, there were no robots that could deepen discussions about work progress and problems. For this reason, there is a demand for improving work efficiency and speeding up problem solving in factories.

[0225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0226] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, and a means for the robot to communicate with workers in real time to improve work efficiency and discuss work progress and problems, thereby making it possible to improve work efficiency in a factory and speed up problem solving.

[0227] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[0228] A "robot" is a mechanical device that operates autonomously according to programmed instructions to perform specific tasks.

[0229] "Studying for exams" refers to learning activities aimed at acquiring the knowledge and skills necessary to pass a particular exam.

[0230] A "means of providing knowledge" is a method or device for conveying information or data to users and deepening their understanding.

[0231] A "theme" is a topic or issue that is the center of discussion or research.

[0232] "Means for conversation and discussion" refers to methods or devices for engaging in dialogue with users and deepening discussion through the exchange of opinions and sharing of information.

[0233] "Means of responding with opinions" refers to a method or device that enables a robot to express its own views or thoughts in response to a user's questions or opinions.

[0234] "Work efficiency" is an index for optimizing the effort and time required to perform a specific task.

[0235] "Means of real-time communication" refers to methods or devices for instantly exchanging information with users and responding quickly.

[0236] "Work progress" is a state or condition that indicates how far a particular piece of work has progressed.

[0237] An "issue" is an obstacle or challenge that occurs in a task or process.

[0238] A "means of discussion" is a method or device for exchanging opinions and sharing information on a particular topic and deepening understanding.

[0239] The system for carrying out the present invention is mainly composed of a robot equipped with artificial intelligence. A specific embodiment of this system will be described below.

[0240] Hardware used

[0241] Robot body: A mechanical device that operates autonomously and communicates with workers (e.g., an articulated robot arm).

[0242] Speech recognition microphone: A device that recognizes the worker's voice and converts it into text (e.g., a microphone compatible with the Google Cloud Speech-to-Text API).

[0243] Display: A device for displaying visual information, if required.

[0244] Software used

[0245] Speech recognition software: Software for converting speech into text (e.g. Google Cloud Speech-to-Text API).

[0246] Generative AI model: An artificial intelligence model for generating appropriate responses to user questions or arguments (e.g., OpenAI’s GPT-4).

[0247] Natural Language Processing Libraries: Libraries for performing text analysis (e.g. NLTK, spaCy).

[0248] Database: A database (e.g. MySQL, PostgreSQL) for storing generated data and user history.

[0249] System Operation

[0250] 1. Voice input: The worker speaks to the robot.

[0251] 2. Speech Recognition: Speech recognition software converts speech into text.

[0252] 3. Text Analysis: Use natural language processing libraries to analyze text and understand user intent.

[0253] 4. Input to the generative AI model: Input a prompt to the generative AI model based on the analysis results.

[0254] 5. Response generation: The generative AI model generates an appropriate response.

[0255] 6. Voice output: The generated response is converted into voice using speech synthesis software and sent back to the worker.

[0256] Examples

[0257] When a worker asks, "What do you think about the efficiency of the new production line?" the system works as follows:

[0258] Worker: "What do you think about the efficiency of the new production line?"

[0259] Robot: "The latest production technology is a quality control system that utilizes AI. This allows for real-time detection of defective products, greatly improving production efficiency."

[0260] Examples of prompt statements

[0261] A user asks, "What do you think about the efficiency of our new production line?" Generate responses that foster discussion about the latest production technologies and efficiency methods.

[0262] In this way, robots can communicate with workers in real time and discuss the progress of work and problems, which will improve work efficiency and speed up problem solving in the factory.

[0263] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0264] Step 1:

[0265] Audio input

[0266] The user talks to the robot.

[0267] Input: User's voice

[0268] Output: Audio data

[0269] Specific Action: A user asks, "What do you think about the efficiency of our new production line?"

[0270] Step 2:

[0271] Speech Recognition

[0272] The server uses voice recognition software to convert the voice data into text.

[0273] Input: Audio data

[0274] Output: Text data

[0275] What happens: Speech recognition software (e.g. Google Cloud Speech-to-Text API) analyzes the voice data and generates text such as, "What do you think about the efficiency of our new production line?"

[0276] Step 3:

[0277] Text Analytics

[0278] The server uses a natural language processing library to analyze the text data and understand the user's intent.

[0279] Input: Text data

[0280] Output: Analysis results (user intent)

[0281] What happens: A natural language processing library (e.g. NLTK, spaCy) analyzes the text data and understands that the user is looking for information on improving production line efficiency.

[0282] Step 4:

[0283] Input to generative AI models

[0284] The server inputs prompts into the generative AI model based on the analysis results.

[0285] Input: Analysis results (user intent)

[0286] Output: Prompt statement

[0287] Specific operation: The server inputs the prompt sentence "The user asked, 'What do you think about the efficiency of the new production line?' Please generate a response that deepens the discussion about the latest production technologies and methods of improving efficiency" into the generative AI model.

[0288] Step 5:

[0289] Response Generation

[0290] A generative AI model generates an appropriate response based on the prompt.

[0291] Input: Prompt statement

[0292] Output: Response text

[0293] How it works: A generative AI model (e.g., OpenAI's GPT-4) analyzes the prompt sentence and generates a response text that reads, "As the latest production technology, there is a quality control system that utilizes AI. This allows defective products to be detected in real time, greatly improving production efficiency."

[0294] Step 6:

[0295] Audio Output

[0296] The server converts the generated response text into speech using speech synthesis software and responds to the user.

[0297] Input: Response text

[0298] Output: Audio data

[0299] Specific operation: The speech synthesis software converts the response text into voice data, and the robot replies to the user, "As the latest production technology, we have a quality control system that utilizes AI. This allows us to detect defective products in real time, greatly improving production efficiency."

[0300] Example 3

[0301] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0302] Conventional robot systems equipped with artificial intelligence only provide information in response to user opinions, and it is difficult to provide additional information or perspectives to deepen the user's opinions. In addition, it is not possible to provide appropriate feedback in real time to the opinions entered by the user, and it is not possible to fully support the user's learning or deepening of discussions.

[0303] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0304] In this invention, the server includes a means for a user to input an opinion, a means for a terminal to transmit the opinion to the server, a means for the server to input the opinion as a prompt sentence to the generative AI model, a means for the generative AI model to generate additional information and viewpoints, a means for the server to transmit the generated information to the terminal, and a means for the terminal to display the information to the user. This makes it possible to provide appropriate feedback to the user's opinion in real time and support the deepening of user learning and discussion.

[0305] "User" means an individual or entity that utilizes the System to input opinions and receive feedback.

[0306] A "terminal" is a device through which a user inputs opinions and communicates with the server, and includes a personal computer, smartphone, etc.

[0307] The "server" is a computer system that receives opinions from users, inputs them as prompt sentences into the generative AI model, and transmits the generated information to the terminal.

[0308] A "generative AI model" is an artificial intelligence model that generates additional information or perspectives based on an input prompt sentence, and includes, for example, models that use natural language processing techniques.

[0309] A "prompt" is a text-based instruction that is input to a generative AI model based on the user's input.

[0310] The "means for displaying information" is a function for visually presenting to the user the information that the terminal receives from the server.

[0311] The present invention is a system in which a user inputs an opinion and provides additional information or perspective to the opinion using a generative AI model. A specific embodiment of the system will be described below.

[0312] First, the user inputs his / her opinion using a terminal (e.g., a PC or smartphone). The opinion input by the user is sent as text data from the terminal to the server. The server converts the received opinion into a prompt sentence. An example of a prompt sentence is "User's opinion: The use of renewable energy should be increased. Please provide additional information or perspectives."

[0313] Next, the server inputs the prompt sentence into a generative AI model (for example, a model using natural language processing technology). The generative AI model generates additional information and perspectives based on the prompt sentence. For example, it might generate content such as, "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar and wind power generation are attracting attention as clean energy sources that do not rely on fossil fuels. In addition, costs are gradually falling due to technological advances in renewable energy."

[0314] The generated information is sent from the server to the terminal. The terminal displays the received information to the user. The user can view this information and deepen his / her opinion.

[0315] In implementing this system, the following hardware and software are used:

[0316] Devices: PCs, smartphones, etc.

[0317] Server: A computer system that receives opinions, generates prompts, inputs them into the generative AI model, and transmits the generated information.

[0318] Generative AI model: An artificial intelligence model that uses natural language processing techniques (e.g., GPT-4)

[0319] As a concrete example, consider the case where a user expresses the opinion that "reducing plastic waste is important." This opinion is input as a prompt sentence to the generative AI model:

[0320] User opinion: Reducing plastic waste is important. Please provide additional information or perspective.

[0321] A generative AI model might generate additional information such as:

[0322] "Reducing plastic waste also helps protect marine ecosystems. Advances in recycling technology are also facilitating the reuse of plastics."

[0323] In this way, the user can obtain new perspectives and information on his / her own opinions. This system supports the user's learning and deepening of discussion. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0324] Step 1:

[0325] The user inputs an opinion.

[0326] The user uses a device (PC or smartphone) to input their opinion into a text box. For example, they might type, "The use of renewable energy should increase." The input opinion is saved as text data on the device.

[0327] Step 2:

[0328] The terminal transmits the opinion to the server.

[0329] The terminal sends the opinions entered by the user to the server as an HTTP request. The input data is sent in text format. The server receives this request and retrieves the opinion data.

[0330] Step 3:

[0331] The server inputs the opinion into the generative AI model as a prompt sentence.

[0332] The server converts the received opinion into a prompt, for example, in the form of "User opinion: The use of renewable energy should be increased. Please provide additional information or perspectives." This prompt is then fed into a generative AI model.

[0333] Step 4:

[0334] Generative AI models generate additional information and perspectives.

[0335] A generative AI model (e.g., a model using natural language processing technology) generates additional information and perspectives based on the prompt sentence. For example, it generates content such as, "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar power generation and wind power generation are attracting attention as clean energy sources that do not depend on fossil fuels. In addition, due to technological advances in renewable energy, costs are gradually decreasing." The generated information is returned to the server as text data.

[0336] Step 5:

[0337] The server transmits the generated information to the terminal.

[0338] The server receives the information obtained from the generative AI model and sends it to the terminal as an HTTP response. The data sent is in text format.

[0339] Step 6:

[0340] The terminal displays the information to the user.

[0341] The terminal displays the information received from the server on the screen. The user can view this information to further develop their own opinion. For example, the following information may be displayed: "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar and wind power generation are attracting attention as clean energy sources that do not depend on fossil fuels. Furthermore, technological advances in renewable energy are gradually reducing costs."

[0342] (Application example 3)

[0343] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0344] Conventional robot systems simply provide responses to user opinions, and lack the ability to provide additional information or perspectives to deepen the user's opinions. In addition, there are limited ways to provide detailed information about products in physical stores, making it difficult for users to obtain sufficient information when selecting products. This creates an issue of limiting the user's purchasing experience.

[0345] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0346] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics of interest to the user, a means for the robot to reply with its own opinion in addition to the user's reply, a means for the robot to provide additional information and viewpoints about products in a physical store, a means for the robot to obtain the user's opinion and generate additional information using a generative AI model, and a means for the robot to present the generated additional information to the user. This allows the user to gain a deeper understanding of the product, improving the purchasing experience.

[0347] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[0348] A "robot" is a mechanical device equipped with artificial intelligence that performs specific tasks automatically.

[0349] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[0350] A "knowledge medium" is a method or technology for conveying information or data to a user.

[0351] A "theme" is a subject matter about a particular topic or issue.

[0352] "Means of discussion" are methods and techniques for interacting with users and exchanging opinions.

[0353] "Means of responding with opinions" refers to methods or technologies that allow a robot to provide its own views in response to a user's questions or opinions.

[0354] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[0355] "Means of providing additional information or perspectives on a product" are methods or technologies that provide more detailed information or a different perspective on the product selected by the user.

[0356] "Means of obtaining user opinions" are the methods and technologies used to collect opinions and questions expressed by users.

[0357] A "generative AI model" is an algorithm or system that enables artificial intelligence to generate new information or answers based on data.

[0358] "Means for generating additional information" are methods or technologies for generating new information or perspectives based on user opinions.

[0359] "Means of presentation to the user" refers to the methods or techniques used to visually or audibly convey the generated information to the user.

[0360] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[0361] System Configuration

[0362] The system consists of an artificial intelligence-enabled robot, smart glasses, an Internet connection, and a server using a generative AI model. Specifically, the system uses the following hardware and software:

[0363] Hardware:

[0364] Smart glasses (e.g. Google Glass (registered trademark))

[0365] Internet connection

[0366] software:

[0367] OpenAI API (generative AI model)

[0368] Python Program

[0369] Processing flow

[0370] 1. Get user feedback:

[0371] Users express their opinions about products through the smart glasses, for example by asking, "Is this product environmentally friendly?"

[0372] 2. Opinion text:

[0373] The smart glasses convert the user's comments into text using voice recognition technology.

[0374] 3. Send to generative AI model:

[0375] The server sends the user's opinion, converted into text, to the OpenAI API. An example of a prompt is as follows:

[0376] What users say about their choice: Is this product eco-friendly?

[0377] Please provide any additional information or perspective.

[0378] 4. Generate additional information:

[0379] The OpenAI API generates additional information and perspectives based on user feedback, such as "This product is made from renewable materials, has a low carbon footprint during manufacturing, and is more energy efficient than other products in the same category."

[0380] 5. User Representation:

[0381] The server transmits the generated additional information to the smart glasses, which present it visually or audibly to the user.

[0382] Examples

[0383] When a user is choosing a product in a physical store and asks through the smart glasses, "Is this product environmentally friendly?", the system works as follows:

[0384] 1. User opinions are converted into text using voice recognition technology.

[0385] 2. Send the converted text to the OpenAI API.

[0386] 3. The OpenAI API generates additional information, such as, "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category."

[0387] 4. The generated information is presented to the user through smart glasses.

[0388] In this way, users can gain a deeper understanding of the products and improve their purchasing experience.

[0389] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0390] Step 1:

[0391] Users express their opinions about products through smart glasses.

[0392] Input: User's spoken opinion (e.g. "Is this product environmentally friendly?")

[0393] Output: Audio data

[0394] What happens: A user speaks a question about a product into the smart glasses.

[0395] Step 2:

[0396] The terminal uses voice recognition technology to convert the user's comments into text format.

[0397] Input: Audio data

[0398] Output: Text data (e.g. "Is this product environmentally friendly?")

[0399] How it works: The voice recognition software built into the smart glasses analyzes the voice data and generates corresponding text data.

[0400] Step 3:

[0401] The server sends the converted user opinions to the generative AI model.

[0402] Input: Text data

[0403] Output: Prompt statement (e.g., "What do you think about the product the user selected: Is this product environmentally friendly? Please provide any additional information or perspective.")

[0404] Specific operation: The server receives the text data and generates a prompt sentence appropriate for the generative AI model.

[0405] Step 4:

[0406] A generative AI model generates additional information and perspectives based on user input.

[0407] Input: prompt statement

[0408] Output: Additional information (e.g. "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category.")

[0409] What it does: The OpenAI API parses the prompt and generates additional relevant information and perspectives.

[0410] Step 5:

[0411] The server transmits the generated additional information to the smart glasses for presentation to the user.

[0412] Input: Additional information

[0413] Output: Information presented to the user (e.g., "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category.")

[0414] Specific operation: The server sends the generated additional information to the smart glasses, which then present the information to the user visually or audibly.

[0415] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0416] "Example 1"

[0417] As one embodiment of the present invention, a system is provided in which a robot equipped with artificial intelligence includes an emotion engine that recognizes the emotion of a user. This emotion engine estimates the emotion of the user from the user's facial expression, tone of voice, and the content of the remarks, and reflects the result in the robot's response. For example, if the robot feels that the user is struggling with learning, it responds by offering words of encouragement or changing the learning content to increase motivation. Also, if the robot feels that the user is excited and arguing, it responds by sharing that excitement and encouraging further discussion.

[0418] "Example 2"

[0419] As another embodiment of the present invention, a system is provided for young people who use SNS. In this system, an emotion engine estimates the user's emotion from the content of posts on the SNS. For example, if a user posts on the SNS that they are feeling stressed, the robot responds by encouraging the user based on that information. Also, if a user posts on the SNS that they are excited and having an argument, the robot responds by deepening the discussion with the user based on that information.

[0420] "Example 3"

[0421] As yet another embodiment of the present invention, a system is provided for elderly people with early dementia. In this system, an emotion engine estimates the emotion of the elderly person from their daily facial expressions and tone of voice, and reflects the result in the robot's response. For example, if it is estimated that the elderly person feels lonely, the robot will become a conversation partner and respond in a way that reduces the elderly person's sense of loneliness. Also, if it is estimated that the elderly person is excited about something, the robot will share that excitement and respond in a way that adds vitality to the elderly person's life.

[0422] The process flow of each embodiment will be described below.

[0423] "Example 1"

[0424] Step 1: The robot initiates a dialogue with the user.

[0425] Step 2: The emotion engine infers emotions from the user's facial expressions, tone of voice, and content of statements.

[0426] Step 3: The robot responds based on the estimated emotions. For example, if the robot senses that the user is struggling with their learning, it can offer encouraging words or change the learning content to increase motivation.

[0427] "Example 2"

[0428] Step 1: The robot analyzes the user's social media posts.

[0429] Step 2: The emotion engine infers the user's emotions from the content of their social media posts.

[0430] Step 3: The robot responds based on the estimated emotions. For example, if a user posts on social media that they are feeling stressed, the robot can respond by encouraging the user based on that information.

[0431] "Example 3"

[0432] Step 1: The robot initiates a dialogue with the elderly person.

[0433] Step 2: The emotion engine estimates the elderly person's emotions from their everyday facial expressions and tone of voice.

[0434] Step 3: The robot responds based on the estimated emotions. For example, if it estimates that an elderly person is feeling lonely, the robot will engage in conversation and respond in a way that reduces the elderly person's sense of loneliness.

[0435] Example 1

[0436] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0437] Conventional learning support robots could not recognize the user's emotions and change their response accordingly, making it difficult to effectively improve the user's motivation to learn. In addition, they could not respond appropriately when the user was confused or excited, which could reduce the efficiency of learning. Furthermore, there was a demand for a system that could provide appropriate answers and solutions to the user's questions and problems.

[0438] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0439] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam study, etc., a means for the robot to talk to and discuss topics that interest the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to include an emotion engine that recognizes the user's emotions, a means for the emotion engine to estimate the user's emotions from the user's facial expressions, tone of voice, and remarks and reflect the results in the robot's response, a means for the robot to provide answers or solutions to the user's questions or problems using a generative AI model, and a means for the robot to generate a response according to the user's emotions. This makes it possible to respond appropriately according to the user's emotions and improve their motivation to learn.

[0440] A "robot equipped with artificial intelligence" is a robot that incorporates artificial intelligence technologies such as machine learning and natural language processing, and can converse with users and provide them with information.

[0441] "Means of providing further knowledge about the content of exam preparation, etc." refers to a function that provides additional information or explanations related to the content the user is studying.

[0442] "A means of talking and discussing topics of interest" refers to a function that enables users to have dialogue, exchange opinions, and discuss topics that interest them.

[0443] "A means for the robot to respond with its own opinion in addition to its own answer" refers to a function in which the robot provides its own views and opinions in addition to the answer to the user's question.

[0444] An "emotion engine" refers to technology for analyzing and inferring emotions from a user's facial expressions, tone of voice, content of statements, etc.

[0445] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers or solutions based on user input.

[0446] "Means for generating a response according to the user's emotions" refers to a function for generating a response appropriate to the user's emotions based on the analysis results of the emotion engine.

[0447] The present invention relates to a system for assisting a user in learning and recognizing emotions by using a robot equipped with artificial intelligence. A specific embodiment of this system will be described below.

[0448] Hardware and Software Configuration

[0449] server

[0450] The server hosts an artificial intelligence model and an emotion engine. The artificial intelligence model uses a general generative AI model. The emotion engine uses a general emotion analysis API, which can analyze user input data and generate appropriate answers or solutions.

[0451] Terminal (robot)

[0452] The terminal is responsible for receiving input from the user and sending it to the server. The terminal is equipped with a touch screen and voice input capabilities, allowing the user to input questions and problems in a natural way. The terminal also has a display and speakers to provide the user with responses received from the server.

[0453] Data processing and calculation

[0454] Sending user input data

[0455] When a user inputs a question or problem into the terminal, the terminal sends the data in text format to the server. For example, if a user inputs "Please tell me how to solve a quadratic equation," the text data is sent to the server.

[0456] Analysis using artificial intelligence models

[0457] The server uses a generative AI model to analyze input data. For example, in response to the input "Teach me how to solve quadratic equations," the generative AI model generates a solution to the quadratic equation. Specifically, the server generates the answer, "There are two ways to solve quadratic equations: completing the square and factoring. For example, if x^2 + 5x + 6 = 0, using the factorization method results in (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3."

[0458] Emotion analysis using emotion engine

[0459] The server sends the user's facial expression, tone of voice, and the content of the speech to the emotion analysis API, which analyzes the user's emotions. For example, if the user is judged to be confused, the result is returned to the server.

[0460] Generating a response

[0461] The server generates an appropriate response based on the answer of the generative AI model and the analysis results of the emotion engine. For example, if the server determines that the user is confused, it adds an encouraging message. Specifically, it generates a message saying, "If there's anything you don't understand, please feel free to ask, and I'll explain other ways too."

[0462] Providing a response

[0463] The server sends the generated response to the terminal as text data. The terminal displays the response received from the server on the screen and reads it out loud. For example, it displays and reads out, "There are two methods for solving quadratic equations: completing the square and factorization. To give a specific example, if x^2 + 5x + 6 = 0, using the factorization method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3. If you don't understand something, feel free to ask, as I will explain other methods as well."

[0464] Examples

[0465] For example, if a user types "How do I solve a quadratic equation?", the following prompt is generated:

[0466] User: How do I solve a quadratic equation?

[0467] Robot: There are several ways to solve quadratic equations, such as completing the square or factoring. For example, if x^2 + 5x + 6 = 0, using factoring gives (x + 2)(x + 3) = 0, which gives the solutions x = -2 and x = -3. Let me know if you have any problems.

[0468] In this way, a system is realized that supports the user's learning and responds according to their emotions.

[0469] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0470] Step 1:

[0471] The user types a question or problem into the terminal.

[0472] Specific action: The user uses the robot's touchscreen or voice input function to input, "Teach me how to solve a quadratic equation."

[0473] Input: The user's question or problem (e.g., "Can you help me solve a quadratic equation?")

[0474] Output: User input as text data

[0475] Step 2:

[0476] The terminal transmits the input data to the server.

[0477] Specific operation: The terminal transmits the user's input as text data to the server, for example, by using an HTTP request.

[0478] Input: User input as text data

[0479] Output: Text data sent to the server

[0480] Step 3:

[0481] The server uses an AI model to analyze the input data and generate an answer or solution.

[0482] Specific operation: The server calls the generative AI model and gives it the prompt "Teach me how to solve a quadratic equation" as input. The generative AI model generates a solution to the quadratic equation.

[0483] Input: User input as text data

[0484] Output: Generated answers or solutions (e.g. "There are two methods for solving quadratic equations: completing the square and factoring. For example, if x^2 + 5x + 6 = 0, using the factoring method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3.")

[0485] Step 4:

[0486] The server uses an emotion engine to analyze the user's emotions.

[0487] Specific operation: The server sends the user's facial expression, tone of voice, and the content of the statement to the emotion analysis API, which analyzes the emotion. For example, if the user is judged to be confused, the result is returned to the server.

[0488] Input: User facial expressions, tone of voice, and what is being said

[0489] Output: Parsed emotion data (e.g. confused)

[0490] Step 5:

[0491] The server generates an appropriate response based on the analysis results.

[0492] Specific operation: In addition to the answer of the generative AI model, the server adds an encouraging message based on the results of the emotion engine, for example, "If you don't understand something, please feel free to ask me, I will explain other ways too."

[0493] Input: Generated answers, solutions, and analyzed emotion data

[0494] Output: Appropriate response (e.g., "There are two ways to solve quadratic equations: completing the square and factoring. For example, if x^2 + 5x + 6 = 0, using the factoring method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3. If you don't understand something, feel free to ask, I'll explain other methods too.")

[0495] Step 6:

[0496] The server generates a response and sends it to the terminal.

[0497] Specific operation: The server sends the generated response to the terminal as text data, for example, as an HTTP response.

[0498] Input: Appropriate response

[0499] Output: Text data sent to the terminal.

[0500] Step 7:

[0501] The terminal provides the response to the user.

[0502] Specific operation: The terminal displays the response received from the server on the screen and reads it out loud. For example, "There are two methods for solving quadratic equations: completing the square and factorization. To give a specific example, if x^2 + 5x + 6 = 0, using the factorization method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3. If you don't understand something, feel free to ask, as I will explain other methods as well." is displayed and read out loud.

[0503] Input: Text data sent to the terminal

[0504] Output: The response provided to the user (both displayed on screen and read aloud)

[0505] (Application example 1)

[0506] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0507] Conventional factory robots were unable to recognize the emotions and conditions of workers, making it difficult for them to provide appropriate support to maximize work efficiency. In addition, they were unable to provide appropriate advice or suggest breaks when workers faced difficult tasks or felt fatigued. This resulted in problems of reduced work efficiency and increased stress for workers.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0509] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam study etc., a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to recognize the worker's emotions and provide appropriate support, a means for the robot to recognize the voice of the worker and perform emotion analysis, and a means for the robot to provide advice to improve work efficiency based on the worker's emotions. This makes it possible to recognize the worker's emotions and state in real time and provide appropriate support and advice.

[0510] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and perform functions such as learning, reasoning, and problem-solving.

[0511] A "robot" is a mechanical device that operates autonomously according to programmed instructions to perform specific tasks.

[0512] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[0513] "Knowledge delivery means" are methods and techniques for presenting users with the information and solutions they need.

[0514] "Discussion tools" are methods and techniques for interacting with users and exchanging opinions on a particular topic.

[0515] "Means of responding" refers to the methods and technologies that a robot uses to communicate its views and suggestions to the user.

[0516] "Means of recognizing emotions" refers to methods or technologies for inferring emotions from a user's facial expressions, tone of voice, content of speech, etc.

[0517] "Means of providing support" refers to methods and technologies for providing appropriate advice and assistance based on the user's condition and needs.

[0518] A "means for recognizing speech" is a method or technology for converting a user's speech into text and understanding its content.

[0519] "Means for performing sentiment analysis" refers to methods or technologies for analyzing emotions from the content and tone of a user's speech and utilizing the results.

[0520] "Means for providing advice" refers to methods or technologies for providing appropriate advice or suggestions based on the user's situation.

[0521] The system for implementing this invention uses a robot equipped with artificial intelligence. The robot is equipped with multiple means for recognizing the emotions of the worker and providing appropriate support. The specific configuration and operation of the system are described below.

[0522] System Configuration

[0523] 1. Hardware

[0524] Camera: Used to capture the facial expressions of the worker.

[0525] Microphone: Used to capture the voice of the worker.

[0526] Robot body: A device equipped with artificial intelligence and used to interact with workers.

[0527] 2. Software

[0528] OpenCV: A library for performing face recognition.

[0529] TENSORFLOW® / Keras: A library for running emotion recognition models.

[0530] SpeechRecognition: A library for performing speech recognition.

[0531] TextBlob: A library for converting speech to text and performing sentiment analysis.

[0532] System Operation

[0533] 1. Emotion recognition

[0534] The worker's facial expressions are captured using a camera and the face is recognized using OpenCV.

[0535] The recognized face images are input into a TensorFlow / Keras model to estimate emotions.

[0536] 2. Speech Recognition and Sentiment Analysis

[0537] The worker's voice is captured using a microphone and converted to text using the SpeechRecognition library.

[0538] The converted text is analyzed using TextBlob to estimate the sentiment.

[0539] 3. Support Provision

[0540] Based on the estimated emotions, the robot will provide appropriate support and advice.

[0541] For example, if the robot estimates that a worker is tired, it will suggest, "We recommend that you take a break."

[0542] If a worker is faced with a difficult task, the robot will provide technical information such as, "Try the following steps to complete this task."

[0543] Examples

[0544] If the worker is tired: The robot will suggest, "It is recommended that you take a break."

[0545] If a worker is faced with a difficult task: the robot will provide technical information such as "Try the following steps to complete this task."

[0546] Examples of prompt statements

[0547] “What suggestions would you make if a worker feels fatigued?”

[0548] "What technical information do you provide to workers when they're facing a difficult task?"

[0549] In this way, the system can recognize the worker's emotions and state in real time and provide appropriate support and advice.

[0550] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0551] Step 1:

[0552] Camera-based facial expression capture

[0553] The server captures the facial expressions of the workers using a camera. The input is the video data from the camera, and the output is the captured image data. Specifically, the camera captures the video in real time and transmits the video data to the server.

[0554] Step 2:

[0555] Facial Recognition

[0556] The server uses OpenCV to recognize faces from the captured image data. The input is the image data obtained in step 1, and the output is the location information of the recognized face. Specifically, it applies the OpenCV face recognition algorithm to identify the location of the face in the image.

[0557] Step 3:

[0558] Emotion Estimation

[0559] The server uses a TensorFlow / Keras model to estimate emotions from the recognized face images. The input is the face position information and image data obtained in step 2, and the output is the estimated emotion label. Specifically, the face part is cut out and preprocessed, and then input to the emotion recognition model to estimate the emotion.

[0560] Step 4:

[0561] Audio capture

[0562] The server captures the voice of the worker using a microphone. The input is the voice data from the microphone, and the output is the captured voice data. Specifically, the microphone captures the voice in real time and transmits the voice data to the server.

[0563] Step 5:

[0564] Speech Recognition

[0565] The server converts the captured voice data into text using the SpeechRecognition library. The input is the voice data obtained in step 4, and the output is the converted text data. Specifically, it applies a speech recognition algorithm to convert the voice data into text.

[0566] Step 6:

[0567] sentiment analysis

[0568] The server uses TextBlob to analyze sentiment from the converted text data. The input is the text data obtained in step 5, and the output is the analyzed sentiment label. Specifically, it applies the sentiment analysis function of TextBlob to estimate the sentiment of the text.

[0569] Step 7:

[0570] Support Offering

[0571] The server provides appropriate support and advice based on the estimated emotion label. The input is the emotion label obtained in step 3 and step 6, and the output is the content of the support and advice to be provided. Specifically, the server selects a pre-defined support message according to the emotion label and presents it to the worker.

[0572] In this way, the system can recognize the worker's emotions and state in real time and provide appropriate support and advice.

[0573] Example 2

[0574] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0575] Conventional robot systems only have limited functionality in terms of interacting with users and analyzing their emotions, making it difficult to meet the diverse needs of users. In addition, there was a lack of systems that could respond appropriately to young people who use social media and elderly people in the early stages of dementia. This resulted in problems of reduced user satisfaction and convenience.

[0576] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0577] In this invention, the server includes a means for performing voice recognition of user comments, a means for generating an appropriate response using a generative AI model, and a means for analyzing SNS posts with an emotion engine, which makes it possible to generate an appropriate response to a user comment and to analyze the user's emotion from the SNS post and take an appropriate action.

[0578] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[0579] A "robot" is a mechanical device equipped with artificial intelligence and capable of performing physical movements and interacting with other people.

[0580] "Contents of exam preparation, etc." refers to knowledge and information related to academic studies and exams.

[0581] A "means of providing knowledge" is a method or technology for conveying information or knowledge to a user.

[0582] "Topics of interest" refer to topics or areas that interest a user.

[0583] "Means of conversation and discussion" refers to methods and techniques for interacting with users and exchanging opinions and holding discussions.

[0584] The "means for responding with the robot's opinion by adding it to its own answer" refers to a method or technology in which the robot adds its own views or opinions to the answer to the user's question.

[0585] A "speech recognition means" is a technology or device for converting speech into text data.

[0586] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate data such as text or voice.

[0587] A "means for generating an appropriate response" is a method or technique for creating an appropriate reply to a user's input.

[0588] "SNS post" refers to posts or comments made by users on social networking services.

[0589] An "emotion engine" is a technology or system for analyzing emotions from text data and inferring emotional states.

[0590] "Analytical means" refers to methods and techniques for analyzing data and understanding its content and characteristics.

[0591] A "means for generating an appropriate response" is a method or technique for creating an appropriate action or response to the user based on the analysis results.

[0592] The present invention is a system that uses a robot equipped with artificial intelligence to communicate with users and analyze posts on SNS. A specific embodiment of this system will be described below.

[0593] Hardware and Software Used

[0594] Hardware: Robot body (e.g. general-purpose robot)

[0595] Software: emotion engines, generative AI models (e.g. GPT-4), speech recognition software (e.g. speech recognition APIs), speech synthesis software (e.g. speech synthesis APIs)

[0596] System Overview

[0597] This system allows users to talk to the robot and discuss topics they are interested in. It also uses an emotion engine to analyze content posted by users on social media and respond appropriately.

[0598] Program Processing

[0599] 1. Start a dialogue with the user

[0600] The user talks to the robot.

[0601] Example: "Tell me about recent advances in science and technology."

[0602] 2. Voice Recognition

[0603] The terminal (the robot itself) records the user's speech using a built-in microphone.

[0604] The server uses voice recognition software (e.g., a voice recognition API) to convert the recorded voice data into text data.

[0605] Example: Generates the text "Tell me about recent advances in science and technology."

[0606] 3. Using generative AI models

[0607] The server uses a generative AI model (e.g. GPT-4) to generate an appropriate response based on the text data.

[0608] Example: "A recent advance in science and technology is the development of quantum computers, which have the potential to dramatically increase the speed of computation."

[0609] 4. Speech synthesis

[0610] The device converts the generated text response into speech using speech synthesis software (e.g., a speech synthesis API) and responds to the user.

[0611] Example: The robot says, "Recent advances in science and technology include the development of quantum computers, which have the potential to dramatically increase the speed of calculations."

[0612] 5. Analysis of social media posts

[0613] A user posts his / her feelings and opinions on a SNS (e.g., social networking service).

[0614] The server retrieves user posts using the SNS API and analyzes them using an emotion engine (e.g., emotion analysis API).

[0615] Example: The content of your post is analyzed to indicate that you are feeling "stressed."

[0616] 6. Generate appropriate responses

[0617] Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[0618] Example: "You've had a tough day today. Time to take a break and refresh yourself." would generate the response.

[0619] 7. Responses by voice synthesis

[0620] The terminal converts the generated text response into speech using speech synthesis software and responds to the user.

[0621] Example: The robot says, "You had a hard time today. Let's take a break and refresh yourself."

[0622] Examples and prompts

[0623] Example 1: Discussion of technological advances

[0624] User: "Tell me about recent advances in science and technology."

[0625] Robot: "A recent advance in science and technology is the development of quantum computers, which have the potential to dramatically increase the speed of computation."

[0626] Example 2: Responses based on social media posts

[0627] User's social media post: "I'm feeling very stressed today"

[0628] Robot: "You've had a hard day today. Let's take a break and refresh yourself."

[0629] Examples of prompt statements

[0630] Prompt: "When the user asks about recent advances in science and technology, generate an answer that provides a discussion of current science and technology trends and their impact."

[0631] In this way, the system can generate appropriate responses based on users' comments and social media posts, deepening the dialogue with users.

[0632] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0633] Step 1:

[0634] The user talks to the robot.

[0635] Input: User's voice input.

[0636] Specific action: The user talks to the robot about a topic that interests them.

[0637] Output: The robot's microphone captures audio data.

[0638] Step 2:

[0639] The device recognizes the user's speech.

[0640] Input: The captured audio data.

[0641] Specific operation: The terminal (the robot itself) records what the user says using the built-in microphone.

[0642] Output: The recorded audio data is sent to the server.

[0643] Step 3:

[0644] The server converts the voice data into text data.

[0645] Input: Recorded audio data.

[0646] Specific operation: The server uses voice recognition software (e.g., a voice recognition API) to convert the voice data into text data.

[0647] Output: Text data is generated.

[0648] Step 4:

[0649] The server uses the generative AI model to generate an appropriate response.

[0650] Input: The generated text data.

[0651] What happens: The server uses a generative AI model (e.g. GPT-4) to generate an appropriate response based on the text data.

[0652] Output: The text data of the response is generated.

[0653] Step 5:

[0654] The terminal vocally responds with the generated response to the user.

[0655] Input: The text data of the response.

[0656] Specific behavior: The device converts the generated text response into speech using speech synthesis software (e.g., a speech synthesis API) and responds to the user.

[0657] Output: The robot responds to the user via voice.

[0658] Step 6:

[0659] A user posts on a social networking site.

[0660] Input: User's social media posts.

[0661] Specific action: A user posts their feelings or opinions on a SNS (e.g., a social networking service).

[0662] Output: Text data posted to SNS.

[0663] Step 7:

[0664] The server analyzes social media posts using an emotion engine.

[0665] Input: Text data posted on SNS.

[0666] Specific operation: The server retrieves user posts using the SNS API and analyzes them using an emotion engine (e.g. emotion analysis API).

[0667] Output: The sentiment analysis results are generated.

[0668] Step 8:

[0669] The server generates an appropriate response based on the analysis results.

[0670] Input: Sentiment analysis results.

[0671] Specific operation: Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[0672] Output: The corresponding text data is generated.

[0673] Step 9:

[0674] The terminal responds to the user by voice with the generated response.

[0675] Input: The corresponding text data.

[0676] What happens: The device uses speech synthesis software to convert the generated text response into speech and responds to the user.

[0677] Output: The robot responds to the user via voice.

[0678] (Application example 2)

[0679] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0680] In modern society, the use of social networking sites is spreading, especially among young people, but it has become a problem that emotions such as stress, anxiety, and excitement on social networking sites affect users' mental health. In addition, since users have limited opportunities to discuss topics that interest them in depth, there is a demand to deepen knowledge and improve the quality of discussions. In order to solve these issues, a system that can properly analyze users' emotions and provide appropriate responses based on them is required.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0682] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics that interest the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to analyze the contents of posts on SNS and estimate the user's emotions, and a means for the robot to provide an appropriate response to the user based on the estimated emotions. This makes it possible to appropriately analyze the user's emotions, support mental health, and deepen knowledge and improve the quality of discussion.

[0683] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[0684] A "robot" is a mechanical device equipped with artificial intelligence that performs specific tasks automatically.

[0685] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[0686] A "knowledge medium" is a method or technique used to provide new information or understanding to a user.

[0687] A "discussion tool" is a method or technique for exchanging opinions with users on a particular topic.

[0688] "Means of responding" refers to the methods or technologies that a robot uses to communicate its views and thoughts to the user.

[0689] "SNS" is an abbreviation for Social Networking Service, a platform for users to share information and communicate online.

[0690] "Means of analyzing post content" refers to methods and technologies for analyzing users' posts on SNS and understanding their content.

[0691] "Means of inferring emotions" refers to methods or technologies for determining a user's emotional state from the content of their posts.

[0692] The "means for providing an appropriate response" refers to a method or technology for providing appropriate advice or support to a user based on the estimated emotions.

[0693] The system for implementing this invention is mainly composed of a robot equipped with artificial intelligence. The robot is equipped with an emotion engine that analyzes the content posted by users on SNS and infers their emotions. It also has a function to provide an appropriate response to the user based on the inferred emotions.

[0694] System Configuration

[0695] The system consists of the following main components:

[0696] 1. Robots equipped with artificial intelligence: They have the ability to interact with users and deepen discussions.

[0697] 2. Emotion engine: Analyzes the content of social media posts and estimates the user's emotions.

[0698] 3. Response Providing Module: Provides an appropriate response to the user based on the estimated sentiment.

[0699] Hardware and Software Used

[0700] Hardware: Smartphones, servers, robots

[0701] Software: Python, requests library, TextBlob library, SNS platform API, emotion engine API

[0702] Data processing and calculation

[0703] 1. Data collection: The server collects user posts using the API of the SNS platform.

[0704] 2. Sentiment analysis: The collected post content is sent to the emotion engine API to analyze the emotions.

[0705] 3. Response delivery: Based on the analysis results, the response delivery module provides appropriate advice and support to users.

[0706] Examples

[0707] For example, if a user posts on social media that "I've been busy at work lately and I'm feeling stressed," the server sends the post to the emotion engine API, which analyzes the emotion of stress. The response module then suggests to the user, "Here are some ways to relax. Take a deep breath and listen to some relaxing music."

[0708] Examples of prompt statements

[0709] A user posted on social media that "I've been busy at work lately and I'm feeling stressed." Analyze this post and suggest an appropriate response to the user.

[0710] In this way, it is possible to properly analyze users' emotions, support their mental health, and deepen knowledge and improve the quality of discussion.

[0711] The flow of the identification process in the application example 2 will be described with reference to FIG.

[0712] Step 1:

[0713] The server collects user posts using the API of the SNS platform. Specifically, it sends an API request specifying the user ID to retrieve the user's latest posts. The input is the user ID, and the output is the user's post.

[0714] Step 2:

[0715] The server sends the collected post content to the emotion engine API and analyzes the emotions. Specifically, it sends the post content in JSON format to the emotion engine API and receives the emotion analysis results. The input is the user's post content, and the output is the emotion analysis results.

[0716] Step 3:

[0717] The server calls the response module based on the emotion analysis result and provides the user with an appropriate response. Specifically, the response module generates appropriate advice or support based on the emotion analysis result and notifies the user. The input is the emotion analysis result, and the output is the advice or support for the user.

[0718] Step 4:

[0719] The user receives and acts on the advice and support provided by the server. Specifically, the user checks the notification from the server and puts into practice the relaxation methods and discussion information. The input is the advice and support from the server, and the output is the user's behavior.

[0720] Step 5:

[0721] The server collects user feedback and uses it to improve the system. Specifically, it collects feedback about how users felt about the advice and support provided, and adjusts the system's algorithm. The input is user feedback, and the output is the improvement of the system.

[0722] Example 3

[0723] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0724] Conventional robots equipped with artificial intelligence could only give simple responses to user opinions, making it difficult to hold deep discussions or provide additional information. They were also unable to respond appropriately to elderly people with early-stage dementia, making it difficult to alleviate their feelings of loneliness and anxiety. This made it difficult to deepen the user's opinions and improve the quality of life for the elderly.

[0725] The identification process by the identification processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes a means for inputting the user's opinion into the generative AI model and providing additional information and viewpoints, a means for estimating the elderly person's emotions from their everyday facial expressions and tone of voice and responding based on the results, and a means for the robot to respond with its own opinion in addition to its own response. This allows the user's opinion to be deepened, and makes it possible to respond appropriately according to the elderly person's emotions.

[0726] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and perform functions such as learning, reasoning, and problem-solving.

[0727] A "robot" is a mechanical device equipped with artificial intelligence and capable of performing physical movements and providing vocal responses.

[0728] A "generative AI model" is an artificial intelligence algorithm that generates new information or responses based on input data.

[0729] A "prompt" is an instruction given to a generative AI model that serves as the basis for the model to generate a response.

[0730] An "emotion engine" is software that estimates human emotions from data such as voice and facial expressions.

[0731] A "user" is a person who operates the system and inputs opinions.

[0732] "Elderly" generally refers to older people, especially those with early symptoms of dementia.

[0733] An "opinion" is a thought or assertion that a user inputs into the system.

[0734] "Additional information or perspectives" refers to supplementary knowledge or different perspectives that the generative AI model provides in response to the user's opinion.

[0735] "Response" refers to the responses and actions that the robot takes toward users and elderly people.

[0736] "Facial expressions" are the movements and expressions of a human face that indicate emotions.

[0737] "Voice tone" refers to characteristics of the voice, such as pitch, strength, and rhythm, and is an element that indicates emotion.

[0738] This invention is a system that uses a robot equipped with artificial intelligence to provide additional information and perspectives to the user's opinions and respond appropriately according to the feelings of the elderly. A specific embodiment of this system will be described below.

[0739] Hardware and Software Used

[0740] Hardware: A high-performance server, the device operated by the user (PC, smartphone, tablet, etc.), and the robot itself (equipped with a voice recognition microphone, camera, and speaker)

[0741] Software: Generative AI models (e.g. GPT-4), emotion engines (e.g. Affectiva), voice recognition software (e.g. Google Speech-to-Text)

[0742] System program processing

[0743] User opinion input and analysis

[0744] A user uses a terminal to input their opinion in text format. For example, the user inputs, "Regarding environmental issues, I think reducing plastic waste is important." The terminal sends this opinion to the server. The server inputs the received opinion into a generative AI model and creates a prompt sentence. For example, it generates a prompt sentence such as, "User's opinion: Regarding environmental issues, I think reducing plastic waste is important. Please provide any additional information or perspective."

[0745] Response generation for generative AI models

[0746] The server receives a response from the generative AI model, for example, "Reducing plastic waste is certainly important. Furthermore, we should also consider promoting recycling and developing alternative materials." The server then sends this response to the robot.

[0747] Robot Response

[0748] The robot will respond to the user with voice based on the response it receives. For example, it might say to the user, "It is certainly important to reduce plastic waste. Furthermore, we should also consider promoting recycling and developing alternative materials."

[0749] Estimating and responding to emotions of the elderly

[0750] The robot uses a camera and microphone to collect the elderly person's everyday facial expressions and tone of voice. The collected data is sent to an emotion engine to estimate their emotions. For example, it may estimate that the elderly person is feeling lonely. Based on the results of the emotion engine, the robot will ask, "How are you doing lately? Is there anything you'd like to talk about?"

[0751] Examples

[0752] User feedback:

[0753] The user types, "Regarding environmental issues, I think reducing plastic waste is important" into a text box displayed on the smartphone screen.

[0754] When the terminal presses the "Submit" button, the opinion data is sent to the server as an HTTP POST request.

[0755] Use of generative AI models:

[0756] The server inputs the opinions into a generative AI model to create a prompt.

[0757] For example, a prompt sentence is generated that reads, "User's opinion: Regarding environmental issues, I think reducing plastic waste is important. Please provide any additional information or perspective."

[0758] Response from the generative AI model:

[0759] The server receives and analyzes the response from the generative AI model.

[0760] For example, you might receive a response like, "Reducing plastic waste is certainly important. In addition, promoting recycling and developing alternative materials should also be considered."

[0761] The robot responds:

[0762] The robot responds through the speaker, "Reducing plastic waste is certainly important. Furthermore, we should also consider promoting recycling and developing alternative materials."

[0763] Estimating and responding to emotions of the elderly:

[0764] The robot captures the elderly person's facial expressions with a camera and collects the tone of their voice with a microphone.

[0765] The collected data is sent to an emotion engine in real time for emotion analysis.

[0766] For example, if the robot estimates that you are feeling lonely, it will ask, "How are you doing lately? Is there anything you'd like to talk about?"

[0767] In this way, the system can provide appropriate information in response to the user's opinion and respond according to the elderly person's feelings. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0768] Step 1:

[0769] The user inputs an opinion.

[0770] The user uses the terminal to input an opinion in text format. For example, the user inputs "Regarding environmental issues, I think reducing plastic waste is important." The input opinion is displayed in a text box on the terminal.

[0771] Step 2:

[0772] The terminal transmits the opinion to the server.

[0773] The terminal sends the inputted opinion to the server as an HTTP POST request. Specifically, the opinion data is sent to the server in JSON format. The input is the user's opinion, and the output is the opinion data sent to the server.

[0774] Step 3:

[0775] The server inputs the opinions into a generative AI model.

[0776] The server creates a prompt sentence to input the received opinion into the generative AI model. For example, it generates a prompt sentence like "User's opinion: Regarding environmental issues, I think reducing plastic waste is important. Please provide additional information or perspectives." The input is the user's opinion data, and the output is the generated prompt sentence.

[0777] Step 4:

[0778] The server receives the response from the generative AI model.

[0779] The server sends a prompt to the generative AI model and receives a response. For example, it receives a response such as, "Reducing plastic waste is certainly important. In addition, we should also consider promoting recycling and developing alternative materials." The input is the prompt, and the output is the response from the generative AI model.

[0780] Step 5:

[0781] The server sends the response to the robot.

[0782] The server sends the response from the generative AI model to the robot. Specifically, it sends the response data using the robot's API. The input is the response from the generative AI model, and the output is the response data sent to the robot.

[0783] Step 6:

[0784] The robot responds to the user.

[0785] The robot responds to the user by voice based on the response it receives. For example, it might say to the user, "Reducing plastic waste is certainly important. Furthermore, we should also consider promoting recycling and developing alternative materials." The input is the response data sent to the robot, and the output is a voice response to the user.

[0786] Step 7:

[0787] A robot estimates the emotions of elderly people.

[0788] The robot collects the elderly person's daily facial expressions and tone of voice using a camera and microphone. The collected data is sent to an emotion engine to estimate their emotions. For example, it estimates that the elderly person feels lonely. The input is the elderly person's facial expressions and tone of voice data, and the output is the estimated emotion data.

[0789] Step 8:

[0790] The robot responds based on emotions.

[0791] Based on the results of the emotion engine, the robot responds according to the elderly person's emotions. For example, if it estimates that the elderly person is feeling lonely, the robot will ask, "How are you doing lately? Is there anything you'd like to talk about?" The input is the estimated emotion data, and the output is a voice response to the elderly person.

[0792] (Application example 3)

[0793] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0794] In modern society, elderly people are becoming increasingly lonely and anxious. It is also important to provide a sense of security in their daily lives, especially for those with early-stage dementia. However, with current technology, it is difficult to grasp the emotions of elderly people in real time and respond appropriately. There is also a lack of means to respond quickly when an abnormality occurs. Furthermore, there is no adequate system in place to provide specific advice to users.

[0795] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam study, etc., a means for the robot to talk to the user and discuss the topic of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to estimate the user's emotion from the user's facial expression and tone of voice using an emotion engine and reflect the result in the robot's response, a means for the robot to send a notification when the robot detects an abnormality, and a means for the robot to provide specific advice to the user using a generative AI model. This makes it possible to grasp the elderly's emotions in real time and take appropriate measures. In addition, it is possible to quickly respond when an abnormality occurs and provide specific advice to the user.

[0796] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and perform functions such as learning, reasoning, and problem-solving.

[0797] A "robot" is a mechanical device equipped with artificial intelligence that performs specific tasks automatically.

[0798] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[0799] "Knowledge delivery means" are methods and technologies used to convey information or data to users and deepen their understanding.

[0800] "Means of conversation and discussion" are methods and techniques for interacting with users and exchanging opinions and holding debates.

[0801] "Means of responding to opinions" refers to methods or technologies that allow a robot to provide its own views or information in response to a user's questions or opinions.

[0802] The "emotion engine" is a technology that infers a user's emotions from their facial expressions and tone of voice, and responds appropriately based on the results.

[0803] "Means of detecting anomalies" are methods or technologies for identifying situations or behaviors that deviate from normal conditions and for issuing warnings or notifications.

[0804] A "notification mechanism" is a method or technique for communicating alerts or information to designated recipients when an anomaly is detected.

[0805] A "generative AI model" is an algorithm or technique that enables artificial intelligence to generate new information or advice based on user input.

[0806] "Means of providing specific advice" are methods or techniques for providing practical, specific advice or suggestions tailored to the user's situation and needs.

[0807] The system for implementing this invention is composed mainly of a robot equipped with artificial intelligence. The robot analyzes the user's facial expressions and tone of voice in real time and estimates the user's emotions using an emotion engine. This allows the robot to respond appropriately according to the user's emotional state.

[0808] Hardware and Software Used

[0809] Hardware: Smartphone, camera, microphone

[0810] Software: Python, OpenCV, Keras, Twilio

[0811] System Configuration

[0812] 1. Emotion recognition:

[0813] It uses your smartphone's camera and microphone to capture your facial expressions and tone of voice.

[0814] Detect faces using OpenCV and analyze facial expressions using Keras emotion recognition models.

[0815] The emotion engine estimates the user's emotions and reflects the results in the robot's response.

[0816] 2. Anomaly detection and notification:

[0817] If emotion recognition detects an abnormality (e.g. anxiety, excitement), Twilio is used to send a notification to family members or security services.

[0818] The notification contains information about the user's current emotional state and required action.

[0819] 3. Generative AI models provide advice:

[0820] It uses generative AI models to generate specific advice based on user input.

[0821] For example, if a user inputs, "Please tell me how to respond if an elderly person feels anxious when alone," the generative AI model will provide specific advice such as, "If an elderly person feels anxious, it is important to first stay calm and listen to them. It is also effective to contact family and friends to give them a sense of security. Also, consider consulting a specialist medical institution if necessary."

[0822] Examples

[0823] For example, if an elderly person with early dementia feels anxious when alone, the robot will respond as follows:

[0824] 1. Using a smartphone camera and microphone to capture a senior's facial expressions and tone of voice.

[0825] 2. Analyze emotions and detect anxiety using OpenCV and Keras.

[0826] 3. Use Twilio to send a notification to a family member saying, "An elderly person is feeling anxious. Please check in."

[0827] 4. Use generative AI models to provide specific advice to seniors, such as, "Are you OK? Is there anything I can help you with?"

[0828] In this way, the present invention can grasp the emotions of the elderly in real time and take appropriate measures. It can also quickly respond to any abnormalities that occur and provide specific advice to the user.

[0829] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0830] Step 1:

[0831] The device (smartphone) uses a camera and microphone to capture the user's facial expressions and tone of voice. The input is real-time video and audio data, and the output is captured video frames and audio clips. In concrete terms, the device captures video frames and audio clips at regular intervals and passes them on to the next processing step.

[0832] Step 2:

[0833] The device uses OpenCV to detect faces from captured video frames. The input is the video frame acquired in step 1, and the output is the area (bounding box) of the detected face. Specifically, the device runs a face detection algorithm to determine the position and size of the face.

[0834] Step 3:

[0835] The device analyzes emotions from the detected face regions using a Keras emotion recognition model. The input is the face regions obtained in step 2, and the output is the estimated emotion (e.g., happiness, sadness, anxiety, etc.). Specifically, the device preprocesses the face regions and inputs them into the emotion recognition model to estimate the emotion.

[0836] Step 4:

[0837] The device determines an appropriate response based on the emotion estimated using the emotion engine. The input is the emotion data obtained in step 3, and the output is the corresponding action (e.g., speaking to the device, sending a notification, etc.). In concrete terms, the device analyzes the emotion data and selects a response based on pre-defined rules.

[0838] Step 5:

[0839] If an abnormality (e.g., anxiety, agitation) is detected, the device uses Twilio to send a notification to a family member or security service. The input is the action determined in step 4, and the output is the sent notification message. In concrete terms, the device calls the Twilio API and sends a message to the specified recipient.

[0840] Step 6:

[0841] The terminal uses the generative AI model to generate specific advice for the user. The input is information about the user's question or situation, and the output is the generated advice. In concrete terms, the terminal passes the user's input as a prompt sentence to the generative AI model and provides the generated advice to the user.

[0842] Step 7:

[0843] The device presents the generated advice to the user and continues the dialogue as necessary. The input is the advice obtained in step 6, and the output is the information presented to the user. In concrete terms, the device conveys the advice to the user by voice or text and waits for the user's response.

[0844] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0845] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0846] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0847] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0848] [Second embodiment]

[0849] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0850] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0851] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0852] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0853] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0854] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0855] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0856] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0857] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0858] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0859] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0860] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0861] "Example 1"

[0862] As one embodiment of the present invention, a robot equipped with artificial intelligence is provided. This robot has a function of providing further knowledge about the contents of exam preparation, etc. For example, when a user is working on a mathematical problem, the robot can provide related mathematical theories and solutions.

[0863] "Example 2"

[0864] The robot also has the ability to engage in conversation and discussion on topics of interest to the user. For example, if the user brings up the topic of advances in science and technology, the robot can engage in a deeper discussion about the latest trends in science and technology and their impact.

[0865] "Example 3"

[0866] In addition, the robot has the ability to respond with its own opinions in addition to the user's own answers. This allows the user to deepen their own opinions. For example, if a user expresses an opinion about an environmental issue, the robot can provide additional information or perspectives on that opinion.

[0867] The process flow of each embodiment will be described below.

[0868] "Example 1"

[0869] Step 1: The user presents the robot with learning content (e.g. a math problem).

[0870] Step 2: The robot acquires knowledge about the relevant learning content from artificial intelligence.

[0871] Step 3: The robot provides the user with the knowledge it has acquired, providing concrete explanations of the relevant mathematical theories and solutions.

[0872] "Example 2"

[0873] Step 1: The user presents the robot with a topic of interest (e.g., advances in science and technology).

[0874] Step 2: The robot gets information about the relevant topic from artificial intelligence.

[0875] Step 3: Based on the information acquired by the robot, the robot will hold a discussion with the user, providing specific details about the latest trends in science and technology and their impact.

[0876] "Example 3"

[0877] Step 1: The user presents the robot with their opinion (e.g., their opinion on environmental issues).

[0878] Step 2: The robot obtains additional information and perspectives on the subject from artificial intelligence.

[0879] Step 3: The robot provides the user with additional information and perspectives that it has obtained, with specific explanations of the additional information and perspectives in response to the user's opinion.

[0880] Example 1

[0881] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0882] Conventional learning support systems have the problem that it is difficult to get an appropriate and prompt answer when a user asks a question about exam preparation or a specific topic. There are also only a limited number of systems that allow users to have in-depth discussions about topics that interest them. Furthermore, there is a lack of functionality for robots to have their own opinions and provide additional information to users.

[0883] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0884] In this invention, the server includes a means for a user to input a question, a means for a terminal to recognize the question, a means for the terminal to transmit the question to the server, a means for the server to analyze the question, a means for the server to generate an answer using a generative AI model, a means for the server to transmit the answer to the terminal, and a means for the terminal to present the answer to the user. This allows the user to obtain an appropriate and prompt answer when asking a question about studying for an exam or a specific topic. In addition, the robot can deeply discuss a topic that interests the user and provide additional information with its own opinion.

[0885] "Means for user input of question" refers to an interface through which a user can input a question by voice or text.

[0886] "Means by which the device recognizes the question" refers to speech recognition software that the device uses to convert the user's speech into text, or the device's ability to directly obtain text input.

[0887] "Means for the terminal to send a question to the server" refers to a communication function for sending text data recognized by the terminal to the server.

[0888] "Means for the server to analyze the question" refers to natural language processing technology used by the server to analyze the text data it receives and understand the intent of the question.

[0889] "Means for the server to generate an answer using a generative AI model" refers to the function of the server using a generative AI model to generate an optimal answer to a question.

[0890] "Means for the server to transmit a response to the terminal" refers to a communication function for transmitting a response generated by the server to the terminal.

[0891] "Means for the terminal to present the answer to the user" refers to an output function for presenting the answer received by the terminal to the user in the form of voice or text.

[0892] A "robot equipped with artificial intelligence" refers to a robot that has built-in artificial intelligence technology and is capable of interacting with users and providing them with information.

[0893] "Means for providing further knowledge about the contents of examination study, etc." refers to a function that provides related knowledge and solutions when a user asks a question related to examination study.

[0894] "Means of talking and discussing topics of interest" refers to the function of the robot to converse and discuss topics that interest the user.

[0895] "A means for the robot to respond with its own opinion in addition to its own answer" refers to a function in which the robot provides its own opinion or additional information in addition to the answer to the user's question.

[0896] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0897] This invention is a system that uses a robot equipped with artificial intelligence to provide appropriate and prompt answers to questions from users about studying for exams or specific topics. A specific embodiment of this system will be described below.

[0898] Hardware and Software Used

[0899] Hardware: High performance processor (e.g. Intel Core i7), camera, microphone, speakers

[0900] Software: generative AI models (e.g. GPT-4), speech recognition software (e.g. Google Speech-to-Text), natural language processing libraries (e.g. spaCy)

[0901] System Configuration

[0902] 1. How the user enters their question:

[0903] The user inputs a question to the robot by voice or text. For example, the user may ask, "Teach me how to solve a quadratic equation."

[0904] 2. How the device recognizes the question:

[0905] The device uses a built-in microphone and speech recognition software to convert the user's speech into text, or, in the case of text input, to capture the text as is.

[0906] 3. How the device sends a query to the server:

[0907] The terminal transmits the recognized text data to the server. At this time, the data is encrypted before being transmitted.

[0908] 4. How the server parses the query:

[0909] The server analyzes the received text data and uses a natural language processing library to understand the intent of the question.

[0910] 5. How the server uses the generative AI model to generate an answer:

[0911] The server uses the generative AI model to generate optimal answers to questions, such as "How do I solve a quadratic equation?", generating a solution using a formula.

[0912] 6. The server sends the response to the terminal by:

[0913] The server then sends the generated response to the terminal. The data is also encrypted when sent.

[0914] 7. The means by which the terminal presents the answer to the user:

[0915] The device will then present the answer to the user either by voice or text, in which case the answer will be read out loud using a speaker.

[0916] Examples

[0917] For example, if a user asks the robot, "Teach me how to solve a quadratic equation," the system operates as follows.

[0918] User: "How do I solve a quadratic equation?"

[0919] Device: Capture audio through the microphone and convert it to text using Google Speech-to-Text: "Can you help me solve a quadratic equation?"

[0920] Terminal: Sends text data to the server.

[0921] Server: Use spaCy to analyze the text and understand the intent of the question.

[0922] Server: Uses GPT-4 to generate the answer, "To solve quadratic equations, use the formula. The formula is x = (-b ± √(b^2 - 4ac)) / 2a."

[0923] Server: Generates the answer and sends it to the device.

[0924] Device: Using the speaker, read to the user, "To solve quadratic equations, use the formula. The formula is x = (-b ± √(b^2 - 4ac)) / 2a."

[0925] Examples of prompt statements

[0926] "When a user asks how to solve a quadratic equation, explain how to solve it using the formula."

[0927] In this way, the robot supports the user's learning.

[0928] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0929] Step 1:

[0930] The user enters a question.

[0931] The user inputs a question to the robot by voice or text. For example, the user may ask, "Please tell me how to solve a quadratic equation." The input is voice or text data.

[0932] Step 2:

[0933] The device recognizes the question.

[0934] The terminal uses a built-in microphone and voice recognition software to convert the user's speech into text. In the case of text input, the terminal obtains the text as is. The input is voice data or text data, and the output is text data.

[0935] Step 3:

[0936] The terminal sends a question to the server.

[0937] The terminal transmits the recognized text data to the server. At this time, the data is encrypted before being transmitted. The input is text data, and the output is encrypted text data.

[0938] Step 4:

[0939] The server parses the question.

[0940] The server analyzes the received text data. It uses a natural language processing library to understand the intent of the question. The input is the encrypted text data, and the output is the analyzed intent of the question.

[0941] Step 5:

[0942] The server generates the answer using a generative AI model.

[0943] The server uses the generative AI model to generate the best answer to the question. For example, for the question "How do I solve a quadratic equation?", it generates a solution using the solution formula. The input is the parsed question intent, and the output is the generated answer.

[0944] Step 6:

[0945] The server sends the response to the terminal.

[0946] The server sends the generated answer to the terminal. The data is also encrypted when sent. The input is the generated answer, and the output is the encrypted answer.

[0947] Step 7:

[0948] The terminal presents the answer to the user.

[0949] The terminal presents the received answer to the user either by voice or text, in the latter case by reading the answer aloud using a speaker. The input is the encrypted answer, and the output is the presented answer by voice or text.

[0950] (Application example 1)

[0951] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0952] Conventional educational support robots were limited to providing knowledge on exam preparation or specific topics, and had the problem of not being able to support the education and training of workers in factories. In addition, since they could not provide real-time knowledge using generative AI models, it was difficult to immediately provide the latest information or specific operating procedures.

[0953] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0954] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to provide knowledge about new machine operation methods and manufacturing processes to workers in a factory, and a means for the robot to obtain and provide related knowledge based on a prompt sentence using a generative AI model. This makes it possible to provide the latest knowledge and specific operating procedures in real time during worker education and training in a factory.

[0955] A "robot equipped with artificial intelligence" is an autonomous mechanical device that incorporates artificial intelligence technologies such as machine learning and natural language processing, and is capable of interacting with users and providing them with knowledge.

[0956] "Means for providing further knowledge about the content of exam preparation, etc." is a function that provides additional information or explanations related to the study content the user is working on.

[0957] "A means to talk and discuss topics of interest" is a function that enables users to engage in dialogue, exchange opinions, and debate on topics that interest them.

[0958] "A means for the robot to respond with its own opinion by adding it to the user's answer" is a function that enables the robot to respond to the user's question by adding its own knowledge and opinions.

[0959] "A means of providing workers in a factory with knowledge about how to operate new machines and manufacturing processes" is a function that provides workers in a factory with information about how to use new machines and manufacturing processes.

[0960] "Means for obtaining and providing relevant knowledge based on a prompt sentence using a generative AI model" refers to a function that uses a generative AI model to generate relevant information based on a prompt sentence entered by a user and provides it to the user.

[0961] The system for implementing this invention is centered around a robot equipped with artificial intelligence. The robot has the function of providing knowledge in various scenarios, such as studying for exams or educating workers in a factory.

[0962] System Configuration

[0963] The system consists of the following main components:

[0964] 1. Robots equipped with artificial intelligence: Equipped with machine learning and natural language processing technology, they can converse with users and provide them with knowledge.

[0965] 2. Generative AI model: Generates relevant knowledge based on the prompt sentence entered by the user.

[0966] 3. Database: Stores information about training content and operating procedures within the factory.

[0967] Hardware and Software Used

[0968] Hardware: Robot body, server, user device (smartphone, tablet, etc.)

[0969] Software: Python, OpenAI API, Database Management System

[0970] Data processing and calculation

[0971] The server sends the prompt sentence entered by the user to the generative AI model to obtain related knowledge, which is then provided to the user through the robot.

[0972] Examples

[0973] For example, if a worker in a factory wants to learn how to operate a new welding machine, the system works as follows.

[0974] 1. User input: A worker says to the robot, "Please teach me the basic principles of welding."

[0975] 2. Prompt generation: The server sends this input as a prompt to the generative AI model.

[0976] Example prompt: "Please provide a detailed explanation of the basic principles of welding."

[0977] 3. Knowledge Acquisition: The generative AI model generates relevant knowledge based on the prompt sentence and returns it to the server.

[0978] 4. Knowledge provision: The server transmits the acquired knowledge to the robot, and the robot provides the knowledge to the worker via voice or display.

[0979] In this way, the system is able to provide up-to-date knowledge and specific operating procedures in real time.

[0980] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0981] Step 1:

[0982] The user inputs a question to the robot. For example, a worker might say, "Please tell me the basic principles of welding." This input is converted into text data through a voice recognition system.

[0983] Step 2:

[0984] The terminal generates a prompt sentence based on the user's input, for example, "Please provide a detailed explanation of the basic principles of welding." This prompt sentence is then sent to the generative AI model.

[0985] Step 3:

[0986] The server sends a prompt to the generative AI model, which generates relevant knowledge based on the prompt, and returns the knowledge to the server as text data.

[0987] Step 4:

[0988] The server receives the generated knowledge and matches it with related information stored in a database, retrieving additional information from the database if necessary.

[0989] Step 5:

[0990] The server transmits the final knowledge data to the robot, which converts the data into a format for providing it to the user, such as voice data or text data for display.

[0991] Step 6:

[0992] The robot provides the knowledge data received from the server to the user, either by explaining it aloud using a speech synthesis system or by displaying text on a display.

[0993] Step 7:

[0994] The user receives the knowledge provided by the robot and uses it to learn or work, and if necessary, can ask further questions to gain additional knowledge.

[0995] Example 2

[0996] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0997] Conventional robot systems have the problem that it is difficult to have in-depth discussions on topics that interest the user. In addition, there is a lack of technology to accurately recognize the user's voice input and generate appropriate responses. Furthermore, there is a need for a system that can speak the generated responses in a natural voice.

[0998] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting a user's voice input into text data using voice recognition software, a means for inputting a prompt sentence into a generative AI model and generating an appropriate response, and a means for converting the generated text response into voice data using voice synthesis software. This enables users to have in-depth discussions on topics that interest them.

[0999] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1000] A "robot" is a mechanical device equipped with artificial intelligence and capable of communicating with humans through physical movements and dialogue.

[1001] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass a particular exam.

[1002] A "knowledge delivery medium" is a method or technique for conveying information or data to a user.

[1003] "Means of discussion" are methods and technologies for exchanging opinions and sharing information through dialogue with users.

[1004] "Speech recognition software" is a program for converting voice data into text data.

[1005] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses based on input text data.

[1006] A "prompt sentence" is a piece of text data input into a generative AI model and is an instruction sentence for generating a response.

[1007] "Speech synthesis software" is a program for converting text data into voice data.

[1008] "Audio data" refers to data that represents audio information in digital form.

[1009] The "means for uttering" refers to a method or technology for playing back the generated voice data through a speaker and conveying information to the user by voice.

[1010] This invention is a system for allowing a user to discuss with a robot a topic that the user is interested in. This system uses a robot equipped with artificial intelligence to carry out a series of processes, including recognizing the user's voice input, generating an appropriate response, and speaking the response aloud.

[1011] Hardware and Software Used

[1012] Hardware: The robot itself (e.g. a typical interactive robot)

[1013] Software: speech recognition software (e.g., a general speech recognition program), generative AI models (e.g., a general generative AI algorithm), speech synthesis software (e.g., a general speech synthesis program)

[1014] System Operation

[1015] 1. User voice input:

[1016] The user talks to the robot, for example, saying, "Tell me about recent advances in science and technology."

[1017] The robot uses a built-in microphone to capture the user's voice.

[1018] 2. Speech to text conversion:

[1019] The terminal (robot) sends the captured voice data to the voice recognition software.

[1020] The server uses voice recognition software to convert the voice data into text data.

[1021] The terminal receives the converted text data.

[1022] 3. Use of generative AI models:

[1023] The device inputs the received text data into the generative AI model.

[1024] The server sends prompts to the generative AI model, which then generates an appropriate response.

[1025] 4. Generate the response:

[1026] The server receives the response from the generative AI model.

[1027] The terminal receives the generated text response from the server.

[1028] 5. Response transcription:

[1029] The terminal sends the received text response to the speech synthesis software.

[1030] The server uses speech synthesis software to convert the text data into voice data.

[1031] The terminal receives the converted voice data.

[1032] 6. Robot response utterance:

[1033] The terminal transmits the received voice data to the robot's speaker.

[1034] The robot speaks the generated voice to the user.

[1035] Examples

[1036] User: "Tell me about recent advances in science and technology."

[1037] Robot: "A recent advance in science and technology is the development of quantum computing. Quantum computers have the potential to rapidly solve problems that cannot be solved by conventional computers."

[1038] Examples of prompt statements

[1039] Please tell me about recent advances in science and technology.

[1040] "Let's discuss the latest developments in AI technology."

[1041] "Tell me about the future of robotics."

[1042] In this way, users can deepen discussions with the robot about topics of interest.

[1043] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1044] Step 1:

[1045] A user speaks to the robot, for example, "Tell me about recent advances in science and technology." The robot uses a built-in microphone to capture the user's voice. The input is the user's voice, and the output is the voice data.

[1046] Step 2:

[1047] The terminal (robot) sends the captured voice data to the voice recognition software. The server uses the voice recognition software to convert the voice data into text data. The input is voice data, and the output is text data. In concrete terms, the voice recognition software analyzes the voice waveform and generates the corresponding text.

[1048] Step 3:

[1049] The terminal receives the converted text data and inputs it into the generative AI model. The server sends a prompt sentence to the generative AI model and generates an appropriate response. The input is the text data, and the output is the generated response text. In concrete terms, the generative AI model analyzes the prompt sentence and generates a response based on related information.

[1050] Step 4:

[1051] The server receives the response from the generative AI model and sends it to the terminal. The terminal receives the generated text response from the server. The input is the generated response text, and the output is the same response text. In specific operations, the server receives the response from the generative AI model and forwards it to the terminal.

[1052] Step 5:

[1053] The terminal sends the received text response to the speech synthesis software. The server uses the speech synthesis software to convert the text data into speech data. The input is text data and the output is speech data. In concrete terms, the speech synthesis software analyzes the text and generates a corresponding speech waveform.

[1054] Step 6:

[1055] The terminal receives the converted voice data and transmits it to the robot's speaker. The robot speaks the generated voice to the user. The input is the voice data, and the output is a voice response to the user. In concrete terms, the robot's speaker plays the voice data and conveys information to the user.

[1056] (Application example 2)

[1057] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1058] Conventional robots are limited to specific tasks and simple dialogue, making it difficult to improve work efficiency through real-time communication and discussion with workers. In addition, there were no robots that could deepen discussions about work progress and problems. For this reason, there is a demand for improving work efficiency and speeding up problem solving in factories.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1060] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, and a means for the robot to communicate with workers in real time to improve work efficiency and discuss work progress and problems, thereby making it possible to improve work efficiency in a factory and speed up problem solving.

[1061] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1062] A "robot" is a mechanical device that operates autonomously according to programmed instructions to perform specific tasks.

[1063] "Studying for exams" refers to learning activities aimed at acquiring the knowledge and skills necessary to pass a particular exam.

[1064] A "means of providing knowledge" is a method or device for conveying information or data to users and deepening their understanding.

[1065] A "theme" is a topic or issue that is the center of discussion or research.

[1066] "Means for conversation and discussion" refers to methods or devices for engaging in dialogue with users and deepening discussion through the exchange of opinions and sharing of information.

[1067] "Means of responding with opinions" refers to a method or device that enables a robot to express its own views or thoughts in response to a user's questions or opinions.

[1068] "Work efficiency" is an index for optimizing the effort and time required to perform a specific task.

[1069] "Means of real-time communication" refers to methods or devices for instantly exchanging information with users and responding quickly.

[1070] "Work progress" is a state or condition that indicates how far a particular piece of work has progressed.

[1071] An "issue" is an obstacle or challenge that occurs in a task or process.

[1072] A "means of discussion" is a method or device for exchanging opinions and sharing information on a particular topic and deepening understanding.

[1073] The system for carrying out the present invention is mainly composed of a robot equipped with artificial intelligence. A specific embodiment of this system will be described below.

[1074] Hardware used

[1075] Robot body: A mechanical device that operates autonomously and communicates with workers (e.g., an articulated robot arm).

[1076] Speech recognition microphone: A device that recognizes the worker's voice and converts it into text (e.g., a microphone compatible with the Google Cloud Speech-to-Text API).

[1077] Display: A device for displaying visual information, if required.

[1078] Software used

[1079] Speech recognition software: Software for converting speech into text (e.g. Google Cloud Speech-to-Text API).

[1080] Generative AI model: An artificial intelligence model for generating appropriate responses to user questions or arguments (e.g., OpenAI’s GPT-4).

[1081] Natural Language Processing Libraries: Libraries for performing text analysis (e.g. NLTK, spaCy).

[1082] Database: A database (e.g. MySQL, PostgreSQL) to store generated data and user history.

[1083] System Operation

[1084] 1. Voice input: The worker speaks to the robot.

[1085] 2. Speech Recognition: Speech recognition software converts speech into text.

[1086] 3. Text Analysis: Use natural language processing libraries to analyze text and understand user intent.

[1087] 4. Input to the generative AI model: Input a prompt to the generative AI model based on the analysis results.

[1088] 5. Response generation: The generative AI model generates an appropriate response.

[1089] 6. Voice output: The generated response is converted into voice using speech synthesis software and sent back to the worker.

[1090] Examples

[1091] When a worker asks, "What do you think about the efficiency of the new production line?" the system works as follows:

[1092] Worker: "What do you think about the efficiency of the new production line?"

[1093] Robot: "The latest production technology is a quality control system that utilizes AI. This allows for real-time detection of defective products, greatly improving production efficiency."

[1094] Examples of prompt statements

[1095] A user asks, "What do you think about the efficiency of our new production line?" Generate responses that foster discussion about the latest production technologies and efficiency methods.

[1096] In this way, robots can communicate with workers in real time and discuss the progress of work and problems, which will improve work efficiency and speed up problem solving in the factory.

[1097] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1098] Step 1:

[1099] Audio input

[1100] The user talks to the robot.

[1101] Input: User's voice

[1102] Output: Audio data

[1103] Specific Action: A user asks, "What do you think about the efficiency of our new production line?"

[1104] Step 2:

[1105] Speech Recognition

[1106] The server uses voice recognition software to convert the voice data into text.

[1107] Input: Audio data

[1108] Output: Text data

[1109] What happens: Speech recognition software (e.g. Google Cloud Speech-to-Text API) analyzes the voice data and generates text such as, "What do you think about the efficiency of our new production line?"

[1110] Step 3:

[1111] Text Analytics

[1112] The server uses a natural language processing library to analyze the text data and understand the user's intent.

[1113] Input: Text data

[1114] Output: Analysis results (user intent)

[1115] What happens: A natural language processing library (e.g. NLTK, spaCy) analyzes the text data and understands that the user is looking for information on improving production line efficiency.

[1116] Step 4:

[1117] Input to generative AI models

[1118] The server inputs prompts into the generative AI model based on the analysis results.

[1119] Input: Analysis results (user intent)

[1120] Output: Prompt statement

[1121] Specific operation: The server inputs the prompt sentence "The user asked, 'What do you think about the efficiency of the new production line?' Please generate a response that deepens the discussion about the latest production technologies and methods of improving efficiency" into the generative AI model.

[1122] Step 5:

[1123] Response Generation

[1124] A generative AI model generates an appropriate response based on the prompt.

[1125] Input: Prompt statement

[1126] Output: Response text

[1127] How it works: A generative AI model (e.g., OpenAI's GPT-4) analyzes the prompt sentence and generates a response text that reads, "As the latest production technology, there is a quality control system that utilizes AI. This allows defective products to be detected in real time, greatly improving production efficiency."

[1128] Step 6:

[1129] Audio Output

[1130] The server converts the generated response text into speech using speech synthesis software and responds to the user.

[1131] Input: Response text

[1132] Output: Audio data

[1133] Specific operation: The speech synthesis software converts the response text into voice data, and the robot replies to the user, "As the latest production technology, we have a quality control system that utilizes AI. This allows us to detect defective products in real time, greatly improving production efficiency."

[1134] Example 3

[1135] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1136] Conventional robot systems equipped with artificial intelligence only provide information in response to user opinions, and it is difficult to provide additional information or perspectives to deepen the user's opinions. In addition, it is not possible to provide appropriate feedback in real time to the opinions entered by the user, and it is not possible to fully support the user's learning or deepening of discussions.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1138] In this invention, the server includes a means for a user to input an opinion, a means for the terminal to transmit the opinion to the server, a means for the server to input the opinion as a prompt sentence to the generative AI model, a means for the generative AI model to generate additional information and viewpoints, a means for the server to transmit the generated information to the terminal, and a means for the terminal to display the information to the user. This makes it possible to provide appropriate feedback to the user's opinion in real time and support the deepening of user learning and discussion.

[1139] "User" means an individual or entity that utilizes the System to input opinions and receive feedback.

[1140] A "terminal" is a device through which a user inputs opinions and communicates with the server, and includes a personal computer, smartphone, etc.

[1141] The "server" is a computer system that receives opinions from users, inputs them as prompt sentences into the generative AI model, and transmits the generated information to the terminal.

[1142] A "generative AI model" is an artificial intelligence model that generates additional information or perspectives based on an input prompt sentence, and includes, for example, models that use natural language processing techniques.

[1143] A "prompt" is a text-based instruction that is input to a generative AI model based on the user's input.

[1144] The "means for displaying information" is a function for visually presenting to the user the information that the terminal receives from the server.

[1145] The present invention is a system in which a user inputs an opinion and provides additional information or perspective for the opinion using a generative AI model. A specific embodiment of the system will be described below.

[1146] First, the user inputs his / her opinion using a terminal (e.g., a PC or smartphone). The opinion input by the user is sent as text data from the terminal to the server. The server converts the received opinion into a prompt sentence. An example of a prompt sentence is "User's opinion: The use of renewable energy should be increased. Please provide additional information or perspectives."

[1147] Next, the server inputs the prompt sentence into a generative AI model (for example, a model using natural language processing technology). The generative AI model generates additional information and perspectives based on the prompt sentence. For example, it might generate content such as, "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar and wind power generation are attracting attention as clean energy sources that do not rely on fossil fuels. In addition, costs are gradually falling due to technological advances in renewable energy."

[1148] The generated information is sent from the server to the terminal. The terminal displays the received information to the user. The user can view this information and deepen his / her own opinion.

[1149] In implementing this system, the following hardware and software are used:

[1150] Devices: PCs, smartphones, etc.

[1151] Server: A computer system that receives opinions, generates prompts, inputs them into the generative AI model, and transmits the generated information.

[1152] Generative AI model: An artificial intelligence model that uses natural language processing techniques (e.g., GPT-4)

[1153] As a concrete example, consider the case where a user expresses the opinion that "reducing plastic waste is important." This opinion is input as a prompt sentence to the generative AI model:

[1154] User opinion: Reducing plastic waste is important. Please provide additional information or perspective.

[1155] A generative AI model might generate additional information such as:

[1156] "Reducing plastic waste also helps protect marine ecosystems. Advances in recycling technology are also facilitating the reuse of plastics."

[1157] In this way, the user can obtain new perspectives and information on his / her own opinions. This system supports the user's learning and deepening of discussion. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1158] Step 1:

[1159] The user inputs an opinion.

[1160] The user uses a device (PC or smartphone) to input their opinion into a text box. For example, they might type, "The use of renewable energy should increase." The input opinion is saved as text data on the device.

[1161] Step 2:

[1162] The terminal transmits the opinion to the server.

[1163] The terminal sends the opinions entered by the user to the server as an HTTP request. The input data is sent in text format. The server receives this request and retrieves the opinion data.

[1164] Step 3:

[1165] The server inputs the opinion into the generative AI model as a prompt sentence.

[1166] The server converts the received opinion into a prompt, for example, in the form of "User opinion: The use of renewable energy should be increased. Please provide additional information or perspectives." This prompt is then fed into a generative AI model.

[1167] Step 4:

[1168] Generative AI models generate additional information and perspectives.

[1169] A generative AI model (e.g., a model using natural language processing technology) generates additional information and perspectives based on the prompt sentence. For example, it generates content such as, "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar power generation and wind power generation are attracting attention as clean energy sources that do not depend on fossil fuels. In addition, due to technological advances in renewable energy, costs are gradually decreasing." The generated information is returned to the server as text data.

[1170] Step 5:

[1171] The server transmits the generated information to the terminal.

[1172] The server receives the information obtained from the generative AI model and sends it to the terminal as an HTTP response. The data sent is in text format.

[1173] Step 6:

[1174] The terminal displays the information to the user.

[1175] The terminal displays the information received from the server on the screen. The user can view this information to further develop their own opinion. For example, the following information may be displayed: "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar and wind power generation are attracting attention as clean energy sources that do not depend on fossil fuels. Furthermore, technological advances in renewable energy are gradually reducing costs."

[1176] (Application example 3)

[1177] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1178] Conventional robot systems simply provide responses to user opinions, and lack the ability to provide additional information or perspectives to deepen the user's opinions. In addition, there are limited ways to provide detailed information about products in physical stores, making it difficult for users to obtain sufficient information when selecting products. This creates an issue of limiting the user's purchasing experience.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1180] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics of interest to the user, a means for the robot to reply with its own opinion in addition to the user's reply, a means for the robot to provide additional information and viewpoints about products in a physical store, a means for the robot to obtain the user's opinion and generate additional information using a generative AI model, and a means for the robot to present the generated additional information to the user. This allows the user to gain a deeper understanding of the product, improving the purchasing experience.

[1181] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1182] A "robot" is a mechanical device equipped with artificial intelligence that performs specific tasks automatically.

[1183] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[1184] A "knowledge medium" is a method or technology for conveying information or data to a user.

[1185] A "theme" is a subject matter about a particular topic or issue.

[1186] "Means of discussion" are methods and techniques for interacting with users and exchanging opinions.

[1187] "Means of responding with opinions" refers to methods or technologies that allow a robot to provide its own views in response to a user's questions or opinions.

[1188] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[1189] "Means of providing additional information or perspectives on a product" are methods or technologies that provide more detailed information or a different perspective on the product selected by the user.

[1190] "Means of obtaining user opinions" are the methods and technologies used to collect opinions and questions expressed by users.

[1191] A "generative AI model" is an algorithm or system that enables artificial intelligence to generate new information or answers based on data.

[1192] "Means for generating additional information" are methods or technologies for generating new information or perspectives based on user opinions.

[1193] "Means of presentation to the user" refers to the methods or techniques used to visually or audibly convey the generated information to the user.

[1194] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[1195] System Configuration

[1196] The system consists of an artificial intelligence-enabled robot, smart glasses, an Internet connection, and a server using a generative AI model. Specifically, the system uses the following hardware and software:

[1197] Hardware:

[1198] Smart glasses (e.g. Google Glass)

[1199] Internet connection

[1200] software:

[1201] OpenAI API (generative AI model)

[1202] Python Program

[1203] Processing flow

[1204] 1. Get user feedback:

[1205] Users express their opinions about products through the smart glasses, for example by asking, "Is this product environmentally friendly?"

[1206] 2. Opinion text:

[1207] The smart glasses convert the user's comments into text using voice recognition technology.

[1208] 3. Send to generative AI model:

[1209] The server sends the user's opinion, converted into text, to the OpenAI API. An example of a prompt is as follows:

[1210] What users say about their choice: Is this product eco-friendly?

[1211] Please provide any additional information or perspective.

[1212] 4. Generate additional information:

[1213] The OpenAI API generates additional information and perspectives based on user feedback, such as "This product is made from renewable materials, has a low carbon footprint during manufacturing, and is more energy efficient than other products in the same category."

[1214] 5. User Presentation:

[1215] The server transmits the generated additional information to the smart glasses, which present it visually or audibly to the user.

[1216] Examples

[1217] When a user is selecting a product in a physical store and asks through the smart glasses, "Is this product environmentally friendly?", the system works as follows:

[1218] 1. User opinions are converted into text using voice recognition technology.

[1219] 2. Send the converted text to the OpenAI API.

[1220] 3. The OpenAI API generates additional information, such as, "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category."

[1221] 4. The generated information is presented to the user through smart glasses.

[1222] In this way, users can gain a deeper understanding of the products and improve their purchasing experience.

[1223] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1224] Step 1:

[1225] Users express their opinions about products through smart glasses.

[1226] Input: User's spoken opinion (e.g. "Is this product environmentally friendly?")

[1227] Output: Audio data

[1228] What happens: A user speaks a question about a product into the smart glasses.

[1229] Step 2:

[1230] The terminal uses voice recognition technology to convert the user's comments into text format.

[1231] Input: Audio data

[1232] Output: Text data (e.g. "Is this product environmentally friendly?")

[1233] How it works: The voice recognition software built into the smart glasses analyzes the voice data and generates corresponding text data.

[1234] Step 3:

[1235] The server sends the converted user opinions to the generative AI model.

[1236] Input: Text data

[1237] Output: Prompt statement (e.g., "What do you think about the product the user selected: Is this product environmentally friendly? Please provide any additional information or perspective.")

[1238] Specific operation: The server receives the text data and generates a prompt sentence appropriate for the generative AI model.

[1239] Step 4:

[1240] A generative AI model generates additional information and perspectives based on user input.

[1241] Input: prompt statement

[1242] Output: Additional information (e.g. "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category.")

[1243] What it does: The OpenAI API parses the prompt and generates additional relevant information and perspectives.

[1244] Step 5:

[1245] The server transmits the generated additional information to the smart glasses for presentation to the user.

[1246] Input: Additional information

[1247] Output: Information presented to the user (e.g., "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category.")

[1248] Specific operation: The server sends the generated additional information to the smart glasses, which then present the information to the user visually or audibly.

[1249] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1250] "Example 1"

[1251] As one embodiment of the present invention, a system is provided in which a robot equipped with artificial intelligence includes an emotion engine that recognizes the emotion of a user. This emotion engine estimates the emotion of the user from the user's facial expression, tone of voice, and the content of the remarks, and reflects the result in the robot's response. For example, if the robot feels that the user is struggling with learning, it responds by offering words of encouragement or changing the learning content to increase motivation. Also, if the robot feels that the user is excited and arguing, it responds by sharing that excitement and encouraging further discussion.

[1252] "Example 2"

[1253] As another embodiment of the present invention, a system is provided for young people who use SNS. In this system, an emotion engine estimates the user's emotion from the content of posts on the SNS. For example, if a user posts on the SNS that they are feeling stressed, the robot responds by encouraging the user based on that information. Also, if a user posts on the SNS that they are excited and having an argument, the robot responds by deepening the discussion with the user based on that information.

[1254] "Example 3"

[1255] As yet another embodiment of the present invention, a system is provided for elderly people with early dementia. In this system, an emotion engine estimates the emotion of the elderly person from their daily facial expressions and tone of voice, and reflects the result in the robot's response. For example, if it is estimated that the elderly person feels lonely, the robot will become a conversation partner and respond in a way that reduces the elderly person's sense of loneliness. Also, if it is estimated that the elderly person is excited about something, the robot will share that excitement and respond in a way that adds vitality to the elderly person's life.

[1256] The process flow of each embodiment will be described below.

[1257] "Example 1"

[1258] Step 1: The robot initiates a dialogue with the user.

[1259] Step 2: The emotion engine infers emotions from the user's facial expressions, tone of voice, and content of speech.

[1260] Step 3: The robot responds based on the estimated emotions. For example, if the robot senses that the user is struggling with their learning, it can offer encouraging words or change the learning content to increase motivation.

[1261] "Example 2"

[1262] Step 1: The robot analyzes the user's social media posts.

[1263] Step 2: The emotion engine infers the user's emotions from the content of their social media posts.

[1264] Step 3: The robot responds based on the estimated emotions. For example, if a user posts on social media that they are feeling stressed, the robot can respond by encouraging the user based on that information.

[1265] "Example 3"

[1266] Step 1: The robot initiates a dialogue with the elderly person.

[1267] Step 2: The emotion engine estimates the elderly person's emotions from their everyday facial expressions and tone of voice.

[1268] Step 3: The robot responds based on the estimated emotions. For example, if it estimates that an elderly person is feeling lonely, the robot will engage in conversation and respond in a way that reduces the elderly person's sense of loneliness.

[1269] Example 1

[1270] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1271] Conventional learning support robots could not recognize the user's emotions and change their response accordingly, making it difficult to effectively improve the user's motivation to learn. In addition, they could not respond appropriately when the user was confused or excited, which could reduce the efficiency of learning. Furthermore, there was a demand for a system that could provide appropriate answers and solutions to the user's questions and problems.

[1272] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1273] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam study, etc., a means for the robot to talk to and discuss topics that interest the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to include an emotion engine that recognizes the user's emotions, a means for the emotion engine to estimate the user's emotions from the user's facial expressions, tone of voice, and remarks and reflect the results in the robot's response, a means for the robot to provide answers or solutions to the user's questions or problems using a generative AI model, and a means for the robot to generate a response according to the user's emotions. This makes it possible to respond appropriately according to the user's emotions and improve their motivation to learn.

[1274] A "robot equipped with artificial intelligence" is a robot that incorporates artificial intelligence technologies such as machine learning and natural language processing, and can converse with users and provide them with information.

[1275] "Means of providing further knowledge about the content of exam preparation, etc." refers to a function that provides additional information or explanations related to the content the user is studying.

[1276] "A means of talking and discussing topics of interest" refers to a function that enables users to have dialogue, exchange opinions, and discuss topics that interest them.

[1277] "A means for the robot to respond with its own opinion in addition to its own answer" refers to a function in which the robot provides its own views and opinions in addition to the answer to the user's question.

[1278] An "emotion engine" refers to technology for analyzing and inferring emotions from a user's facial expressions, tone of voice, content of statements, etc.

[1279] A "generative AI model" refers to an artificial intelligence model that generates appropriate answers or solutions based on user input.

[1280] "Means for generating a response according to the user's emotions" refers to a function for generating a response appropriate to the user's emotions based on the analysis results of the emotion engine.

[1281] The present invention relates to a system for assisting a user in learning and recognizing emotions by using a robot equipped with artificial intelligence. A specific embodiment of this system will be described below.

[1282] Hardware and Software Configuration

[1283] server

[1284] The server hosts an artificial intelligence model and an emotion engine. The artificial intelligence model uses a general generative AI model. The emotion engine uses a general emotion analysis API, which can analyze user input data and generate appropriate answers or solutions.

[1285] Terminal (robot)

[1286] The terminal is responsible for receiving input from the user and sending it to the server. The terminal is equipped with a touch screen and voice input capabilities, allowing the user to input questions and problems in a natural way. The terminal also has a display and speakers to provide the user with responses received from the server.

[1287] Data processing and calculation

[1288] Sending user input data

[1289] When a user inputs a question or problem into the terminal, the terminal sends the data in text format to the server. For example, if a user inputs "Teach me how to solve a quadratic equation," the text data is sent to the server.

[1290] Analysis using artificial intelligence models

[1291] The server uses a generative AI model to analyze input data. For example, in response to the input "Teach me how to solve quadratic equations," the generative AI model generates a solution to the quadratic equation. Specifically, the server generates the answer, "There are two ways to solve quadratic equations: completing the square and factoring. For example, if x^2 + 5x + 6 = 0, using the factorization method results in (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3."

[1292] Emotion analysis using emotion engine

[1293] The server sends the user's facial expression, tone of voice, and the content of the speech to the emotion analysis API, which analyzes the user's emotions. For example, if the user is judged to be confused, the result is returned to the server.

[1294] Generating a response

[1295] The server generates an appropriate response based on the answer of the generative AI model and the analysis results of the emotion engine. For example, if the server determines that the user is confused, it adds an encouraging message. Specifically, it generates a message saying, "If there's anything you don't understand, please feel free to ask, and I'll explain other ways too."

[1296] Providing a response

[1297] The server sends the generated response to the terminal as text data. The terminal displays the response received from the server on the screen and reads it out loud. For example, it displays and reads out, "There are two methods for solving quadratic equations: completing the square and factorization. To give a specific example, if x^2 + 5x + 6 = 0, using the factorization method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3. If you don't understand something, feel free to ask, as I will explain other methods as well."

[1298] Examples

[1299] For example, if a user types "How do I solve a quadratic equation?", the following prompt is generated:

[1300] User: How do I solve a quadratic equation?

[1301] Robot: There are several ways to solve quadratic equations, such as completing the square or factoring. For example, if x^2 + 5x + 6 = 0, using factoring gives (x + 2)(x + 3) = 0, which gives the solutions x = -2 and x = -3. Let me know if you have any problems.

[1302] In this way, a system is realized that supports the user's learning and responds according to their emotions.

[1303] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1304] Step 1:

[1305] The user types a question or problem into the terminal.

[1306] Specific action: The user uses the robot's touchscreen or voice input function to input, "Teach me how to solve a quadratic equation."

[1307] Input: The user's question or problem (e.g., "Can you help me solve a quadratic equation?")

[1308] Output: User input as text data

[1309] Step 2:

[1310] The terminal transmits the input data to the server.

[1311] Specific operation: The terminal transmits the user's input as text data to the server, for example, by using an HTTP request.

[1312] Input: User input as text data

[1313] Output: Text data sent to the server

[1314] Step 3:

[1315] The server uses an AI model to analyze the input data and generate an answer or solution.

[1316] Specific operation: The server calls the generative AI model and gives it the prompt "Teach me how to solve a quadratic equation" as input. The generative AI model generates a solution to the quadratic equation.

[1317] Input: User input as text data

[1318] Output: Generated answers or solutions (e.g. "There are two methods for solving quadratic equations: completing the square and factoring. For example, if x^2 + 5x + 6 = 0, using the factoring method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3.")

[1319] Step 4:

[1320] The server uses an emotion engine to analyze the user's emotions.

[1321] Specific operation: The server sends the user's facial expression, tone of voice, and the content of the statement to the emotion analysis API, which analyzes the emotion. For example, if the user is judged to be confused, the result is returned to the server.

[1322] Input: User facial expressions, tone of voice, and what is being said

[1323] Output: Parsed emotion data (e.g. confused)

[1324] Step 5:

[1325] The server generates an appropriate response based on the analysis results.

[1326] Specific operation: In addition to the answer of the generative AI model, the server adds an encouraging message based on the results of the emotion engine, for example, "If you don't understand something, please feel free to ask me, I will explain other ways too."

[1327] Input: Generated answers, solutions, and analyzed emotion data

[1328] Output: Appropriate response (e.g., "There are two ways to solve quadratic equations: completing the square and factoring. For example, if x^2 + 5x + 6 = 0, using the factoring method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3. If you don't understand something, feel free to ask, I'll explain other methods too.")

[1329] Step 6:

[1330] The server generates a response and sends it to the terminal.

[1331] Specific operation: The server sends the generated response to the terminal as text data, for example, as an HTTP response.

[1332] Input: Appropriate response

[1333] Output: Text data sent to the terminal.

[1334] Step 7:

[1335] The terminal provides the response to the user.

[1336] Specific operation: The terminal displays the response received from the server on the screen and reads it out loud. For example, "There are two methods for solving quadratic equations: completing the square and factorization. To give a specific example, if x^2 + 5x + 6 = 0, using the factorization method gives (x + 2)(x + 3) = 0, and the solutions are x = -2 and x = -3. If you don't understand something, feel free to ask, as I will explain other methods as well." is displayed and read out loud.

[1337] Input: Text data sent to the terminal

[1338] Output: The response provided to the user (both displayed on screen and read aloud)

[1339] (Application example 1)

[1340] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1341] Conventional factory robots were unable to recognize the emotions and conditions of workers, making it difficult for them to provide appropriate support to maximize work efficiency. In addition, they were unable to provide appropriate advice or suggest breaks when workers faced difficult tasks or felt fatigued. This resulted in problems of reduced work efficiency and increased stress for workers.

[1342] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1343] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam study etc., a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to recognize the worker's emotions and provide appropriate support, a means for the robot to recognize the voice of the worker and perform emotion analysis, and a means for the robot to provide advice to improve work efficiency based on the worker's emotions. This makes it possible to recognize the worker's emotions and state in real time and provide appropriate support and advice.

[1344] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and perform functions such as learning, reasoning, and problem-solving.

[1345] A "robot" is a mechanical device that operates autonomously according to programmed instructions to perform specific tasks.

[1346] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[1347] "Knowledge delivery means" are methods and techniques for presenting users with the information and solutions they need.

[1348] "Discussion tools" are methods and techniques for interacting with users and exchanging opinions on a particular topic.

[1349] "Means of responding" refers to the methods and technologies that a robot uses to communicate its views and suggestions to the user.

[1350] "Means of recognizing emotions" refers to methods or technologies for inferring emotions from a user's facial expressions, tone of voice, content of speech, etc.

[1351] "Means of providing support" refers to methods and technologies for providing appropriate advice and assistance based on the user's condition and needs.

[1352] A "means for recognizing speech" is a method or technology for converting a user's speech into text and understanding its content.

[1353] "Means for performing sentiment analysis" refers to methods or technologies for analyzing emotions from the content and tone of a user's speech and utilizing the results.

[1354] "Means for providing advice" refers to methods or technologies for providing appropriate advice or suggestions based on the user's situation.

[1355] The system for implementing this invention uses a robot equipped with artificial intelligence. The robot is equipped with multiple means for recognizing the emotions of the worker and providing appropriate support. The specific configuration and operation of the system are described below.

[1356] System Configuration

[1357] 1. Hardware

[1358] Camera: Used to capture the facial expressions of the worker.

[1359] Microphone: Used to capture the voice of the worker.

[1360] Robot body: A device equipped with artificial intelligence and used to interact with workers.

[1361] 2. Software

[1362] OpenCV: A library for performing face recognition.

[1363] TensorFlow / Keras: Libraries for running emotion recognition models.

[1364] SpeechRecognition: A library for performing speech recognition.

[1365] TextBlob: A library for converting speech to text and performing sentiment analysis.

[1366] System Operation

[1367] 1. Emotion recognition

[1368] The worker's facial expressions are captured using a camera and the face is recognized using OpenCV.

[1369] The recognized face images are input into a TensorFlow / Keras model to estimate emotions.

[1370] 2. Speech Recognition and Sentiment Analysis

[1371] The worker's voice is captured using a microphone and converted to text using the SpeechRecognition library.

[1372] The converted text is analyzed using TextBlob to estimate the sentiment.

[1373] 3. Support Provision

[1374] Based on the estimated emotions, the robot will provide appropriate support and advice.

[1375] For example, if the robot estimates that a worker is tired, it will suggest, "We recommend that you take a break."

[1376] If a worker is faced with a difficult task, the robot will provide technical information such as, "Try the following steps to complete this task."

[1377] Examples

[1378] If the worker is tired: The robot will suggest, "It is recommended that you take a break."

[1379] If a worker is faced with a difficult task: the robot will provide technical information such as "Try the following steps to complete this task."

[1380] Examples of prompt statements

[1381] “What suggestions would you make if a worker feels fatigued?”

[1382] "What technical information do you provide to workers when they're facing a difficult task?"

[1383] In this way, the system can recognize the worker's emotions and state in real time and provide appropriate support and advice.

[1384] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1385] Step 1:

[1386] Camera-based facial expression capture

[1387] The server captures the facial expressions of the workers using a camera. The input is the video data from the camera, and the output is the captured image data. Specifically, the camera captures the video in real time and transmits the video data to the server.

[1388] Step 2:

[1389] Facial Recognition

[1390] The server uses OpenCV to recognize faces from the captured image data. The input is the image data obtained in step 1, and the output is the location information of the recognized face. Specifically, it applies the OpenCV face recognition algorithm to identify the location of the face in the image.

[1391] Step 3:

[1392] Emotion Estimation

[1393] The server uses a TensorFlow / Keras model to estimate emotions from the recognized face images. The input is the face position information and image data obtained in step 2, and the output is the estimated emotion label. Specifically, the face part is cut out and preprocessed, and then input to the emotion recognition model to estimate the emotion.

[1394] Step 4:

[1395] Audio capture

[1396] The server captures the voice of the worker using a microphone. The input is the voice data from the microphone, and the output is the captured voice data. Specifically, the microphone captures the voice in real time and transmits the voice data to the server.

[1397] Step 5:

[1398] Speech Recognition

[1399] The server converts the captured voice data into text using the SpeechRecognition library. The input is the voice data obtained in step 4, and the output is the converted text data. Specifically, it applies a speech recognition algorithm to convert the voice data into text.

[1400] Step 6:

[1401] sentiment analysis

[1402] The server uses TextBlob to analyze sentiment from the converted text data. The input is the text data obtained in step 5, and the output is the analyzed sentiment label. Specifically, it applies the sentiment analysis function of TextBlob to estimate the sentiment of the text.

[1403] Step 7:

[1404] Support Offering

[1405] The server provides appropriate support and advice based on the estimated emotion label. The input is the emotion label obtained in step 3 and step 6, and the output is the content of the support and advice to be provided. Specifically, the server selects a pre-defined support message according to the emotion label and presents it to the worker.

[1406] In this way, the system can recognize the worker's emotions and state in real time and provide appropriate support and advice.

[1407] Example 2

[1408] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1409] Conventional robot systems only have limited functionality in terms of interacting with users and analyzing their emotions, making it difficult to meet the diverse needs of users. In addition, there was a lack of systems that could respond appropriately to young people who use social media and elderly people with early dementia. This resulted in problems of reduced user satisfaction and convenience.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1411] In this invention, the server includes a means for performing voice recognition of user comments, a means for generating an appropriate response using a generative AI model, and a means for analyzing SNS posts with an emotion engine, which makes it possible to generate an appropriate response to a user comment and to analyze the user's emotion from the SNS post and take an appropriate action.

[1412] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1413] A "robot" is a mechanical device equipped with artificial intelligence and capable of performing physical movements and interacting with other people.

[1414] "Contents of exam preparation, etc." refers to knowledge and information related to academic studies and exams.

[1415] A "means of providing knowledge" is a method or technology for conveying information or knowledge to a user.

[1416] "Topics of interest" refer to topics or areas that interest a user.

[1417] "Means for conversation and discussion" refers to methods and techniques for interacting with users and exchanging opinions and holding discussions.

[1418] The "means for responding with the robot's opinion by adding it to its own answer" refers to a method or technology in which the robot adds its own views or opinions to the answer to the user's question.

[1419] A "speech recognition means" is a technology or device for converting speech into text data.

[1420] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate data such as text or voice.

[1421] A "means for generating an appropriate response" is a method or technique for creating an appropriate reply to a user's input.

[1422] "SNS post" refers to posts or comments made by users on social networking services.

[1423] An "emotion engine" is a technology or system for analyzing emotions from text data and inferring emotional states.

[1424] "Analytical means" refers to methods and techniques for analyzing data and understanding its content and characteristics.

[1425] A "means for generating an appropriate response" is a method or technique for creating an appropriate action or response to the user based on the analysis results.

[1426] The present invention is a system that uses a robot equipped with artificial intelligence to communicate with users and analyze posts on SNS. A specific embodiment of this system will be described below.

[1427] Hardware and Software Used

[1428] Hardware: Robot body (e.g. general-purpose robot)

[1429] Software: emotion engines, generative AI models (e.g. GPT-4), speech recognition software (e.g. speech recognition APIs), speech synthesis software (e.g. speech synthesis APIs)

[1430] System Overview

[1431] This system allows users to talk to the robot and discuss topics they are interested in. It also uses an emotion engine to analyze content posted by users on social media and respond appropriately.

[1432] Program Processing

[1433] 1. Start a dialogue with the user

[1434] The user talks to the robot.

[1435] Example: "Tell me about recent advances in science and technology."

[1436] 2. Voice Recognition

[1437] The terminal (the robot itself) records the user's speech using a built-in microphone.

[1438] The server uses voice recognition software (e.g., a voice recognition API) to convert the recorded voice data into text data.

[1439] Example: Generates the text "Tell me about recent advances in science and technology."

[1440] 3. Using generative AI models

[1441] The server uses a generative AI model (e.g. GPT-4) to generate an appropriate response based on the text data.

[1442] Example: "A recent advance in science and technology is the development of quantum computers, which have the potential to dramatically increase the speed of computation."

[1443] 4. Speech synthesis

[1444] The device converts the generated text response into speech using speech synthesis software (e.g., a speech synthesis API) and responds to the user.

[1445] Example: The robot says, "Recent advances in science and technology include the development of quantum computers, which have the potential to dramatically increase the speed of calculations."

[1446] 5. Analysis of social media posts

[1447] A user posts his / her feelings and opinions on a SNS (e.g., social networking service).

[1448] The server retrieves user posts using the SNS API and analyzes them using an emotion engine (e.g., emotion analysis API).

[1449] Example: The content of your post is analyzed to indicate that you are feeling "stressed."

[1450] 6. Generate appropriate responses

[1451] Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[1452] Example: "You've had a tough day today. Time to take a break and refresh yourself." would generate the response.

[1453] 7. Responses by voice synthesis

[1454] The terminal converts the generated text response into speech using speech synthesis software and responds to the user.

[1455] Example: The robot says, "You had a hard time today. Let's take a break and refresh yourself."

[1456] Examples and prompts

[1457] Example 1: Discussion of technological advances

[1458] User: "Tell me about recent advances in science and technology."

[1459] Robot: "A recent advance in science and technology is the development of quantum computers, which have the potential to dramatically increase the speed of computation."

[1460] Example 2: Responses based on social media posts

[1461] User's social media post: "I'm feeling very stressed today"

[1462] Robot: "You've had a hard day today. Let's take a break and refresh yourself."

[1463] Examples of prompt statements

[1464] Prompt: "When the user asks about recent advances in science and technology, generate an answer that provides a discussion of current science and technology trends and their impact."

[1465] In this way, the system can generate appropriate responses based on users' comments and social media posts, deepening the dialogue with users.

[1466] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1467] Step 1:

[1468] The user talks to the robot.

[1469] Input: User's voice input.

[1470] Specific action: The user talks to the robot about a topic that interests them.

[1471] Output: The robot's microphone captures audio data.

[1472] Step 2:

[1473] The device recognizes the user's speech.

[1474] Input: The captured audio data.

[1475] Specific operation: The terminal (the robot itself) records what the user says using the built-in microphone.

[1476] Output: The recorded audio data is sent to the server.

[1477] Step 3:

[1478] The server converts the voice data into text data.

[1479] Input: Recorded audio data.

[1480] Specific operation: The server uses voice recognition software (e.g., a voice recognition API) to convert the voice data into text data.

[1481] Output: Text data is generated.

[1482] Step 4:

[1483] The server uses the generative AI model to generate an appropriate response.

[1484] Input: The generated text data.

[1485] What happens: The server uses a generative AI model (e.g. GPT-4) to generate an appropriate response based on the text data.

[1486] Output: The text data of the response is generated.

[1487] Step 5:

[1488] The terminal vocally responds with the generated response to the user.

[1489] Input: The text data of the response.

[1490] Specific behavior: The device converts the generated text response into speech using speech synthesis software (e.g., a speech synthesis API) and responds to the user.

[1491] Output: The robot responds to the user via voice.

[1492] Step 6:

[1493] A user posts on a social networking site.

[1494] Input: User's social media posts.

[1495] Specific action: A user posts their feelings or opinions on a SNS (e.g., a social networking service).

[1496] Output: Text data posted to SNS.

[1497] Step 7:

[1498] The server analyzes social media posts using an emotion engine.

[1499] Input: Text data posted on SNS.

[1500] Specific operation: The server retrieves user posts using the SNS API and analyzes them using an emotion engine (e.g. emotion analysis API).

[1501] Output: The sentiment analysis results are generated.

[1502] Step 8:

[1503] The server generates an appropriate response based on the analysis results.

[1504] Input: Sentiment analysis results.

[1505] Specific operation: Based on the analysis results, the server uses a generative AI model to generate an appropriate response.

[1506] Output: The corresponding text data is generated.

[1507] Step 9:

[1508] The terminal responds to the user by voice with the generated response.

[1509] Input: The corresponding text data.

[1510] What happens: The device uses speech synthesis software to convert the generated text response into speech and responds to the user.

[1511] Output: The robot responds to the user via voice.

[1512] (Application example 2)

[1513] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1514] In modern society, the use of social networking sites is spreading, especially among young people, but it has become a problem that emotions such as stress, anxiety, and excitement on social networking sites affect users' mental health. In addition, since users have limited opportunities to discuss topics that interest them in depth, there is a demand to deepen knowledge and improve the quality of discussions. In order to solve these issues, a system that can properly analyze users' emotions and provide appropriate responses based on them is required.

[1515] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1516] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics that interest the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to analyze the contents of posts on SNS and estimate the user's emotions, and a means for the robot to provide an appropriate response to the user based on the estimated emotions. This makes it possible to appropriately analyze the user's emotions, support mental health, and deepen knowledge and improve the quality of discussion.

[1517] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1518] A "robot" is a mechanical device equipped with artificial intelligence that performs specific tasks automatically.

[1519] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[1520] A "knowledge medium" is a method or technique used to provide new information or understanding to a user.

[1521] A "discussion tool" is a method or technique for exchanging opinions with users on a particular topic.

[1522] "Means of responding" refers to the methods or technologies that a robot uses to communicate its views and thoughts to the user.

[1523] "SNS" is an abbreviation for Social Networking Service, a platform for users to share information and communicate online.

[1524] "Means of analyzing post content" refers to methods and technologies for analyzing users' posts on SNS and understanding their content.

[1525] "Means of inferring emotions" refers to methods or technologies for determining a user's emotional state from the content of their posts.

[1526] The "means for providing an appropriate response" refers to a method or technology for providing appropriate advice or support to a user based on the estimated emotions.

[1527] The system for implementing this invention is mainly composed of a robot equipped with artificial intelligence. The robot is equipped with an emotion engine that analyzes the content posted by users on SNS and infers their emotions. It also has a function to provide an appropriate response to the user based on the inferred emotions.

[1528] System Configuration

[1529] The system consists of the following main components:

[1530] 1. Robots equipped with artificial intelligence: They have the ability to interact with users and deepen discussions.

[1531] 2. Emotion engine: Analyzes the content of social media posts and estimates the user's emotions.

[1532] 3. Response Providing Module: Provides an appropriate response to the user based on the estimated sentiment.

[1533] Hardware and Software Used

[1534] Hardware: Smartphones, servers, robots

[1535] Software: Python, requests library, TextBlob library, SNS platform API, emotion engine API

[1536] Data processing and calculation

[1537] 1. Data collection: The server collects user posts using the API of the SNS platform.

[1538] 2. Sentiment analysis: The collected post content is sent to the emotion engine API to analyze the emotions.

[1539] 3. Response delivery: Based on the analysis results, the response delivery module provides appropriate advice and support to users.

[1540] Examples

[1541] For example, if a user posts on social media that "I've been busy at work lately and I'm feeling stressed," the server sends the post to the emotion engine API, which analyzes the emotion of stress. The response module then suggests to the user, "Here are some ways to relax. Take a deep breath and listen to some relaxing music."

[1542] Examples of prompt statements

[1543] A user posted on social media that "I've been busy at work lately and I'm feeling stressed." Analyze this post and suggest an appropriate response to the user.

[1544] In this way, it is possible to properly analyze users' emotions, support their mental health, and deepen knowledge and improve the quality of discussion.

[1545] The flow of the identification process in the application example 2 will be described with reference to FIG.

[1546] Step 1:

[1547] The server collects user posts using the API of the SNS platform. Specifically, it sends an API request specifying the user ID to retrieve the user's latest posts. The input is the user ID, and the output is the user's posts.

[1548] Step 2:

[1549] The server sends the collected post content to the emotion engine API and analyzes the emotions. Specifically, it sends the post content in JSON format to the emotion engine API and receives the emotion analysis results. The input is the user's post content, and the output is the emotion analysis results.

[1550] Step 3:

[1551] The server calls the response module based on the emotion analysis result and provides the user with an appropriate response. Specifically, the response module generates appropriate advice or support based on the emotion analysis result and notifies the user. The input is the emotion analysis result, and the output is the advice or support for the user.

[1552] Step 4:

[1553] The user receives and acts on the advice and support provided by the server. Specifically, the user checks the notification from the server and puts into practice the relaxation methods and discussion information. The input is the advice and support from the server, and the output is the user's behavior.

[1554] Step 5:

[1555] The server collects user feedback and uses it to improve the system. Specifically, it collects feedback about how users felt about the advice and support provided, and adjusts the system's algorithm. The input is user feedback, and the output is the improvement of the system.

[1556] Example 3

[1557] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1558] Conventional robots equipped with artificial intelligence could only give simple responses to user opinions, making it difficult to hold deep discussions or provide additional information. They were also unable to respond appropriately to elderly people with early-stage dementia, making it difficult to alleviate their feelings of loneliness and anxiety. This made it difficult to deepen the user's opinions and improve the quality of life for the elderly.

[1559] The identification process by the identification processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means. In this invention, the server includes a means for inputting the user's opinion into the generative AI model and providing additional information and viewpoints, a means for estimating the elderly person's emotions from their everyday facial expressions and tone of voice and responding based on the results, and a means for the robot to respond with its own opinion in addition to its own response. This allows the user's opinion to be deepened, and makes it possible to respond appropriately according to the elderly person's emotions.

[1560] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and perform functions such as learning, reasoning, and problem-solving.

[1561] A "robot" is a mechanical device equipped with artificial intelligence and capable of performing physical movements and providing vocal responses.

[1562] A "generative AI model" is an artificial intelligence algorithm that generates new information or responses based on input data.

[1563] A "prompt" is an instruction given to a generative AI model that serves as the basis for the model to generate a response.

[1564] An "emotion engine" is software that estimates human emotions from data such as voice and facial expressions.

[1565] A "user" is a person who operates the system and inputs opinions.

[1566] "Elderly" generally refers to older people, especially those with early symptoms of dementia.

[1567] An "opinion" is a thought or assertion that a user inputs into the system.

[1568] "Additional information or perspectives" refers to supplementary knowledge or different perspectives that the generative AI model provides in response to the user's opinion.

[1569] "Response" refers to the responses and actions that the robot takes toward users and elderly people.

[1570] "Facial expressions" are the movements and expressions of a human face that indicate emotions.

[1571] "Voice tone" refers to characteristics of the voice, such as pitch, strength, and rhythm, and is an element that indicates emotion.

[1572] This invention is a system that uses a robot equipped with artificial intelligence to provide additional information and perspectives to the user's opinions and respond appropriately according to the feelings of the elderly. A specific embodiment of this system will be described below.

[1573] Hardware and Software Used

[1574] Hardware: A high-performance server, the device operated by the user (PC, smartphone, tablet, etc.), and the robot itself (equipped with a voice recognition microphone, camera, and speaker)

[1575] Software: Generative AI models (e.g. GPT-4), emotion engines (e.g. Affectiva), voice recognition software (e.g. Google Speech-to-Text)

[1576] System program processing

[1577] User opinion input and analysis

[1578] A user uses a terminal to input their opinion in text format. For example, the user inputs, "Regarding environmental issues, I think reducing plastic waste is important." The terminal sends this opinion to the server. The server inputs the received opinion into a generative AI model and creates a prompt sentence. For example, it generates a prompt sentence such as, "User's opinion: Regarding environmental issues, I think reducing plastic waste is important. Please provide any additional information or perspective."

[1579] Response generation for generative AI models

[1580] The server receives a response from the generative AI model, for example, "Reducing plastic waste is certainly important. Furthermore, we should also consider promoting recycling and developing alternative materials." The server then sends this response to the robot.

[1581] Robot Response

[1582] The robot will respond to the user with voice based on the response it receives. For example, it might say to the user, "It is certainly important to reduce plastic waste. Furthermore, we should also consider promoting recycling and developing alternative materials."

[1583] Estimating and responding to emotions of the elderly

[1584] The robot uses a camera and microphone to collect the elderly person's everyday facial expressions and tone of voice. The collected data is sent to an emotion engine to estimate their emotions. For example, it may estimate that the elderly person is feeling lonely. Based on the results of the emotion engine, the robot will ask, "How are you doing lately? Is there anything you'd like to talk about?"

[1585] Examples

[1586] User feedback:

[1587] The user types, "Regarding environmental issues, I think reducing plastic waste is important" into a text box displayed on the smartphone screen.

[1588] When the terminal presses the "Send" button, the opinion data is sent to the server as an HTTP POST request.

[1589] Use of generative AI models:

[1590] The server inputs the opinions into a generative AI model to create a prompt.

[1591] For example, a prompt sentence such as "User's opinion: Regarding environmental issues, I think reducing plastic waste is important. Please provide additional information or perspectives." is generated.

[1592] Response from the generative AI model:

[1593] The server receives and analyzes the response from the generative AI model.

[1594] For example, you might receive a response like, "Reducing plastic waste is certainly important. In addition, promoting recycling and developing alternative materials should also be considered."

[1595] The robot responds:

[1596] The robot responds through the speaker, "Reducing plastic waste is certainly important. Furthermore, we should also consider promoting recycling and developing alternative materials."

[1597] Estimating and responding to emotions of the elderly:

[1598] The robot captures the elderly person's facial expressions with a camera and collects the tone of their voice with a microphone.

[1599] The collected data is sent to an emotion engine in real time for emotion analysis.

[1600] For example, if the robot estimates that you are feeling lonely, it will ask, "How are you doing lately? Is there anything you'd like to talk about?"

[1601] In this way, the system can provide appropriate information in response to the user's opinion and respond according to the elderly person's feelings. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1602] Step 1:

[1603] The user inputs an opinion.

[1604] The user uses the terminal to input an opinion in text format. For example, the user inputs "Regarding environmental issues, I think reducing plastic waste is important." The input opinion is displayed in a text box on the terminal.

[1605] Step 2:

[1606] The terminal transmits the opinion to the server.

[1607] The terminal sends the inputted opinion to the server as an HTTP POST request. Specifically, the opinion data is sent to the server in JSON format. The input is the user's opinion, and the output is the opinion data sent to the server.

[1608] Step 3:

[1609] The server inputs the opinions into a generative AI model.

[1610] The server creates a prompt sentence to input the received opinion into the generative AI model. For example, it generates a prompt sentence like "User's opinion: Regarding environmental issues, I think reducing plastic waste is important. Please provide additional information or perspectives." The input is the user's opinion data, and the output is the generated prompt sentence.

[1611] Step 4:

[1612] The server receives the response from the generative AI model.

[1613] The server sends a prompt to the generative AI model and receives a response. For example, it receives a response such as, "Reducing plastic waste is certainly important. In addition, we should also consider promoting recycling and developing alternative materials." The input is the prompt, and the output is the response from the generative AI model.

[1614] Step 5:

[1615] The server sends the response to the robot.

[1616] The server sends the response from the generative AI model to the robot. Specifically, it sends the response data using the robot's API. The input is the response from the generative AI model, and the output is the response data sent to the robot.

[1617] Step 6:

[1618] The robot responds to the user.

[1619] The robot responds to the user by voice based on the response it receives. For example, it might say to the user, "Reducing plastic waste is certainly important. Furthermore, we should also consider promoting recycling and developing alternative materials." The input is the response data sent to the robot, and the output is a voice response to the user.

[1620] Step 7:

[1621] A robot estimates the emotions of elderly people.

[1622] The robot collects the elderly person's daily facial expressions and tone of voice using a camera and microphone. The collected data is sent to an emotion engine to estimate their emotions. For example, it estimates that the elderly person feels lonely. The input is the elderly person's facial expressions and tone of voice data, and the output is the estimated emotion data.

[1623] Step 8:

[1624] The robot responds based on emotions.

[1625] Based on the results of the emotion engine, the robot responds according to the elderly person's emotions. For example, if it estimates that the elderly person is feeling lonely, the robot will ask, "How are you doing lately? Is there anything you'd like to talk about?" The input is the estimated emotion data, and the output is a voice response to the elderly person.

[1626] (Application example 3)

[1627] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1628] In modern society, elderly people are becoming increasingly lonely and anxious. It is also important to provide a sense of security in their daily lives, especially for those with early-stage dementia. However, with current technology, it is difficult to grasp the emotions of elderly people in real time and respond appropriately. There is also a lack of means to respond quickly when an abnormality occurs. Furthermore, there is no adequate system in place to provide specific advice to users.

[1629] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means. In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam study, etc., a means for the robot to talk to the user and discuss the topic of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to estimate the user's emotion from the user's facial expression and tone of voice using an emotion engine and reflect the result in the robot's response, a means for the robot to send a notification when the robot detects an abnormality, and a means for the robot to provide specific advice to the user using a generative AI model. This makes it possible to grasp the elderly's emotions in real time and take appropriate measures. In addition, it is possible to quickly respond when an abnormality occurs and provide specific advice to the user.

[1630] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and perform functions such as learning, reasoning, and problem-solving.

[1631] A "robot" is a mechanical device equipped with artificial intelligence that performs specific tasks automatically.

[1632] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[1633] "Knowledge delivery means" are methods and technologies used to convey information or data to users and deepen their understanding.

[1634] "Means of conversation and discussion" are methods and techniques for interacting with users and exchanging opinions and holding debates.

[1635] "Means of responding to opinions" refers to methods or technologies that allow a robot to provide its own views or information in response to a user's questions or opinions.

[1636] The "emotion engine" is a technology that infers a user's emotions from their facial expressions and tone of voice, and responds appropriately based on the results.

[1637] "Means of detecting anomalies" are methods or technologies for identifying situations or behaviors that deviate from normal conditions and for issuing warnings or notifications.

[1638] A "notification mechanism" is a method or technique for communicating alerts or information to designated recipients when an anomaly is detected.

[1639] A "generative AI model" is an algorithm or technique that enables artificial intelligence to generate new information or advice based on user input.

[1640] "Means of providing specific advice" are methods or techniques for providing practical, specific advice or suggestions tailored to the user's situation and needs.

[1641] The system for implementing this invention is composed mainly of a robot equipped with artificial intelligence. The robot analyzes the user's facial expressions and tone of voice in real time and estimates the user's emotions using an emotion engine. This allows the robot to respond appropriately according to the user's emotional state.

[1642] Hardware and Software Used

[1643] Hardware: Smartphone, camera, microphone

[1644] Software: Python, OpenCV, Keras, Twilio

[1645] System Configuration

[1646] 1. Emotion recognition:

[1647] It uses your smartphone's camera and microphone to capture your facial expressions and tone of voice.

[1648] Detect faces using OpenCV and analyze facial expressions using Keras emotion recognition models.

[1649] The emotion engine estimates the user's emotions and reflects the results in the robot's response.

[1650] 2. Anomaly detection and notification:

[1651] If emotion recognition detects an abnormality (e.g. anxiety, excitement), Twilio is used to send a notification to family members or security services.

[1652] The notification contains information about the user's current emotional state and required action.

[1653] 3. Generative AI models provide advice:

[1654] It uses generative AI models to generate specific advice based on user input.

[1655] For example, if a user inputs, "Please tell me how to respond if an elderly person feels anxious when alone," the generative AI model will provide specific advice such as, "If an elderly person feels anxious, it is important to first stay calm and listen to them. It is also effective to contact family and friends to give them a sense of security. Also, consider consulting a specialist medical institution if necessary."

[1656] Examples

[1657] For example, if an elderly person with early dementia feels anxious when alone, the robot will respond as follows:

[1658] 1. Using a smartphone camera and microphone to capture a senior's facial expressions and tone of voice.

[1659] 2. Analyze emotions and detect anxiety using OpenCV and Keras.

[1660] 3. Use Twilio to send a notification to a family member saying, "An elderly person is feeling anxious. Please check in."

[1661] 4. Use generative AI models to provide specific advice to seniors, such as, "Are you OK? Is there anything I can help you with?"

[1662] In this way, the present invention can grasp the emotions of the elderly in real time and take appropriate measures. It can also quickly respond to any abnormalities that occur and provide specific advice to the user.

[1663] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1664] Step 1:

[1665] The device (smartphone) uses a camera and microphone to capture the user's facial expressions and tone of voice. The input is real-time video and audio data, and the output is captured video frames and audio clips. In concrete terms, the device captures video frames and audio clips at regular intervals and passes them on to the next processing step.

[1666] Step 2:

[1667] The device uses OpenCV to detect faces from captured video frames. The input is the video frame acquired in step 1, and the output is the area (bounding box) of the detected face. Specifically, the device runs a face detection algorithm to determine the position and size of the face.

[1668] Step 3:

[1669] The device analyzes emotions from the detected face regions using a Keras emotion recognition model. The input is the face regions obtained in step 2, and the output is the estimated emotion (e.g., happiness, sadness, anxiety, etc.). Specifically, the device preprocesses the face regions and inputs them into the emotion recognition model to estimate the emotion.

[1670] Step 4:

[1671] The device determines an appropriate response based on the emotion estimated using the emotion engine. The input is the emotion data obtained in step 3, and the output is the corresponding action (e.g., speaking to the device, sending a notification, etc.). In concrete terms, the device analyzes the emotion data and selects a response based on pre-defined rules.

[1672] Step 5:

[1673] If an abnormality (e.g., anxiety, agitation) is detected, the device uses Twilio to send a notification to a family member or security service. The input is the action determined in step 4, and the output is the sent notification message. In concrete terms, the device calls the Twilio API and sends a message to the specified recipient.

[1674] Step 6:

[1675] The terminal uses the generative AI model to generate specific advice for the user. The input is information about the user's question or situation, and the output is the generated advice. In concrete terms, the terminal passes the user's input as a prompt sentence to the generative AI model and provides the generated advice to the user.

[1676] Step 7:

[1677] The device presents the generated advice to the user and continues the dialogue as necessary. The input is the advice obtained in step 6, and the output is the information presented to the user. In concrete terms, the device conveys the advice to the user by voice or text and waits for the user's response.

[1678] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1679] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1680] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1681] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1682] [Third embodiment]

[1683] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1684] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1685] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1686] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1687] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1688] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1689] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1690] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1691] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1692] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1693] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1694] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1695] "Example 1"

[1696] As one embodiment of the present invention, a robot equipped with artificial intelligence is provided. This robot has a function of providing further knowledge about the contents of exam preparation, etc. For example, when a user is working on a mathematical problem, the robot can provide related mathematical theories and solutions.

[1697] "Example 2"

[1698] The robot also has the ability to engage in conversation and discussion on topics of interest to the user. For example, if the user brings up the topic of advances in science and technology, the robot can engage in a deeper discussion about the latest trends in science and technology and their impact.

[1699] "Example 3"

[1700] In addition, the robot has the ability to respond with its own opinions in addition to the user's own answers. This allows the user to deepen their own opinions. For example, if a user expresses an opinion about an environmental issue, the robot can provide additional information or perspectives on that opinion.

[1701] The process flow of each embodiment will be described below.

[1702] "Example 1"

[1703] Step 1: The user presents the robot with learning content (e.g. a math problem).

[1704] Step 2: The robot acquires knowledge about the relevant learning content from artificial intelligence.

[1705] Step 3: The robot provides the user with the knowledge it has acquired, providing concrete explanations of the relevant mathematical theories and solutions.

[1706] "Example 2"

[1707] Step 1: The user presents the robot with a topic of interest (e.g., advances in science and technology).

[1708] Step 2: The robot gets information about the relevant topic from artificial intelligence.

[1709] Step 3: Based on the information acquired by the robot, the robot will hold a discussion with the user, providing specific details about the latest trends in science and technology and their impact.

[1710] "Example 3"

[1711] Step 1: The user presents the robot with their opinion (e.g., their opinion on environmental issues).

[1712] Step 2: The robot obtains additional information and perspectives on the subject from artificial intelligence.

[1713] Step 3: The robot provides the user with additional information and perspectives that it has obtained, with specific explanations of the additional information and perspectives in response to the user's opinion.

[1714] Example 1

[1715] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1716] Conventional learning support systems have the problem that it is difficult to get an appropriate and prompt answer when a user asks a question about exam preparation or a specific topic. There are also only a limited number of systems that allow users to have in-depth discussions about topics that interest them. Furthermore, there is a lack of functionality for robots to have their own opinions and provide additional information to users.

[1717] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1718] In this invention, the server includes a means for a user to input a question, a means for a terminal to recognize the question, a means for the terminal to transmit the question to the server, a means for the server to analyze the question, a means for the server to generate an answer using a generative AI model, a means for the server to transmit the answer to the terminal, and a means for the terminal to present the answer to the user. This allows the user to obtain an appropriate and prompt answer when asking a question about studying for an exam or a specific topic. In addition, the robot can deeply discuss a topic that interests the user and provide additional information with its own opinion.

[1719] "Means for user input of question" refers to an interface through which a user can input a question by voice or text.

[1720] "Means by which the device recognizes the question" refers to speech recognition software that the device uses to convert the user's speech into text, or the device's ability to directly obtain text input.

[1721] "Means for the terminal to send a question to the server" refers to a communication function for sending text data recognized by the terminal to the server.

[1722] "Means for the server to analyze the question" refers to natural language processing technology used by the server to analyze the text data it receives and understand the intent of the question.

[1723] "Means for the server to generate an answer using a generative AI model" refers to the function of the server using a generative AI model to generate an optimal answer to a question.

[1724] "Means for the server to transmit a response to the terminal" refers to a communication function for transmitting a response generated by the server to the terminal.

[1725] "Means for the terminal to present the answer to the user" refers to an output function for presenting the answer received by the terminal to the user in the form of voice or text.

[1726] A "robot equipped with artificial intelligence" refers to a robot that has built-in artificial intelligence technology and is capable of interacting with users and providing them with information.

[1727] "Means for providing further knowledge about the contents of examination study, etc." refers to a function that provides related knowledge and solutions when a user asks a question related to examination study.

[1728] "Means of talking and discussing topics of interest" refers to the function of the robot to converse and discuss topics that interest the user.

[1729] "A means for the robot to respond with its own opinion in addition to its own answer" refers to a function in which the robot provides its own opinion or additional information in addition to the answer to the user's question.

[1730] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1731] This invention is a system that uses a robot equipped with artificial intelligence to provide appropriate and prompt answers to questions from users about studying for exams or specific topics. A specific embodiment of this system will be described below.

[1732] Hardware and Software Used

[1733] Hardware: High performance processor (e.g. Intel Core i7), camera, microphone, speakers

[1734] Software: generative AI models (e.g. GPT-4), speech recognition software (e.g. Google Speech-to-Text), natural language processing libraries (e.g. spaCy)

[1735] System Configuration

[1736] 1. How the user enters their question:

[1737] The user inputs a question to the robot by voice or text. For example, the user may ask, "Teach me how to solve a quadratic equation."

[1738] 2. How the device recognizes the question:

[1739] The device uses a built-in microphone and speech recognition software to convert the user's speech into text, or, in the case of text input, to capture the text as is.

[1740] 3. How the device sends a query to the server:

[1741] The terminal transmits the recognized text data to the server. At this time, the data is encrypted before being transmitted.

[1742] 4. How the server parses the query:

[1743] The server analyzes the received text data and uses a natural language processing library to understand the intent of the question.

[1744] 5. How the server uses the generative AI model to generate an answer:

[1745] The server uses the generative AI model to generate optimal answers to questions, such as "How do I solve a quadratic equation?", generating a solution using a formula.

[1746] 6. The server sends the response to the terminal by:

[1747] The server then sends the generated response to the terminal. The data is also encrypted when sent.

[1748] 7. The means by which the terminal presents the answer to the user:

[1749] The device will then present the answer to the user either by voice or text, in which case the answer will be read out loud using a speaker.

[1750] Examples

[1751] For example, if a user asks the robot, "Teach me how to solve a quadratic equation," the system operates as follows.

[1752] User: "How do I solve a quadratic equation?"

[1753] Device: Capture audio through the microphone and convert it to text using Google Speech-to-Text: "Can you help me solve a quadratic equation?"

[1754] Terminal: Sends text data to the server.

[1755] Server: Use spaCy to analyze the text and understand the intent of the question.

[1756] Server: Uses GPT-4 to generate the answer, "To solve quadratic equations, use the formula. The formula is x = (-b ± √(b^2 - 4ac)) / 2a."

[1757] Server: Generates the answer and sends it to the device.

[1758] Device: Using the speaker, read to the user, "To solve quadratic equations, use the formula. The formula is x = (-b ± √(b^2 - 4ac)) / 2a."

[1759] Examples of prompt statements

[1760] "When a user asks how to solve a quadratic equation, explain how to solve it using the formula."

[1761] In this way, the robot supports the user's learning.

[1762] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1763] Step 1:

[1764] The user enters a question.

[1765] The user inputs a question to the robot by voice or text. For example, the user may ask, "Please tell me how to solve a quadratic equation." The input is voice or text data.

[1766] Step 2:

[1767] The device recognizes the question.

[1768] The terminal uses a built-in microphone and voice recognition software to convert the user's speech into text. In the case of text input, the terminal obtains the text as is. The input is voice data or text data, and the output is text data.

[1769] Step 3:

[1770] The terminal sends a question to the server.

[1771] The terminal transmits the recognized text data to the server. At this time, the data is encrypted before being transmitted. The input is text data, and the output is encrypted text data.

[1772] Step 4:

[1773] The server parses the question.

[1774] The server analyzes the received text data. It uses a natural language processing library to understand the intent of the question. The input is the encrypted text data, and the output is the analyzed intent of the question.

[1775] Step 5:

[1776] The server generates the answer using a generative AI model.

[1777] The server uses the generative AI model to generate the best answer to the question. For example, for the question "How do I solve a quadratic equation?", it generates a solution using the solution formula. The input is the parsed question intent, and the output is the generated answer.

[1778] Step 6:

[1779] The server sends the response to the terminal.

[1780] The server sends the generated answer to the terminal. The data is also encrypted when sent. The input is the generated answer, and the output is the encrypted answer.

[1781] Step 7:

[1782] The terminal presents the answer to the user.

[1783] The terminal presents the received answer to the user either by voice or text, in the latter case by reading the answer aloud using a speaker. The input is the encrypted answer, and the output is the presented answer by voice or text.

[1784] (Application example 1)

[1785] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1786] Conventional educational support robots were limited to providing knowledge on exam preparation or specific topics, and had the problem of not being able to support the education and training of workers in factories. In addition, since they could not provide real-time knowledge using generative AI models, it was difficult to immediately provide the latest information or specific operating procedures.

[1787] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1788] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, a means for the robot to provide knowledge about new machine operation methods and manufacturing processes to workers in a factory, and a means for the robot to obtain and provide related knowledge based on a prompt sentence using a generative AI model. This makes it possible to provide the latest knowledge and specific operating procedures in real time during worker education and training in a factory.

[1789] A "robot equipped with artificial intelligence" is an autonomous mechanical device that incorporates artificial intelligence technologies such as machine learning and natural language processing, and is capable of interacting with users and providing them with knowledge.

[1790] "Means of providing further knowledge about the content of exam preparation, etc." is a function that provides additional information or explanations related to the study content the user is working on.

[1791] "A means to talk to and discuss topics of interest" is a function that enables users to engage in dialogue, exchange opinions, and debate on topics that interest them.

[1792] "A means for the robot to respond with its own opinion by adding it to the user's answer" is a function that enables the robot to respond to the user's question by adding its own knowledge and opinions.

[1793] "A means of providing workers in a factory with knowledge about how to operate new machines and manufacturing processes" is a function that provides workers in a factory with information about how to use new machines and manufacturing processes.

[1794] "Means for obtaining and providing relevant knowledge based on a prompt sentence using a generative AI model" refers to a function that uses a generative AI model to generate relevant information based on a prompt sentence entered by a user and provides it to the user.

[1795] The system for implementing this invention is centered around a robot equipped with artificial intelligence. The robot has the function of providing knowledge in various scenarios, such as studying for exams or educating workers in a factory.

[1796] System Configuration

[1797] The system consists of the following main components:

[1798] 1. Robots equipped with artificial intelligence: Equipped with machine learning and natural language processing technology, they can converse with users and provide them with knowledge.

[1799] 2. Generative AI model: Generates relevant knowledge based on the prompt sentence entered by the user.

[1800] 3. Database: Stores information about training content and operating procedures within the factory.

[1801] Hardware and Software Used

[1802] Hardware: Robot body, server, user device (smartphone, tablet, etc.)

[1803] Software: Python, OpenAI API, Database Management System

[1804] Data processing and calculation

[1805] The server sends the prompt sentence entered by the user to the generative AI model to obtain related knowledge, which is then provided to the user through the robot.

[1806] Examples

[1807] For example, if a worker in a factory wants to learn how to operate a new welding machine, the system works as follows.

[1808] 1. User input: A worker says to the robot, "Please teach me the basic principles of welding."

[1809] 2. Prompt generation: The server sends this input as a prompt to the generative AI model.

[1810] Example prompt: "Please provide a detailed explanation of the basic principles of welding."

[1811] 3. Knowledge Acquisition: The generative AI model generates relevant knowledge based on the prompt sentence and returns it to the server.

[1812] 4. Knowledge provision: The server transmits the acquired knowledge to the robot, and the robot provides the knowledge to the worker via voice or display.

[1813] In this way, the system is able to provide up-to-date knowledge and specific operating procedures in real time.

[1814] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1815] Step 1:

[1816] The user inputs a question to the robot. For example, a worker might say, "Please tell me the basic principles of welding." This input is converted into text data through a voice recognition system.

[1817] Step 2:

[1818] The terminal generates a prompt sentence based on the user's input, for example, "Please provide a detailed explanation of the basic principles of welding." This prompt sentence is then sent to the generative AI model.

[1819] Step 3:

[1820] The server sends a prompt to the generative AI model, which generates relevant knowledge based on the prompt, and returns the knowledge to the server as text data.

[1821] Step 4:

[1822] The server receives the generated knowledge and matches it with related information stored in a database, retrieving additional information from the database if necessary.

[1823] Step 5:

[1824] The server transmits the final knowledge data to the robot, which converts the data into a format for providing it to the user, such as voice data or text data for display.

[1825] Step 6:

[1826] The robot provides the knowledge data received from the server to the user, either by explaining it aloud using a speech synthesis system or by displaying text on a display.

[1827] Step 7:

[1828] The user receives the knowledge provided by the robot and uses it to learn or work, and if necessary, can ask further questions to gain additional knowledge.

[1829] Example 2

[1830] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1831] Conventional robot systems have the problem that it is difficult to have in-depth discussions on topics that interest the user. In addition, there is a lack of technology to accurately recognize the user's voice input and generate appropriate responses. Furthermore, there is a need for a system that can speak the generated responses in a natural voice.

[1832] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for converting a user's voice input into text data using voice recognition software, a means for inputting a prompt sentence into a generative AI model and generating an appropriate response, and a means for converting the generated text response into voice data using voice synthesis software. This enables users to have in-depth discussions on topics that interest them.

[1833] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1834] A "robot" is a mechanical device equipped with artificial intelligence and capable of communicating with humans through physical movements and dialogue.

[1835] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass a particular exam.

[1836] A "knowledge delivery medium" is a method or technique for conveying information or data to a user.

[1837] "Means of discussion" are methods and technologies for exchanging opinions and sharing information through dialogue with users.

[1838] "Speech recognition software" is a program for converting voice data into text data.

[1839] A "generative AI model" is an artificial intelligence algorithm that generates appropriate responses based on input text data.

[1840] A "prompt sentence" is a piece of text data input into a generative AI model and is an instruction sentence for generating a response.

[1841] "Speech synthesis software" is a program for converting text data into voice data.

[1842] "Audio data" refers to data that represents audio information in digital form.

[1843] The "means for uttering" refers to a method or technology for playing back the generated voice data through a speaker and conveying information to the user by voice.

[1844] This invention is a system for allowing a user to discuss with a robot a topic that the user is interested in. This system uses a robot equipped with artificial intelligence to carry out a series of processes, including recognizing the user's voice input, generating an appropriate response, and speaking the response aloud.

[1845] Hardware and Software Used

[1846] Hardware: The robot itself (e.g. a typical interactive robot)

[1847] Software: speech recognition software (e.g., a general speech recognition program), generative AI models (e.g., a general generative AI algorithm), speech synthesis software (e.g., a general speech synthesis program)

[1848] System Operation

[1849] 1. User voice input:

[1850] The user talks to the robot, for example, saying, "Tell me about recent advances in science and technology."

[1851] The robot uses a built-in microphone to capture the user's voice.

[1852] 2. Speech to text conversion:

[1853] The terminal (robot) sends the captured voice data to the voice recognition software.

[1854] The server uses voice recognition software to convert the voice data into text data.

[1855] The terminal receives the converted text data.

[1856] 3. Use of generative AI models:

[1857] The device inputs the received text data into the generative AI model.

[1858] The server sends prompts to the generative AI model, which then generates an appropriate response.

[1859] 4. Generate the response:

[1860] The server receives the response from the generative AI model.

[1861] The terminal receives the generated text response from the server.

[1862] 5. Response transcription:

[1863] The terminal sends the received text response to the speech synthesis software.

[1864] The server uses speech synthesis software to convert the text data into voice data.

[1865] The terminal receives the converted voice data.

[1866] 6. Robot response utterance:

[1867] The terminal transmits the received voice data to the robot's speaker.

[1868] The robot speaks the generated voice to the user.

[1869] Examples

[1870] User: "Tell me about recent advances in science and technology."

[1871] Robot: "A recent advance in science and technology is the development of quantum computing. Quantum computers have the potential to rapidly solve problems that cannot be solved by conventional computers."

[1872] Examples of prompt statements

[1873] Please tell me about recent advances in science and technology.

[1874] "Let's discuss the latest developments in AI technology."

[1875] "Tell me about the future of robotics."

[1876] In this way, users can deepen discussions with the robot about topics of interest.

[1877] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1878] Step 1:

[1879] A user speaks to the robot, for example, "Tell me about recent advances in science and technology." The robot uses a built-in microphone to capture the user's voice. The input is the user's voice, and the output is the voice data.

[1880] Step 2:

[1881] The terminal (robot) sends the captured voice data to the voice recognition software. The server uses the voice recognition software to convert the voice data into text data. The input is voice data, and the output is text data. In concrete terms, the voice recognition software analyzes the voice waveform and generates the corresponding text.

[1882] Step 3:

[1883] The terminal receives the converted text data and inputs it into the generative AI model. The server sends a prompt sentence to the generative AI model and generates an appropriate response. The input is the text data, and the output is the generated response text. In concrete terms, the generative AI model analyzes the prompt sentence and generates a response based on related information.

[1884] Step 4:

[1885] The server receives the response from the generative AI model and sends it to the terminal. The terminal receives the generated text response from the server. The input is the generated response text, and the output is the same response text. In specific operations, the server receives the response from the generative AI model and forwards it to the terminal.

[1886] Step 5:

[1887] The terminal sends the received text response to the speech synthesis software. The server uses the speech synthesis software to convert the text data into speech data. The input is text data and the output is speech data. In concrete terms, the speech synthesis software analyzes the text and generates a corresponding speech waveform.

[1888] Step 6:

[1889] The terminal receives the converted voice data and transmits it to the robot's speaker. The robot speaks the generated voice to the user. The input is the voice data, and the output is a voice response to the user. In concrete terms, the robot's speaker plays the voice data and conveys information to the user.

[1890] (Application example 2)

[1891] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1892] Conventional robots are limited to specific tasks and simple dialogue, making it difficult to improve work efficiency through real-time communication and discussion with workers. In addition, there were no robots that could deepen discussions about work progress and problems. For this reason, there is a demand for improving work efficiency and speeding up problem solving in factories.

[1893] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1894] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics of interest to the robot, a means for the robot to reply with its own opinion in addition to its own reply, and a means for the robot to communicate with workers in real time to improve work efficiency and discuss work progress and problems, thereby making it possible to improve work efficiency in a factory and speed up problem solving.

[1895] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[1896] A "robot" is a mechanical device that operates autonomously according to programmed instructions to perform specific tasks.

[1897] "Studying for exams" refers to learning activities aimed at acquiring the knowledge and skills necessary to pass a particular exam.

[1898] A "means of providing knowledge" is a method or device for conveying information or data to users and deepening their understanding.

[1899] A "theme" is a topic or issue that is the center of discussion or research.

[1900] "Means for conversation and discussion" refers to methods or devices for engaging in dialogue with users and deepening discussion through the exchange of opinions and sharing of information.

[1901] "Means of responding with opinions" refers to a method or device that enables a robot to express its own views or thoughts in response to a user's questions or opinions.

[1902] "Work efficiency" is an index for optimizing the effort and time required to perform a specific task.

[1903] "Means of real-time communication" refers to methods or devices for instantly exchanging information with users and responding quickly.

[1904] "Work progress" is a state or condition that indicates how far a particular piece of work has progressed.

[1905] An "issue" is an obstacle or challenge that occurs in a task or process.

[1906] A "means of discussion" is a method or device for exchanging opinions and sharing information on a particular topic and deepening understanding.

[1907] The system for carrying out the present invention is mainly composed of a robot equipped with artificial intelligence. A specific embodiment of this system will be described below.

[1908] Hardware used

[1909] Robot body: A mechanical device that operates autonomously and communicates with workers (e.g., an articulated robot arm).

[1910] Speech recognition microphone: A device that recognizes the worker's voice and converts it into text (e.g., a microphone compatible with the Google Cloud Speech-to-Text API).

[1911] Display: A device for displaying visual information, if required.

[1912] Software used

[1913] Speech recognition software: Software for converting speech into text (e.g. Google Cloud Speech-to-Text API).

[1914] Generative AI model: An artificial intelligence model for generating appropriate responses to user questions or arguments (e.g., OpenAI’s GPT-4).

[1915] Natural Language Processing Libraries: Libraries for performing text analysis (e.g. NLTK, spaCy).

[1916] Database: A database (e.g. MySQL, PostgreSQL) to store generated data and user history.

[1917] System Operation

[1918] 1. Voice input: The worker speaks to the robot.

[1919] 2. Speech Recognition: Speech recognition software converts speech into text.

[1920] 3. Text Analysis: Use natural language processing libraries to analyze text and understand user intent.

[1921] 4. Input to the generative AI model: Input a prompt to the generative AI model based on the analysis results.

[1922] 5. Response generation: The generative AI model generates an appropriate response.

[1923] 6. Voice output: The generated response is converted into voice using speech synthesis software and sent back to the worker.

[1924] Examples

[1925] When a worker asks, "What do you think about the efficiency of the new production line?" the system works as follows:

[1926] Worker: "What do you think about the efficiency of the new production line?"

[1927] Robot: "The latest production technology is a quality control system that utilizes AI. This allows for real-time detection of defective products, greatly improving production efficiency."

[1928] Examples of prompt statements

[1929] A user asks, "What do you think about the efficiency of our new production line?" Generate responses that foster discussion about the latest production technologies and efficiency methods.

[1930] In this way, robots can communicate with workers in real time and discuss the progress of work and problems, which will improve work efficiency and speed up problem solving in the factory.

[1931] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1932] Step 1:

[1933] Audio input

[1934] The user talks to the robot.

[1935] Input: User's voice

[1936] Output: Audio data

[1937] Specific Action: A user asks, "What do you think about the efficiency of our new production line?"

[1938] Step 2:

[1939] Speech Recognition

[1940] The server uses voice recognition software to convert the voice data into text.

[1941] Input: Audio data

[1942] Output: Text data

[1943] What happens: Speech recognition software (e.g. Google Cloud Speech-to-Text API) analyzes the voice data and generates text such as, "What do you think about the efficiency of our new production line?"

[1944] Step 3:

[1945] Text Analytics

[1946] The server uses a natural language processing library to analyze the text data and understand the user's intent.

[1947] Input: Text data

[1948] Output: Analysis results (user intent)

[1949] What happens: A natural language processing library (e.g. NLTK, spaCy) analyzes the text data and understands that the user is looking for information on improving production line efficiency.

[1950] Step 4:

[1951] Input to generative AI models

[1952] The server inputs prompts into the generative AI model based on the analysis results.

[1953] Input: Analysis results (user intent)

[1954] Output: Prompt statement

[1955] Specific operation: The server inputs the prompt sentence "The user asked, 'What do you think about the efficiency of the new production line?' Please generate a response that deepens the discussion about the latest production technologies and methods of improving efficiency" into the generative AI model.

[1956] Step 5:

[1957] Response Generation

[1958] A generative AI model generates an appropriate response based on the prompt.

[1959] Input: Prompt statement

[1960] Output: Response text

[1961] How it works: A generative AI model (e.g., OpenAI's GPT-4) analyzes the prompt sentence and generates a response text that reads, "As the latest production technology, there is a quality control system that utilizes AI. This allows defective products to be detected in real time, greatly improving production efficiency."

[1962] Step 6:

[1963] Audio Output

[1964] The server converts the generated response text into speech using speech synthesis software and responds to the user.

[1965] Input: Response text

[1966] Output: Audio data

[1967] Specific operation: The speech synthesis software converts the response text into voice data, and the robot replies to the user, "As the latest production technology, we have a quality control system that utilizes AI. This allows us to detect defective products in real time, greatly improving production efficiency."

[1968] Example 3

[1969] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1970] Conventional robot systems equipped with artificial intelligence only provide information in response to user opinions, and it is difficult to provide additional information or perspectives to deepen the user's opinions. In addition, it is not possible to provide appropriate feedback in real time to the opinions entered by the user, and it is not possible to fully support the user's learning or deepening of discussions.

[1971] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1972] In this invention, the server includes a means for a user to input an opinion, a means for a terminal to transmit the opinion to the server, a means for the server to input the opinion as a prompt sentence to the generative AI model, a means for the generative AI model to generate additional information and viewpoints, a means for the server to transmit the generated information to the terminal, and a means for the terminal to display the information to the user. This makes it possible to provide appropriate feedback to the user's opinion in real time and support the deepening of user learning and discussion.

[1973] "User" means an individual or entity that utilizes the System to input opinions and receive feedback.

[1974] A "terminal" is a device through which a user inputs opinions and communicates with the server, and includes a personal computer, smartphone, etc.

[1975] The "server" is a computer system that receives opinions from users, inputs them as prompt sentences into the generative AI model, and transmits the generated information to the terminal.

[1976] A "generative AI model" is an artificial intelligence model that generates additional information or perspectives based on an input prompt sentence, and includes, for example, models that use natural language processing techniques.

[1977] A "prompt" is a text-based instruction that is input to a generative AI model based on the user's input.

[1978] The "means for displaying information" is a function for visually presenting to the user the information that the terminal receives from the server.

[1979] The present invention is a system in which a user inputs an opinion and provides additional information or perspective to the opinion using a generative AI model. A specific embodiment of the system will be described below.

[1980] First, the user inputs his / her opinion using a terminal (e.g., a PC or smartphone). The opinion input by the user is sent as text data from the terminal to the server. The server converts the received opinion into a prompt sentence. An example of a prompt sentence is "User's opinion: The use of renewable energy should be increased. Please provide additional information or perspectives."

[1981] Next, the server inputs the prompt sentence into a generative AI model (for example, a model using natural language processing technology). The generative AI model generates additional information and perspectives based on the prompt sentence. For example, it might generate content such as, "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar and wind power generation are attracting attention as clean energy sources that do not rely on fossil fuels. In addition, costs are gradually falling due to technological advances in renewable energy."

[1982] The generated information is sent from the server to the terminal. The terminal displays the received information to the user. The user can view this information and deepen his / her opinion.

[1983] In implementing this system, the following hardware and software are used:

[1984] Devices: PCs, smartphones, etc.

[1985] Server: A computer system that receives opinions, generates prompts, inputs them into the generative AI model, and transmits the generated information.

[1986] Generative AI model: An artificial intelligence model that uses natural language processing techniques (e.g., GPT-4)

[1987] As a concrete example, consider the case where a user expresses the opinion that "reducing plastic waste is important." This opinion is input as a prompt sentence to the generative AI model:

[1988] User opinion: Reducing plastic waste is important. Please provide additional information or perspective.

[1989] A generative AI model might generate additional information such as:

[1990] "Reducing plastic waste also helps protect marine ecosystems. Advances in recycling technology are also facilitating the reuse of plastics."

[1991] In this way, the user can obtain new perspectives and information on his / her own opinions. This system supports the user's learning and deepening of discussion. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1992] Step 1:

[1993] The user inputs an opinion.

[1994] The user uses a device (PC or smartphone) to input their opinion into a text box. For example, they might type, "The use of renewable energy should increase." The input opinion is saved as text data on the device.

[1995] Step 2:

[1996] The terminal transmits the opinion to the server.

[1997] The terminal sends the opinions entered by the user to the server as an HTTP request. The input data is sent in text format. The server receives this request and retrieves the opinion data.

[1998] Step 3:

[1999] The server inputs the opinion into the generative AI model as a prompt sentence.

[2000] The server converts the received opinion into a prompt, for example, in the form of "User opinion: The use of renewable energy should be increased. Please provide additional information or perspectives." This prompt is then fed into a generative AI model.

[2001] Step 4:

[2002] Generative AI models generate additional information and perspectives.

[2003] A generative AI model (e.g., a model using natural language processing technology) generates additional information and perspectives based on the prompt sentence. For example, it generates content such as, "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar power generation and wind power generation are attracting attention as clean energy sources that do not depend on fossil fuels. In addition, due to technological advances in renewable energy, costs are gradually decreasing." The generated information is returned to the server as text data.

[2004] Step 5:

[2005] The server transmits the generated information to the terminal.

[2006] The server receives the information obtained from the generative AI model and sends it to the terminal as an HTTP response. The data sent is in text format.

[2007] Step 6:

[2008] The terminal displays the information to the user.

[2009] The terminal displays the information received from the server on the screen. The user can view this information to further develop their own opinion. For example, the following information may be displayed: "The use of renewable energy contributes greatly to reducing greenhouse gas emissions. In particular, solar and wind power generation are attracting attention as clean energy sources that do not depend on fossil fuels. Furthermore, technological advances in renewable energy are gradually reducing costs."

[2010] (Application example 3)

[2011] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[2012] Conventional robot systems simply provide responses to user opinions, and lack the ability to provide additional information or perspectives to deepen the user's opinions. In addition, there are limited ways to provide detailed information about products in physical stores, making it difficult for users to obtain sufficient information when selecting products. This creates an issue of limiting the user's purchasing experience.

[2013] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[2014] In this invention, the server includes a robot equipped with artificial intelligence, a means for the robot to provide further knowledge about the contents of exam preparation, etc., a means for the robot to talk to and discuss topics of interest to the user, a means for the robot to reply with its own opinion in addition to the user's reply, a means for the robot to provide additional information and viewpoints about products in a physical store, a means for the robot to obtain the user's opinion and generate additional information using a generative AI model, and a means for the robot to present the generated additional information to the user. This allows the user to gain a deeper understanding of the product, improving the purchasing experience.

[2015] "Artificial intelligence" is the technology that enables computer systems to mimic human intelligence and learn, reason, and self-correct.

[2016] A "robot" is a mechanical device equipped with artificial intelligence that performs specific tasks automatically.

[2017] "Studying for exams" is the activity of learning the knowledge and skills necessary to pass an exam.

[2018] A "knowledge medium" is a method or technology for conveying information or data to a user.

[2019] A "theme" is a subject matter about a particular topic or issue.

[2020] "Means of discussion" are methods and techniques for interacting with users and exchanging opinions.

[2021] "Means of responding with opinions" refers to methods or technologies that allow a robot to provide its own views in response to a user's questions or opinions.

[2022] A "brick and mortar store" is a store that exists in a physical location and offers goods and services.

[2023] "Means of providing additional information or perspectives on a product" are methods or technologies that provide more detailed information or a different perspective on the product selected by the user.

[2024] "Means of obtaining user opinions" are the methods and technologies used to collect opinions and questions expressed by users.

[2025] A "generative AI model" is an algorithm or system that enables artificial intelligence to generate new information or answers based on data.

[2026] "Means for generating additional information" are methods or technologies for generating new information or perspectives based on user opinions.

[2027] "Means of presentation to the user" refers to the methods or techniques used to visually or audibly convey the generated information to the user.

[2028] The following system configuration and processing procedure will be described as an embodiment of the present invention.

[2029] System Configuration

[2030] The system consists of an artificial intelligence-enabled robot, smart glasses, an Internet connection, and a server using a generative AI model. Specifically, the system uses the following hardware and software:

[2031] Hardware:

[2032] Smart glasses (e.g. Google Glass)

[2033] Internet connection

[2034] software:

[2035] OpenAI API (generative AI model)

[2036] Python Program

[2037] Processing flow

[2038] 1. Get user feedback:

[2039] Users express their opinions about products through the smart glasses, for example by asking, "Is this product environmentally friendly?"

[2040] 2. Opinion text:

[2041] The smart glasses convert the user's comments into text using voice recognition technology.

[2042] 3. Send to generative AI model:

[2043] The server sends the user's opinion, converted into text, to the OpenAI API. An example of a prompt is as follows:

[2044] What users say about their choice: Is this product eco-friendly?

[2045] Please provide any additional information or perspective.

[2046] 4. Generate additional information:

[2047] The OpenAI API generates additional information and perspectives based on user feedback, such as "This product is made from renewable materials, has a low carbon footprint during manufacturing, and is more energy efficient than other products in the same category."

[2048] 5. User Representation:

[2049] The server transmits the generated additional information to the smart glasses, which present it visually or audibly to the user.

[2050] Examples

[2051] When a user is choosing a product in a physical store and asks through the smart glasses, "Is this product environmentally friendly?", the system works as follows:

[2052] 1. User opinions are converted into text using voice recognition technology.

[2053] 2. Send the converted text to the OpenAI API.

[2054] 3. The OpenAI API generates additional information, such as, "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category."

[2055] 4. The generated information is presented to the user through smart glasses.

[2056] In this way, users can gain a deeper understanding of the products and improve their purchasing experience.

[2057] The flow of the specific process in the application example 3 will be described with reference to FIG.

[2058] Step 1:

[2059] Users express their opinions about products through smart glasses.

[2060] Input: User's spoken opinion (e.g. "Is this product environmentally friendly?")

[2061] Output: Audio data

[2062] What happens: A user speaks a question about a product into the smart glasses.

[2063] Step 2:

[2064] The terminal uses voice recognition technology to convert the user's comments into text format.

[2065] Input: Audio data

[2066] Output: Text data (e.g. "Is this product environmentally friendly?")

[2067] How it works: The voice recognition software built into the smart glasses analyzes the voice data and generates corresponding text data.

[2068] Step 3:

[2069] The server sends the converted user opinions to the generative AI model.

[2070] Input: Text data

[2071] Output: Prompt statement (e.g., "What do you think about the product the user selected: Is this product environmentally friendly? Please provide any additional information or perspective.")

[2072] Specific operation: The server receives the text data and generates a prompt sentence appropriate for the generative AI model.

[2073] Step 4:

[2074] A generative AI model generates additional information and perspectives based on user input.

[2075] Input: prompt statement

[2076] Output: Additional information (e.g. "This product is made from renewable materials and has a low carbon footprint during manufacturing. It is also more energy efficient than other products in the same category.")

[2077] What it does: The OpenAI API parses the prompt and generates additional relevant information and perspectives.

[2078] Step 5:

[2079] The server transmits the generated additional information to the smart glasses for presentation to the user.

[2080] Input: Additional information

[2081] Out...

Claims

1. a means for a user to input a question; means for the terminal to recognize said question; means for the terminal to transmit the query to a server; means for the server to analyze a query; means for the server to generate answers to the questions using a generative AI model; means for the server to transmit the response to the terminal; means for the terminal to present the answer to a user; A robot equipped with artificial intelligence and communicating the answer to the user; A means for the robot to provide the user with further knowledge about the contents of exam preparation; means for the robot to engage in conversation with the user and discuss a topic of interest to the user; A means for the robot to reply with an opinion that the robot has in a form of adding to the reply of the robot; A system including:

2. The system according to claim 1 , wherein the robot is provided to a young generation that utilizes social networking services.

3. The system according to claim 1 , wherein the robot is provided for elderly people in the early stages of dementia.

Images