system

Abstract

We provide the system. [Solution] A means of supporting education using machine learning that enables two-way dialogue using a first language and a second language, A means for creating individual learning plans based on the user's native language and target language, A means of providing users with simulated dialogue using a virtual reality environment, A means of collecting and analyzing user learning progress information and providing real-time feedback, A means of displaying subtitles through the user interface to make the dialogue easier to understand, A means of using a robotic device that supports language learning through dialogue in daily life within a home environment, A system that includes means for analyzing a user's question using speech recognition and natural language processing technologies and generating appropriate linguistic expressions.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional language learning systems are mainly operated in a single language environment, making it difficult for beginners and learners unaccustomed to foreign languages to take the first step. As a result, there are problems such as taking a long time in language acquisition and a decrease in motivation. In addition, due to the lack of a function to flexibly customize the learning plan according to the progress and needs of individual learners, an efficient learning environment is not provided.

Means for Solving the Problems

[0005] This invention provides an artificial intelligence-powered language learning support system capable of two-way dialogue using a first language and a second language. This system generates individualized learning plans based on the user's native language and the language they wish to learn, and further provides realistic learning opportunities through dialogue training in a virtual reality environment. It also analyzes the user's learning progress data in real time and provides appropriate feedback to achieve an optimal learning process for each learner. Furthermore, the system enhances comprehension by displaying watermarked subtitles in the user interface to provide visual support.

[0006] Artificial intelligence is a technology in which computer systems imitate human knowledge and behavior through learning and pattern recognition to solve problems and make decisions.

[0007] Language learning is the process of acquiring the ability to listen to, speak, read, and write a particular language.

[0008] "Two-way dialogue" is a form of reciprocal communication between two parties, where information is exchanged back and forth to deepen understanding.

[0009] A "virtual reality environment" is a technology that allows users to have an immersive experience within a computer-generated visual and auditory environment.

[0010] A "learning plan" refers to the specific schedule and content of learning activities that are planned to suit the individual learner's needs and goals.

[0011] "Subtitles" are a function that displays audio information as text on the screen in video and audio content.

[0012] "Feedback" refers to evaluations and suggestions given regarding specific actions or outcomes, which contribute to improving future behavior.

[0013] "Progress data" refers to chronologically managed information that shows the progress of learners or projects. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] [[ID=1,9]]In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system for effectively supporting language learning through artificial intelligence technology. This system achieves a personalized learning experience tailored to each individual learner through the coordinated operation of a server, terminal, and user. The roles of each component are described below.

[0036] The server is built around an artificial intelligence engine that can use both the learner's native language and the second language they wish to learn. This engine enables two-way dialogue, analyzing voice input from the user and generating appropriate responses. The server also manages progress data from many learners and processes individual progress to dynamically generate the optimal learning plan.

[0037] The terminal functions as a user interface, receiving and displaying data sent from the server. In particular, it generates interactive dialogue scenes using a virtual reality environment, providing the user with a realistic learning experience. The terminal acquires the user's voice through a microphone and sends it to the server, while playing back the server's response with audio and subtitles.

[0038] The user initiates a conversation with the AI ​​tutor through this system. When the user asks a question in their native language, the device receives it, processes it on the server, and then returns a response in the target language to the user. To aid understanding, subtitles of the conversation are displayed on the device screen in real time. This method makes it easier for users to become familiar with a second language, even from the basics, and allows them to progress in their learning.

[0039] As a concrete example, let's say a user uses this system to learn English for travel. A virtual cafe scene unfolds on the device, and the user tells the AI ​​teacher in their native language, "I'd like to order a coffee." The server receives this phrase, generates the appropriate English expression, "I would like to order a coffee," and returns it to the device. The user listens to the English expression, and at the same time, the phrase is displayed as subtitles, helping them understand pronunciation and grammar. In this way, the user repeatedly practices what they have learned in a way that is relevant to real-life situations, and acquires the language in a natural manner.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The user starts up their device and logs into the learning application. The user selects the language and level they want to learn on their device. The selection information is sent from the device to the server.

[0043] Step 2:

[0044] Based on the user's selection, the server retrieves data for the appropriate learning mode and virtual environment. The server then creates a pre-configured learning plan and sends it to the terminal.

[0045] Step 3:

[0046] The terminal sets up the virtual reality environment based on data received from the server. The user is notified on the terminal's screen or via audio when it is ready.

[0047] Step 4:

[0048] The user begins a conversation with the AI ​​teacher through the device. The user asks for simple phrases in the language they want to learn in their native language and inputs them into the device via the microphone.

[0049] Step 5:

[0050] The device sends the user's voice input to the server in real time. The server analyzes this data and generates translations and responses using an AI language model.

[0051] Step 6:

[0052] The server sends the generated translation and audio data to the terminal, providing the user with an appropriate response. For English phrases, the terminal will play the audio "I would like to order a coffee."

[0053] Step 7:

[0054] The device displays subtitles on the screen along with the response. These subtitles are updated in real time to aid user understanding.

[0055] Step 8:

[0056] The user continues practicing the dialogue by asking questions and confirming details. Feedback on differences in pronunciation and intonation is displayed on the device screen.

[0057] Step 9:

[0058] The server collects user progress and performance data and saves the information for the next session. Based on the analysis results, it adjusts the learning plan and provides advice.

[0059] By repeating the above steps, users will gradually improve their language skills.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] Traditional language learning systems often lack sufficient two-way communication and personalized learning, making it difficult for learners to acquire a language efficiently. Furthermore, the lack of real-time feedback and self-paced virtual environment dialogue training hindered the improvement of actual communication skills.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for supporting language learning using an intelligent control device capable of two-way dialogue using a first language and a second language, means for creating an individualized learning plan based on the user's native language and the language to be learned, and means for integrating speech recognition and translation functions to perform language conversion and response generation in real time. This enables learners to effectively acquire a language at their own pace through interactive, practical dialogue while receiving appropriate feedback in real time.

[0065] An "intelligent control device" is a device that uses artificial intelligence technology to perform advanced processing related to language conversion and dialogue generation.

[0066] "Supporting language learning" means providing users with various functions and tools to efficiently acquire their target language.

[0067] An "individualized learning plan" is a curriculum that includes learning procedures and content optimized according to each learner's progress and needs.

[0068] A "virtual reality environment" is a virtual space created using computer technology, through which users can engage in dialogue training in a manner close to reality.

[0069] "Dialogue practice" is an activity that simulates two-way language communication for the purpose of language acquisition.

[0070] "Learning progress data" refers to information that quantifies or records the user's progress and achievements in language learning.

[0071] "Instant reporting" means providing learners with real-time feedback.

[0072] "Displaying explanatory text" means showing textual information on the screen to help the user better understand the content of the conversation.

[0073] "Speech recognition" is the process of analyzing speech input through a microphone and converting it into digital text information.

[0074] A "translation function" is a technology that converts information expressed in one language into another language.

[0075] "Response generation" refers to creating a response in the appropriate language based on user input.

[0076] A "generative AI model" is an artificial intelligence technology or algorithm used in natural language processing to generate contextually appropriate text.

[0077] "Optimizing according to context" means adjusting the information provided to best suit the flow of the conversation and the user's needs.

[0078] This invention operates based on a language learning support system using an intelligent control device. The main components are a server, a terminal, and a user.

[0079] The server functions as a central hub for supporting language learning. Specifically, it uses speech recognition software (e.g., a common cloud-based speech analysis API) to convert audio data sent by the user into text. Next, it uses translation functionality to convert the user's native language to a second language (e.g., a general-purpose translation API), and leverages generative AI models (e.g., an open-source natural language generation engine) to generate appropriate responses for the user. Furthermore, by performing these processes in real time, it provides an advanced language learning experience.

[0080] The terminal communicates with the server as a user interface. Specifically, it sends audio data to the server and plays back the server's response in both audio and subtitle format. The terminal utilizes a virtual reality environment (e.g., a typical VR engine) to provide the user with immersive dialogue practice, allowing the user to experience practical scenarios.

[0081] Users utilize this system to advance their language learning. As an example of a prompt, a user might input "Tell me a phrase to order food at a restaurant" into the system. The device then sends the audio to the server, which generates an appropriate response such as "Can I order some food, please?", which the device then plays back to the user. The user can listen to the audio and deepen their understanding of the phrase through subtitles displayed on the screen.

[0082] As a concrete example, if a user wants to practice dialogue in a travel scenario, the device will create a virtual airport scene, simulating a situation where the user asks the question "Where is the boarding gate?" at the entrance. The server will translate this to "Where is the boarding gate?" and provide the user with this sentence as practice material. Through such realistic scenarios, the system provides an environment where users can acquire language naturally.

[0083] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0084] Step 1:

[0085] The user uses the device's microphone to input a question in their native language. The input audio is converted into digital data and received by the device. The device then appropriately encodes the audio data and prepares it for transmission to the server.

[0086] Step 2:

[0087] The terminal transmits digitized audio data to a server via the internet. Upon receiving this data, the server uses speech recognition software to convert it into text data. This conversion process involves natural language processing, which analyzes the audio signal and transforms it into corresponding text.

[0088] Step 3:

[0089] The server passes the text obtained through speech recognition to a translation engine, which converts it from the user's native language to a second language. For example, the Japanese phrase "I would like to order a coffee" is translated into English. The translated text is then used directly as input to the generative AI model.

[0090] Step 4:

[0091] The server feeds the translated second-language text into a generation AI model, which then generates a context-based, natural response. Here, the AI ​​complements the context based on the user's past usage history and known dialogue patterns to create an appropriate response. For example, it generates a dialogue model to approach the request, "I want to order coffee."

[0092] Step 5:

[0093] The generated response is translated back into the user's native language and sent to the speech synthesis process. The server applies speech synthesis technology to convert the text response into audio data and prepares to send the result to the terminal. The speech synthesis process generates synthesized speech from the text and outputs it as a digital audio file.

[0094] Step 6:

[0095] The server sends the audio file and response text together to the terminal. The terminal receives this, displays it as subtitles on the screen, and plays the audio. The user can learn how to use the language in real time by reviewing this output. The on-screen subtitles are displayed in real time to aid understanding and help the user follow the flow of the conversation.

[0096] (Application Example 1)

[0097] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0098] In language learning, especially for beginners, dialogue-based practice and application in concrete everyday situations are crucial. However, current online learning tools and materials struggle to replicate real-life interactions and make it difficult to receive personalized feedback in real time. The challenge lies in solving these problems and providing an effective and immersive language learning experience.

[0099] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0100] This invention includes a server that provides means for supporting education using machine learning capable of two-way dialogue using a first language and a second language, means for using a robotic device that supports language learning through dialogue in daily life in a home environment, and means for analyzing the user's questions and generating appropriate language expressions using speech recognition and natural language processing technologies. This enables the user to learn a language through natural dialogue in the home and receive immediate feedback, thereby effectively improving their language skills.

[0101] The terms "first language and second language" refer to the user's native language, which they already speak, and the target language they are trying to learn.

[0102] "Two-way dialogue" refers to a form of linguistic exchange in which the user and the system communicate with each other.

[0103] "Education using machine learning" is a method that utilizes machine learning technology to provide educational programs optimized for individual learners.

[0104] A "robot device that supports language learning through dialogue in daily life within a home environment" is a robot used in the home that supports the user's language learning through everyday conversation.

[0105] "Speech recognition technology" is a technology that receives voice input from a user, analyzes its content, and converts it into understandable digital data.

[0106] "Natural language processing technology" is a technology that enables computers to understand, generate, and analyze human language, making natural dialogue between users and systems possible.

[0107] "Generating appropriate language expressions" is the process of creating correct words and phrases in the target language based on user input.

[0108] The system that realizes this invention consists of three components: a server, a terminal, and a user. The server supports education by centering on a machine learning engine that conducts two-way dialogue using the learner's native language and the target language. Specifically, it can analyze the user's voice input and generate appropriate responses by utilizing speech recognition technology and natural language processing technology. To this end, the server uses Google® Cloud Speech-to-Text API and OpenAI® generative AI models.

[0109] Users utilize a robotic device specifically designed for language learning within their everyday home environment. This device uses a built-in microphone to collect audio and transmit it to a server. The server's response is played back as audio through a speaker, while subtitles of the dialogue are displayed on the screen. This facilitates language comprehension through both audio and visual means.

[0110] On the device side, the accuracy of the user's pronunciation and vocabulary is evaluated in real time, and learning is adaptively adjusted. New learning content is dynamically generated and provided based on the user's past dialogue history and learning performance.

[0111] As a concrete example, a user can ask a robotic device how to say "crack an egg" in English while cooking. The server receives this and generates the phrase "Crack the egg," providing immediate feedback with both audio and subtitles. The user can then check the correct expression and practice their pronunciation.

[0112] Examples of prompts include, "Translate this sentence into English: I am studying Japanese," and "Tell me how to respond to the following dialogue in Spanish: 'Please give me some bread.'" These operations create an environment in which learners can naturally improve their language skills.

[0113] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0114] Step 1:

[0115] The user provides voice input to the robotic device within their home. The user's speech is captured by the robot's microphone. The input data is the user's voice signal.

[0116] Step 2:

[0117] The device uses speech recognition technology to convert the user's voice signal into digital text data. This process utilizes the Google Cloud Speech-to-Text API to analyze and transcribe the voice data. The output is the text data of what the user said.

[0118] Step 3:

[0119] The server receives the generated text data and performs the corresponding translation using natural language processing technology. In this process, OpenAI's generative AI model is used to convert the text data into the target language and generate a response with appropriate grammar and expression. The output is the translated text data.

[0120] Step 4:

[0121] The translated text data is sent back to the device and converted into audio data using speech synthesis technology. The user confirms the correct pronunciation of the target language through the audio played from the device's speaker. The audio data is then output.

[0122] Step 5:

[0123] Simultaneously, subtitles of the dialogue are displayed on the device's screen in real time. This allows users to visually confirm the learning content. The displayed subtitle data is the previously generated translated text data.

[0124] Step 6:

[0125] The server dynamically generates the next learning content using the user's past learning performance data. This data includes previous interaction history and current progress. Based on this, it adjusts the individual learning plan to provide the user with the optimal learning experience. The output is the newly generated learning plan data.

[0126] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0127] This invention provides a system that incorporates emotion recognition functionality to improve the language learning experience. This system identifies the user's emotions in real time and adaptively adjusts the learning plan and dialogue content to create a more personalized learning environment. The system's components and specific processing details are described below.

[0128] The server is equipped with an emotion engine that analyzes emotional data from the user's facial expressions and voice. The emotion engine uses machine learning algorithms to classify the user's emotional state and adjusts the dialogue content based on that information. The server also works in conjunction with existing AI language models to provide two-way language learning support.

[0129] The device is equipped with sensors that detect the user's biometric signals and collect data. When the user uses the device, it sends emotional data to the server in real time and returns feedback to the user based on that data. This enables natural conversations that are in line with the user's emotions.

[0130] The user initiates a conversation with the AI ​​tutor through a device equipped with emotion recognition capabilities. When the user asks a question or makes a statement, the device simultaneously captures not only the content of the statement but also the tone of voice and facial expressions. The server analyzes this data to determine the user's emotional state in real time.

[0131] As a concrete example, consider a scenario where a user is learning the pronunciation of a new word from an AI teacher, makes a mistake, and displays an expression of frustration. The device detects this expression, and the server uses its emotion engine to determine that the user is feeling "frustrated." Based on this, the server provides the device with a gentle, encouraging message and feedback prompting it to try again. It can also temporarily change the learning plan and switch to easier tasks if necessary.

[0132] Through this system, users can gain a learning experience that resonates with their emotions, and are expected to efficiently improve their language skills while maintaining their motivation.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] The user starts up the device and logs into the learning application. The user selects the language to learn, and the device sends this information to the server.

[0136] Step 2:

[0137] The server prepares a learning plan and emotion engine based on the user's selection and sends the necessary data to the device. The server performs the initial setup for the conversation and notifies the device when it is ready.

[0138] Step 3:

[0139] The device performs the initial setup of the virtual reality environment and starts the system so that the user can begin learning. The device is equipped with sensors to detect the user's facial expressions and voice.

[0140] Step 4:

[0141] The user begins a conversation with the AI ​​teacher. As the user speaks, the device captures facial expression data in real time along with the audio and sends it to the server.

[0142] Step 5:

[0143] The server analyzes the received audio data using an AI language model to generate appropriate responses to the user's statements. Simultaneously, the emotion engine analyzes facial expression data to identify the user's emotional state.

[0144] Step 6:

[0145] The server sends a response to the terminal along with a corresponding action based on the user's emotional state. For example, if the user shows signs of anxiety, the server will select a message that includes encouragement.

[0146] Step 7:

[0147] The device plays the response received from the server to the user as audio and displays subtitles on the screen. Emotion-based feedback is also displayed on the screen simultaneously, and learning suggestions are made according to the situation.

[0148] Step 8:

[0149] Users receive the feedback they receive and act upon it to incorporate it into their next interaction and learning plan. They deepen their understanding by attempting to engage in further dialogue on areas of concern.

[0150] Step 9:

[0151] The server accumulates user learning progress and sentiment data, and creates an optimized plan for the next learning session. Based on the analysis results, further fine-tuning of the learning process is performed.

[0152] Through these steps, users can improve their language skills in an optimal learning environment that is sensitive to their emotions.

[0153] (Example 2)

[0154] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0155] Traditional language learning systems have the drawback of only being able to respond to the user's superficial learning progress and lacking the flexibility to adapt to the user's emotional state. This can lead to decreased motivation and difficulty in efficient learning. Furthermore, individual learning plans are not sufficiently personalized, creating a need for a learning environment optimized for each user.

[0156] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0157] In this invention, the server includes means for analyzing emotional information from the user's facial expressions and voice and adaptively adjusting the learning plan and dialogue content based on those emotions; means for providing information generated based on the emotional analysis results and generating appropriate feedback using a generated information processing model; and means for dynamically constructing the next learning content based on the user's past dialogue records and learning efficiency. This makes it possible to provide a personalized learning experience that is in line with the user's emotional state.

[0158] "Language learning" refers to activities that users engage in to acquire a new language, and is a process of improving comprehension and expression through the use of sound, text, and other means.

[0159] An "information processing device" refers to a computer system that can take data as input and output necessary information through processing.

[0160] A "learning plan" is an educational design that sets specific content and timeframes to achieve the user's learning objectives.

[0161] A "virtual environment" is a computer simulation space created using digital technology, allowing users to have an experience similar to reality.

[0162] "Learning progress information" refers to data that shows how well a user understands the content and has achieved their goals during the learning process.

[0163] A "user interface" is a set of components that enable interaction between a system and a user, and includes screen displays and input methods.

[0164] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions and voice.

[0165] "Sentiment analysis" is the process of identifying and classifying a user's emotional state using machine learning algorithms and data processing.

[0166] A "generative information processing model" is a system that has an algorithm for generating a specific output based on input data.

[0167] "Feedback" refers to the responses, suggestions, and evaluations that a system provides to a user.

[0168] "Dialogue records" are data from past conversations and interactions, and they serve as the foundation for constructing new learning content.

[0169] "Learning efficiency" is an indicator that represents the speed and effort required for users to achieve their set learning goals.

[0170] This invention aims to improve the effectiveness of language learning by constructing an information processing system that recognizes user emotions in real time and provides corresponding feedback. The specific configuration and operation of the system are described below.

[0171] The server is equipped with an emotion engine and a generative information processing model. The emotion engine analyzes facial and voice data sent from the user's device and uses machine learning algorithms to identify the user's emotional information. For example, if a user shows frustration while learning a new word and practicing pronunciation, the emotion engine will classify this as "frustration." The server uses this emotional information to input into the generative AI model, which generates the most appropriate feedback for the user. This feedback is generated based on prompts and may include encouraging messages such as, "It's okay, let's try again."

[0172] Meanwhile, the device is equipped with sensors to acquire the user's biometric signals. When the user begins language learning, the device uses its camera and microphone to collect data such as facial expressions and voice tone, and transmits it to the server in real time. This allows for the immediate adjustment of a personalized learning plan based on the user's emotional information.

[0173] Through this system, users can learn while receiving personalized support tailored to their emotional state. For example, the system can dynamically construct the content of the next dialogue based on past conversation history. Through this approach, users are expected to efficiently improve their language skills.

[0174] An example of a prompt message is, "Generate the best feedback message for when the user is feeling frustrated." Based on such prompt messages, the server generates and provides a message that is sensitive to the user's emotional state.

[0175] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0176] Step 1:

[0177] The device captures the user's facial expressions and voice in real time. It uses sensors to capture the user's face with a camera and record their voice with a microphone. This data serves as input for analyzing the user's emotions.

[0178] Step 2:

[0179] The device sends the collected facial expression data and audio data to the server. The data sent includes image frames and audio files, which are the materials needed for analysis on the server.

[0180] Step 3:

[0181] The server uses an emotion engine to analyze the user's emotions. It applies machine learning algorithms to the received data to classify the emotional information. For example, it identifies emotional states such as "satisfied," "frustrated," and "questionable." This process yields the emotion analysis results.

[0182] Step 4:

[0183] The server generates feedback using a generative AI model based on the sentiment analysis results. The prompt "Generate the best feedback message for when the user is feeling frustrated." is input to the generative AI model, and a specific recommendation message is output.

[0184] Step 5:

[0185] The server sends the generated feedback message to the terminal. The message is formatted in a user-friendly format and prepared as data for display on the terminal.

[0186] Step 6:

[0187] The device displays feedback messages received from the server to the user. It provides immediate feedback to the user by displaying text messages on the screen and issuing audio alerts as needed. This process allows the user to receive helpful guidance that responds to changes in their emotions.

[0188] (Application Example 2)

[0189] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0190] When learning a language, instruction and feedback tailored to individual emotional states are limited, resulting in a problem where users cannot receive education that addresses their own feelings. Furthermore, a uniform approach risks diminishing learners' motivation and may not optimize learning efficiency. As a result, there is a challenge in providing flexible learning support that meets the individual needs of users.

[0191] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0192] In this invention, the server includes means for supporting language learning using a computer capable of two-way dialogue using a first language and a second language, means for recognizing emotions and adaptively adjusting the learning plan and dialogue content based on the emotional state, and means for a home device to provide emotionally responsive feedback in voice. This makes it possible to respond to the user's emotional state and provide an optimal learning experience.

[0193] A "computer" is an electronic device that automatically performs data processing and operations, and in particular refers to a device that supports language learning using artificial intelligence.

[0194] "Means of supporting language learning" refer to technological means that enable users to effectively acquire a language, such as providing interactive dialogue and learning plans.

[0195] An "individualized learning plan" refers to a customized learning program created based on the user's native language and the language they are learning.

[0196] A "virtual reality environment" is a simulation environment created using computer graphics and sensor technology to allow users to experience reality.

[0197] "Learning progress data" refers to data that represents the progress and results recorded by users during the process of language learning, and is analyzable information.

[0198] A "user interface" refers to a device or display screen that allows a user and a system to exchange information with each other.

[0199] "Recognizing emotions" refers to technology that analyzes biosignals such as a user's facial expressions and voice to determine their current emotional state.

[0200] "Adjusting learning plans and dialogue content based on emotional state" refers to the process of modifying the learning process or providing individually tailored feedback in response to recognized emotions.

[0201] "Household appliances" refer to robots and electronic devices used in the home, and are generally devices that perform various tasks to assist users.

[0202] "Providing feedback via voice" refers to the process of giving advice and evaluations via voice based on the learner's actions and responses.

[0203] To implement this invention, a computer is first configured as hardware equipped with speech recognition and facial expression analysis functions. Specifically, a home-use device equipped with a camera and microphone is required. This device acquires the user's speech and facial expressions in real time, and emotion recognition software in the cloud analyzes this data.

[0204] The server is equipped with an emotion recognition engine that identifies the user's emotional state based on data sent from the device and generates appropriate feedback. This feedback is dynamically generated by a generative AI model as language learning content suitable for the specific emotion. The device then delivers this feedback to the user via voice.

[0205] For example, if a user makes a sulky face while practicing English pronunciation, the calculator can generate feedback such as, "Calm down a bit and try again. Let's work together until you get it!" This allows the user to concentrate on learning with peace of mind.

[0206] An example of a prompt to input into a generative AI model is, "Generate a message that provides positive feedback when the user mispronounces something." This prompt serves as the foundation for learning support that is empathetic to the user's emotions, helping to improve the quality of learning.

[0207] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0208] Step 1:

[0209] The device acquires the user's voice and facial expressions in real time through its camera and microphone. Audio and video data are obtained as input. The device then prepares this data to send to the server.

[0210] Step 2:

[0211] The server inputs the audio and video data received from the terminal into the emotion recognition engine for analysis. The emotion recognition engine uses a machine learning algorithm to identify the user's emotional state. This process yields a classification result of the emotional state as output.

[0212] Step 3:

[0213] The server initiates the process of inputting a prompt message into the generative AI model based on the classification results of the emotional state. The prompt message used is, "Generate a message that provides positive feedback when the user mispronounces something." This process generates an appropriate message.

[0214] Step 4:

[0215] The server converts the generated feedback message into audio data and sends it to the terminal. This process uses a text-to-speech (TTS) engine to generate audio that is easy for the user to understand.

[0216] Step 5:

[0217] The device delivers audio feedback received from the server to the user via a playback device. By listening to this feedback, the user can receive support tailored to their learning process.

[0218] Step 6:

[0219] The user incorporates feedback from their device and continues learning. Performing new learning actions or responses restarts the process from step 1.

[0220] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0221] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0222] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0223] [Second Embodiment]

[0224] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0225] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0226] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0227] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0228] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0230] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0231] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0232] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0233] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0234] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0235] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0236] This invention provides a system for effectively supporting language learning through artificial intelligence technology. This system achieves a personalized learning experience tailored to each individual learner through the coordinated operation of a server, terminal, and user. The roles of each component are described below.

[0237] The server is built around an artificial intelligence engine that can use both the learner's native language and the second language they wish to learn. This engine enables two-way dialogue, analyzing voice input from the user and generating appropriate responses. The server also manages progress data from many learners and processes individual progress to dynamically generate the optimal learning plan.

[0238] The terminal functions as a user interface, receiving and displaying data sent from the server. In particular, it generates interactive dialogue scenes using a virtual reality environment, providing the user with a realistic learning experience. The terminal acquires the user's voice through a microphone and sends it to the server, while playing back the server's response with audio and subtitles.

[0239] The user initiates a conversation with the AI ​​tutor through this system. When the user asks a question in their native language, the device receives it, processes it on the server, and then returns a response in the target language to the user. To aid understanding, subtitles of the conversation are displayed on the device screen in real time. This method makes it easier for users to become familiar with a second language, even from the basics, and allows them to progress in their learning.

[0240] As a concrete example, let's say a user uses this system to learn English for travel. A virtual cafe scene unfolds on the device, and the user tells the AI ​​teacher in their native language, "I'd like to order a coffee." The server receives this phrase, generates the appropriate English expression, "I would like to order a coffee," and returns it to the device. The user listens to the English expression, and at the same time, the phrase is displayed as subtitles, helping them understand pronunciation and grammar. In this way, the user repeatedly practices what they have learned in a way that is relevant to real-life situations, and acquires the language in a natural manner.

[0241] The following describes the processing flow.

[0242] Step 1:

[0243] The user starts up their device and logs into the learning application. The user selects the language and level they want to learn on their device. The selection information is sent from the device to the server.

[0244] Step 2:

[0245] Based on the user's selection, the server retrieves data for the appropriate learning mode and virtual environment. The server then creates a pre-configured learning plan and sends it to the terminal.

[0246] Step 3:

[0247] The terminal sets up the virtual reality environment based on data received from the server. The user is notified on the terminal's screen or via audio when it is ready.

[0248] Step 4:

[0249] The user begins a conversation with the AI ​​teacher through the device. The user asks for simple phrases in the language they want to learn in their native language and inputs them into the device via the microphone.

[0250] Step 5:

[0251] The device sends the user's voice input to the server in real time. The server analyzes this data and generates translations and responses using an AI language model.

[0252] Step 6:

[0253] The server sends the generated translation and audio data to the terminal, providing the user with an appropriate response. For English phrases, the terminal will play the audio "I would like to order a coffee."

[0254] Step 7:

[0255] The device displays subtitles on the screen along with the response. These subtitles are updated in real time to aid user understanding.

[0256] Step 8:

[0257] The user continues practicing the dialogue by asking questions and confirming details. Feedback on differences in pronunciation and intonation is displayed on the device screen.

[0258] Step 9:

[0259] The server collects user progress and performance data and saves the information for the next session. Based on the analysis results, it adjusts the learning plan and provides advice.

[0260] By repeating the above steps, users will gradually improve their language skills.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0263] Traditional language learning systems often lack sufficient two-way communication and personalized learning, making it difficult for learners to acquire a language efficiently. Furthermore, the lack of real-time feedback and self-paced virtual environment dialogue training hindered the improvement of actual communication skills.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for supporting language learning using an intelligent control device capable of two-way dialogue using a first language and a second language, means for creating an individualized learning plan based on the user's native language and the language to be learned, and means for integrating speech recognition and translation functions to perform language conversion and response generation in real time. This enables learners to effectively acquire a language at their own pace through interactive, practical dialogue while receiving appropriate feedback in real time.

[0266] An "intelligent control device" is a device that uses artificial intelligence technology to perform advanced processing related to language conversion and dialogue generation.

[0267] "Supporting language learning" means providing users with various functions and tools to efficiently acquire their target language.

[0268] An "individualized learning plan" is a curriculum that includes learning procedures and content optimized according to each learner's progress and needs.

[0269] A "virtual reality environment" is a virtual space created using computer technology, through which users can engage in dialogue training in a manner close to reality.

[0270] "Dialogue practice" is an activity that simulates two-way language communication for the purpose of language acquisition.

[0271] "Learning progress data" refers to information that quantifies or records the user's progress and achievements in language learning.

[0272] "Instant reporting" means providing learners with real-time feedback.

[0273] "Displaying explanatory text" means showing textual information on the screen to help the user better understand the content of the conversation.

[0274] "Speech recognition" is the process of analyzing speech input through a microphone and converting it into digital text information.

[0275] A "translation function" is a technology that converts information expressed in one language into another language.

[0276] "Response generation" refers to creating a response in the appropriate language based on user input.

[0277] A "generative AI model" is an artificial intelligence technology or algorithm used in natural language processing to generate contextually appropriate text.

[0278] "Optimizing according to context" means adjusting the information provided to best suit the flow of the conversation and the user's needs.

[0279] This invention operates based on a language learning support system using an intelligent control device. The main components are a server, a terminal, and a user.

[0280] The server functions as a central hub for supporting language learning. Specifically, it uses speech recognition software (e.g., a common cloud-based speech analysis API) to convert audio data sent by the user into text. Next, it uses translation functionality to convert the user's native language to a second language (e.g., a general-purpose translation API), and leverages generative AI models (e.g., an open-source natural language generation engine) to generate appropriate responses for the user. Furthermore, by performing these processes in real time, it provides an advanced language learning experience.

[0281] The terminal communicates with the server as a user interface. Specifically, it sends audio data to the server and plays back the server's response in both audio and subtitle format. The terminal utilizes a virtual reality environment (e.g., a typical VR engine) to provide the user with immersive dialogue practice, allowing the user to experience practical scenarios.

[0282] The user uses this system to advance language learning. As an example of a prompt sentence, the user inputs "Teach me the phrases for ordering food at a restaurant." into the system. At this time, the terminal sends the voice to the server, and the server generates an appropriate response such as "Can I order some food, please?" and the terminal plays it back to the user. The user can deepen the understanding of the phrase by listening to the voice and through the subtitles displayed on the screen.

[0283] As a specific example, when the user wants to practice conversations in a travel scenario, the terminal constructs a virtual airport scene and reproduces the situation where the user asks "Where is the boarding gate?" at the entrance. The server converts it to "Where is the boarding gate?" and provides this sentence to the user as practice material. Through such real scenarios, an environment is provided where the user can acquire the language in a natural form.

[0284] The flow of the specific process in Example 1 will be described using FIG. 11.

[0285] Step 1:

[0286] The user uses the microphone of the terminal to input a question in the mother tongue. The input voice is converted into digital data and captured by the terminal. The terminal appropriately encodes the voice data and prepares to send it to the server.

[0287] Step 2:

[0288] The terminal sends the digitized voice data to the server via the Internet. When the server receives this data, it uses voice recognition software to convert it into text data. In this conversion process, natural language processing is performed to analyze the voice signal and convert it into the corresponding text.

[0289] Step 3: <​The server passes the text obtained through speech recognition to a translation engine, which converts it from the user's native language to a second language. For example, the Japanese phrase "I would like to order a coffee" is translated into English. The translated text is then used directly as input to the generative AI model.

[0291] Step 4:

[0292] The server feeds the translated second-language text into a generation AI model, which then generates a context-based, natural response. Here, the AI ​​complements the context based on the user's past usage history and known dialogue patterns to create an appropriate response. For example, it generates a dialogue model to approach the request, "I want to order coffee."

[0293] Step 5:

[0294] The generated response is translated back into the user's native language and sent to the speech synthesis process. The server applies speech synthesis technology to convert the text response into audio data and prepares to send the result to the terminal. The speech synthesis process generates synthesized speech from the text and outputs it as a digital audio file.

[0295] Step 6:

[0296] The server sends the audio file and response text together to the terminal. The terminal receives this, displays it as subtitles on the screen, and plays the audio. The user can learn how to use the language in real time by reviewing this output. The on-screen subtitles are displayed in real time to aid understanding and help the user follow the flow of the conversation.

[0297] (Application Example 1)

[0298] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0299] In language learning, especially for beginners, dialogue-based practice and application in concrete everyday situations are crucial. However, current online learning tools and materials struggle to replicate real-life interactions and make it difficult to receive personalized feedback in real time. The challenge lies in solving these problems and providing an effective and immersive language learning experience.

[0300] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0301] This invention includes a server that provides means for supporting education using machine learning capable of two-way dialogue using a first language and a second language, means for using a robotic device that supports language learning through dialogue in daily life in a home environment, and means for analyzing the user's questions and generating appropriate language expressions using speech recognition and natural language processing technologies. This enables the user to learn a language through natural dialogue in the home and receive immediate feedback, thereby effectively improving their language skills.

[0302] The terms "first language and second language" refer to the user's native language, which they already speak, and the target language they are trying to learn.

[0303] "Two-way dialogue" refers to a form of linguistic exchange in which the user and the system communicate with each other.

[0304] "Education using machine learning" is a method that utilizes machine learning technology to provide educational programs optimized for individual learners.

[0305] A "robot device that supports language learning through dialogue in daily life within a home environment" is a robot used in the home that supports the user's language learning through everyday conversation.

[0306] "Voice recognition technology" refers to the technology that receives a user's voice input and converts it into digital data that can be analyzed and understood by a computer.

[0307] "Natural language processing technology" refers to the technology for a computer to understand, generate, and analyze human language, enabling natural interaction between a user and a system.

[0308] "Generating appropriate language expressions" refers to the process of creating correct words and phrases in the target language according to a user's input.

[0309] The system that realizes this invention is composed of three parties: a server, a terminal, and a user. The server supports education centered around a machine learning engine that conducts two-way conversations using the learner's native language and the target language for learning. Specifically, by leveraging voice recognition technology and natural language processing technology, it is possible to analyze a user's voice input and generate appropriate responses. For this purpose, the server uses the Google Cloud Speech-to-Text API and the generative AI model of OpenAI.

[0310] The user uses a robot device specialized for language learning in the home environment where they spend their daily lives. This device collects voice using the built-in microphone and sends it to the server. The response from the server is reproduced as voice through a speaker, and at the same time, subtitles of the conversation are displayed on the display. This supports language understanding through both voice and vision.

[0311] On the terminal side, it evaluates the accuracy of the user's pronunciation and phrases in real time and adaptively adjusts the learning. Based on the user's past conversation history and learning performance, it dynamically generates and provides new learning content. [[ID=2�]]

[0312] As a concrete example, a user can ask a robotic device how to say "crack an egg" in English while cooking. The server receives this and generates the phrase "Crack the egg," providing immediate feedback with both audio and subtitles. The user can then check the correct expression and practice their pronunciation.

[0313] Examples of prompts include, "Translate this sentence into English: I am studying Japanese," and "Tell me how to respond to the following dialogue in Spanish: 'Please give me some bread.'" These operations create an environment in which learners can naturally improve their language skills.

[0314] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0315] Step 1:

[0316] The user provides voice input to the robotic device within their home. The user's speech is captured by the robot's microphone. The input data is the user's voice signal.

[0317] Step 2:

[0318] The device uses speech recognition technology to convert the user's voice signal into digital text data. This process utilizes the Google Cloud Speech-to-Text API to analyze and transcribe the voice data. The output is the text data of what the user said.

[0319] Step 3:

[0320] The server receives the generated text data and performs the corresponding translation using natural language processing technology. In this process, OpenAI's generative AI model is used to convert the text data into the target language and generate a response with appropriate grammar and expression. The output is the translated text data.

[0321] Step 4:

[0322] The translated text data is sent back to the device and converted into audio data using speech synthesis technology. The user confirms the correct pronunciation of the target language through the audio played from the device's speaker. The audio data is then output.

[0323] Step 5:

[0324] Simultaneously, subtitles of the dialogue are displayed on the device's screen in real time. This allows users to visually confirm the learning content. The displayed subtitle data is the previously generated translated text data.

[0325] Step 6:

[0326] The server dynamically generates the next learning content using the user's past learning performance data. This data includes previous interaction history and current progress. Based on this, it adjusts the individual learning plan to provide the user with the optimal learning experience. The output is the newly generated learning plan data.

[0327] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0328] This invention provides a system that incorporates emotion recognition functionality to improve the language learning experience. This system identifies the user's emotions in real time and adaptively adjusts the learning plan and dialogue content to create a more personalized learning environment. The system's components and specific processing details are described below.

[0329] The server is equipped with an emotion engine that analyzes emotional data from the user's facial expressions and voice. The emotion engine uses machine learning algorithms to classify the user's emotional state and adjusts the dialogue content based on that information. The server also works in conjunction with existing AI language models to provide two-way language learning support.

[0330] The device is equipped with sensors that detect the user's biometric signals and collect data. When the user uses the device, it sends emotional data to the server in real time and returns feedback to the user based on that data. This enables natural conversations that are in line with the user's emotions.

[0331] The user initiates a conversation with the AI ​​tutor through a device equipped with emotion recognition capabilities. When the user asks a question or makes a statement, the device simultaneously captures not only the content of the statement but also the tone of voice and facial expressions. The server analyzes this data to determine the user's emotional state in real time.

[0332] As a concrete example, consider a scenario where a user is learning the pronunciation of a new word from an AI teacher, makes a mistake, and displays an expression of frustration. The device detects this expression, and the server uses its emotion engine to determine that the user is feeling "frustrated." Based on this, the server provides the device with a gentle, encouraging message and feedback prompting it to try again. It can also temporarily change the learning plan and switch to easier tasks if necessary.

[0333] Through this system, users can gain a learning experience that resonates with their emotions, and are expected to efficiently improve their language skills while maintaining their motivation.

[0334] The following describes the processing flow.

[0335] Step 1:

[0336] The user starts up the device and logs into the learning application. The user selects the language to learn, and the device sends this information to the server.

[0337] Step 2:

[0338] The server prepares a learning plan and emotion engine based on the user's selection and sends the necessary data to the device. The server performs the initial setup for the conversation and notifies the device when it is ready.

[0339] Step 3:

[0340] The device performs the initial setup of the virtual reality environment and starts the system so that the user can begin learning. The device is equipped with sensors to detect the user's facial expressions and voice.

[0341] Step 4:

[0342] The user begins a conversation with the AI ​​teacher. As the user speaks, the device captures facial expression data in real time along with the audio and sends it to the server.

[0343] Step 5:

[0344] The server analyzes the received audio data using an AI language model to generate appropriate responses to the user's statements. Simultaneously, the emotion engine analyzes facial expression data to identify the user's emotional state.

[0345] Step 6:

[0346] The server sends a response to the terminal along with a corresponding action based on the user's emotional state. For example, if the user shows signs of anxiety, the server will select a message that includes encouragement.

[0347] Step 7:

[0348] The device plays the response received from the server to the user as audio and displays subtitles on the screen. Emotion-based feedback is also displayed on the screen simultaneously, and learning suggestions are made according to the situation.

[0349] Step 8:

[0350] Users receive the feedback they receive and act upon it to incorporate it into their next interaction and learning plan. They deepen their understanding by attempting to engage in further dialogue on areas of concern.

[0351] Step 9:

[0352] The server accumulates user learning progress and sentiment data, and creates an optimized plan for the next learning session. Based on the analysis results, further fine-tuning of the learning process is performed.

[0353] Through these steps, users can improve their language skills in an optimal learning environment that is sensitive to their emotions.

[0354] (Example 2)

[0355] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0356] Traditional language learning systems have the drawback of only being able to respond to the user's superficial learning progress and lacking the flexibility to adapt to the user's emotional state. This can lead to decreased motivation and difficulty in efficient learning. Furthermore, individual learning plans are not sufficiently personalized, creating a need for a learning environment optimized for each user.

[0357] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0358] In this invention, the server includes means for analyzing emotional information from the user's facial expressions and voice and adaptively adjusting the learning plan and dialogue content based on those emotions; means for providing information generated based on the emotional analysis results and generating appropriate feedback using a generated information processing model; and means for dynamically constructing the next learning content based on the user's past dialogue records and learning efficiency. This makes it possible to provide a personalized learning experience that is in line with the user's emotional state.

[0359] "Language learning" refers to activities that users engage in to acquire a new language, and is a process of improving comprehension and expression through the use of sound, text, and other means.

[0360] An "information processing device" refers to a computer system that can take data as input and output necessary information through processing.

[0361] A "learning plan" is an educational design that sets specific content and timeframes to achieve the user's learning objectives.

[0362] A "virtual environment" is a computer simulation space created using digital technology, allowing users to have an experience similar to reality.

[0363] "Learning progress information" refers to data that shows how well a user understands the content and has achieved their goals during the learning process.

[0364] A "user interface" is a set of components that enable interaction between a system and a user, and includes screen displays and input methods.

[0365] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions and voice.

[0366] "Sentiment analysis" is the process of identifying and classifying a user's emotional state using machine learning algorithms and data processing.

[0367] A "generative information processing model" is a system that has an algorithm for generating a specific output based on input data.

[0368] "Feedback" refers to the responses, suggestions, and evaluations that a system provides to a user.

[0369] "Dialogue records" are data from past conversations and interactions, and they serve as the foundation for constructing new learning content.

[0370] "Learning efficiency" is an indicator that represents the speed and effort required for users to achieve their set learning goals.

[0371] This invention aims to improve the effectiveness of language learning by constructing an information processing system that recognizes user emotions in real time and provides corresponding feedback. The specific configuration and operation of the system are described below.

[0372] The server is equipped with an emotion engine and a generative information processing model. The emotion engine analyzes facial and voice data sent from the user's device and uses machine learning algorithms to identify the user's emotional information. For example, if a user shows frustration while learning a new word and practicing pronunciation, the emotion engine will classify this as "frustration." The server uses this emotional information to input into the generative AI model, which generates the most appropriate feedback for the user. This feedback is generated based on prompts and may include encouraging messages such as, "It's okay, let's try again."

[0373] Meanwhile, the device is equipped with sensors to acquire the user's biometric signals. When the user begins language learning, the device uses its camera and microphone to collect data such as facial expressions and voice tone, and transmits it to the server in real time. This allows for the immediate adjustment of a personalized learning plan based on the user's emotional information.

[0374] Through this system, users can learn while receiving personalized support tailored to their emotional state. For example, the system can dynamically construct the content of the next dialogue based on past conversation history. Through this approach, users are expected to efficiently improve their language skills.

[0375] An example of a prompt message is, "Generate the best feedback message for when the user is feeling frustrated." Based on such prompt messages, the server generates and provides a message that is sensitive to the user's emotional state.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] The device captures the user's facial expressions and voice in real time. It uses sensors to capture the user's face with a camera and record their voice with a microphone. This data serves as input for analyzing the user's emotions.

[0379] Step 2:

[0380] The device sends the collected facial expression data and audio data to the server. The data sent includes image frames and audio files, which are the materials needed for analysis on the server.

[0381] Step 3:

[0382] The server uses an emotion engine to analyze the user's emotions. It applies machine learning algorithms to the received data to classify the emotional information. For example, it identifies emotional states such as "satisfied," "frustrated," and "questionable." This process yields the emotion analysis results.

[0383] Step 4:

[0384] The server generates feedback using a generative AI model based on the sentiment analysis results. The prompt "Generate the best feedback message for when the user is feeling frustrated." is input to the generative AI model, and a specific recommendation message is output.

[0385] Step 5:

[0386] The server sends the generated feedback message to the terminal. The message is formatted in a user-friendly format and prepared as data for display on the terminal.

[0387] Step 6:

[0388] The device displays feedback messages received from the server to the user. It provides immediate feedback to the user by displaying text messages on the screen and issuing audio alerts as needed. This process allows the user to receive helpful guidance that responds to changes in their emotions.

[0389] (Application Example 2)

[0390] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0391] When learning a language, instruction and feedback tailored to individual emotional states are limited, resulting in a problem where users cannot receive education that addresses their own feelings. Furthermore, a uniform approach risks diminishing learners' motivation and may not optimize learning efficiency. As a result, there is a challenge in providing flexible learning support that meets the individual needs of users.

[0392] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0393] In this invention, the server includes means for supporting language learning using a computer capable of two-way dialogue using a first language and a second language, means for recognizing emotions and adaptively adjusting the learning plan and dialogue content based on the emotional state, and means for a home device to provide emotionally responsive feedback in voice. This makes it possible to respond to the user's emotional state and provide an optimal learning experience.

[0394] A "computer" is an electronic device that automatically performs data processing and operations, and in particular refers to a device that supports language learning using artificial intelligence.

[0395] "Means of supporting language learning" refer to technological means that enable users to effectively acquire a language, such as providing interactive dialogue and learning plans.

[0396] An "individualized learning plan" refers to a customized learning program created based on the user's native language and the language they are learning.

[0397] A "virtual reality environment" is a simulation environment created using computer graphics and sensor technology to allow users to experience reality.

[0398] "Learning progress data" refers to data that represents the progress and results recorded by users during the process of language learning, and is analyzable information.

[0399] A "user interface" refers to a device or display screen that allows a user and a system to exchange information with each other.

[0400] "Recognizing emotions" refers to technology that analyzes biosignals such as a user's facial expressions and voice to determine their current emotional state.

[0401] "Adjusting learning plans and dialogue content based on emotional state" refers to the process of modifying the learning process or providing individually tailored feedback in response to recognized emotions.

[0402] "Household appliances" refer to robots and electronic devices used in the home, and are generally devices that perform various tasks to assist users.

[0403] "Providing feedback via voice" refers to the process of giving advice and evaluations via voice based on the learner's actions and responses.

[0404] To implement this invention, a computer is first configured as hardware equipped with speech recognition and facial expression analysis functions. Specifically, a home-use device equipped with a camera and microphone is required. This device acquires the user's speech and facial expressions in real time, and emotion recognition software in the cloud analyzes this data.

[0405] The server is equipped with an emotion recognition engine that identifies the user's emotional state based on data sent from the device and generates appropriate feedback. This feedback is dynamically generated by a generative AI model as language learning content suitable for the specific emotion. The device then delivers this feedback to the user via voice.

[0406] For example, if a user makes a sulky face while practicing English pronunciation, the calculator can generate feedback such as, "Calm down a bit and try again. Let's work together until you get it!" This allows the user to concentrate on learning with peace of mind.

[0407] An example of a prompt to input into a generative AI model is, "Generate a message that provides positive feedback when the user mispronounces something." This prompt serves as the foundation for learning support that is empathetic to the user's emotions, helping to improve the quality of learning.

[0408] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0409] Step 1:

[0410] The device acquires the user's voice and facial expressions in real time through its camera and microphone. Audio and video data are obtained as input. The device then prepares this data to send to the server.

[0411] Step 2:

[0412] The server inputs the audio and video data received from the terminal into the emotion recognition engine for analysis. The emotion recognition engine uses a machine learning algorithm to identify the user's emotional state. This process yields a classification result of the emotional state as output.

[0413] Step 3:

[0414] The server initiates the process of inputting a prompt message into the generative AI model based on the classification results of the emotional state. The prompt message used is, "Generate a message that provides positive feedback when the user mispronounces something." This process generates an appropriate message.

[0415] Step 4:

[0416] The server converts the generated feedback message into audio data and sends it to the terminal. This process uses a text-to-speech (TTS) engine to generate audio that is easy for the user to understand.

[0417] Step 5:

[0418] The device delivers audio feedback received from the server to the user via a playback device. By listening to this feedback, the user can receive support tailored to their learning process.

[0419] Step 6:

[0420] The user incorporates feedback from their device and continues learning. Performing new learning actions or responses restarts the process from step 1.

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0422] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0424] [Third Embodiment]

[0425] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0426] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0427] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0429] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0431] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0432] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0433] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0436] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0437] This invention provides a system for effectively supporting language learning through artificial intelligence technology. This system achieves a personalized learning experience tailored to each individual learner through the coordinated operation of a server, terminal, and user. The roles of each component are described below.

[0438] The server is built around an artificial intelligence engine that can use both the learner's native language and the second language they wish to learn. This engine enables two-way dialogue, analyzing voice input from the user and generating appropriate responses. The server also manages progress data from many learners and processes individual progress to dynamically generate the optimal learning plan.

[0439] The terminal functions as a user interface, receiving and displaying data sent from the server. In particular, it generates interactive dialogue scenes using a virtual reality environment, providing the user with a realistic learning experience. The terminal acquires the user's voice through a microphone and sends it to the server, while playing back the server's response with audio and subtitles.

[0440] The user initiates a conversation with the AI ​​tutor through this system. When the user asks a question in their native language, the device receives it, processes it on the server, and then returns a response in the target language to the user. To aid understanding, subtitles of the conversation are displayed on the device screen in real time. This method makes it easier for users to become familiar with a second language, even from the basics, and allows them to progress in their learning.

[0441] As a concrete example, let's say a user uses this system to learn English for travel. A virtual cafe scene unfolds on the device, and the user tells the AI ​​teacher in their native language, "I'd like to order a coffee." The server receives this phrase, generates the appropriate English expression, "I would like to order a coffee," and returns it to the device. The user listens to the English expression, and at the same time, the phrase is displayed as subtitles, helping them understand pronunciation and grammar. In this way, the user repeatedly practices what they have learned in a way that is relevant to real-life situations, and acquires the language in a natural manner.

[0442] The following describes the processing flow.

[0443] Step 1:

[0444] The user starts up their device and logs into the learning application. The user selects the language and level they want to learn on their device. The selection information is sent from the device to the server.

[0445] Step 2:

[0446] Based on the user's selection, the server retrieves data for the appropriate learning mode and virtual environment. The server then creates a pre-configured learning plan and sends it to the terminal.

[0447] Step 3:

[0448] The terminal sets up the virtual reality environment based on data received from the server. The user is notified on the terminal's screen or via audio when it is ready.

[0449] Step 4:

[0450] The user begins a conversation with the AI ​​teacher through the device. The user asks for simple phrases in the language they want to learn in their native language and inputs them into the device via the microphone.

[0451] Step 5:

[0452] The device sends the user's voice input to the server in real time. The server analyzes this data and generates translations and responses using an AI language model.

[0453] Step 6:

[0454] The server sends the generated translation and audio data to the terminal, providing the user with an appropriate response. For English phrases, the terminal will play the audio "I would like to order a coffee."

[0455] Step 7:

[0456] The device displays subtitles on the screen along with the response. These subtitles are updated in real time to aid user understanding.

[0457] Step 8:

[0458] The user continues practicing the dialogue by asking questions and confirming details. Feedback on differences in pronunciation and intonation is displayed on the device screen.

[0459] Step 9:

[0460] The server collects user progress and performance data and saves the information for the next session. Based on the analysis results, it adjusts the learning plan and provides advice.

[0461] By repeating the above steps, users will gradually improve their language skills.

[0462] (Example 1)

[0463] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0464] Traditional language learning systems often lack sufficient two-way communication and personalized learning, making it difficult for learners to acquire a language efficiently. Furthermore, the lack of real-time feedback and self-paced virtual environment dialogue training hindered the improvement of actual communication skills.

[0465] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0466] In this invention, the server includes means for supporting language learning using an intelligent control device capable of two-way dialogue using a first language and a second language, means for creating an individualized learning plan based on the user's native language and the language to be learned, and means for integrating speech recognition and translation functions to perform language conversion and response generation in real time. This enables learners to effectively acquire a language at their own pace through interactive, practical dialogue while receiving appropriate feedback in real time.

[0467] An "intelligent control device" is a device that uses artificial intelligence technology to perform advanced processing related to language conversion and dialogue generation.

[0468] "Supporting language learning" means providing users with various functions and tools to efficiently acquire their target language.

[0469] An "individualized learning plan" is a curriculum that includes learning procedures and content optimized according to each learner's progress and needs.

[0470] A "virtual reality environment" is a virtual space created using computer technology, through which users can engage in dialogue training in a manner close to reality.

[0471] "Dialogue practice" is an activity that simulates two-way language communication for the purpose of language acquisition.

[0472] "Learning progress data" refers to information that quantifies or records the user's progress and achievements in language learning.

[0473] "Instant reporting" means providing learners with real-time feedback.

[0474] "Displaying explanatory text" means showing textual information on the screen to help the user better understand the content of the conversation.

[0475] "Speech recognition" is the process of analyzing speech input through a microphone and converting it into digital text information.

[0476] A "translation function" is a technology that converts information expressed in one language into another language.

[0477] "Response generation" refers to creating a response in the appropriate language based on user input.

[0478] A "generative AI model" is an artificial intelligence technology or algorithm used in natural language processing to generate contextually appropriate text.

[0479] "Optimizing according to context" means adjusting the information provided to best suit the flow of the conversation and the user's needs.

[0480] This invention operates based on a language learning support system using an intelligent control device. The main components are a server, a terminal, and a user.

[0481] The server functions as a central hub for supporting language learning. Specifically, it uses speech recognition software (e.g., a common cloud-based speech analysis API) to convert audio data sent by the user into text. Next, it uses translation functionality to convert the user's native language to a second language (e.g., a general-purpose translation API), and leverages generative AI models (e.g., an open-source natural language generation engine) to generate appropriate responses for the user. Furthermore, by performing these processes in real time, it provides an advanced language learning experience.

[0482] The terminal communicates with the server as a user interface. Specifically, it sends audio data to the server and plays back the server's response in both audio and subtitle format. The terminal utilizes a virtual reality environment (e.g., a typical VR engine) to provide the user with immersive dialogue practice, allowing the user to experience practical scenarios.

[0483] Users utilize this system to advance their language learning. As an example of a prompt, a user might input "Tell me a phrase to order food at a restaurant" into the system. The device then sends the audio to the server, which generates an appropriate response such as "Can I order some food, please?", which the device then plays back to the user. The user can listen to the audio and deepen their understanding of the phrase through subtitles displayed on the screen.

[0484] As a concrete example, if a user wants to practice dialogue in a travel scenario, the device will create a virtual airport scene, simulating a situation where the user asks the question "Where is the boarding gate?" at the entrance. The server will translate this to "Where is the boarding gate?" and provide the user with this sentence as practice material. Through such realistic scenarios, the system provides an environment where users can acquire language naturally.

[0485] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0486] Step 1:

[0487] The user uses the device's microphone to input a question in their native language. The input audio is converted into digital data and received by the device. The device then appropriately encodes the audio data and prepares it for transmission to the server.

[0488] Step 2:

[0489] The terminal transmits digitized audio data to a server via the internet. Upon receiving this data, the server uses speech recognition software to convert it into text data. This conversion process involves natural language processing, which analyzes the audio signal and transforms it into corresponding text.

[0490] Step 3:

[0491] The server passes the text obtained through speech recognition to a translation engine, which converts it from the user's native language to a second language. For example, the Japanese phrase "I would like to order a coffee" is translated into English. The translated text is then used directly as input to the generative AI model.

[0492] Step 4:

[0493] The server feeds the translated second-language text into a generation AI model, which then generates a context-based, natural response. Here, the AI ​​complements the context based on the user's past usage history and known dialogue patterns to create an appropriate response. For example, it generates a dialogue model to approach the request, "I want to order coffee."

[0494] Step 5:

[0495] The generated response is translated back into the user's native language and sent to the speech synthesis process. The server applies speech synthesis technology to convert the text response into audio data and prepares to send the result to the terminal. The speech synthesis process generates synthesized speech from the text and outputs it as a digital audio file.

[0496] Step 6:

[0497] The server sends the audio file and response text together to the terminal. The terminal receives this, displays it as subtitles on the screen, and plays the audio. The user can learn how to use the language in real time by reviewing this output. The on-screen subtitles are displayed in real time to aid understanding and help the user follow the flow of the conversation.

[0498] (Application Example 1)

[0499] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0500] In language learning, especially for beginners, dialogue-based practice and application in concrete everyday situations are crucial. However, current online learning tools and materials struggle to replicate real-life interactions and make it difficult to receive personalized feedback in real time. The challenge lies in solving these problems and providing an effective and immersive language learning experience.

[0501] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0502] This invention includes a server that provides means for supporting education using machine learning capable of two-way dialogue using a first language and a second language, means for using a robotic device that supports language learning through dialogue in daily life in a home environment, and means for analyzing the user's questions and generating appropriate language expressions using speech recognition and natural language processing technologies. This enables the user to learn a language through natural dialogue in the home and receive immediate feedback, thereby effectively improving their language skills.

[0503] The terms "first language and second language" refer to the user's native language, which they already speak, and the target language they are trying to learn.

[0504] "Two-way dialogue" refers to a form of linguistic exchange in which the user and the system communicate with each other.

[0505] "Education using machine learning" is a method that utilizes machine learning technology to provide educational programs optimized for individual learners.

[0506] A "robot device that supports language learning through dialogue in daily life within a home environment" is a robot used in the home that supports the user's language learning through everyday conversation.

[0507] "Speech recognition technology" is a technology that receives voice input from a user, analyzes its content, and converts it into understandable digital data.

[0508] "Natural language processing technology" is a technology that enables computers to understand, generate, and analyze human language, making natural dialogue between users and systems possible.

[0509] "Generating appropriate language expressions" is the process of creating correct words and phrases in the target language based on user input.

[0510] The system that realizes this invention consists of three parties: a server, a terminal, and a user. The server supports education by centering on a machine learning engine that conducts two-way dialogue using the learner's native language and the target language. Specifically, it can analyze the user's voice input and generate appropriate responses by utilizing speech recognition technology and natural language processing technology. To this end, the server uses the Google Cloud Speech-to-Text API and OpenAI's generative AI model.

[0511] Users utilize a robotic device specifically designed for language learning within their everyday home environment. This device uses a built-in microphone to collect audio and transmit it to a server. The server's response is played back as audio through a speaker, while subtitles of the dialogue are displayed on the screen. This facilitates language comprehension through both audio and visual means.

[0512] On the device side, the accuracy of the user's pronunciation and vocabulary is evaluated in real time, and learning is adaptively adjusted. New learning content is dynamically generated and provided based on the user's past dialogue history and learning performance.

[0513] As a concrete example, a user can ask a robotic device how to say "crack an egg" in English while cooking. The server receives this and generates the phrase "Crack the egg," providing immediate feedback with both audio and subtitles. The user can then check the correct expression and practice their pronunciation.

[0514] Examples of prompts include, "Translate this sentence into English: I am studying Japanese," and "Tell me how to respond to the following dialogue in Spanish: 'Please give me some bread.'" These operations create an environment in which learners can naturally improve their language skills.

[0515] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0516] Step 1:

[0517] The user provides voice input to the robotic device within their home. The user's speech is captured by the robot's microphone. The input data is the user's voice signal.

[0518] Step 2:

[0519] The device uses speech recognition technology to convert the user's voice signal into digital text data. This process utilizes the Google Cloud Speech-to-Text API to analyze and transcribe the voice data. The output is the text data of what the user said.

[0520] Step 3:

[0521] The server receives the generated text data and performs the corresponding translation using natural language processing technology. In this process, OpenAI's generative AI model is used to convert the text data into the target language and generate a response with appropriate grammar and expression. The output is the translated text data.

[0522] Step 4:

[0523] The translated text data is sent back to the device and converted into audio data using speech synthesis technology. The user confirms the correct pronunciation of the target language through the audio played from the device's speaker. The audio data is then output.

[0524] Step 5:

[0525] Simultaneously, subtitles of the dialogue are displayed on the device's screen in real time. This allows users to visually confirm the learning content. The displayed subtitle data is the previously generated translated text data.

[0526] Step 6:

[0527] The server dynamically generates the next learning content using the user's past learning performance data. This data includes previous interaction history and current progress. Based on this, it adjusts the individual learning plan to provide the user with the optimal learning experience. The output is the newly generated learning plan data.

[0528] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0529] This invention provides a system that incorporates emotion recognition functionality to improve the language learning experience. This system identifies the user's emotions in real time and adaptively adjusts the learning plan and dialogue content to create a more personalized learning environment. The system's components and specific processing details are described below.

[0530] The server is equipped with an emotion engine that analyzes emotional data from the user's facial expressions and voice. The emotion engine uses machine learning algorithms to classify the user's emotional state and adjusts the dialogue content based on that information. The server also works in conjunction with existing AI language models to provide two-way language learning support.

[0531] The device is equipped with sensors that detect the user's biometric signals and collect data. When the user uses the device, it sends emotional data to the server in real time and returns feedback to the user based on that data. This enables natural conversations that are in line with the user's emotions.

[0532] The user initiates a conversation with the AI ​​tutor through a device equipped with emotion recognition capabilities. When the user asks a question or makes a statement, the device simultaneously captures not only the content of the statement but also the tone of voice and facial expressions. The server analyzes this data to determine the user's emotional state in real time.

[0533] As a concrete example, consider a scenario where a user is learning the pronunciation of a new word from an AI teacher, makes a mistake, and displays an expression of frustration. The device detects this expression, and the server uses its emotion engine to determine that the user is feeling "frustrated." Based on this, the server provides the device with a gentle, encouraging message and feedback prompting it to try again. It can also temporarily change the learning plan and switch to easier tasks if necessary.

[0534] Through this system, users can gain a learning experience that resonates with their emotions, and are expected to efficiently improve their language skills while maintaining their motivation.

[0535] The following describes the processing flow.

[0536] Step 1:

[0537] The user starts up the device and logs into the learning application. The user selects the language to learn, and the device sends this information to the server.

[0538] Step 2:

[0539] The server prepares a learning plan and emotion engine based on the user's selection and sends the necessary data to the device. The server performs the initial setup for the conversation and notifies the device when it is ready.

[0540] Step 3:

[0541] The device performs the initial setup of the virtual reality environment and starts the system so that the user can begin learning. The device is equipped with sensors to detect the user's facial expressions and voice.

[0542] Step 4:

[0543] The user begins a conversation with the AI ​​teacher. As the user speaks, the device captures facial expression data in real time along with the audio and sends it to the server.

[0544] Step 5:

[0545] The server analyzes the received audio data using an AI language model to generate appropriate responses to the user's statements. Simultaneously, the emotion engine analyzes facial expression data to identify the user's emotional state.

[0546] Step 6:

[0547] The server sends a response to the terminal along with a corresponding action based on the user's emotional state. For example, if the user shows signs of anxiety, the server will select a message that includes encouragement.

[0548] Step 7:

[0549] The device plays the response received from the server to the user as audio and displays subtitles on the screen. Emotion-based feedback is also displayed on the screen simultaneously, and learning suggestions are made according to the situation.

[0550] Step 8:

[0551] Users receive the feedback they receive and act upon it to incorporate it into their next interaction and learning plan. They deepen their understanding by attempting to engage in further dialogue on areas of concern.

[0552] Step 9:

[0553] The server accumulates user learning progress and sentiment data, and creates an optimized plan for the next learning session. Based on the analysis results, further fine-tuning of the learning process is performed.

[0554] Through these steps, users can improve their language skills in an optimal learning environment that is sensitive to their emotions.

[0555] (Example 2)

[0556] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0557] Traditional language learning systems have the drawback of only being able to respond to the user's superficial learning progress and lacking the flexibility to adapt to the user's emotional state. This can lead to decreased motivation and difficulty in efficient learning. Furthermore, individual learning plans are not sufficiently personalized, creating a need for a learning environment optimized for each user.

[0558] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0559] In this invention, the server includes means for analyzing emotional information from the user's facial expressions and voice and adaptively adjusting the learning plan and dialogue content based on those emotions; means for providing information generated based on the emotional analysis results and generating appropriate feedback using a generated information processing model; and means for dynamically constructing the next learning content based on the user's past dialogue records and learning efficiency. This makes it possible to provide a personalized learning experience that is in line with the user's emotional state.

[0560] "Language learning" refers to activities that users engage in to acquire a new language, and is a process of improving comprehension and expression through the use of sound, text, and other means.

[0561] An "information processing device" refers to a computer system that can take data as input and output necessary information through processing.

[0562] A "learning plan" is an educational design that sets specific content and timeframes to achieve the user's learning objectives.

[0563] A "virtual environment" is a computer simulation space created using digital technology, allowing users to have an experience similar to reality.

[0564] "Learning progress information" refers to data that shows how well a user understands the content and has achieved their goals during the learning process.

[0565] A "user interface" is a set of components that enable interaction between a system and a user, and includes screen displays and input methods.

[0566] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions and voice.

[0567] "Sentiment analysis" is the process of identifying and classifying a user's emotional state using machine learning algorithms and data processing.

[0568] A "generative information processing model" is a system that has an algorithm for generating a specific output based on input data.

[0569] "Feedback" refers to the responses, suggestions, and evaluations that a system provides to a user.

[0570] "Dialogue records" are data from past conversations and interactions, and they serve as the foundation for constructing new learning content.

[0571] "Learning efficiency" is an indicator that represents the speed and effort required for users to achieve their set learning goals.

[0572] This invention aims to improve the effectiveness of language learning by constructing an information processing system that recognizes user emotions in real time and provides corresponding feedback. The specific configuration and operation of the system are described below.

[0573] The server is equipped with an emotion engine and a generative information processing model. The emotion engine analyzes facial and voice data sent from the user's device and uses machine learning algorithms to identify the user's emotional information. For example, if a user shows frustration while learning a new word and practicing pronunciation, the emotion engine will classify this as "frustration." The server uses this emotional information to input into the generative AI model, which generates the most appropriate feedback for the user. This feedback is generated based on prompts and may include encouraging messages such as, "It's okay, let's try again."

[0574] Meanwhile, the device is equipped with sensors to acquire the user's biometric signals. When the user begins language learning, the device uses its camera and microphone to collect data such as facial expressions and voice tone, and transmits it to the server in real time. This allows for the immediate adjustment of a personalized learning plan based on the user's emotional information.

[0575] Through this system, users can learn while receiving personalized support tailored to their emotional state. For example, the system can dynamically construct the content of the next dialogue based on past conversation history. Through this approach, users are expected to efficiently improve their language skills.

[0576] An example of a prompt message is, "Generate the best feedback message for when the user is feeling frustrated." Based on such prompt messages, the server generates and provides a message that is sensitive to the user's emotional state.

[0577] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0578] Step 1:

[0579] The device captures the user's facial expressions and voice in real time. It uses sensors to capture the user's face with a camera and record their voice with a microphone. This data serves as input for analyzing the user's emotions.

[0580] Step 2:

[0581] The device sends the collected facial expression data and audio data to the server. The data sent includes image frames and audio files, which are the materials needed for analysis on the server.

[0582] Step 3:

[0583] The server uses an emotion engine to analyze the user's emotions. It applies machine learning algorithms to the received data to classify the emotional information. For example, it identifies emotional states such as "satisfied," "frustrated," and "questionable." This process yields the emotion analysis results.

[0584] Step 4:

[0585] The server generates feedback using a generative AI model based on the sentiment analysis results. The prompt "Generate the best feedback message for when the user is feeling frustrated." is input to the generative AI model, and a specific recommendation message is output.

[0586] Step 5:

[0587] The server sends the generated feedback message to the terminal. The message is formatted in a user-friendly format and prepared as data for display on the terminal.

[0588] Step 6:

[0589] The device displays feedback messages received from the server to the user. It provides immediate feedback to the user by displaying text messages on the screen and issuing audio alerts as needed. This process allows the user to receive helpful guidance that responds to changes in their emotions.

[0590] (Application Example 2)

[0591] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0592] When learning a language, instruction and feedback tailored to individual emotional states are limited, resulting in a problem where users cannot receive education that addresses their own feelings. Furthermore, a uniform approach risks diminishing learners' motivation and may not optimize learning efficiency. As a result, there is a challenge in providing flexible learning support that meets the individual needs of users.

[0593] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0594] In this invention, the server includes means for supporting language learning using a computer capable of two-way dialogue using a first language and a second language, means for recognizing emotions and adaptively adjusting the learning plan and dialogue content based on the emotional state, and means for a home device to provide emotionally responsive feedback in voice. This makes it possible to respond to the user's emotional state and provide an optimal learning experience.

[0595] A "computer" is an electronic device that automatically performs data processing and operations, and in particular refers to a device that supports language learning using artificial intelligence.

[0596] "Means of supporting language learning" refer to technological means that enable users to effectively acquire a language, such as providing interactive dialogue and learning plans.

[0597] An "individualized learning plan" refers to a customized learning program created based on the user's native language and the language they are learning.

[0598] A "virtual reality environment" is a simulation environment created using computer graphics and sensor technology to allow users to experience reality.

[0599] "Learning progress data" refers to data that represents the progress and results recorded by users during the process of language learning, and is analyzable information.

[0600] A "user interface" refers to a device or display screen that allows a user and a system to exchange information with each other.

[0601] "Recognizing emotions" refers to technology that analyzes biosignals such as a user's facial expressions and voice to determine their current emotional state.

[0602] "Adjusting learning plans and dialogue content based on emotional state" refers to the process of modifying the learning process or providing individually tailored feedback in response to recognized emotions.

[0603] "Household appliances" refer to robots and electronic devices used in the home, and are generally devices that perform various tasks to assist users.

[0604] "Providing feedback via voice" refers to the process of giving advice and evaluations via voice based on the learner's actions and responses.

[0605] To implement this invention, a computer is first configured as hardware equipped with speech recognition and facial expression analysis functions. Specifically, a home-use device equipped with a camera and microphone is required. This device acquires the user's speech and facial expressions in real time, and emotion recognition software in the cloud analyzes this data.

[0606] The server is equipped with an emotion recognition engine that identifies the user's emotional state based on data sent from the device and generates appropriate feedback. This feedback is dynamically generated by a generative AI model as language learning content suitable for the specific emotion. The device then delivers this feedback to the user via voice.

[0607] For example, if a user makes a sulky face while practicing English pronunciation, the calculator can generate feedback such as, "Calm down a bit and try again. Let's work together until you get it!" This allows the user to concentrate on learning with peace of mind.

[0608] An example of a prompt to input into a generative AI model is, "Generate a message that provides positive feedback when the user mispronounces something." This prompt serves as the foundation for learning support that is empathetic to the user's emotions, helping to improve the quality of learning.

[0609] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0610] Step 1:

[0611] The device acquires the user's voice and facial expressions in real time through its camera and microphone. Audio and video data are obtained as input. The device then prepares this data to send to the server.

[0612] Step 2:

[0613] The server inputs the audio and video data received from the terminal into the emotion recognition engine for analysis. The emotion recognition engine uses a machine learning algorithm to identify the user's emotional state. This process yields a classification result of the emotional state as output.

[0614] Step 3:

[0615] The server initiates the process of inputting a prompt message into the generative AI model based on the classification results of the emotional state. The prompt message used is, "Generate a message that provides positive feedback when the user mispronounces something." This process generates an appropriate message.

[0616] Step 4:

[0617] The server converts the generated feedback message into audio data and sends it to the terminal. This process uses a text-to-speech (TTS) engine to generate audio that is easy for the user to understand.

[0618] Step 5:

[0619] The device delivers audio feedback received from the server to the user via a playback device. By listening to this feedback, the user can receive support tailored to their learning process.

[0620] Step 6:

[0621] The user incorporates feedback from their device and continues learning. Performing new learning actions or responses restarts the process from step 1.

[0622] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0623] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0624] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0625] [Fourth Embodiment]

[0626] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0627] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0628] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0629] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0630] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0631] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0632] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0633] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0634] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0635] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0636] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0637] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0638] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] This invention provides a system for effectively supporting language learning through artificial intelligence technology. This system achieves a personalized learning experience tailored to each individual learner through the coordinated operation of a server, terminal, and user. The roles of each component are described below.

[0640] The server is built around an artificial intelligence engine that can use both the learner's native language and the second language they wish to learn. This engine enables two-way dialogue, analyzing voice input from the user and generating appropriate responses. The server also manages progress data from many learners and processes individual progress to dynamically generate the optimal learning plan.

[0641] The terminal functions as a user interface, receiving and displaying data sent from the server. In particular, it generates interactive dialogue scenes using a virtual reality environment, providing the user with a realistic learning experience. The terminal acquires the user's voice through a microphone and sends it to the server, while playing back the server's response with audio and subtitles.

[0642] The user initiates a conversation with the AI ​​tutor through this system. When the user asks a question in their native language, the device receives it, processes it on the server, and then returns a response in the target language to the user. To aid understanding, subtitles of the conversation are displayed on the device screen in real time. This method makes it easier for users to become familiar with a second language, even from the basics, and allows them to progress in their learning.

[0643] As a concrete example, let's say a user uses this system to learn English for travel. A virtual cafe scene unfolds on the device, and the user tells the AI ​​teacher in their native language, "I'd like to order a coffee." The server receives this phrase, generates the appropriate English expression, "I would like to order a coffee," and returns it to the device. The user listens to the English expression, and at the same time, the phrase is displayed as subtitles, helping them understand pronunciation and grammar. In this way, the user repeatedly practices what they have learned in a way that is relevant to real-life situations, and acquires the language in a natural manner.

[0644] The following describes the processing flow.

[0645] Step 1:

[0646] The user starts up their device and logs into the learning application. The user selects the language and level they want to learn on their device. The selection information is sent from the device to the server.

[0647] Step 2:

[0648] Based on the user's selection, the server retrieves data for the appropriate learning mode and virtual environment. The server then creates a pre-configured learning plan and sends it to the terminal.

[0649] Step 3:

[0650] The terminal sets up the virtual reality environment based on data received from the server. The user is notified on the terminal's screen or via audio when it is ready.

[0651] Step 4:

[0652] The user begins a conversation with the AI ​​teacher through the device. The user asks for simple phrases in the language they want to learn in their native language and inputs them into the device via the microphone.

[0653] Step 5:

[0654] The device sends the user's voice input to the server in real time. The server analyzes this data and generates translations and responses using an AI language model.

[0655] Step 6:

[0656] The server sends the generated translation and audio data to the terminal, providing the user with an appropriate response. For English phrases, the terminal will play the audio "I would like to order a coffee."

[0657] Step 7:

[0658] The device displays subtitles on the screen along with the response. These subtitles are updated in real time to aid user understanding.

[0659] Step 8:

[0660] The user continues practicing the dialogue by asking questions and confirming details. Feedback on differences in pronunciation and intonation is displayed on the device screen.

[0661] Step 9:

[0662] The server collects user progress and performance data and saves the information for the next session. Based on the analysis results, it adjusts the learning plan and provides advice.

[0663] By repeating the above steps, users will gradually improve their language skills.

[0664] (Example 1)

[0665] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0666] Traditional language learning systems often lack sufficient two-way communication and personalized learning, making it difficult for learners to acquire a language efficiently. Furthermore, the lack of real-time feedback and self-paced virtual environment dialogue training hindered the improvement of actual communication skills.

[0667] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0668] In this invention, the server includes means for supporting language learning using an intelligent control device capable of two-way dialogue using a first language and a second language, means for creating an individualized learning plan based on the user's native language and the language to be learned, and means for integrating speech recognition and translation functions to perform language conversion and response generation in real time. This enables learners to effectively acquire a language at their own pace through interactive, practical dialogue while receiving appropriate feedback in real time.

[0669] An "intelligent control device" is a device that uses artificial intelligence technology to perform advanced processing related to language conversion and dialogue generation.

[0670] "Supporting language learning" means providing users with various functions and tools to efficiently acquire their target language.

[0671] An "individualized learning plan" is a curriculum that includes learning procedures and content optimized according to each learner's progress and needs.

[0672] A "virtual reality environment" is a virtual space created using computer technology, through which users can engage in dialogue training in a manner close to reality.

[0673] "Dialogue practice" is an activity that simulates two-way language communication for the purpose of language acquisition.

[0674] "Learning progress data" refers to information that quantifies or records the user's progress and achievements in language learning.

[0675] "Instant reporting" means providing learners with real-time feedback.

[0676] "Displaying explanatory text" means showing textual information on the screen to help the user better understand the content of the conversation.

[0677] "Speech recognition" is the process of analyzing speech input through a microphone and converting it into digital text information.

[0678] A "translation function" is a technology that converts information expressed in one language into another language.

[0679] "Response generation" refers to creating a response in the appropriate language based on user input.

[0680] A "generative AI model" is an artificial intelligence technology or algorithm used in natural language processing to generate contextually appropriate text.

[0681] "Optimizing according to context" means adjusting the information provided to best suit the flow of the conversation and the user's needs.

[0682] This invention operates based on a language learning support system using an intelligent control device. The main components are a server, a terminal, and a user.

[0683] The server functions as a central hub for supporting language learning. Specifically, it uses speech recognition software (e.g., a common cloud-based speech analysis API) to convert audio data sent by the user into text. Next, it uses translation functionality to convert the user's native language to a second language (e.g., a general-purpose translation API), and leverages generative AI models (e.g., an open-source natural language generation engine) to generate appropriate responses for the user. Furthermore, by performing these processes in real time, it provides an advanced language learning experience.

[0684] The terminal communicates with the server as a user interface. Specifically, it sends audio data to the server and plays back the server's response in both audio and subtitle format. The terminal utilizes a virtual reality environment (e.g., a typical VR engine) to provide the user with immersive dialogue practice, allowing the user to experience practical scenarios.

[0685] Users utilize this system to advance their language learning. As an example of a prompt, a user might input "Tell me a phrase to order food at a restaurant" into the system. The device then sends the audio to the server, which generates an appropriate response such as "Can I order some food, please?", which the device then plays back to the user. The user can listen to the audio and deepen their understanding of the phrase through subtitles displayed on the screen.

[0686] As a concrete example, if a user wants to practice dialogue in a travel scenario, the device will create a virtual airport scene, simulating a situation where the user asks the question "Where is the boarding gate?" at the entrance. The server will translate this to "Where is the boarding gate?" and provide the user with this sentence as practice material. Through such realistic scenarios, the system provides an environment where users can acquire language naturally.

[0687] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0688] Step 1:

[0689] The user uses the device's microphone to input a question in their native language. The input audio is converted into digital data and received by the device. The device then appropriately encodes the audio data and prepares it for transmission to the server.

[0690] Step 2:

[0691] The terminal transmits digitized audio data to a server via the internet. Upon receiving this data, the server uses speech recognition software to convert it into text data. This conversion process involves natural language processing, which analyzes the audio signal and transforms it into corresponding text.

[0692] Step 3:

[0693] The server passes the text obtained through speech recognition to a translation engine, which converts it from the user's native language to a second language. For example, the Japanese phrase "I would like to order a coffee" is translated into English. The translated text is then used directly as input to the generative AI model.

[0694] Step 4:

[0695] The server feeds the translated second-language text into a generation AI model, which then generates a context-based, natural response. Here, the AI ​​complements the context based on the user's past usage history and known dialogue patterns to create an appropriate response. For example, it generates a dialogue model to approach the request, "I want to order coffee."

[0696] Step 5:

[0697] The generated response is translated back into the user's native language and sent to the speech synthesis process. The server applies speech synthesis technology to convert the text response into audio data and prepares to send the result to the terminal. The speech synthesis process generates synthesized speech from the text and outputs it as a digital audio file.

[0698] Step 6:

[0699] The server sends the audio file and response text together to the terminal. The terminal receives this, displays it as subtitles on the screen, and plays the audio. The user can learn how to use the language in real time by reviewing this output. The on-screen subtitles are displayed in real time to aid understanding and help the user follow the flow of the conversation.

[0700] (Application Example 1)

[0701] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0702] In language learning, especially for beginners, dialogue-based practice and application in concrete everyday situations are crucial. However, current online learning tools and materials struggle to replicate real-life interactions and make it difficult to receive personalized feedback in real time. The challenge lies in solving these problems and providing an effective and immersive language learning experience.

[0703] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0704] This invention includes a server that provides means for supporting education using machine learning capable of two-way dialogue using a first language and a second language, means for using a robotic device that supports language learning through dialogue in daily life in a home environment, and means for analyzing the user's questions and generating appropriate language expressions using speech recognition and natural language processing technologies. This enables the user to learn a language through natural dialogue in the home and receive immediate feedback, thereby effectively improving their language skills.

[0705] The terms "first language and second language" refer to the user's native language, which they already speak, and the target language they are trying to learn.

[0706] "Two-way dialogue" refers to a form of linguistic exchange in which the user and the system communicate with each other.

[0707] "Education using machine learning" is a method that utilizes machine learning technology to provide educational programs optimized for individual learners.

[0708] A "robot device that supports language learning through dialogue in daily life within a home environment" is a robot used in the home that supports the user's language learning through everyday conversation.

[0709] "Speech recognition technology" is a technology that receives voice input from a user, analyzes its content, and converts it into understandable digital data.

[0710] "Natural language processing technology" is a technology that enables computers to understand, generate, and analyze human language, making natural dialogue between users and systems possible.

[0711] "Generating appropriate language expressions" is the process of creating correct words and phrases in the target language based on user input.

[0712] The system that realizes this invention consists of three parties: a server, a terminal, and a user. The server supports education by centering on a machine learning engine that conducts two-way dialogue using the learner's native language and the target language. Specifically, it can analyze the user's voice input and generate appropriate responses by utilizing speech recognition technology and natural language processing technology. To this end, the server uses the Google Cloud Speech-to-Text API and OpenAI's generative AI model.

[0713] Users utilize a robotic device specifically designed for language learning within their everyday home environment. This device uses a built-in microphone to collect audio and transmit it to a server. The server's response is played back as audio through a speaker, while subtitles of the dialogue are displayed on the screen. This facilitates language comprehension through both audio and visual means.

[0714] On the device side, the accuracy of the user's pronunciation and vocabulary is evaluated in real time, and learning is adaptively adjusted. New learning content is dynamically generated and provided based on the user's past dialogue history and learning performance.

[0715] As a concrete example, a user can ask a robotic device how to say "crack an egg" in English while cooking. The server receives this and generates the phrase "Crack the egg," providing immediate feedback with both audio and subtitles. The user can then check the correct expression and practice their pronunciation.

[0716] Examples of prompts include, "Translate this sentence into English: I am studying Japanese," and "Tell me how to respond to the following dialogue in Spanish: 'Please give me some bread.'" These operations create an environment in which learners can naturally improve their language skills.

[0717] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0718] Step 1:

[0719] The user provides voice input to the robotic device within their home. The user's speech is captured by the robot's microphone. The input data is the user's voice signal.

[0720] Step 2:

[0721] The device uses speech recognition technology to convert the user's voice signal into digital text data. This process utilizes the Google Cloud Speech-to-Text API to analyze and transcribe the voice data. The output is the text data of what the user said.

[0722] Step 3:

[0723] The server receives the generated text data and performs the corresponding translation using natural language processing technology. In this process, OpenAI's generative AI model is used to convert the text data into the target language and generate a response with appropriate grammar and expression. The output is the translated text data.

[0724] Step 4:

[0725] The translated text data is sent back to the device and converted into audio data using speech synthesis technology. The user confirms the correct pronunciation of the target language through the audio played from the device's speaker. The audio data is then output.

[0726] Step 5:

[0727] Simultaneously, subtitles of the dialogue are displayed on the device's screen in real time. This allows users to visually confirm the learning content. The displayed subtitle data is the previously generated translated text data.

[0728] Step 6:

[0729] The server dynamically generates the next learning content using the user's past learning performance data. This data includes previous interaction history and current progress. Based on this, it adjusts the individual learning plan to provide the user with the optimal learning experience. The output is the newly generated learning plan data.

[0730] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0731] This invention provides a system that incorporates emotion recognition functionality to improve the language learning experience. This system identifies the user's emotions in real time and adaptively adjusts the learning plan and dialogue content to create a more personalized learning environment. The system's components and specific processing details are described below.

[0732] The server is equipped with an emotion engine that analyzes emotional data from the user's facial expressions and voice. The emotion engine uses machine learning algorithms to classify the user's emotional state and adjusts the dialogue content based on that information. The server also works in conjunction with existing AI language models to provide two-way language learning support.

[0733] The device is equipped with sensors that detect the user's biometric signals and collect data. When the user uses the device, it sends emotional data to the server in real time and returns feedback to the user based on that data. This enables natural conversations that are in line with the user's emotions.

[0734] The user initiates a conversation with the AI ​​tutor through a device equipped with emotion recognition capabilities. When the user asks a question or makes a statement, the device simultaneously captures not only the content of the statement but also the tone of voice and facial expressions. The server analyzes this data to determine the user's emotional state in real time.

[0735] As a concrete example, consider a scenario where a user is learning the pronunciation of a new word from an AI teacher, makes a mistake, and displays an expression of frustration. The device detects this expression, and the server uses its emotion engine to determine that the user is feeling "frustrated." Based on this, the server provides the device with a gentle, encouraging message and feedback prompting it to try again. It can also temporarily change the learning plan and switch to easier tasks if necessary.

[0736] Through this system, users can gain a learning experience that resonates with their emotions, and are expected to efficiently improve their language skills while maintaining their motivation.

[0737] The following describes the processing flow.

[0738] Step 1:

[0739] The user starts up the device and logs into the learning application. The user selects the language to learn, and the device sends this information to the server.

[0740] Step 2:

[0741] The server prepares a learning plan and emotion engine based on the user's selection and sends the necessary data to the device. The server performs the initial setup for the conversation and notifies the device when it is ready.

[0742] Step 3:

[0743] The device performs the initial setup of the virtual reality environment and starts the system so that the user can begin learning. The device is equipped with sensors to detect the user's facial expressions and voice.

[0744] Step 4:

[0745] The user begins a conversation with the AI ​​teacher. As the user speaks, the device captures facial expression data in real time along with the audio and sends it to the server.

[0746] Step 5:

[0747] The server analyzes the received audio data using an AI language model to generate appropriate responses to the user's statements. Simultaneously, the emotion engine analyzes facial expression data to identify the user's emotional state.

[0748] Step 6:

[0749] The server sends a response to the terminal along with a corresponding action based on the user's emotional state. For example, if the user shows signs of anxiety, the server will select a message that includes encouragement.

[0750] Step 7:

[0751] The device plays the response received from the server to the user as audio and displays subtitles on the screen. Emotion-based feedback is also displayed on the screen simultaneously, and learning suggestions are made according to the situation.

[0752] Step 8:

[0753] Users receive the feedback they receive and act upon it to incorporate it into their next interaction and learning plan. They deepen their understanding by attempting to engage in further dialogue on areas of concern.

[0754] Step 9:

[0755] The server accumulates user learning progress and sentiment data, and creates an optimized plan for the next learning session. Based on the analysis results, further fine-tuning of the learning process is performed.

[0756] Through these steps, users can improve their language skills in an optimal learning environment that is sensitive to their emotions.

[0757] (Example 2)

[0758] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0759] Traditional language learning systems have the drawback of only being able to respond to the user's superficial learning progress and lacking the flexibility to adapt to the user's emotional state. This can lead to decreased motivation and difficulty in efficient learning. Furthermore, individual learning plans are not sufficiently personalized, creating a need for a learning environment optimized for each user.

[0760] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0761] In this invention, the server includes means for analyzing emotional information from the user's facial expressions and voice and adaptively adjusting the learning plan and dialogue content based on those emotions; means for providing information generated based on the emotional analysis results and generating appropriate feedback using a generated information processing model; and means for dynamically constructing the next learning content based on the user's past dialogue records and learning efficiency. This makes it possible to provide a personalized learning experience that is in line with the user's emotional state.

[0762] "Language learning" refers to activities that users engage in to acquire a new language, and is a process of improving comprehension and expression through the use of sound, text, and other means.

[0763] An "information processing device" refers to a computer system that can take data as input and output necessary information through processing.

[0764] A "learning plan" is an educational design that sets specific content and timeframes to achieve the user's learning objectives.

[0765] A "virtual environment" is a computer simulation space created using digital technology, allowing users to have an experience similar to reality.

[0766] "Learning progress information" refers to data that shows how well a user understands the content and has achieved their goals during the learning process.

[0767] A "user interface" is a set of components that enable interaction between a system and a user, and includes screen displays and input methods.

[0768] "Emotional information" refers to data that indicates the emotional state of a user, obtained from their facial expressions and voice.

[0769] "Sentiment analysis" is the process of identifying and classifying a user's emotional state using machine learning algorithms and data processing.

[0770] A "generative information processing model" is a system that has an algorithm for generating a specific output based on input data.

[0771] "Feedback" refers to the responses, suggestions, and evaluations that a system provides to a user.

[0772] "Dialogue records" are data from past conversations and interactions, and they serve as the foundation for constructing new learning content.

[0773] "Learning efficiency" is an indicator that represents the speed and effort required for users to achieve their set learning goals.

[0774] This invention aims to improve the effectiveness of language learning by constructing an information processing system that recognizes user emotions in real time and provides corresponding feedback. The specific configuration and operation of the system are described below.

[0775] The server is equipped with an emotion engine and a generative information processing model. The emotion engine analyzes facial and voice data sent from the user's device and uses machine learning algorithms to identify the user's emotional information. For example, if a user shows frustration while learning a new word and practicing pronunciation, the emotion engine will classify this as "frustration." The server uses this emotional information to input into the generative AI model, which generates the most appropriate feedback for the user. This feedback is generated based on prompts and may include encouraging messages such as, "It's okay, let's try again."

[0776] Meanwhile, the device is equipped with sensors to acquire the user's biometric signals. When the user begins language learning, the device uses its camera and microphone to collect data such as facial expressions and voice tone, and transmits it to the server in real time. This allows for the immediate adjustment of a personalized learning plan based on the user's emotional information.

[0777] Through this system, users can learn while receiving personalized support tailored to their emotional state. For example, the system can dynamically construct the content of the next dialogue based on past conversation history. Through this approach, users are expected to efficiently improve their language skills.

[0778] An example of a prompt message is, "Generate the best feedback message for when the user is feeling frustrated." Based on such prompt messages, the server generates and provides a message that is sensitive to the user's emotional state.

[0779] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0780] Step 1:

[0781] The device captures the user's facial expressions and voice in real time. It uses sensors to capture the user's face with a camera and record their voice with a microphone. This data serves as input for analyzing the user's emotions.

[0782] Step 2:

[0783] The device sends the collected facial expression data and audio data to the server. The data sent includes image frames and audio files, which are the materials needed for analysis on the server.

[0784] Step 3:

[0785] The server uses an emotion engine to analyze the user's emotions. It applies machine learning algorithms to the received data to classify the emotional information. For example, it identifies emotional states such as "satisfied," "frustrated," and "questionable." This process yields the emotion analysis results.

[0786] Step 4:

[0787] The server generates feedback using a generative AI model based on the sentiment analysis results. The prompt "Generate the best feedback message for when the user is feeling frustrated." is input to the generative AI model, and a specific recommendation message is output.

[0788] Step 5:

[0789] The server sends the generated feedback message to the terminal. The message is formatted in a user-friendly format and prepared as data for display on the terminal.

[0790] Step 6:

[0791] The device displays feedback messages received from the server to the user. It provides immediate feedback to the user by displaying text messages on the screen and issuing audio alerts as needed. This process allows the user to receive helpful guidance that responds to changes in their emotions.

[0792] (Application Example 2)

[0793] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0794] When learning a language, instruction and feedback tailored to individual emotional states are limited, resulting in a problem where users cannot receive education that addresses their own feelings. Furthermore, a uniform approach risks diminishing learners' motivation and may not optimize learning efficiency. As a result, there is a challenge in providing flexible learning support that meets the individual needs of users.

[0795] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0796] In this invention, the server includes means for supporting language learning using a computer capable of two-way dialogue using a first language and a second language, means for recognizing emotions and adaptively adjusting the learning plan and dialogue content based on the emotional state, and means for a home device to provide emotionally responsive feedback in voice. This makes it possible to respond to the user's emotional state and provide an optimal learning experience.

[0797] A "computer" is an electronic device that automatically performs data processing and operations, and in particular refers to a device that supports language learning using artificial intelligence.

[0798] "Means of supporting language learning" refer to technological means that enable users to effectively acquire a language, such as providing interactive dialogue and learning plans.

[0799] An "individualized learning plan" refers to a customized learning program created based on the user's native language and the language they are learning.

[0800] A "virtual reality environment" is a simulation environment created using computer graphics and sensor technology to allow users to experience reality.

[0801] "Learning progress data" refers to data that represents the progress and results recorded by users during the process of language learning, and is analyzable information.

[0802] A "user interface" refers to a device or display screen that allows a user and a system to exchange information with each other.

[0803] "Recognizing emotions" refers to technology that analyzes biosignals such as a user's facial expressions and voice to determine their current emotional state.

[0804] "Adjusting learning plans and dialogue content based on emotional state" refers to the process of modifying the learning process or providing individually tailored feedback in response to recognized emotions.

[0805] "Household appliances" refer to robots and electronic devices used in the home, and are generally devices that perform various tasks to assist users.

[0806] "Providing feedback via voice" refers to the process of giving advice and evaluations via voice based on the learner's actions and responses.

[0807] To implement this invention, a computer is first configured as hardware equipped with speech recognition and facial expression analysis functions. Specifically, a home-use device equipped with a camera and microphone is required. This device acquires the user's speech and facial expressions in real time, and emotion recognition software in the cloud analyzes this data.

[0808] The server is equipped with an emotion recognition engine that identifies the user's emotional state based on data sent from the device and generates appropriate feedback. This feedback is dynamically generated by a generative AI model as language learning content suitable for the specific emotion. The device then delivers this feedback to the user via voice.

[0809] For example, if a user makes a sulky face while practicing English pronunciation, the calculator can generate feedback such as, "Calm down a bit and try again. Let's work together until you get it!" This allows the user to concentrate on learning with peace of mind.

[0810] An example of a prompt to input into a generative AI model is, "Generate a message that provides positive feedback when the user mispronounces something." This prompt serves as the foundation for learning support that is empathetic to the user's emotions, helping to improve the quality of learning.

[0811] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0812] Step 1:

[0813] The device acquires the user's voice and facial expressions in real time through its camera and microphone. Audio and video data are obtained as input. The device then prepares this data to send to the server.

[0814] Step 2:

[0815] The server inputs the audio and video data received from the terminal into the emotion recognition engine for analysis. The emotion recognition engine uses a machine learning algorithm to identify the user's emotional state. This process yields a classification result of the emotional state as output.

[0816] Step 3:

[0817] The server initiates the process of inputting a prompt message into the generative AI model based on the classification results of the emotional state. The prompt message used is, "Generate a message that provides positive feedback when the user mispronounces something." This process generates an appropriate message.

[0818] Step 4:

[0819] The server converts the generated feedback message into audio data and sends it to the terminal. This process uses a text-to-speech (TTS) engine to generate audio that is easy for the user to understand.

[0820] Step 5:

[0821] The device delivers audio feedback received from the server to the user via a playback device. By listening to this feedback, the user can receive support tailored to their learning process.

[0822] Step 6:

[0823] The user incorporates feedback from their device and continues learning. Performing new learning actions or responses restarts the process from step 1.

[0824] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0825] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0827] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0828] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0829] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0830] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0831] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0832] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0833] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0834] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0835] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0836] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0837] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0838] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0839] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0840] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0841] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0842] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0843] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0844] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0845] The following is further disclosed regarding the embodiments described above.

[0846] (Claim 1)

[0847] A means of supporting language learning using artificial intelligence capable of two-way dialogue using a first language and a second language,

[0848] A means for creating individual learning plans based on the user's native language and target language,

[0849] A means of providing users with dialogue training using a virtual reality environment,

[0850] A means of collecting and analyzing user learning progress data and providing real-time feedback,

[0851] A system that includes means of displaying subtitles through the user interface to make the content of the dialogue easier to understand.

[0852] (Claim 2)

[0853] The system according to claim 1, which evaluates the accuracy of the user's pronunciation and grammar and adaptively adjusts the learning process.

[0854] (Claim 3)

[0855] The system according to claim 1, which dynamically generates the next learning content based on the user's past dialogue history and learning performance.

[0856] "Example 1"

[0857] (Claim 1)

[0858] A means of supporting language learning using an intelligent control device capable of two-way dialogue using a first language and a second language,

[0859] A means for creating individual learning plans based on the user's native language and target language,

[0860] A means of providing users with dialogue practice using a virtual reality environment,

[0861] A means of collecting and analyzing user learning progress data and reporting it instantly,

[0862] A means of displaying explanatory text through the user interface to make it easier to understand the content of the dialogue,

[0863] A means of integrating speech recognition and translation functions to perform language conversion and response generation in real time,

[0864] A system that includes means for optimizing dialogue content in context using a generative AI model.

[0865] (Claim 2)

[0866] The system according to claim 1, which evaluates the accuracy of the user's pronunciation and grammar and adaptively adjusts the learning process.

[0867] (Claim 3)

[0868] The system according to claim 1, which dynamically generates the next learning content based on the user's past dialogue history and learning performance.

[0869] "Application Example 1"

[0870] (Claim 1)

[0871] A means of supporting education using machine learning that enables two-way dialogue using a first language and a second language,

[0872] A means for creating individual learning plans based on the user's native language and target language,

[0873] A means of providing users with simulated dialogue using a virtual reality environment,

[0874] A means of collecting and analyzing user learning progress information and providing real-time feedback,

[0875] A means of displaying subtitles through the user interface to make the dialogue easier to understand,

[0876] A means of using a robotic device that supports language learning through dialogue in daily life within a home environment,

[0877] A system that includes means for analyzing a user's question using speech recognition and natural language processing technologies and generating appropriate linguistic expressions.

[0878] (Claim 2)

[0879] The system according to claim 1, which evaluates the accuracy of the user's pronunciation and grammar and adaptively adjusts the learning process.

[0880] (Claim 3)

[0881] The system according to claim 1, which dynamically generates the next learning content based on the user's past dialogue history and learning performance.

[0882] "Example 2 of combining an emotion engine"

[0883] (Claim 1)

[0884] A means of supporting language learning using an information processing device capable of two-way dialogue using a first language and a second language,

[0885] A means for creating individual learning plans based on the user's native language and target language,

[0886] A means of providing users with dialogue training using a virtual environment,

[0887] A means for collecting and analyzing user learning progress information and providing real-time responses,

[0888] A means of displaying subtitles through the user interface to make the dialogue easier to understand,

[0889] A means of analyzing emotional information from the user's facial expressions and voice, and adaptively adjusting the learning plan and dialogue content based on those emotions,

[0890] A system that provides information generated based on emotion analysis results and includes means for generating appropriate feedback using a generated information processing model.

[0891] (Claim 2)

[0892] The system according to claim 1, which evaluates the accuracy of the user's pronunciation and grammar and adaptively adjusts the learning process.

[0893] (Claim 3)

[0894] The system according to claim 1, which dynamically constructs the next learning content based on the user's past dialogue records and learning efficiency.

[0895] "Application example 2 when combining with an emotional engine"

[0896] (Claim 1)

[0897] A means of supporting language learning using a computer capable of two-way dialogue using a first language and a second language,

[0898] A means for creating individual learning plans based on the user's native language and target language,

[0899] A means of providing users with dialogue training using a virtual reality environment,

[0900] A means of collecting and analyzing user learning progress data and returning information in real time,

[0901] A means of displaying subtitles through the user interface to make the dialogue easier to understand,

[0902] A means of recognizing emotions and adaptively adjusting learning plans and dialogue content based on emotional states,

[0903] A system that includes means by which a home appliance provides emotionally responsive feedback via voice.

[0904] (Claim 2)

[0905] The system according to claim 1, which evaluates the accuracy of the user's pronunciation and grammar and adaptively adjusts the learning process.

[0906] (Claim 3)

[0907] The system according to claim 1, which dynamically generates the next learning content based on the user's past dialogue history and learning performance. [Explanation of symbols]

[0908] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of supporting education using machine learning that enables two-way dialogue using a first language and a second language, A means for creating individual learning plans based on the user's native language and target language, A means of providing users with simulated dialogue using a virtual reality environment, A means of collecting and analyzing user learning progress information and providing real-time feedback, A means of displaying subtitles through the user interface to make the dialogue easier to understand, A means of using a robotic device that supports language learning through dialogue in daily life within a home environment, A system that includes means for analyzing a user's question using speech recognition and natural language processing technologies and generating appropriate linguistic expressions.

2. The system according to claim 1, which evaluates the accuracy of the user's pronunciation and grammar and adaptively adjusts the learning process.

3. The system according to claim 1, which dynamically generates the next learning content based on the user's past dialogue history and learning performance.

Images