system

Abstract

We provide the system. [Solution] A means of collecting user voice and text data and evaluating their abilities and progress, A means for generating an individually optimized learning plan based on the aforementioned evaluation, A means of providing the user with the aforementioned learning plan and providing real-time feedback according to their progress, A means of accessing diverse learning materials and selecting learning resources according to interests and skill levels, A means of analyzing the user's learning history, identifying weaknesses, and providing suggestions to encourage efficient improvement, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern busy living environments, it is difficult for language learners to effectively ensure high-quality learning time. In particular, it is difficult to find a customized learning method according to each individual's learning pace and interests. For this reason, many learners face problems such as setbacks in advancing their learning plans or being unable to effectively improve the necessary skills.

Means for Solving the Problems

[0005] This invention provides a system that collects user voice and text data and evaluates their abilities and progress in detail. Based on the obtained data, it generates an individually optimized learning plan and provides it to the user in real time. It also selects diverse learning materials according to the user's interests and skill level, identifies weaknesses, and provides suggestions to encourage efficient improvement. As a result, users can optimize their learning time, maintain sustained motivation, and have access to a learning environment available 34 hours a day, 365 days a year.

[0006] A "user" is an individual who uses this system to learn a language.

[0007] "Audio and text data" refers to information about spoken and written words that users input into the system.

[0008] "Ability and progress" refers to the user's current skill level and their progress in learning those skills.

[0009] A "personally optimized learning plan" is a combination of a learning schedule and materials that has been tailored based on the user's characteristics and goals.

[0010] "Real-time feedback" refers to suggestions for evaluation and correction that are provided immediately to the user during the learning process.

[0011] "Diverse learning materials" refers to various types of learning resources and content that are useful for language learning.

[0012] "Interest and skill level" refers to the depth of knowledge and current level of proficiency in areas of interest or specific fields that the user wishes to learn.

[0013] A "weakness" is an aspect of language skills that the user has not yet fully understood or mastered.

[0014] "Suggestions for improvement" refer to practical advice and practice methods designed to enhance the user's learning efficiency. [Brief explanation of the drawing]

[0015] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] Shows an emotion map to which multiple emotions are mapped. [Figure 10] Shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The AI ​​language learning system of the present invention aims to improve the language abilities of users. This system improves learning efficiency by allowing users to input voice and text data and providing personalized learning plans based on that data.

[0037] Specifically, the user inputs audio of the language they want to practice via a device. The device sends this audio data to a server, which analyzes the data using speech recognition technology. Based on the analyzed data, the user's pronunciation ability and comprehension level are evaluated.

[0038] Based on the evaluation results, the server creates a learning plan tailored to the user. This plan is automatically optimized, taking into account the user's past learning history, current skill level, goals, interests, and other factors. The generated plan is sent to the device and can be accessed by the user at any time.

[0039] Furthermore, the server has the capability to provide real-time feedback. Every time a user practices pronunciation or grammar, the server immediately analyzes it and displays the results through the device. This allows users to quickly identify and correct their weaknesses.

[0040] The system also provides access to a variety of learning materials. News articles, audio materials, literary works, and other materials are selected according to the user's interests and skill level. The selected materials are delivered to the device, allowing the user to progress with their learning on a daily basis.

[0041] Furthermore, the server periodically re-evaluates the user's learning data and provides practice suggestions as needed. For example, if a user has weaknesses in listening comprehension, the server will recommend specific audio materials. In this way, the system supports the user's progress and enables more efficient language learning.

[0042] This system allows users to experience personalized learning regardless of time or location, improving the speed and accuracy of language acquisition.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The user launches the application using their device and selects voice input mode. The user then speaks the words or phrases they want to practice into the microphone.

[0046] Step 2:

[0047] The device collects the user's voice data, converts it to digital data, and then sends it to the server. The voice data is transferred using a secure communication protocol.

[0048] Step 3:

[0049] The server uses a speech recognition API to analyze the received audio data. The audio data is converted into text data, and its pronunciation accuracy and intonation are evaluated.

[0050] Step 4:

[0051] The server evaluates the user's pronunciation ability based on the analysis results and generates real-time feedback. The feedback details specific errors and areas for improvement in the user's pronunciation.

[0052] Step 5:

[0053] The server automatically generates a personalized learning plan that takes into account the user's past learning history and current ability assessment. This plan includes recommended learning materials and goals to be achieved.

[0054] Step 6:

[0055] The server sends the generated learning plan to the device, allowing the user to begin learning according to the plan. The device then notifies the user that a new learning plan is available.

[0056] Step 7:

[0057] Users receive real-time feedback via their device and improve their learning by correcting the displayed errors. They repeatedly practice pronunciation while referring to the feedback.

[0058] Step 8:

[0059] The server periodically analyzes the user's learning activity to identify their weaknesses. It then suggests practice methods and learning materials for improvement as needed.

[0060] Step 9:

[0061] The server selects and delivers a variety of learning materials to the user's device based on their interests and skill level. Users can then use these materials to broaden their learning scope and enhance their abilities.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] Traditional language learning systems struggle to effectively optimize for individual user abilities and learning progress, and have limitations in real-time feedback and resource selection. Therefore, users find it difficult to find a learning method that suits them, and there is a need for improved learning efficiency.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan using a generative AI model, and means for performing analysis and providing feedback in real time according to the progress. This enables the user to establish a learning plan optimized for them and efficiently improve their language skills.

[0067] "User" refers to an individual who uses this system for language learning.

[0068] "Voice and text data" refers to spoken and written text provided by the user, which will be used as input data for the system.

[0069] Evaluating "ability" refers to the process of analyzing the current state of a user's language skills and identifying their strengths and weaknesses.

[0070] "Progress" is an indicator that shows the degree of improvement or achievement a user has made since starting language learning.

[0071] A "generative AI model" refers to a system that includes an algorithm for automatically generating an optimal learning plan for a user using artificial intelligence.

[0072] A "personally optimized learning plan" refers to a plan that provides learning methods and materials optimized based on each user's abilities and progress.

[0073] "Analysis and feedback" refers to the process of evaluating the user's pronunciation and grammar in real time and returning the results to the user.

[0074] "Educational resources" is a general term for various teaching materials and educational content intended for language learning.

[0075] A "communication protocol" is a technology that defines a set of rules and procedures used when sending and receiving data over a network.

[0076] A "display device" refers to a monitor or screen used to provide users with visual information, such as real-time feedback.

[0077] This AI language learning system aims to improve the user's language abilities. Specific embodiments of the present invention are described below.

[0078] The user uses their device to record audio data of the language they want to learn. Using the audio recording application on the device, the user generates the audio as digital data and sends this data to the server. The HTTP protocol is generally used for transmission.

[0079] The server uses speech recognition software to analyze the received audio data. For example, it uses widely available speech recognition services on the internet to convert speech to text. Based on the recognized text data, the server evaluates the user's pronunciation and comprehension. The evaluation results are further processed by a generative AI model, and a personalized learning plan is automatically generated.

[0080] The generated learning plan is optimized based on the user's characteristics and past learning history, and is provided to the user via the device. The user can then proceed with their daily learning based on this plan. Each time the user practices, the device receives real-time feedback from the server and presents it to the user visually.

[0081] Furthermore, the server periodically re-evaluates the user's learning data and generates new learning suggestions. For example, if the user has difficulty with listening skills, appropriate audio materials will be recommended. In this way, the system efficiently supports the user's learning.

[0082] Examples of specific cases and prompt statements

[0083] As a concrete example, consider a scenario where a user is studying Spanish. The user uses their device to record the phrase "Hola, ¿cómo estás?" (Hello, how are you?). The server performs speech recognition to determine if the pronunciation is correct. Based on the results, the server generates an appropriate practice plan and provides specific feedback for improving pronunciation.

[0084] As an example of a prompt, the following instructions can be entered into the generative AI model:

[0085] "Please recommend listening and pronunciation materials focused on beginner levels for users learning English."

[0086] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0087] Step 1:

[0088] The user records the audio of the language they want to learn using their device. This recording is done using an audio recording application on the device, and the audio data is output in digital format.

[0089] Step 2:

[0090] The terminal sends digital audio data to the server. HTTP is used as the communication protocol. The output digital audio data is then input to the server in a format that it can process.

[0091] Step 3:

[0092] The server passes the received audio data to the speech recognition engine. This engine converts the audio into text. The speech recognition engine outputs the recognized text data as the result of the analysis.

[0093] Step 4:

[0094] The server analyzes the text data obtained through speech recognition and evaluates the user's pronunciation and comprehension. Here, data calculations are performed to measure the accuracy of the user's pronunciation. The evaluation results are then output.

[0095] Step 5:

[0096] Based on the evaluation results and past learning history, the server generates an individually optimized learning plan using a generated AI model. The generated learning plan is output as data in JSON format.

[0097] Step 6:

[0098] The server sends the generated learning plan to the terminal. The terminal visually displays the received learning plan to the user, making it accessible to the user. The optimized learning content is output through the display device.

[0099] Step 7:

[0100] When a user practices pronunciation or grammar, a feedback request is sent to the server in real time via the device. The server immediately analyzes the data and generates feedback data. The feedback is output and sent back to the device.

[0101] Step 8:

[0102] The device receives feedback from the server and presents it visually to the user. The feedback includes specific instructions to help the user improve their pronunciation and comprehension.

[0103] Step 9:

[0104] The server periodically re-evaluates the user's learning data and generates new practice suggestions as needed. Based on this information, new learning material suggestions are output.

[0105] (Application Example 1)

[0106] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0107] Many language learning systems struggle to provide a personalized learning experience because they cannot offer real-time conversation practice tailored to the user's current location or places they visit. Furthermore, their lack of ability to provide immediate pronunciation evaluation and geographical-based feedback makes it difficult for learners to improve their language skills when needed.

[0108] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0109] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan, and means for providing conversational phrases and information relevant to the user based on geographical information. This enables realistic language learning tailored to the places the user visits, providing a more flexible and effective learning experience.

[0110] "User" refers to an individual or group that uses a language learning system.

[0111] "Audio and text data" refers to audio and text information entered or provided by the user, and is the data that forms the basis for evaluation and feedback.

[0112] "Means of evaluating ability and progress" refers to a system that analyzes a user's language skills and learning status and provides objective numerical data and results.

[0113] A "personally optimized learning plan" is the process of creating a learning program that is best suited to the individual, based on the user's evaluation results and learning history.

[0114] "Learning resources" is a general term for information sources that users utilize for learning, including news articles, audio materials, and literary works.

[0115] "Geographic-based conversational phrases" are phrases used to provide conversational content relevant to the user's current location or destination.

[0116] "Pronunciation scoring and evaluation" is a process that analyzes a user's pronunciation and provides a numerical representation of their accuracy and fluency.

[0117] A "means of providing real-time feedback" refers to a system that quickly returns evaluation results and suggestions for improvement in response to user input.

[0118] A description of embodiments for carrying out this invention will be given.

[0119] The server receives data when the user inputs voice or text data. For voice data, the server uses speech recognition software (e.g., Google® Cloud Speech-to-Text or IBM Watson®) to convert the input speech into text. Using this converted text data, a machine learning model (e.g., Tensorflow® or PyTorch) evaluates the user's language ability. Based on the evaluation, the server generates an individually optimized learning plan.

[0120] The device displays a learning plan received from the server to the user. The learning plan includes conversational phrases and information based on geographical location, providing content relevant to the user's current location. The device also has a real-time feedback function. This feedback, based on information obtained from pronunciation scoring and evaluation, immediately informs the user of areas for improvement.

[0121] Users can progressively improve their language skills by following the provided learning plan. For example, users can practice conversational phrases needed when visiting a city hall. If a user wants to simulate a situation such as "requesting the issuance of a resident registration certificate," the device will present appropriate phrases and prompt the user to practice pronunciation.

[0122] An example of a prompt sentence for a generative AI model is: "Please create sentences that provide everyday conversational phrases for tourists at a local supermarket. Also, please provide information on recommended phrases for pronunciation practice."

[0123] Thus, the system in this invention provides a learning experience tailored to the user and improves language ability.

[0124] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0125] Step 1:

[0126] The user inputs audio data of the language to be learned into the device. This input audio data is sent by the device to the server. The server then receives the audio data to begin the speech recognition process.

[0127] Step 2:

[0128] The server converts the received audio data into text data using speech recognition software such as Google Cloud Speech-to-Text or IBM Watson. This process provides the user's pronunciation information in text format.

[0129] Step 3:

[0130] The server inputs the converted text data into machine learning models such as TensorFlow or PyTorch to evaluate the user's language ability. This evaluation process analyzes the text data and calculates metrics such as pronunciation scores and grammatical comprehension.

[0131] Step 4:

[0132] The server combines evaluation results, the user's learning history, and geographical information to generate a personalized learning plan. This learning plan includes adapted conversational phrases and relevant information.

[0133] Step 5:

[0134] The device displays a learning plan sent from the server to the user. Based on the information presented, the user engages in geographically relevant conversation practice. Specifically, a screen is displayed for practicing phrases tailored to the destination.

[0135] Step 6:

[0136] When the user inputs voice again and practices the conversation, the device sends the audio to the server. The server evaluates it and generates immediate feedback.

[0137] Step 7:

[0138] The terminal displays feedback from the server to the user in real time. Based on this feedback, the user can correct their mistakes and improve their language skills.

[0139] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0140] The system of this invention provides a more effective and adaptive learning experience by combining an emotion engine with the user's language learning. The emotion engine has the function of analyzing the user's voice and facial expression data and evaluating the current emotional state in real time. This makes it possible to provide feedback and customize learning content based on the user's emotions.

[0141] When a user inputs voice data using a device, the camera and microphone simultaneously collect emotional data. The device sends this data to a server, which uses an emotion engine to analyze the user's emotional state. The analyzed emotional data reflects the user's state of excitement, frustration, concentration, and other emotions they experience while learning.

[0142] The server takes the user's emotional state into consideration and creates an individually optimized learning plan. This plan includes dynamic adjustments, such as presenting more challenging materials when the user is highly motivated, or sending encouraging messages when motivation is low.

[0143] Furthermore, the server provides users with visual or auditory feedback from the emotion engine. For example, if a user is feeling tired, the system will suggest a break or provide relaxing content to help them continue learning. These settings allow users to experience a learning environment that takes their emotions into account, leading to more fulfilling language learning.

[0144] For example, when a user is practicing speaking online, the system captures their facial expressions in conjunction with their voice. If the emotion engine detects that the user is nervous, the server displays a video on the user's device guiding them on how to relax. In this way, the emotion engine-based approach provides users with an environment where they can focus on improving their skills while receiving emotional support.

[0145] The following describes the processing flow.

[0146] Step 1:

[0147] The user launches a learning application on their device and selects voice input mode. The user then pronounces a specific phrase as practice. The device simultaneously collects audio and video data using its microphone and camera.

[0148] Step 2:

[0149] The device sends the collected audio and video data to the server. This transmission uses an encrypted communication protocol, ensuring the data is delivered securely to the server.

[0150] Step 3:

[0151] The server processes the received audio data using speech recognition technology to evaluate pronunciation and intonation. Simultaneously, an emotion engine analyzes video data to identify the user's emotional state from their facial expressions.

[0152] Step 4:

[0153] The server comprehensively evaluates the user's language ability and emotional state based on the analysis results. If tension or stress is detected, appropriate countermeasures are considered based on the results of the emotion engine.

[0154] Step 5:

[0155] The server considers the user's current state and past learning history to generate an individually optimized learning plan. This plan includes content adjustments and recommendations for learning materials that take motivation into account.

[0156] Step 6:

[0157] The server generates a learning plan and feedback, which is then sent to the device and presented to the user. The device displays the feedback visually and audibly to the user in an easy-to-understand manner.

[0158] Step 7:

[0159] The system uses user feedback provided via their device to correct mistakes and start new exercises. Users adjust the pace and content as needed based on the feedback.

[0160] Step 8:

[0161] The server continuously monitors the user's emotions and learning progress, providing additional resources and encouraging messages as needed. This approach ensures that users are always supported and maintain an optimal learning experience.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] Conventional language learning systems have been insufficient in providing effective feedback and optimizing learning plans that take into account the user's emotional state, resulting in challenges in improving learning efficiency and motivation. Furthermore, they have struggled to flexibly respond to the individual learning needs and emotional states of users.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes means for collecting the user's voice, visual, and text information and evaluating their abilities and progress; means for generating an individually optimized learning plan based on the evaluation and the user's emotional state; and means for providing the learning plan to the user and providing real-time adaptive feedback according to their progress and emotional state. This makes it possible to provide an effective and individually optimized learning environment that takes the user's emotional state into account.

[0167] "Audio, visual, and text information" is a general term for data that includes the characteristics of the user's voice, facial expressions, and written or spoken text.

[0168] "Evaluation" is the act of determining a user's current learning ability, progress, and emotional state by analyzing collected data.

[0169] A "personally optimized learning plan" refers to learning methods and content that are customized based on the user's specific needs, ability level, and emotional state.

[0170] "Adaptive feedback" refers to responses or advice that are modified or adjusted according to the user's real-time learning progress and emotional state.

[0171] "Learning resources" refers to educational materials, tools, and other sources of information provided to users for language learning.

[0172] "Communication procedures" refer to a set of protocols and rules used to send and receive data between a user and a server.

[0173] "Display device" refers to a device or apparatus used to transmit visual or auditory information to a user.

[0174] The present invention provides an advanced method that combines an emotion engine to enrich the user's language learning experience. Specifically, it realizes an individually optimized learning environment through interaction between the server, terminal, and user.

[0175] Users begin learning using their devices, during which the device's built-in voice input device and camera collect the user's voice and visual information. This data is automatically sent to a server for advanced analysis. The server uses an emotion engine, voice analysis, and facial expression analysis technology to evaluate the user's emotional state. This evaluation result is analyzed along with the user's learning progress, ability level, and motivation.

[0176] Based on the analyzed data, the server presents a personalized learning plan to the user. The difficulty level and content are selected to best suit the user's current emotional state. For example, if the user is feeling stressed, they may be presented with simple tasks or relaxing videos to alleviate that stress. Furthermore, the server provides real-time feedback and dynamically adjusts the learning plan according to the user's emotions.

[0177] For example, if the emotion engine detects tension or fatigue from the audio and facial expressions captured by the device while the user is practicing speaking, the server will present a video on relaxation techniques or display a message of encouragement. In this way, users can efficiently improve their skills while receiving emotional support.

[0178] An example of a prompt to a generative AI model is, "Suggest content to provide when the user's emotional state is anxious." This prompt allows the AI ​​model to generate and provide the user with the most suitable relaxation content or tasks.

[0179] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0180] Step 1:

[0181] When a user begins learning on the device, it uses its built-in microphone and camera to collect the user's voice and facial expression data. This input data includes the user's voice tone, speaking pace, and facial muscle movements. The device transmits this data to a server in real time, providing foundational data for the next analysis step.

[0182] Step 2:

[0183] The server analyzes the received audio and facial expression data. This process uses a generative AI model to detect emotions from the audio and facial recognition technology to evaluate the emotional state from the facial expressions. This data processing yields an output of the emotional state the user is currently experiencing. Specifically, when a user is nervous, their voice tone may be higher and their face may appear tense.

[0184] Step 3:

[0185] The server generates an optimal learning plan for the user based on the analyzed emotional state. The generated plan includes learning materials and simple tasks to help the user relax if they are feeling stressed. This plan is automatically adjusted by the generating AI model, taking into account the user's current ability level and learning goals. The output includes a list of specific tasks and recommended learning materials.

[0186] Step 4:

[0187] The server provides the user with the generated learning plan as feedback, presenting it visually or audibly via the device. Based on the feedback received, the user performs the next learning step. At this stage, for example, relaxing music may be played, or simple questions may be displayed on the screen. The user's responses are collected again as new input data and sent back to the server.

[0188] Step 5:

[0189] The server monitors user feedback and adjusts the learning plan in real time as needed. If it determines that the user's progress has improved, it increases the difficulty level of the learning materials or introduces different types of challenges. This cycle is repeated, ensuring that users always receive an optimized learning experience.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0192] Traditional language learning systems often provide a uniform learning plan without considering the user's emotional state, which can reduce learning efficiency. In particular, despite the significant impact emotions can have on learning, there is a lack of means to customize the learning experience based on these factors. Furthermore, there is a problem in providing appropriate feedback when motivation is low or users are experiencing stress.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes means for collecting the user's voice and visual data and analyzing their emotional state; means for dynamically adjusting an individually optimized learning plan based on the emotional state; and means for providing the adjusted learning plan to the user and providing real-time feedback according to the emotional state. This makes it possible to provide an appropriate learning experience that is in line with the user's emotions and to improve learning effectiveness.

[0195] "Voice data" refers to information about the user's voice that is analyzed using speech recognition technology.

[0196] "Visual data" refers to information about images or videos acquired to capture the user's facial expressions and movements.

[0197] "Emotional state" refers to the results of a real-time assessment of the user's psychological or emotional state.

[0198] A "personally optimized learning plan" refers to learning content and teaching methods that are dynamically adjusted based on the user's abilities and emotional state.

[0199] "Feedback" refers to evaluations or advice provided to the user during learning, and is delivered in real time in response to their emotional state.

[0200] "Educational resources" refer to educational materials and learning content that are accessible to support the user's learning.

[0201] "Learning history" refers to information that includes records of the user's learning activities to date, as well as their level of achievement and challenges based on those activities.

[0202] "Emotional history" refers to a record of a user's past emotional states and is data used to adjust learning.

[0203] A "wireless communication protocol" is a set of standards and procedures that define the communication methods used to transfer data wirelessly.

[0204] A "human-machine interface" is a device or means that enables the exchange of information between a user and a system.

[0205] This invention implements a language learning system equipped with an emotion engine using a consumer robot. The system mainly includes hardware for processing audio and visual data, and software for performing emotion analysis.

[0206] Specifically, the robot is equipped with a high-quality microphone and camera to acquire audio and visual data from the user. The audio data is converted into text data using a speech recognition system such as the Google Speech-to-Text API. The visual data is analyzed in real time using facial recognition systems such as OpenCV and Microsoft® Face API.

[0207] The server processes this data using an emotion engine (e.g., Python's EmotionRecognition library) to evaluate the user's emotional state. Based on the evaluation, it generates an individually optimized learning plan and dynamically adjusts it. The learning plan and feedback are provided to the user in an appropriate format, and the server presents this to the user via a human-machine interface.

[0208] For example, if the emotion engine detects that a user is stressed while learning a language, the system will play a relaxing video on the robot's display. This allows the user to continue learning in a more relaxed state.

[0209] When using generative AI models to generate content and adjust learning, prompts such as "Generate feedback that suggests relaxation when the user is tired" are used. This prompt supports the generation of appropriate feedback based on the user's emotional state.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The device acquires audio and visual data from the user in real time. Input consists of the user's speech and facial expressions, and data is collected using a microphone and camera. The collected data is sent to a server, where the audio data is prepared for conversion into text data.

[0213] Step 2:

[0214] The server converts the input audio data into text data using a speech recognition system. It utilizes technologies such as the Google Speech-to-Text API to convert speech into text strings and generate the text data necessary for subsequent processing. The output is text data representing what the user said.

[0215] Step 3:

[0216] The server analyzes the user's emotional state using visual data. Using OpenCV and the Microsoft Face API, it reads the user's facial expressions from the acquired visual data and inputs them into the emotion analysis engine. The output is data indicating the analyzed emotional state of the user.

[0217] Step 4:

[0218] The server uses an emotion analysis engine to evaluate the user's emotional state and generate an individually optimized learning plan. For example, it uses the Python EmotionRecognition library to process emotional data in real time and adjust the learning plan based on the user's psychological state. The output is the adjusted learning plan.

[0219] Step 5:

[0220] The server provides the generated learning plan to the user via the terminal. The user's display shows the tailored learning materials and information, and provides voice-based feedback as needed. The input is the learning plan generated in step 4, and the output is the learning content presented to the user.

[0221] Step 6:

[0222] Users can progress through their learning based on the presented learning content. Based on the results, feedback is provided in real time, and prompts are generated using a generative AI model as needed. For example, a prompt such as "Generate feedback suggesting relaxation if the user is tired" might be used. The output is appropriate feedback tailored to the user's learning progress.

[0223] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0224] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0225] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0226] [Second Embodiment]

[0227] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0228] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0229] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0230] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0231] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0232] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0233] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0234] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0235] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0236] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0237] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0238] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0239] The AI ​​language learning system of the present invention aims to improve the language abilities of users. This system improves learning efficiency by allowing users to input voice and text data and providing personalized learning plans based on that data.

[0240] Specifically, the user inputs audio of the language they want to practice via a device. The device sends this audio data to a server, which analyzes the data using speech recognition technology. Based on the analyzed data, the user's pronunciation ability and comprehension level are evaluated.

[0241] Based on the evaluation results, the server creates a learning plan tailored to the user. This plan is automatically optimized, taking into account the user's past learning history, current skill level, goals, interests, and other factors. The generated plan is sent to the device and can be accessed by the user at any time.

[0242] Furthermore, the server has the capability to provide real-time feedback. Every time a user practices pronunciation or grammar, the server immediately analyzes it and displays the results through the device. This allows users to quickly identify and correct their weaknesses.

[0243] The system also provides access to a variety of learning materials. News articles, audio materials, literary works, and other materials are selected according to the user's interests and skill level. The selected materials are delivered to the device, allowing the user to progress with their learning on a daily basis.

[0244] Furthermore, the server periodically re-evaluates the user's learning data and provides practice suggestions as needed. For example, if a user has weaknesses in listening comprehension, the server will recommend specific audio materials. In this way, the system supports the user's progress and enables more efficient language learning.

[0245] This system allows users to experience personalized learning regardless of time or location, improving the speed and accuracy of language acquisition.

[0246] The following describes the processing flow.

[0247] Step 1:

[0248] The user launches the application using their device and selects voice input mode. The user then speaks the words or phrases they want to practice into the microphone.

[0249] Step 2:

[0250] The device collects the user's voice data, converts it to digital data, and then sends it to the server. The voice data is transferred using a secure communication protocol.

[0251] Step 3:

[0252] The server uses a speech recognition API to analyze the received audio data. The audio data is converted into text data, and its pronunciation accuracy and intonation are evaluated.

[0253] Step 4:

[0254] The server evaluates the user's pronunciation ability based on the analysis results and generates real-time feedback. The feedback details specific errors and areas for improvement in the user's pronunciation.

[0255] Step 5:

[0256] The server automatically generates a personalized learning plan that takes into account the user's past learning history and current ability assessment. This plan includes recommended learning materials and goals to be achieved.

[0257] Step 6:

[0258] The server sends the generated learning plan to the device, allowing the user to begin learning according to the plan. The device then notifies the user that a new learning plan is available.

[0259] Step 7:

[0260] Users receive real-time feedback via their device and improve their learning by correcting the displayed errors. They repeatedly practice pronunciation while referring to the feedback.

[0261] Step 8:

[0262] The server periodically analyzes the user's learning activity to identify their weaknesses. It then suggests practice methods and learning materials for improvement as needed.

[0263] Step 9:

[0264] The server selects and delivers a variety of learning materials to the user's device based on their interests and skill level. Users can then use these materials to broaden their learning scope and enhance their abilities.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] Traditional language learning systems struggle to effectively optimize for individual user abilities and learning progress, and have limitations in real-time feedback and resource selection. Therefore, users find it difficult to find a learning method that suits them, and there is a need for improved learning efficiency.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan using a generative AI model, and means for performing analysis and providing feedback in real time according to the progress. This enables the user to establish a learning plan optimized for them and efficiently improve their language skills.

[0270] "User" refers to an individual who uses this system for language learning.

[0271] "Voice and text data" refers to spoken and written text provided by the user, which will be used as input data for the system.

[0272] Evaluating "ability" refers to the process of analyzing the current state of a user's language skills and identifying their strengths and weaknesses.

[0273] "Progress" is an indicator that shows the degree of improvement or achievement a user has made since starting language learning.

[0274] A "generative AI model" refers to a system that includes an algorithm for automatically generating an optimal learning plan for a user using artificial intelligence.

[0275] A "personally optimized learning plan" refers to a plan that provides learning methods and materials optimized based on each user's abilities and progress.

[0276] "Analysis and feedback" refers to the process of evaluating the user's pronunciation and grammar in real time and returning the results to the user.

[0277] "Educational resources" is a general term for various teaching materials and educational content intended for language learning.

[0278] A "communication protocol" is a technology that defines a set of rules and procedures used when sending and receiving data over a network.

[0279] A "display device" refers to a monitor or screen used to provide users with visual information, such as real-time feedback.

[0280] This AI language learning system aims to improve the user's language abilities. Specific embodiments of the present invention are described below.

[0281] The user uses their device to record audio data of the language they want to learn. Using the audio recording application on the device, the user generates the audio as digital data and sends this data to the server. The HTTP protocol is generally used for transmission.

[0282] The server uses speech recognition software to analyze the received voice data. For example, to convert voice to text, it utilizes a widely used speech recognition service on the Internet. Based on the recognized text data, the server evaluates the user's pronunciation and comprehension. The evaluation results are further processed by the generative AI model to automatically generate an individually optimized learning plan.

[0283] The generated learning plan is optimized based on the user's characteristics and past learning history and is provided to the user via the terminal. The user can proceed with daily learning based on this learning plan. Each time the user practices, the terminal receives real-time feedback from the server and visually presents it to the user.

[0284] Furthermore, the server periodically re-evaluates the user's learning data and generates new learning proposals. For example, if there are issues with listening skills, appropriate audio teaching materials are recommended. In this way, the system efficiently supports the user's learning.

[0285] Examples of specific cases and prompt sentences

[0286] As a specific example, consider the situation where a user is studying Spanish. The user uses the terminal to record the phrase "Hola, ¿cómo estás? (Hello, how are you?)" The server performs speech recognition and determines whether the pronunciation is correct. Based on the result, the server generates an appropriate practice plan and provides specific feedback for pronunciation improvement.

[0287] As an example of a prompt sentence, the following instructions can be input into the generative AI model:

[0288] "Please recommend listening and pronunciation teaching materials focused on the beginner level for the user to learn English."

[0289] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0290] Step 1:

[0291] The user records the audio of the language they want to learn using their device. This recording is done using an audio recording application on the device, and the audio data is output in digital format.

[0292] Step 2:

[0293] The terminal sends digital audio data to the server. HTTP is used as the communication protocol. The output digital audio data is then input to the server in a format that it can process.

[0294] Step 3:

[0295] The server passes the received audio data to the speech recognition engine. This engine converts the audio into text. The speech recognition engine outputs the recognized text data as the result of the analysis.

[0296] Step 4:

[0297] The server analyzes the text data obtained through speech recognition and evaluates the user's pronunciation and comprehension. Here, data calculations are performed to measure the accuracy of the user's pronunciation. The evaluation results are then output.

[0298] Step 5:

[0299] Based on the evaluation results and past learning history, the server generates an individually optimized learning plan using a generated AI model. The generated learning plan is output as data in JSON format.

[0300] Step 6:

[0301] The server transmits the generated learning plan to the terminal. The terminal visually displays the received learning plan to the user to make it accessible to the user. Through the display device, the optimized learning content is output.

[0302] Step 7:

[0303] When the user practices pronunciation and grammar, a feedback request is sent to the server in real time via the terminal. The server immediately analyzes the data and generates feedback data. The feedback is output and sent to the terminal.

[0304] Step 8:

[0305] The terminal receives the feedback from the server and visually presents it to the user. The feedback contains specific guidance content for the user to improve pronunciation and comprehension.

[0306] Step 9:

[0307] The server periodically re-evaluates the user's learning data and generates new practice proposals if necessary. Based on this information, proposals for new learning materials are output.

[0308] (Application Example 1)

[0309] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0310] Many language learning systems have the problem that they cannot provide real-time conversation practice according to the user's current location or the place they visit, and it is difficult to provide an individually optimized learning experience. Also, due to the lack of the ability to immediately provide feedback based on pronunciation evaluation and geographical information, it is difficult for learners to improve appropriate language skills in the necessary situations.

[0311] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0312] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan, and means for providing conversational phrases and information relevant to the user based on geographical information. This enables realistic language learning tailored to the places the user visits, providing a more flexible and effective learning experience.

[0313] "User" refers to an individual or group that uses a language learning system.

[0314] "Audio and text data" refers to audio and text information entered or provided by the user, and is the data that forms the basis for evaluation and feedback.

[0315] "Means of evaluating ability and progress" refers to a system that analyzes a user's language skills and learning status and provides objective numerical data and results.

[0316] A "personally optimized learning plan" is the process of creating a learning program that is best suited to the individual, based on the user's evaluation results and learning history.

[0317] "Learning resources" is a general term for information sources that users utilize for learning, including news articles, audio materials, and literary works.

[0318] "Geographic-based conversational phrases" are phrases used to provide conversational content relevant to the user's current location or destination.

[0319] "Pronunciation scoring and evaluation" is a process that analyzes a user's pronunciation and provides a numerical representation of their accuracy and fluency.

[0320] A "means of providing real-time feedback" refers to a system that quickly returns evaluation results and suggestions for improvement in response to user input.

[0321] A description of embodiments for carrying out this invention will be given.

[0322] The server's role is to receive data when the user inputs voice or text data. For voice data, the server utilizes speech recognition software (e.g., Google Cloud Speech-to-Text or IBM Watson) to convert the input speech into text. Using this converted text data, a machine learning model (such as TensorFlow or PyTorch) evaluates the user's language ability. Based on the evaluation, the server generates an individually optimized learning plan.

[0323] The device displays a learning plan received from the server to the user. The learning plan includes conversational phrases and information based on geographical location, providing content relevant to the user's current location. The device also has a real-time feedback function. This feedback, based on information obtained from pronunciation scoring and evaluation, immediately informs the user of areas for improvement.

[0324] Users can progressively improve their language skills by following the provided learning plan. For example, users can practice conversational phrases needed when visiting a city hall. If a user wants to simulate a situation such as "requesting the issuance of a resident registration certificate," the device will present appropriate phrases and prompt the user to practice pronunciation.

[0325] An example of a prompt sentence for a generative AI model is: "Please create sentences that provide everyday conversational phrases for tourists at a local supermarket. Also, please provide information on recommended phrases for pronunciation practice."

[0326] Thus, the system in this invention provides a learning experience tailored to the user and improves language ability.

[0327] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0328] Step 1:

[0329] The user inputs audio data of the language to be learned into the device. This input audio data is sent by the device to the server. The server then receives the audio data to begin the speech recognition process.

[0330] Step 2:

[0331] The server converts the received audio data into text data using speech recognition software such as Google Cloud Speech-to-Text or IBM Watson. This process provides the user's pronunciation information in text format.

[0332] Step 3:

[0333] The server inputs the converted text data into machine learning models such as TensorFlow or PyTorch to evaluate the user's language ability. This evaluation process analyzes the text data and calculates metrics such as pronunciation scores and grammatical comprehension.

[0334] Step 4:

[0335] The server combines evaluation results, the user's learning history, and geographical information to generate a personalized learning plan. This learning plan includes adapted conversational phrases and relevant information.

[0336] Step 5:

[0337] The device displays a learning plan sent from the server to the user. Based on the information presented, the user engages in geographically relevant conversation practice. Specifically, a screen is displayed for practicing phrases tailored to the destination.

[0338] Step 6:

[0339] When the user inputs voice again and practices the conversation, the device sends the audio to the server. The server evaluates it and generates immediate feedback.

[0340] Step 7:

[0341] The terminal displays feedback from the server to the user in real time. Based on this feedback, the user can correct their mistakes and improve their language skills.

[0342] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0343] The system of this invention provides a more effective and adaptive learning experience by combining an emotion engine with the user's language learning. The emotion engine has the function of analyzing the user's voice and facial expression data and evaluating the current emotional state in real time. This makes it possible to provide feedback and customize learning content based on the user's emotions.

[0344] When a user inputs voice data using a device, the camera and microphone simultaneously collect emotional data. The device sends this data to a server, which uses an emotion engine to analyze the user's emotional state. The analyzed emotional data reflects the user's state of excitement, frustration, concentration, and other emotions they experience while learning.

[0345] The server takes the user's emotional state into consideration and creates an individually optimized learning plan. This plan includes dynamic adjustments, such as presenting more challenging materials when the user is highly motivated, or sending encouraging messages when motivation is low.

[0346] Furthermore, the server provides users with visual or auditory feedback from the emotion engine. For example, if a user is feeling tired, the system will suggest a break or provide relaxing content to help them continue learning. These settings allow users to experience a learning environment that takes their emotions into account, leading to more fulfilling language learning.

[0347] For example, when a user is practicing speaking online, the system captures their facial expressions in conjunction with their voice. If the emotion engine detects that the user is nervous, the server displays a video on the user's device guiding them on how to relax. In this way, the emotion engine-based approach provides users with an environment where they can focus on improving their skills while receiving emotional support.

[0348] The following describes the processing flow.

[0349] Step 1:

[0350] The user launches a learning application on their device and selects voice input mode. The user then pronounces a specific phrase as practice. The device simultaneously collects audio and video data using its microphone and camera.

[0351] Step 2:

[0352] The device sends the collected audio and video data to the server. This transmission uses an encrypted communication protocol, ensuring the data is delivered securely to the server.

[0353] Step 3:

[0354] The server processes the received audio data using speech recognition technology to evaluate pronunciation and intonation. Simultaneously, an emotion engine analyzes video data to identify the user's emotional state from their facial expressions.

[0355] Step 4:

[0356] The server comprehensively evaluates the user's language ability and emotional state based on the analysis results. If tension or stress is detected, appropriate countermeasures are considered based on the results of the emotion engine.

[0357] Step 5:

[0358] The server considers the user's current state and past learning history to generate an individually optimized learning plan. This plan includes content adjustments and recommendations for learning materials that take motivation into account.

[0359] Step 6:

[0360] The server generates a learning plan and feedback, which is then sent to the device and presented to the user. The device displays the feedback visually and audibly to the user in an easy-to-understand manner.

[0361] Step 7:

[0362] The system uses user feedback provided via their device to correct mistakes and start new exercises. Users adjust the pace and content as needed based on the feedback.

[0363] Step 8:

[0364] The server continuously monitors the user's emotions and learning progress, providing additional resources and encouraging messages as needed. This approach ensures that users are always supported and maintain an optimal learning experience.

[0365] (Example 2)

[0366] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0367] Conventional language learning systems have been insufficient in providing effective feedback and optimizing learning plans that take into account the user's emotional state, resulting in challenges in improving learning efficiency and motivation. Furthermore, they have struggled to flexibly respond to the individual learning needs and emotional states of users.

[0368] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0369] In this invention, the server includes means for collecting the user's voice, visual, and text information and evaluating their abilities and progress; means for generating an individually optimized learning plan based on the evaluation and the user's emotional state; and means for providing the learning plan to the user and providing real-time adaptive feedback according to their progress and emotional state. This makes it possible to provide an effective and individually optimized learning environment that takes the user's emotional state into account.

[0370] "Audio, visual, and text information" is a general term for data that includes the characteristics of the user's voice, facial expressions, and written or spoken text.

[0371] "Evaluation" is the act of determining a user's current learning ability, progress, and emotional state by analyzing collected data.

[0372] A "personally optimized learning plan" refers to learning methods and content that are customized based on the user's specific needs, ability level, and emotional state.

[0373] "Adaptive feedback" refers to responses or advice that are modified or adjusted according to the user's real-time learning progress and emotional state.

[0374] "Learning resources" refers to educational materials, tools, and other sources of information provided to users for language learning.

[0375] "Communication procedures" refer to a set of protocols and rules used to send and receive data between a user and a server.

[0376] "Display device" refers to a device or apparatus used to transmit visual or auditory information to a user.

[0377] The present invention provides an advanced method that combines an emotion engine to enrich the user's language learning experience. Specifically, it realizes an individually optimized learning environment through interaction between the server, terminal, and user.

[0378] Users begin learning using their devices, during which the device's built-in voice input device and camera collect the user's voice and visual information. This data is automatically sent to a server for advanced analysis. The server uses an emotion engine, voice analysis, and facial expression analysis technology to evaluate the user's emotional state. This evaluation result is analyzed along with the user's learning progress, ability level, and motivation.

[0379] Based on the analyzed data, the server presents a personalized learning plan to the user. The difficulty level and content are selected to best suit the user's current emotional state. For example, if the user is feeling stressed, they may be presented with simple tasks or relaxing videos to alleviate that stress. Furthermore, the server provides real-time feedback and dynamically adjusts the learning plan according to the user's emotions.

[0380] For example, if the emotion engine detects tension or fatigue from the audio and facial expressions captured by the device while the user is practicing speaking, the server will present a video on relaxation techniques or display a message of encouragement. In this way, users can efficiently improve their skills while receiving emotional support.

[0381] An example of a prompt to a generative AI model is, "Suggest content to provide when the user's emotional state is anxious." This prompt allows the AI ​​model to generate and provide the user with the most suitable relaxation content or tasks.

[0382] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0383] Step 1:

[0384] When a user begins learning on the device, it uses its built-in microphone and camera to collect the user's voice and facial expression data. This input data includes the user's voice tone, speaking pace, and facial muscle movements. The device transmits this data to a server in real time, providing foundational data for the next analysis step.

[0385] Step 2:

[0386] The server analyzes the received audio and facial expression data. This process uses a generative AI model to detect emotions from the audio and facial recognition technology to evaluate the emotional state from the facial expressions. This data processing yields an output of the emotional state the user is currently experiencing. Specifically, when a user is nervous, their voice tone may be higher and their face may appear tense.

[0387] Step 3:

[0388] The server generates an optimal learning plan for the user based on the analyzed emotional state. The generated plan includes learning materials and simple tasks to help the user relax if they are feeling stressed. This plan is automatically adjusted by the generating AI model, taking into account the user's current ability level and learning goals. The output includes a list of specific tasks and recommended learning materials.

[0389] Step 4:

[0390] The server provides the user with the generated learning plan as feedback, presenting it visually or audibly via the device. Based on the feedback received, the user performs the next learning step. At this stage, for example, relaxing music may be played, or simple questions may be displayed on the screen. The user's responses are collected again as new input data and sent back to the server.

[0391] Step 5:

[0392] The server monitors user feedback and adjusts the learning plan in real time as needed. If it determines that the user's progress has improved, it increases the difficulty level of the learning materials or introduces different types of challenges. This cycle is repeated, ensuring that users always receive an optimized learning experience.

[0393] (Application Example 2)

[0394] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0395] Traditional language learning systems often provide a uniform learning plan without considering the user's emotional state, which can reduce learning efficiency. In particular, despite the significant impact emotions can have on learning, there is a lack of means to customize the learning experience based on these factors. Furthermore, there is a problem in providing appropriate feedback when motivation is low or users are experiencing stress.

[0396] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0397] In this invention, the server includes means for collecting the user's voice and visual data and analyzing their emotional state; means for dynamically adjusting an individually optimized learning plan based on the emotional state; and means for providing the adjusted learning plan to the user and providing real-time feedback according to the emotional state. This makes it possible to provide an appropriate learning experience that is in line with the user's emotions and to improve learning effectiveness.

[0398] "Voice data" refers to information about the user's voice that is analyzed using speech recognition technology.

[0399] "Visual data" refers to information about images or videos acquired to capture the user's facial expressions and movements.

[0400] "Emotional state" refers to the results of a real-time assessment of the user's psychological or emotional state.

[0401] A "personally optimized learning plan" refers to learning content and teaching methods that are dynamically adjusted based on the user's abilities and emotional state.

[0402] "Feedback" refers to evaluations or advice provided to the user during learning, and is delivered in real time in response to their emotional state.

[0403] "Educational resources" refer to educational materials and learning content that are accessible to support the user's learning.

[0404] "Learning history" refers to information that includes records of the user's learning activities to date, as well as their level of achievement and challenges based on those activities.

[0405] "Emotional history" refers to a record of a user's past emotional states and is data used to adjust learning.

[0406] A "wireless communication protocol" is a set of standards and procedures that define the communication methods used to transfer data wirelessly.

[0407] A "human-machine interface" is a device or means that enables the exchange of information between a user and a system.

[0408] This invention implements a language learning system equipped with an emotion engine using a consumer robot. The system mainly includes hardware for processing audio and visual data, and software for performing emotion analysis.

[0409] Specifically, the robot is equipped with a high-quality microphone and camera to acquire audio and visual data from the user. The audio data is converted into text data using a speech recognition system such as the Google Speech-to-Text API. The visual data is analyzed in real time by facial recognition systems such as OpenCV and the Microsoft Face API.

[0410] The server processes this data using an emotion engine (e.g., Python's EmotionRecognition library) to evaluate the user's emotional state. Based on the evaluation, it generates an individually optimized learning plan and dynamically adjusts it. The learning plan and feedback are provided to the user in an appropriate format, and the server presents this to the user via a human-machine interface.

[0411] For example, if the emotion engine detects that a user is stressed while learning a language, the system will play a relaxing video on the robot's display. This allows the user to continue learning in a more relaxed state.

[0412] When using generative AI models to generate content and adjust learning, prompts such as "Generate feedback that suggests relaxation when the user is tired" are used. This prompt supports the generation of appropriate feedback based on the user's emotional state.

[0413] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0414] Step 1:

[0415] The device acquires audio and visual data from the user in real time. Input consists of the user's speech and facial expressions, and data is collected using a microphone and camera. The collected data is sent to a server, where the audio data is prepared for conversion into text data.

[0416] Step 2:

[0417] The server converts the input audio data into text data using a speech recognition system. It utilizes technologies such as the Google Speech-to-Text API to convert speech into text strings and generate the text data necessary for subsequent processing. The output is text data representing what the user said.

[0418] Step 3:

[0419] The server analyzes the user's emotional state using visual data. Using OpenCV and the Microsoft Face API, it reads the user's facial expressions from the acquired visual data and inputs them into the emotion analysis engine. The output is data indicating the analyzed emotional state of the user.

[0420] Step 4:

[0421] The server uses an emotion analysis engine to evaluate the user's emotional state and generate an individually optimized learning plan. For example, it uses the Python EmotionRecognition library to process emotional data in real time and adjust the learning plan based on the user's psychological state. The output is the adjusted learning plan.

[0422] Step 5:

[0423] The server provides the generated learning plan to the user via the terminal. The user's display shows the tailored learning materials and information, and provides voice-based feedback as needed. The input is the learning plan generated in step 4, and the output is the learning content presented to the user.

[0424] Step 6:

[0425] Users can progress through their learning based on the presented learning content. Based on the results, feedback is provided in real time, and prompts are generated using a generative AI model as needed. For example, a prompt such as "Generate feedback suggesting relaxation if the user is tired" might be used. The output is appropriate feedback tailored to the user's learning progress.

[0426] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0427] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0428] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0429] [Third Embodiment]

[0430] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0431] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0432] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0433] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0434] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0435] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0436] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0437] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0438] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0439] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0440] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0441] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0442] The AI ​​language learning system of the present invention aims to improve the language abilities of users. This system improves learning efficiency by allowing users to input voice and text data and providing personalized learning plans based on that data.

[0443] Specifically, the user inputs audio of the language they want to practice via a device. The device sends this audio data to a server, which analyzes the data using speech recognition technology. Based on the analyzed data, the user's pronunciation ability and comprehension level are evaluated.

[0444] Based on the evaluation results, the server creates a learning plan tailored to the user. This plan is automatically optimized, taking into account the user's past learning history, current skill level, goals, interests, and other factors. The generated plan is sent to the device and can be accessed by the user at any time.

[0445] Furthermore, the server has the capability to provide real-time feedback. Every time a user practices pronunciation or grammar, the server immediately analyzes it and displays the results through the device. This allows users to quickly identify and correct their weaknesses.

[0446] The system also provides access to a variety of learning materials. News articles, audio materials, literary works, and other materials are selected according to the user's interests and skill level. The selected materials are delivered to the device, allowing the user to progress with their learning on a daily basis.

[0447] Furthermore, the server periodically re-evaluates the user's learning data and provides practice suggestions as needed. For example, if a user has weaknesses in listening comprehension, the server will recommend specific audio materials. In this way, the system supports the user's progress and enables more efficient language learning.

[0448] This system allows users to experience personalized learning regardless of time or location, improving the speed and accuracy of language acquisition.

[0449] The following describes the processing flow.

[0450] Step 1:

[0451] The user launches the application using their device and selects voice input mode. The user then speaks the words or phrases they want to practice into the microphone.

[0452] Step 2:

[0453] The device collects the user's voice data, converts it to digital data, and then sends it to the server. The voice data is transferred using a secure communication protocol.

[0454] Step 3:

[0455] The server uses a speech recognition API to analyze the received audio data. The audio data is converted into text data, and its pronunciation accuracy and intonation are evaluated.

[0456] Step 4:

[0457] The server evaluates the user's pronunciation ability based on the analysis results and generates real-time feedback. The feedback details specific errors and areas for improvement in the user's pronunciation.

[0458] Step 5:

[0459] The server automatically generates a personalized learning plan that takes into account the user's past learning history and current ability assessment. This plan includes recommended learning materials and goals to be achieved.

[0460] Step 6:

[0461] The server sends the generated learning plan to the device, allowing the user to begin learning according to the plan. The device then notifies the user that a new learning plan is available.

[0462] Step 7:

[0463] Users receive real-time feedback via their device and improve their learning by correcting the displayed errors. They repeatedly practice pronunciation while referring to the feedback.

[0464] Step 8:

[0465] The server periodically analyzes the user's learning activity to identify their weaknesses. It then suggests practice methods and learning materials for improvement as needed.

[0466] Step 9:

[0467] The server selects and delivers a variety of learning materials to the user's device based on their interests and skill level. Users can then use these materials to broaden their learning scope and enhance their abilities.

[0468] (Example 1)

[0469] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0470] Traditional language learning systems struggle to effectively optimize for individual user abilities and learning progress, and have limitations in real-time feedback and resource selection. Therefore, users find it difficult to find a learning method that suits them, and there is a need for improved learning efficiency.

[0471] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0472] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan using a generative AI model, and means for performing analysis and providing feedback in real time according to the progress. This enables the user to establish a learning plan optimized for them and efficiently improve their language skills.

[0473] "User" refers to an individual who uses this system for language learning.

[0474] "Voice and text data" refers to spoken and written text provided by the user, which will be used as input data for the system.

[0475] Evaluating "ability" refers to the process of analyzing the current state of a user's language skills and identifying their strengths and weaknesses.

[0476] "Progress" is an indicator that shows the degree of improvement or achievement a user has made since starting language learning.

[0477] A "generative AI model" refers to a system that includes an algorithm for automatically generating an optimal learning plan for a user using artificial intelligence.

[0478] A "personally optimized learning plan" refers to a plan that provides learning methods and materials optimized based on each user's abilities and progress.

[0479] "Analysis and feedback" refers to the process of evaluating the user's pronunciation and grammar in real time and returning the results to the user.

[0480] "Educational resources" is a general term for various teaching materials and educational content intended for language learning.

[0481] A "communication protocol" is a technology that defines a set of rules and procedures used when sending and receiving data over a network.

[0482] A "display device" refers to a monitor or screen used to provide users with visual information, such as real-time feedback.

[0483] This AI language learning system aims to improve the user's language abilities. Specific embodiments of the present invention are described below.

[0484] The user uses their device to record audio data of the language they want to learn. Using the audio recording application on the device, the user generates the audio as digital data and sends this data to the server. The HTTP protocol is generally used for transmission.

[0485] The server uses speech recognition software to analyze the received audio data. For example, it uses widely available speech recognition services on the internet to convert speech to text. Based on the recognized text data, the server evaluates the user's pronunciation and comprehension. The evaluation results are further processed by a generative AI model, and a personalized learning plan is automatically generated.

[0486] The generated learning plan is optimized based on the user's characteristics and past learning history, and is provided to the user via the device. The user can then proceed with their daily learning based on this plan. Each time the user practices, the device receives real-time feedback from the server and presents it to the user visually.

[0487] Furthermore, the server periodically re-evaluates the user's learning data and generates new learning suggestions. For example, if the user has difficulty with listening skills, appropriate audio materials will be recommended. In this way, the system efficiently supports the user's learning.

[0488] Examples of specific cases and prompt statements

[0489] As a concrete example, consider a scenario where a user is studying Spanish. The user uses their device to record the phrase "Hola, ¿cómo estás?" (Hello, how are you?). The server performs speech recognition to determine if the pronunciation is correct. Based on the results, the server generates an appropriate practice plan and provides specific feedback for improving pronunciation.

[0490] As an example of a prompt, the following instructions can be entered into the generative AI model:

[0491] "Please recommend listening and pronunciation materials focused on beginner levels for users learning English."

[0492] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0493] Step 1:

[0494] The user records the audio of the language they want to learn using their device. This recording is done using an audio recording application on the device, and the audio data is output in digital format.

[0495] Step 2:

[0496] The terminal sends digital audio data to the server. HTTP is used as the communication protocol. The output digital audio data is then input to the server in a format that it can process.

[0497] Step 3:

[0498] The server passes the received audio data to the speech recognition engine. This engine converts the audio into text. The speech recognition engine outputs the recognized text data as the result of the analysis.

[0499] Step 4:

[0500] The server analyzes the text data obtained through speech recognition and evaluates the user's pronunciation and comprehension. Here, data calculations are performed to measure the accuracy of the user's pronunciation. The evaluation results are then output.

[0501] Step 5:

[0502] Based on the evaluation results and past learning history, the server generates an individually optimized learning plan using a generated AI model. The generated learning plan is output as data in JSON format.

[0503] Step 6:

[0504] The server sends the generated learning plan to the terminal. The terminal visually displays the received learning plan to the user, making it accessible to the user. The optimized learning content is output through the display device.

[0505] Step 7:

[0506] When a user practices pronunciation or grammar, a feedback request is sent to the server in real time via the device. The server immediately analyzes the data and generates feedback data. The feedback is output and sent back to the device.

[0507] Step 8:

[0508] The device receives feedback from the server and presents it visually to the user. The feedback includes specific instructions to help the user improve their pronunciation and comprehension.

[0509] Step 9:

[0510] The server periodically re-evaluates the user's learning data and generates new practice suggestions as needed. Based on this information, new learning material suggestions are output.

[0511] (Application Example 1)

[0512] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] Many language learning systems struggle to provide a personalized learning experience because they cannot offer real-time conversation practice tailored to the user's current location or places they visit. Furthermore, their lack of ability to provide immediate pronunciation evaluation and geographical-based feedback makes it difficult for learners to improve their language skills when needed.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0515] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan, and means for providing conversational phrases and information relevant to the user based on geographical information. This enables realistic language learning tailored to the places the user visits, providing a more flexible and effective learning experience.

[0516] "User" refers to an individual or group that uses a language learning system.

[0517] "Audio and text data" refers to audio and text information entered or provided by the user, and is the data that forms the basis for evaluation and feedback.

[0518] "Means of evaluating ability and progress" refers to a system that analyzes a user's language skills and learning status and provides objective numerical data and results.

[0519] A "personally optimized learning plan" is the process of creating a learning program that is best suited to the individual, based on the user's evaluation results and learning history.

[0520] "Learning resources" is a general term for information sources that users utilize for learning, including news articles, audio materials, and literary works.

[0521] "Geographic-based conversational phrases" are phrases used to provide conversational content relevant to the user's current location or destination.

[0522] "Pronunciation scoring and evaluation" is a process that analyzes a user's pronunciation and provides a numerical representation of their accuracy and fluency.

[0523] A "means of providing real-time feedback" refers to a system that quickly returns evaluation results and suggestions for improvement in response to user input.

[0524] A description of embodiments for carrying out this invention will be given.

[0525] The server's role is to receive data when the user inputs voice or text data. For voice data, the server utilizes speech recognition software (e.g., Google Cloud Speech-to-Text or IBM Watson) to convert the input speech into text. Using this converted text data, a machine learning model (such as TensorFlow or PyTorch) evaluates the user's language ability. Based on the evaluation, the server generates an individually optimized learning plan.

[0526] The device displays a learning plan received from the server to the user. The learning plan includes conversational phrases and information based on geographical location, providing content relevant to the user's current location. The device also has a real-time feedback function. This feedback, based on information obtained from pronunciation scoring and evaluation, immediately informs the user of areas for improvement.

[0527] Users can progressively improve their language skills by following the provided learning plan. For example, users can practice conversational phrases needed when visiting a city hall. If a user wants to simulate a situation such as "requesting the issuance of a resident registration certificate," the device will present appropriate phrases and prompt the user to practice pronunciation.

[0528] An example of a prompt sentence for a generative AI model is: "Please create sentences that provide everyday conversational phrases for tourists at a local supermarket. Also, please provide information on recommended phrases for pronunciation practice."

[0529] Thus, the system in this invention provides a learning experience tailored to the user and improves language ability.

[0530] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0531] Step 1:

[0532] The user inputs audio data of the language to be learned into the device. This input audio data is sent by the device to the server. The server then receives the audio data to begin the speech recognition process.

[0533] Step 2:

[0534] The server converts the received audio data into text data using speech recognition software such as Google Cloud Speech-to-Text or IBM Watson. This process provides the user's pronunciation information in text format.

[0535] Step 3:

[0536] The server inputs the converted text data into machine learning models such as TensorFlow or PyTorch to evaluate the user's language ability. This evaluation process analyzes the text data and calculates metrics such as pronunciation scores and grammatical comprehension.

[0537] Step 4:

[0538] The server combines evaluation results, the user's learning history, and geographical information to generate a personalized learning plan. This learning plan includes adapted conversational phrases and relevant information.

[0539] Step 5:

[0540] The device displays a learning plan sent from the server to the user. Based on the information presented, the user engages in geographically relevant conversation practice. Specifically, a screen is displayed for practicing phrases tailored to the destination.

[0541] Step 6:

[0542] When the user inputs voice again and practices the conversation, the device sends the audio to the server. The server evaluates it and generates immediate feedback.

[0543] Step 7:

[0544] The terminal displays feedback from the server to the user in real time. Based on this feedback, the user can correct their mistakes and improve their language skills.

[0545] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0546] The system of this invention provides a more effective and adaptive learning experience by combining an emotion engine with the user's language learning. The emotion engine has the function of analyzing the user's voice and facial expression data and evaluating the current emotional state in real time. This makes it possible to provide feedback and customize learning content based on the user's emotions.

[0547] When a user inputs voice data using a device, the camera and microphone simultaneously collect emotional data. The device sends this data to a server, which uses an emotion engine to analyze the user's emotional state. The analyzed emotional data reflects the user's state of excitement, frustration, concentration, and other emotions they experience while learning.

[0548] The server takes the user's emotional state into consideration and creates an individually optimized learning plan. This plan includes dynamic adjustments, such as presenting more challenging materials when the user is highly motivated, or sending encouraging messages when motivation is low.

[0549] Furthermore, the server provides users with visual or auditory feedback from the emotion engine. For example, if a user is feeling tired, the system will suggest a break or provide relaxing content to help them continue learning. These settings allow users to experience a learning environment that takes their emotions into account, leading to more fulfilling language learning.

[0550] For example, when a user is practicing speaking online, the system captures their facial expressions in conjunction with their voice. If the emotion engine detects that the user is nervous, the server displays a video on the user's device guiding them on how to relax. In this way, the emotion engine-based approach provides users with an environment where they can focus on improving their skills while receiving emotional support.

[0551] The following describes the processing flow.

[0552] Step 1:

[0553] The user launches a learning application on their device and selects voice input mode. The user then pronounces a specific phrase as practice. The device simultaneously collects audio and video data using its microphone and camera.

[0554] Step 2:

[0555] The device sends the collected audio and video data to the server. This transmission uses an encrypted communication protocol, ensuring the data is delivered securely to the server.

[0556] Step 3:

[0557] The server processes the received audio data using speech recognition technology to evaluate pronunciation and intonation. Simultaneously, an emotion engine analyzes video data to identify the user's emotional state from their facial expressions.

[0558] Step 4:

[0559] The server comprehensively evaluates the user's language ability and emotional state based on the analysis results. If tension or stress is detected, appropriate countermeasures are considered based on the results of the emotion engine.

[0560] Step 5:

[0561] The server considers the user's current state and past learning history to generate an individually optimized learning plan. This plan includes content adjustments and recommendations for learning materials that take motivation into account.

[0562] Step 6:

[0563] The server generates a learning plan and feedback, which is then sent to the device and presented to the user. The device displays the feedback visually and audibly to the user in an easy-to-understand manner.

[0564] Step 7:

[0565] The system uses user feedback provided via their device to correct mistakes and start new exercises. Users adjust the pace and content as needed based on the feedback.

[0566] Step 8:

[0567] The server continuously monitors the user's emotions and learning progress, providing additional resources and encouraging messages as needed. This approach ensures that users are always supported and maintain an optimal learning experience.

[0568] (Example 2)

[0569] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0570] Conventional language learning systems have been insufficient in providing effective feedback and optimizing learning plans that take into account the user's emotional state, resulting in challenges in improving learning efficiency and motivation. Furthermore, they have struggled to flexibly respond to the individual learning needs and emotional states of users.

[0571] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0572] In this invention, the server includes means for collecting the user's voice, visual, and text information and evaluating their abilities and progress; means for generating an individually optimized learning plan based on the evaluation and the user's emotional state; and means for providing the learning plan to the user and providing real-time adaptive feedback according to their progress and emotional state. This makes it possible to provide an effective and individually optimized learning environment that takes the user's emotional state into account.

[0573] "Audio, visual, and text information" is a general term for data that includes the characteristics of the user's voice, facial expressions, and written or spoken text.

[0574] "Evaluation" is the act of determining a user's current learning ability, progress, and emotional state by analyzing collected data.

[0575] A "personally optimized learning plan" refers to learning methods and content that are customized based on the user's specific needs, ability level, and emotional state.

[0576] "Adaptive feedback" refers to responses or advice that are modified or adjusted according to the user's real-time learning progress and emotional state.

[0577] "Learning resources" refers to educational materials, tools, and other sources of information provided to users for language learning.

[0578] "Communication procedures" refer to a set of protocols and rules used to send and receive data between a user and a server.

[0579] "Display device" refers to a device or apparatus used to transmit visual or auditory information to a user.

[0580] The present invention provides an advanced method that combines an emotion engine to enrich the user's language learning experience. Specifically, it realizes an individually optimized learning environment through interaction between the server, terminal, and user.

[0581] Users begin learning using their devices, during which the device's built-in voice input device and camera collect the user's voice and visual information. This data is automatically sent to a server for advanced analysis. The server uses an emotion engine, voice analysis, and facial expression analysis technology to evaluate the user's emotional state. This evaluation result is analyzed along with the user's learning progress, ability level, and motivation.

[0582] Based on the analyzed data, the server presents a personalized learning plan to the user. The difficulty level and content are selected to best suit the user's current emotional state. For example, if the user is feeling stressed, they may be presented with simple tasks or relaxing videos to alleviate that stress. Furthermore, the server provides real-time feedback and dynamically adjusts the learning plan according to the user's emotions.

[0583] For example, if the emotion engine detects tension or fatigue from the audio and facial expressions captured by the device while the user is practicing speaking, the server will present a video on relaxation techniques or display a message of encouragement. In this way, users can efficiently improve their skills while receiving emotional support.

[0584] An example of a prompt to a generative AI model is, "Suggest content to provide when the user's emotional state is anxious." This prompt allows the AI ​​model to generate and provide the user with the most suitable relaxation content or tasks.

[0585] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0586] Step 1:

[0587] When a user begins learning on the device, it uses its built-in microphone and camera to collect the user's voice and facial expression data. This input data includes the user's voice tone, speaking pace, and facial muscle movements. The device transmits this data to a server in real time, providing foundational data for the next analysis step.

[0588] Step 2:

[0589] The server analyzes the received audio and facial expression data. This process uses a generative AI model to detect emotions from the audio and facial recognition technology to evaluate the emotional state from the facial expressions. This data processing yields an output of the emotional state the user is currently experiencing. Specifically, when a user is nervous, their voice tone may be higher and their face may appear tense.

[0590] Step 3:

[0591] The server generates an optimal learning plan for the user based on the analyzed emotional state. The generated plan includes learning materials and simple tasks to help the user relax if they are feeling stressed. This plan is automatically adjusted by the generating AI model, taking into account the user's current ability level and learning goals. The output includes a list of specific tasks and recommended learning materials.

[0592] Step 4:

[0593] The server provides the user with the generated learning plan as feedback, presenting it visually or audibly via the device. Based on the feedback received, the user performs the next learning step. At this stage, for example, relaxing music may be played, or simple questions may be displayed on the screen. The user's responses are collected again as new input data and sent back to the server.

[0594] Step 5:

[0595] The server monitors user feedback and adjusts the learning plan in real time as needed. If it determines that the user's progress has improved, it increases the difficulty level of the learning materials or introduces different types of challenges. This cycle is repeated, ensuring that users always receive an optimized learning experience.

[0596] (Application Example 2)

[0597] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0598] Traditional language learning systems often provide a uniform learning plan without considering the user's emotional state, which can reduce learning efficiency. In particular, despite the significant impact emotions can have on learning, there is a lack of means to customize the learning experience based on these factors. Furthermore, there is a problem in providing appropriate feedback when motivation is low or users are experiencing stress.

[0599] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0600] In this invention, the server includes means for collecting the user's voice and visual data and analyzing their emotional state; means for dynamically adjusting an individually optimized learning plan based on the emotional state; and means for providing the adjusted learning plan to the user and providing real-time feedback according to the emotional state. This makes it possible to provide an appropriate learning experience that is in line with the user's emotions and to improve learning effectiveness.

[0601] "Voice data" refers to information about the user's voice that is analyzed using speech recognition technology.

[0602] "Visual data" refers to information about images or videos acquired to capture the user's facial expressions and movements.

[0603] "Emotional state" refers to the results of a real-time assessment of the user's psychological or emotional state.

[0604] A "personally optimized learning plan" refers to learning content and teaching methods that are dynamically adjusted based on the user's abilities and emotional state.

[0605] "Feedback" refers to evaluations or advice provided to the user during learning, and is delivered in real time in response to their emotional state.

[0606] "Educational resources" refer to educational materials and learning content that are accessible to support the user's learning.

[0607] "Learning history" refers to information that includes records of the user's learning activities to date, as well as their level of achievement and challenges based on those activities.

[0608] "Emotional history" refers to a record of a user's past emotional states and is data used to adjust learning.

[0609] A "wireless communication protocol" is a set of standards and procedures that define the communication methods used to transfer data wirelessly.

[0610] A "human-machine interface" is a device or means that enables the exchange of information between a user and a system.

[0611] This invention implements a language learning system equipped with an emotion engine using a consumer robot. The system mainly includes hardware for processing audio and visual data, and software for performing emotion analysis.

[0612] Specifically, the robot is equipped with a high-quality microphone and camera to acquire audio and visual data from the user. The audio data is converted into text data using a speech recognition system such as the Google Speech-to-Text API. The visual data is analyzed in real time by facial recognition systems such as OpenCV and the Microsoft Face API.

[0613] The server processes this data using an emotion engine (e.g., Python's EmotionRecognition library) to evaluate the user's emotional state. Based on the evaluation, it generates an individually optimized learning plan and dynamically adjusts it. The learning plan and feedback are provided to the user in an appropriate format, and the server presents this to the user via a human-machine interface.

[0614] For example, if the emotion engine detects that a user is stressed while learning a language, the system will play a relaxing video on the robot's display. This allows the user to continue learning in a more relaxed state.

[0615] When using generative AI models to generate content and adjust learning, prompts such as "Generate feedback that suggests relaxation when the user is tired" are used. This prompt supports the generation of appropriate feedback based on the user's emotional state.

[0616] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0617] Step 1:

[0618] The device acquires audio and visual data from the user in real time. Input consists of the user's speech and facial expressions, and data is collected using a microphone and camera. The collected data is sent to a server, where the audio data is prepared for conversion into text data.

[0619] Step 2:

[0620] The server converts the input audio data into text data using a speech recognition system. It utilizes technologies such as the Google Speech-to-Text API to convert speech into text strings and generate the text data necessary for subsequent processing. The output is text data representing what the user said.

[0621] Step 3:

[0622] The server analyzes the user's emotional state using visual data. Using OpenCV and the Microsoft Face API, it reads the user's facial expressions from the acquired visual data and inputs them into the emotion analysis engine. The output is data indicating the analyzed emotional state of the user.

[0623] Step 4:

[0624] The server uses an emotion analysis engine to evaluate the user's emotional state and generate an individually optimized learning plan. For example, it uses the Python EmotionRecognition library to process emotional data in real time and adjust the learning plan based on the user's psychological state. The output is the adjusted learning plan.

[0625] Step 5:

[0626] The server provides the generated learning plan to the user via the terminal. The user's display shows the tailored learning materials and information, and provides voice-based feedback as needed. The input is the learning plan generated in step 4, and the output is the learning content presented to the user.

[0627] Step 6:

[0628] Users can progress through their learning based on the presented learning content. Based on the results, feedback is provided in real time, and prompts are generated using a generative AI model as needed. For example, a prompt such as "Generate feedback suggesting relaxation if the user is tired" might be used. The output is appropriate feedback tailored to the user's learning progress.

[0629] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0630] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0631] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0632] [Fourth Embodiment]

[0633] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0634] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0635] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0636] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0637] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0638] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0639] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0640] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0641] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0642] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0643] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0644] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0645] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0646] The AI ​​language learning system of the present invention aims to improve the language abilities of users. This system improves learning efficiency by allowing users to input voice and text data and providing personalized learning plans based on that data.

[0647] Specifically, the user inputs audio of the language they want to practice via a device. The device sends this audio data to a server, which analyzes the data using speech recognition technology. Based on the analyzed data, the user's pronunciation ability and comprehension level are evaluated.

[0648] Based on the evaluation results, the server creates a learning plan tailored to the user. This plan is automatically optimized, taking into account the user's past learning history, current skill level, goals, interests, and other factors. The generated plan is sent to the device and can be accessed by the user at any time.

[0649] Furthermore, the server has the capability to provide real-time feedback. Every time a user practices pronunciation or grammar, the server immediately analyzes it and displays the results through the device. This allows users to quickly identify and correct their weaknesses.

[0650] The system also provides access to a variety of learning materials. News articles, audio materials, literary works, and other materials are selected according to the user's interests and skill level. The selected materials are delivered to the device, allowing the user to progress with their learning on a daily basis.

[0651] Furthermore, the server periodically re-evaluates the user's learning data and provides practice suggestions as needed. For example, if a user has weaknesses in listening comprehension, the server will recommend specific audio materials. In this way, the system supports the user's progress and enables more efficient language learning.

[0652] This system allows users to experience personalized learning regardless of time or location, improving the speed and accuracy of language acquisition.

[0653] The following describes the processing flow.

[0654] Step 1:

[0655] The user launches the application using their device and selects voice input mode. The user then speaks the words or phrases they want to practice into the microphone.

[0656] Step 2:

[0657] The device collects the user's voice data, converts it to digital data, and then sends it to the server. The voice data is transferred using a secure communication protocol.

[0658] Step 3:

[0659] The server uses a speech recognition API to analyze the received audio data. The audio data is converted into text data, and its pronunciation accuracy and intonation are evaluated.

[0660] Step 4:

[0661] The server evaluates the user's pronunciation ability based on the analysis results and generates real-time feedback. The feedback details specific errors and areas for improvement in the user's pronunciation.

[0662] Step 5:

[0663] The server automatically generates a personalized learning plan that takes into account the user's past learning history and current ability assessment. This plan includes recommended learning materials and goals to be achieved.

[0664] Step 6:

[0665] The server sends the generated learning plan to the device, allowing the user to begin learning according to the plan. The device then notifies the user that a new learning plan is available.

[0666] Step 7:

[0667] Users receive real-time feedback via their device and improve their learning by correcting the displayed errors. They repeatedly practice pronunciation while referring to the feedback.

[0668] Step 8:

[0669] The server periodically analyzes the user's learning activity to identify their weaknesses. It then suggests practice methods and learning materials for improvement as needed.

[0670] Step 9:

[0671] The server selects and delivers a variety of learning materials to the user's device based on their interests and skill level. Users can then use these materials to broaden their learning scope and enhance their abilities.

[0672] (Example 1)

[0673] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] Traditional language learning systems struggle to effectively optimize for individual user abilities and learning progress, and have limitations in real-time feedback and resource selection. Therefore, users find it difficult to find a learning method that suits them, and there is a need for improved learning efficiency.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0676] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan using a generative AI model, and means for performing analysis and providing feedback in real time according to the progress. This enables the user to establish a learning plan optimized for them and efficiently improve their language skills.

[0677] "User" refers to an individual who uses this system for language learning.

[0678] "Voice and text data" refers to spoken and written text provided by the user, which will be used as input data for the system.

[0679] Evaluating "ability" refers to the process of analyzing the current state of a user's language skills and identifying their strengths and weaknesses.

[0680] "Progress" is an indicator that shows the degree of improvement or achievement a user has made since starting language learning.

[0681] A "generative AI model" refers to a system that includes an algorithm for automatically generating an optimal learning plan for a user using artificial intelligence.

[0682] A "personally optimized learning plan" refers to a plan that provides learning methods and materials optimized based on each user's abilities and progress.

[0683] "Analysis and feedback" refers to the process of evaluating the user's pronunciation and grammar in real time and returning the results to the user.

[0684] "Educational resources" is a general term for various teaching materials and educational content intended for language learning.

[0685] A "communication protocol" is a technology that defines a set of rules and procedures used when sending and receiving data over a network.

[0686] A "display device" refers to a monitor or screen used to provide users with visual information, such as real-time feedback.

[0687] This AI language learning system aims to improve the user's language abilities. Specific embodiments of the present invention are described below.

[0688] The user uses their device to record audio data of the language they want to learn. Using the audio recording application on the device, the user generates the audio as digital data and sends this data to the server. The HTTP protocol is generally used for transmission.

[0689] The server uses speech recognition software to analyze the received audio data. For example, it uses widely available speech recognition services on the internet to convert speech to text. Based on the recognized text data, the server evaluates the user's pronunciation and comprehension. The evaluation results are further processed by a generative AI model, and a personalized learning plan is automatically generated.

[0690] The generated learning plan is optimized based on the user's characteristics and past learning history, and is provided to the user via the device. The user can then proceed with their daily learning based on this plan. Each time the user practices, the device receives real-time feedback from the server and presents it to the user visually.

[0691] Furthermore, the server periodically re-evaluates the user's learning data and generates new learning suggestions. For example, if the user has difficulty with listening skills, appropriate audio materials will be recommended. In this way, the system efficiently supports the user's learning.

[0692] Examples of specific cases and prompt statements

[0693] As a concrete example, consider a scenario where a user is studying Spanish. The user uses their device to record the phrase "Hola, ¿cómo estás?" (Hello, how are you?). The server performs speech recognition to determine if the pronunciation is correct. Based on the results, the server generates an appropriate practice plan and provides specific feedback for improving pronunciation.

[0694] As an example of a prompt, the following instructions can be entered into the generative AI model:

[0695] "Please recommend listening and pronunciation materials focused on beginner levels for users learning English."

[0696] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0697] Step 1:

[0698] The user records the audio of the language they want to learn using their device. This recording is done using an audio recording application on the device, and the audio data is output in digital format.

[0699] Step 2:

[0700] The terminal sends digital audio data to the server. HTTP is used as the communication protocol. The output digital audio data is then input to the server in a format that it can process.

[0701] Step 3:

[0702] The server passes the received audio data to the speech recognition engine. This engine converts the audio into text. The speech recognition engine outputs the recognized text data as the result of the analysis.

[0703] Step 4:

[0704] The server analyzes the text data obtained through speech recognition and evaluates the user's pronunciation and comprehension. Here, data calculations are performed to measure the accuracy of the user's pronunciation. The evaluation results are then output.

[0705] Step 5:

[0706] Based on the evaluation results and past learning history, the server generates an individually optimized learning plan using a generated AI model. The generated learning plan is output as data in JSON format.

[0707] Step 6:

[0708] The server sends the generated learning plan to the terminal. The terminal visually displays the received learning plan to the user, making it accessible to the user. The optimized learning content is output through the display device.

[0709] Step 7:

[0710] When a user practices pronunciation or grammar, a feedback request is sent to the server in real time via the device. The server immediately analyzes the data and generates feedback data. The feedback is output and sent back to the device.

[0711] Step 8:

[0712] The device receives feedback from the server and presents it visually to the user. The feedback includes specific instructions to help the user improve their pronunciation and comprehension.

[0713] Step 9:

[0714] The server periodically re-evaluates the user's learning data and generates new practice suggestions as needed. Based on this information, new learning material suggestions are output.

[0715] (Application Example 1)

[0716] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0717] Many language learning systems struggle to provide a personalized learning experience because they cannot offer real-time conversation practice tailored to the user's current location or places they visit. Furthermore, their lack of ability to provide immediate pronunciation evaluation and geographical-based feedback makes it difficult for learners to improve their language skills when needed.

[0718] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0719] In this invention, the server includes means for collecting the user's voice and text data and evaluating their abilities and progress, means for generating an individually optimized learning plan, and means for providing conversational phrases and information relevant to the user based on geographical information. This enables realistic language learning tailored to the places the user visits, providing a more flexible and effective learning experience.

[0720] "User" refers to an individual or group that uses a language learning system.

[0721] "Audio and text data" refers to audio and text information entered or provided by the user, and is the data that forms the basis for evaluation and feedback.

[0722] "Means of evaluating ability and progress" refers to a system that analyzes a user's language skills and learning status and provides objective numerical data and results.

[0723] A "personally optimized learning plan" is the process of creating a learning program that is best suited to the individual, based on the user's evaluation results and learning history.

[0724] "Learning resources" is a general term for information sources that users utilize for learning, including news articles, audio materials, and literary works.

[0725] "Geographic-based conversational phrases" are phrases used to provide conversational content relevant to the user's current location or destination.

[0726] "Pronunciation scoring and evaluation" is a process that analyzes a user's pronunciation and provides a numerical representation of their accuracy and fluency.

[0727] A "means of providing real-time feedback" refers to a system that quickly returns evaluation results and suggestions for improvement in response to user input.

[0728] A description of embodiments for carrying out this invention will be given.

[0729] The server's role is to receive data when the user inputs voice or text data. For voice data, the server utilizes speech recognition software (e.g., Google Cloud Speech-to-Text or IBM Watson) to convert the input speech into text. Using this converted text data, a machine learning model (such as TensorFlow or PyTorch) evaluates the user's language ability. Based on the evaluation, the server generates an individually optimized learning plan.

[0730] The device displays a learning plan received from the server to the user. The learning plan includes conversational phrases and information based on geographical location, providing content relevant to the user's current location. The device also has a real-time feedback function. This feedback, based on information obtained from pronunciation scoring and evaluation, immediately informs the user of areas for improvement.

[0731] Users can progressively improve their language skills by following the provided learning plan. For example, users can practice conversational phrases needed when visiting a city hall. If a user wants to simulate a situation such as "requesting the issuance of a resident registration certificate," the device will present appropriate phrases and prompt the user to practice pronunciation.

[0732] An example of a prompt sentence for a generative AI model is: "Please create sentences that provide everyday conversational phrases for tourists at a local supermarket. Also, please provide information on recommended phrases for pronunciation practice."

[0733] Thus, the system in this invention provides a learning experience tailored to the user and improves language ability.

[0734] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0735] Step 1:

[0736] The user inputs audio data of the language to be learned into the device. This input audio data is sent by the device to the server. The server then receives the audio data to begin the speech recognition process.

[0737] Step 2:

[0738] The server converts the received audio data into text data using speech recognition software such as Google Cloud Speech-to-Text or IBM Watson. This process provides the user's pronunciation information in text format.

[0739] Step 3:

[0740] The server inputs the converted text data into machine learning models such as TensorFlow or PyTorch to evaluate the user's language ability. This evaluation process analyzes the text data and calculates metrics such as pronunciation scores and grammatical comprehension.

[0741] Step 4:

[0742] The server combines evaluation results, the user's learning history, and geographical information to generate a personalized learning plan. This learning plan includes adapted conversational phrases and relevant information.

[0743] Step 5:

[0744] The device displays a learning plan sent from the server to the user. Based on the information presented, the user engages in geographically relevant conversation practice. Specifically, a screen is displayed for practicing phrases tailored to the destination.

[0745] Step 6:

[0746] When the user inputs voice again and practices the conversation, the device sends the audio to the server. The server evaluates it and generates immediate feedback.

[0747] Step 7:

[0748] The terminal displays feedback from the server to the user in real time. Based on this feedback, the user can correct their mistakes and improve their language skills.

[0749] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0750] The system of this invention provides a more effective and adaptive learning experience by combining an emotion engine with the user's language learning. The emotion engine has the function of analyzing the user's voice and facial expression data and evaluating the current emotional state in real time. This makes it possible to provide feedback and customize learning content based on the user's emotions.

[0751] When a user inputs voice data using a device, the camera and microphone simultaneously collect emotional data. The device sends this data to a server, which uses an emotion engine to analyze the user's emotional state. The analyzed emotional data reflects the user's state of excitement, frustration, concentration, and other emotions they experience while learning.

[0752] The server takes the user's emotional state into consideration and creates an individually optimized learning plan. This plan includes dynamic adjustments, such as presenting more challenging materials when the user is highly motivated, or sending encouraging messages when motivation is low.

[0753] Furthermore, the server provides users with visual or auditory feedback from the emotion engine. For example, if a user is feeling tired, the system will suggest a break or provide relaxing content to help them continue learning. These settings allow users to experience a learning environment that takes their emotions into account, leading to more fulfilling language learning.

[0754] For example, when a user is practicing speaking online, the system captures their facial expressions in conjunction with their voice. If the emotion engine detects that the user is nervous, the server displays a video on the user's device guiding them on how to relax. In this way, the emotion engine-based approach provides users with an environment where they can focus on improving their skills while receiving emotional support.

[0755] The following describes the processing flow.

[0756] Step 1:

[0757] The user launches a learning application on their device and selects voice input mode. The user then pronounces a specific phrase as practice. The device simultaneously collects audio and video data using its microphone and camera.

[0758] Step 2:

[0759] The device sends the collected audio and video data to the server. This transmission uses an encrypted communication protocol, ensuring the data is delivered securely to the server.

[0760] Step 3:

[0761] The server processes the received audio data using speech recognition technology to evaluate pronunciation and intonation. Simultaneously, an emotion engine analyzes video data to identify the user's emotional state from their facial expressions.

[0762] Step 4:

[0763] The server comprehensively evaluates the user's language ability and emotional state based on the analysis results. If tension or stress is detected, appropriate countermeasures are considered based on the results of the emotion engine.

[0764] Step 5:

[0765] The server considers the user's current state and past learning history to generate an individually optimized learning plan. This plan includes content adjustments and recommendations for learning materials that take motivation into account.

[0766] Step 6:

[0767] The server generates a learning plan and feedback, which is then sent to the device and presented to the user. The device displays the feedback visually and audibly to the user in an easy-to-understand manner.

[0768] Step 7:

[0769] The system uses user feedback provided via their device to correct mistakes and start new exercises. Users adjust the pace and content as needed based on the feedback.

[0770] Step 8:

[0771] The server continuously monitors the user's emotions and learning progress, providing additional resources and encouraging messages as needed. This approach ensures that users are always supported and maintain an optimal learning experience.

[0772] (Example 2)

[0773] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0774] Conventional language learning systems have been insufficient in providing effective feedback and optimizing learning plans that take into account the user's emotional state, resulting in challenges in improving learning efficiency and motivation. Furthermore, they have struggled to flexibly respond to the individual learning needs and emotional states of users.

[0775] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0776] In this invention, the server includes means for collecting the user's voice, visual, and text information and evaluating their abilities and progress; means for generating an individually optimized learning plan based on the evaluation and the user's emotional state; and means for providing the learning plan to the user and providing real-time adaptive feedback according to their progress and emotional state. This makes it possible to provide an effective and individually optimized learning environment that takes the user's emotional state into account.

[0777] "Audio, visual, and text information" is a general term for data that includes the characteristics of the user's voice, facial expressions, and written or spoken text.

[0778] "Evaluation" is the act of determining a user's current learning ability, progress, and emotional state by analyzing collected data.

[0779] A "personally optimized learning plan" refers to learning methods and content that are customized based on the user's specific needs, ability level, and emotional state.

[0780] "Adaptive feedback" refers to responses or advice that are modified or adjusted according to the user's real-time learning progress and emotional state.

[0781] "Learning resources" refers to educational materials, tools, and other sources of information provided to users for language learning.

[0782] "Communication procedures" refer to a set of protocols and rules used to send and receive data between a user and a server.

[0783] "Display device" refers to a device or apparatus used to transmit visual or auditory information to a user.

[0784] The present invention provides an advanced method that combines an emotion engine to enrich the user's language learning experience. Specifically, it realizes an individually optimized learning environment through interaction between the server, terminal, and user.

[0785] Users begin learning using their devices, during which the device's built-in voice input device and camera collect the user's voice and visual information. This data is automatically sent to a server for advanced analysis. The server uses an emotion engine, voice analysis, and facial expression analysis technology to evaluate the user's emotional state. This evaluation result is analyzed along with the user's learning progress, ability level, and motivation.

[0786] Based on the analyzed data, the server presents a personalized learning plan to the user. The difficulty level and content are selected to best suit the user's current emotional state. For example, if the user is feeling stressed, they may be presented with simple tasks or relaxing videos to alleviate that stress. Furthermore, the server provides real-time feedback and dynamically adjusts the learning plan according to the user's emotions.

[0787] For example, if the emotion engine detects tension or fatigue from the audio and facial expressions captured by the device while the user is practicing speaking, the server will present a video on relaxation techniques or display a message of encouragement. In this way, users can efficiently improve their skills while receiving emotional support.

[0788] An example of a prompt to a generative AI model is, "Suggest content to provide when the user's emotional state is anxious." This prompt allows the AI ​​model to generate and provide the user with the most suitable relaxation content or tasks.

[0789] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0790] Step 1:

[0791] When a user begins learning on the device, it uses its built-in microphone and camera to collect the user's voice and facial expression data. This input data includes the user's voice tone, speaking pace, and facial muscle movements. The device transmits this data to a server in real time, providing foundational data for the next analysis step.

[0792] Step 2:

[0793] The server analyzes the received audio and facial expression data. This process uses a generative AI model to detect emotions from the audio and facial recognition technology to evaluate the emotional state from the facial expressions. This data processing yields an output of the emotional state the user is currently experiencing. Specifically, when a user is nervous, their voice tone may be higher and their face may appear tense.

[0794] Step 3:

[0795] The server generates an optimal learning plan for the user based on the analyzed emotional state. The generated plan includes learning materials and simple tasks to help the user relax if they are feeling stressed. This plan is automatically adjusted by the generating AI model, taking into account the user's current ability level and learning goals. The output includes a list of specific tasks and recommended learning materials.

[0796] Step 4:

[0797] The server provides the user with the generated learning plan as feedback, presenting it visually or audibly via the device. Based on the feedback received, the user performs the next learning step. At this stage, for example, relaxing music may be played, or simple questions may be displayed on the screen. The user's responses are collected again as new input data and sent back to the server.

[0798] Step 5:

[0799] The server monitors user feedback and adjusts the learning plan in real time as needed. If it determines that the user's progress has improved, it increases the difficulty level of the learning materials or introduces different types of challenges. This cycle is repeated, ensuring that users always receive an optimized learning experience.

[0800] (Application Example 2)

[0801] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0802] Traditional language learning systems often provide a uniform learning plan without considering the user's emotional state, which can reduce learning efficiency. In particular, despite the significant impact emotions can have on learning, there is a lack of means to customize the learning experience based on these factors. Furthermore, there is a problem in providing appropriate feedback when motivation is low or users are experiencing stress.

[0803] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0804] In this invention, the server includes means for collecting the user's voice and visual data and analyzing their emotional state; means for dynamically adjusting an individually optimized learning plan based on the emotional state; and means for providing the adjusted learning plan to the user and providing real-time feedback according to the emotional state. This makes it possible to provide an appropriate learning experience that is in line with the user's emotions and to improve learning effectiveness.

[0805] "Voice data" refers to information about the user's voice that is analyzed using speech recognition technology.

[0806] "Visual data" refers to information about images or videos acquired to capture the user's facial expressions and movements.

[0807] "Emotional state" refers to the results of a real-time assessment of the user's psychological or emotional state.

[0808] A "personally optimized learning plan" refers to learning content and teaching methods that are dynamically adjusted based on the user's abilities and emotional state.

[0809] "Feedback" refers to evaluations or advice provided to the user during learning, and is delivered in real time in response to their emotional state.

[0810] "Educational resources" refer to educational materials and learning content that are accessible to support the user's learning.

[0811] "Learning history" refers to information that includes records of the user's learning activities to date, as well as their level of achievement and challenges based on those activities.

[0812] "Emotional history" refers to a record of a user's past emotional states and is data used to adjust learning.

[0813] A "wireless communication protocol" is a set of standards and procedures that define the communication methods used to transfer data wirelessly.

[0814] A "human-machine interface" is a device or means that enables the exchange of information between a user and a system.

[0815] This invention implements a language learning system equipped with an emotion engine using a consumer robot. The system mainly includes hardware for processing audio and visual data, and software for performing emotion analysis.

[0816] Specifically, the robot is equipped with a high-quality microphone and camera to acquire audio and visual data from the user. The audio data is converted into text data using a speech recognition system such as the Google Speech-to-Text API. The visual data is analyzed in real time by facial recognition systems such as OpenCV and the Microsoft Face API.

[0817] The server processes this data using an emotion engine (e.g., Python's EmotionRecognition library) to evaluate the user's emotional state. Based on the evaluation, it generates an individually optimized learning plan and dynamically adjusts it. The learning plan and feedback are provided to the user in an appropriate format, and the server presents this to the user via a human-machine interface.

[0818] For example, if the emotion engine detects that a user is stressed while learning a language, the system will play a relaxing video on the robot's display. This allows the user to continue learning in a more relaxed state.

[0819] When using generative AI models to generate content and adjust learning, prompts such as "Generate feedback that suggests relaxation when the user is tired" are used. This prompt supports the generation of appropriate feedback based on the user's emotional state.

[0820] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0821] Step 1:

[0822] The device acquires audio and visual data from the user in real time. Input consists of the user's speech and facial expressions, and data is collected using a microphone and camera. The collected data is sent to a server, where the audio data is prepared for conversion into text data.

[0823] Step 2:

[0824] The server converts the input audio data into text data using a speech recognition system. It utilizes technologies such as the Google Speech-to-Text API to convert speech into text strings and generate the text data necessary for subsequent processing. The output is text data representing what the user said.

[0825] Step 3:

[0826] The server analyzes the user's emotional state using visual data. Using OpenCV and the Microsoft Face API, it reads the user's facial expressions from the acquired visual data and inputs them into the emotion analysis engine. The output is data indicating the analyzed emotional state of the user.

[0827] Step 4:

[0828] The server uses an emotion analysis engine to evaluate the user's emotional state and generate an individually optimized learning plan. For example, it uses the Python EmotionRecognition library to process emotional data in real time and adjust the learning plan based on the user's psychological state. The output is the adjusted learning plan.

[0829] Step 5:

[0830] The server provides the generated learning plan to the user via the terminal. The user's display shows the tailored learning materials and information, and provides voice-based feedback as needed. The input is the learning plan generated in step 4, and the output is the learning content presented to the user.

[0831] Step 6:

[0832] Users can progress through their learning based on the presented learning content. Based on the results, feedback is provided in real time, and prompts are generated using a generative AI model as needed. For example, a prompt such as "Generate feedback suggesting relaxation if the user is tired" might be used. The output is appropriate feedback tailored to the user's learning progress.

[0833] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0834] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0835] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0836] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0837] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0838] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0839] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0840] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0841] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0842] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0843] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0844] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0845] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0846] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0847] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0848] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0849] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0850] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0851] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0852] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0853] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0854] The following is further disclosed regarding the embodiments described above.

[0855] (Claim 1)

[0856] A means of collecting user voice and text data and evaluating their abilities and progress,

[0857] A means for generating an individually optimized learning plan based on the aforementioned evaluation,

[0858] A means of providing the user with the aforementioned learning plan and providing real-time feedback according to their progress,

[0859] A means of accessing diverse learning materials and selecting learning resources according to interests and skill levels,

[0860] A means of analyzing the user's learning history, identifying weaknesses, and providing suggestions to encourage efficient improvement,

[0861] A system that includes this.

[0862] (Claim 2)

[0863] The system according to claim 1, further comprising means for using a communication protocol to receive the aforementioned voice and text data.

[0864] (Claim 3)

[0865] The system according to claim 1, further comprising a display device for visually presenting the real-time feedback.

[0866] "Example 1"

[0867] (Claim 1)

[0868] A means of collecting user voice and text data and evaluating their abilities and progress,

[0869] A means for generating an individually optimized learning plan using a generated AI model based on the aforementioned evaluation,

[0870] A means for providing the user with the aforementioned learning plan and for performing real-time analysis and feedback according to progress,

[0871] A means of accessing diverse educational resources and selecting learning resources according to interests and skill levels,

[0872] A method for analyzing received data using speech recognition technology, identifying weaknesses, and making suggestions to promote efficient improvement,

[0873] A system that includes this.

[0874] (Claim 2)

[0875] The system according to claim 1, further comprising means for using a communication protocol to receive voice and text data.

[0876] (Claim 3)

[0877] The system according to claim 1, further comprising a display device for visually presenting real-time feedback.

[0878] "Application Example 1"

[0879] (Claim 1)

[0880] A means of collecting user voice and text data and evaluating their abilities and progress,

[0881] A means for generating an individually optimized learning plan based on the aforementioned evaluation,

[0882] A means of providing the user with the aforementioned learning plan and providing real-time feedback according to their progress,

[0883] A means of accessing diverse learning materials and selecting learning resources according to interests and skill levels,

[0884] A means of analyzing the user's learning history, identifying weaknesses, and providing suggestions to encourage efficient improvement,

[0885] Means of providing conversational phrases and information relevant to the user based on geographical information,

[0886] A means of conducting pronunciation scoring and evaluation and providing feedback,

[0887] A system that includes this.

[0888] (Claim 2)

[0889] The system according to claim 1, further comprising means for using a communication protocol to receive the aforementioned voice and text data.

[0890] (Claim 3)

[0891] The system according to claim 1, further comprising a display device for visually presenting the real-time feedback, and for presenting content related to geographical information.

[0892] "Example 2 of combining an emotion engine"

[0893] (Claim 1)

[0894] A means of collecting user voice, visual, and text information and evaluating their abilities and progress,

[0895] A means for generating an individually optimized learning plan based on the aforementioned evaluation and the user's emotional state,

[0896] A means for providing the user with the aforementioned learning plan and for providing adaptive feedback in real time according to progress and emotional state,

[0897] Means for accessing diverse learning resources and selecting learning information sources according to interests, abilities, and emotional states,

[0898] A means of analyzing the user's learning history and emotional data to identify weaknesses and provide suggestions for efficient improvement,

[0899] A system that includes this.

[0900] (Claim 2)

[0901] The system according to claim 1, further comprising means for using communication procedures to receive audio, visual, and text information.

[0902] (Claim 3)

[0903] The system according to claim 1, further comprising a display device for visually and audibly presenting the real-time feedback.

[0904] "Application example 2 when combining with an emotional engine"

[0905] (Claim 1)

[0906] A means for collecting user voice and visual data and analyzing their emotional state,

[0907] A means for dynamically adjusting the learning plan, which is individually optimized based on the aforementioned emotional state,

[0908] A means of providing the user with the adjusted learning plan and providing real-time feedback according to their emotional state,

[0909] A means of accessing diverse educational resources and selecting learning resources that match emotional and skill levels,

[0910] A means of analyzing a user's learning history and emotional history to make suggestions for efficiently improving their learning,

[0911] A system that includes this.

[0912] (Claim 2)

[0913] The system according to claim 1, further comprising means for using a wireless communication protocol to receive the aforementioned audio data and visual data.

[0914] (Claim 3)

[0915] The system according to claim 1, further comprising a human-machine interface for presenting the aforementioned feedback visually and audibly. [Explanation of Symbols]

[0916] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting user voice and text data and evaluating their abilities and progress, A means for generating an individually optimized learning plan based on the aforementioned evaluation, A means of providing the user with the aforementioned learning plan and providing real-time feedback according to their progress, A means of accessing diverse learning materials and selecting learning resources according to interests and skill levels, A means of analyzing the user's learning history, identifying weaknesses, and providing suggestions to encourage efficient improvement, A system that includes this.

2. The system according to claim 1, further comprising means for using a communication protocol to receive the aforementioned voice and text data.

3. The system according to claim 1, further comprising a display device for visually presenting the real-time feedback.

Images