system

Abstract

The system according to this embodiment aims to efficiently learn phrases and words that were not understood while watching a drama. [Solution] The system according to the embodiment comprises a detection unit, a generation unit, a question unit, and a provision unit. The detection unit detects phrases and words that the user did not understand while watching a drama. The generation unit generates an interactive quiz based on the phrases and words detected by the detection unit. The question unit presents the quiz generated by the generation unit in real time using a chatbot function. The provision unit provides answers and explanations for the quiz presented by the question unit.

Description

Technical Field

[0006] , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, there is a problem that there is a lack of means for efficiently learning phrases and words that could not be understood during drama viewing.

[0005] The system according to the embodiment aims to efficiently learn phrases and words that could not be understood during drama viewing.

Means for Solving the Problems

[0006] The system according to this embodiment comprises a detection unit, a generation unit, a question unit, and a provision unit. The detection unit detects phrases and words that the user did not understand while watching a drama. The generation unit generates interactive quizzes based on the phrases and words detected by the detection unit. The question unit presents the quizzes generated by the generation unit in real time using a chatbot function. The provision unit provides answers and explanations for the quizzes presented by the question unit. [Effects of the Invention]

[0007] The system according to this embodiment allows for efficient learning of phrases and words that were not understood while watching a drama. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10]This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F manages communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 1 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 comprises a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The receiving device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The language learning support system according to an embodiment of the present invention is a system that supports language learning while watching a drama and after watching it, utilizing generative AI technology. In this system, when a user watches a drama, the generative AI analyzes the content of the drama being watched and automatically detects phrases and words that the user did not understand. Next, the generative AI generates an interactive quiz based on the phrases and words it detected. The quiz is mainly composed of phrases and words that the user did not understand, allowing the user to learn their meaning and usage. The generated quiz is presented in real time using a chatbot function, and the user can answer the quiz through the chatbot and receive answers and explanations. This system allows users to advance language learning while watching dramas, promoting retention of the learned content. Furthermore, by utilizing a chatbot, a real-time interactive learning experience is provided, increasing user motivation. For example, by answering quizzes while watching a drama, users can gain a deeper understanding of the content they are watching and improve the learning effect. Specifically, when a user watches a drama, the generative AI uses speech recognition technology to analyze the user's reactions and identify phrases and words that the user did not understand. Next, based on the phrases and words identified by the generating AI, interactive quizzes are created, including multiple-choice questions and questions asking for word meanings. The generated quizzes are presented in real time using the chatbot function, and users can answer the quizzes through the chatbot and receive answers and explanations. This system allows users to learn a language while watching dramas, promoting retention of the learned material. Furthermore, the use of the chatbot provides a real-time, interactive learning experience, increasing user motivation. In summary, this language learning support system allows users to learn a language while watching dramas, promoting retention of the learned material.

[0029] The language learning support system according to this embodiment comprises a detection unit, a generation unit, a question unit, and a provision unit. The detection unit detects phrases and words that the user did not understand while watching a drama. The detection unit analyzes the user's reactions, for example, using speech recognition technology, to identify the phrases and words that were not understood. The generation unit generates interactive quizzes based on the phrases and words detected by the detection unit. The generation unit generates interactive quizzes, for example, multiple-choice questions and questions asking for the meaning of words. The question unit presents the quizzes generated by the generation unit in real time using a chatbot function. The question unit presents the quizzes in real time using a chatbot function. The provision unit provides answers and explanations for the quizzes presented by the question unit. The provision unit provides answers and explanations for the quizzes, for example, through a chatbot. As a result, the language learning support system according to this embodiment allows the user to progress with language learning while watching a drama, and promotes the retention of learned content.

[0030] The detection unit detects phrases and words that the user did not understand while watching a drama. Specifically, it uses speech recognition technology to analyze the user's reactions and identify the phrases and words that were not understood. The speech recognition technology analyzes the user's speech and reactions in real time and evaluates the level of understanding of specific phrases and words. For example, if the user pauses or rewinds when they hear a particular phrase, it is determined that the user did not understand that phrase. It can also analyze what the user says and detect questions or doubts about specific words or phrases. Furthermore, the detection unit can use eye-tracking technology to measure the time the user spends looking at subtitles or specific parts of the screen and evaluate the level of understanding. This allows the detection unit to accurately identify where the user had difficulty understanding and use this information to support learning in the next step.

[0031] The generation unit generates interactive quizzes based on phrases and words detected by the detection unit. Specifically, it generates interactive quizzes such as multiple-choice questions and questions asking for word meanings. The generation unit uses AI to generate optimal quizzes according to the user's level of understanding and learning progress. For example, for phrases the user did not understand, it generates multiple-choice questions asking for the meaning of that phrase. In questions asking for word meanings, it generates questions that present the definition and usage of the word, and ask the user to select the correct meaning. Furthermore, the generation unit can analyze the user's past learning history and performance data to provide quizzes optimized for each individual user. As a result, the generation unit can provide effective learning support to improve the user's understanding.

[0032] The question-generating unit presents quizzes generated by the generation unit in real time using a chatbot function. Specifically, it uses a chatbot function to present quizzes in real time. The question-generating unit presents quizzes at appropriate times while the user is watching the drama, supporting the user's learning. For example, by presenting a quiz about a phrase that the user did not understand immediately after it appears, the learning effect can be enhanced. Furthermore, the question-generating unit can dynamically adjust the next quiz according to the user's answer, providing learning tailored to the user's level of understanding. In this way, the question-generating unit helps users effectively advance their language learning while watching the drama.

[0033] The service provider provides answers and explanations for quizzes created by the question provider. Specifically, it provides answers and explanations for quizzes via a chatbot. After a user answers a quiz, the service provider provides the correct answer and its explanation in real time to deepen the user's understanding. For example, after a user answers a multiple-choice question, it provides a detailed explanation of the correct answer and the reason why. In questions asking for the meaning of a word, it presents the correct meaning and examples of how the word is used, helping the user understand the word in a real-world context. Furthermore, the service provider can analyze the user's answer history, evaluate their level of understanding of specific fields and themes, and provide feedback that is useful for the next learning step. In this way, the service provider can help users effectively retain what they have learned and maximize the results of their language learning.

[0034] The detection unit can analyze the user's responses using speech recognition technology and identify phrases or words that were not understood. For example, when a user is watching a drama, the generation AI uses speech recognition technology to analyze the user's responses and identify phrases or words that were not understood. This allows for the accurate identification of phrases or words that were not understood by analyzing the user's responses. Some or all of the above processing in the detection unit may be performed using the generation AI, or it may be performed without the generation AI. For example, the detection unit can input the user's voice data into the generation AI, which can then analyze the voice data to identify phrases or words that were not understood.

[0035] The generation unit can generate interactive quizzes, such as multiple-choice questions and questions asking for the meaning of words. For example, the generation unit generates interactive quizzes, such as multiple-choice questions and questions asking for the meaning of words, based on phrases and words detected by the detection unit. This allows for the generation of effective quizzes based on phrases and words that the user did not understand. Some or all of the above-described processes in the generation unit may be performed using a generation AI, or they may be performed without a generation AI. For example, the generation unit can input phrases and words identified by the detection unit into a generation AI, and the generation AI can generate an interactive quiz.

[0036] The question-generating unit can generate quizzes in real time using a chatbot function. For example, the question-generating unit can generate quizzes in real time using the chatbot function. This allows users to answer quizzes in real time, enhancing the learning effect. Some or all of the above-described processes in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input quizzes generated by the generation unit into an AI, which can then use a chatbot function to generate quizzes in real time.

[0037] The service provider can provide answers and explanations to quizzes through a chatbot. For example, the service provider can provide answers and explanations to quizzes created by the question provider through a chatbot. This allows users to receive answers and explanations to quizzes in real time, deepening their understanding of the learning material. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input answers and explanations to quizzes created by the question provider into an AI, which can then provide answers and explanations through a chatbot.

[0038] The detection unit can analyze the user's viewing history and prioritize the detection of phrases and words that the user previously did not understand. For example, based on the user's viewing history of dramas, the detection unit can list phrases and words that the user frequently did not understand and prioritize their detection. The detection unit can also prioritize the detection of phrases and words that the user previously did not understand in dramas of a specific genre when the user is watching that genre. Furthermore, the detection unit can record phrases and words that the user did not understand for each episode the user has watched and prioritize their detection during subsequent viewings. This allows the detection unit to prioritize the detection of phrases and words that the user did not understand based on their past viewing history. Some or all of the above processing in the detection unit may be performed using a generation AI, or it may be performed without a generation AI. For example, the detection unit can input the user's viewing history data into a generation AI, which can then analyze the viewing history and prioritize the detection of phrases and words that the user did not understand.

[0039] The detection unit can apply different detection algorithms to each scene in the drama to detect phrases and words appropriate to the scene. For example, in an action scene, the detection unit can apply an algorithm that prioritizes detecting phrases and words related to actions. In a romance scene, the detection unit can also apply an algorithm that prioritizes detecting phrases and words related to emotional expression. Furthermore, in a comedy scene, the detection unit can apply an algorithm that prioritizes detecting phrases and words related to humor and jokes. This allows for the detection of appropriate phrases and words for each scene. Some or all of the above processing in the detection unit may be performed using a generative AI, or it may be performed without a generative AI. For example, the detection unit can input drama scene data into a generative AI, which can then apply different detection algorithms to each scene to detect phrases and words.

[0040] The detection unit can prioritize the detection of region-specific phrases and words based on the user's geographical location information. For example, if the user is in a specific region, the detection unit will prioritize the detection of the local dialect and unique expressions. Furthermore, if the user is traveling, the detection unit can prioritize the detection of phrases and words commonly used in the visited region. Additionally, if the user is in a different country, the detection unit can prioritize the detection of phrases and words related to the culture and customs of that country. This prioritizes the detection of region-specific phrases and words, thereby enhancing the user's learning effectiveness. Some or all of the above processing in the detection unit may be performed using a generative AI, or it may be performed without a generative AI. For example, the detection unit can input the user's geographical location information into a generative AI, which can then prioritize the detection of region-specific phrases and words.

[0041] The detection unit can analyze a user's social media activity and detect relevant phrases and words. For example, the detection unit can detect relevant phrases and words based on phrases and words that the user frequently uses on social media. The detection unit can also analyze the content of posts from accounts that the user follows and detect relevant phrases and words. Furthermore, the detection unit can detect relevant phrases and words based on topics in online communities that the user participates in. This allows for the detection of relevant phrases and words based on social media activity. Some or all of the above processing in the detection unit may be performed using a generative AI, or it may be performed without a generative AI. For example, the detection unit can input the user's social media activity data into a generative AI, which can then analyze the social media activity and detect relevant phrases and words.

[0042] The generation unit can select the optimal quiz format by referring to the user's learning history when generating quizzes. For example, the generation unit can select the optimal quiz format based on the user's past quiz answer history. The generation unit can also prioritize selecting quiz formats in which the user excels. Furthermore, the generation unit can avoid selecting quiz formats in which the user struggles. This allows the generation unit to provide the optimal quiz format based on the user's learning history. Some or all of the above processing in the generation unit may be performed using a generation AI, or it may be performed without a generation AI. For example, the generation unit can input the user's learning history data into a generation AI, which can then analyze the learning history and select the optimal quiz format.

[0043] The generation unit can apply different quiz formats depending on the genre of the drama when generating quizzes. For example, in the case of an action drama, the generation unit can apply quiz formats related to action. In the case of a romance drama, the generation unit can also apply quiz formats related to emotional expression. Furthermore, in the case of a comedy drama, the generation unit can apply quiz formats related to humor and jokes. This allows for the provision of appropriate quiz formats according to the genre of the drama. Some or all of the above processing in the generation unit may be performed using a generation AI, or it may be performed without a generation AI. For example, the generation unit can input drama genre data into a generation AI, and the generation AI can generate quizzes by applying quiz formats appropriate to the genre.

[0044] The generation unit can adjust the frequency of quiz presentations based on the user's viewing time when generating quizzes. For example, if the user is watching at night, the generation AI can set a lower frequency. The generation unit can also set a higher frequency if the user is watching during the day. Furthermore, if the user is watching on weekends, the generation AI can set a moderate frequency. This allows quizzes to be presented at an appropriate frequency depending on the user's viewing time. Some or all of the above processing in the generation unit may be performed using the generation AI, or it may be performed without using the generation AI. For example, the generation unit can input user viewing time data into the generation AI, and the generation AI can adjust the frequency of quiz presentations based on the viewing time.

[0045] The generation unit can customize the quiz content based on the user's interests when generating quizzes. For example, if the user is interested in a particular genre, the generation unit can generate a quiz related to that genre. Furthermore, if the user is interested in a particular character, the generation unit can generate a quiz related to that character. In addition, if the user is interested in a particular episode, the generation unit can generate a quiz related to that episode. This allows for the provision of appropriate quiz content according to the user's interests. Some or all of the above-described processes in the generation unit may be performed using a generation AI, or they may be performed without a generation AI. For example, the generation unit can input user interest data into a generation AI, which can then customize the quiz content based on those interests.

[0046] The question-generating unit can select the optimal question format by referring to the user's past quiz answer history when presenting a question. For example, the question-generating unit can present questions in a similar format based on the format of questions the user has answered correctly in the past. It can also present questions in a different format based on the format of questions the user has answered incorrectly in the past. Furthermore, the question-generating unit can prioritize presenting questions from genres in which the user has excelled in the past. This allows the system to provide the optimal question format based on the user's past quiz answer history. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input the user's past quiz answer history data into a generating AI, which can then analyze the answer history and select the optimal question format.

[0047] The question-generating unit can select the optimal question format by considering the user's device information when presenting questions. For example, if the user is using a smartphone, the unit can provide a question format that matches the screen size. Furthermore, if the user is using a tablet, the unit can provide a question format optimized for a larger screen. Additionally, if the user is using a smartwatch, the unit can provide a concise and highly visible question format. This allows the system to provide the optimal question format according to the user's device information. Some or all of the above processing in the question-generating unit may be performed using AI, or not. For example, the question-generating unit can input the user's device information into a generating AI, which can then analyze the device information and select the optimal question format.

[0048] The question-generating unit can adjust its question presentation method when presenting questions, taking into account the user's viewing environment (e.g., volume, screen brightness). For example, if the user has set the volume low, the question-generating unit can provide a question presentation method that includes many visual hints. It can also provide a question presentation method with high contrast if the user has set the screen brightness low. Furthermore, if the user is using headphones, the question-generating unit can provide a question presentation method that includes many audio elements. This allows the system to provide the optimal question presentation method according to the user's viewing environment. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input the user's viewing environment data into a generating AI, which can then analyze the viewing environment and select the optimal question presentation method.

[0049] The question-generating unit can analyze the user's viewing behavior (e.g., pausing, rewinding) and adjust the timing of question presentation. For example, the generation AI can present a quiz when the user pauses. The question-generating unit can also present a quiz related to the rewound portion when the user rewinds. Furthermore, if the user fast-forwards, the question-generating unit can later present a quiz related to the fast-forwarded portion. This allows quizzes to be presented at appropriate times according to the user's viewing behavior. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input user viewing behavior data into the generation AI, which can then analyze the behavior and adjust the timing of question presentation.

[0050] The service provider can select the optimal delivery method by referring to the user's past learning history when providing answers and explanations. For example, the service provider can provide explanations in a similar format to those the user found easy to understand in the past. Alternatively, the service provider can avoid explanation formats that the user found difficult in the past and provide explanations in a different format. Furthermore, the service provider can prioritize providing explanations in genres that the user has excelled in the past. This allows the service provider to provide the optimal delivery method based on the user's past learning history. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input the user's past learning history data into a generating AI, which can then analyze the learning history and select the optimal delivery method.

[0051] The information provider can provide additional relevant information based on the user's viewing history when providing answers and explanations. For example, the information provider can provide background information related to the drama episodes the user has watched. It can also provide cultural information related to the scenes the user has watched. Furthermore, it can provide detailed information related to the characters the user has watched. This allows the information provider to provide additional relevant information based on the user's viewing history. Some or all of the above processing in the information provider may be performed using AI or not. For example, the information provider can input the user's viewing history data into a generating AI, which can then analyze the viewing history and provide additional relevant information.

[0052] The service provider can select the optimal format for providing answers and explanations, taking into account the user's device information. For example, if the user is using a smartphone, the service provider can provide a format that matches the screen size. Furthermore, if the user is using a tablet, the service provider can provide a format optimized for a larger screen. Additionally, if the user is using a smartwatch, the service provider can provide a concise and highly visible format. This allows the service provider to provide the optimal format according to the user's device information. Some or all of the above processing in the service provider may be performed using AI, or not. For example, the service provider can input the user's device information into a generating AI, which can then analyze the device information and select the optimal format.

[0053] The service provider can analyze the user's viewing behavior (e.g., pausing, rewinding) when providing answers and explanations and adjust the timing of the provision. For example, the service provider can have the generating AI provide the answers and explanations when the user pauses. The service provider can also provide answers and explanations related to the rewound portion when the user rewinds. Furthermore, if the user fast-forwards, the service provider can later provide answers and explanations related to the fast-forwarded portion. This allows the service provider to provide answers and explanations at the appropriate timing according to the user's viewing behavior. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input user viewing behavior data into the generating AI, which can then analyze the behavior and adjust the timing of the provision.

[0054] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0055] A language learning support system can track a user's learning progress and provide personalized learning plans. For example, a detection unit records phrases and words the user previously did not understand and evaluates their learning progress. Next, a generation unit generates quizzes suggesting what the user should learn next, based on their progress. Furthermore, a delivery unit can provide a learning plan tailored to the user's progress via a chatbot. This allows users to learn efficiently at their own pace.

[0056] The language learning support system can provide customized feedback according to the user's learning style. For example, the detection unit determines whether the user is a visual or auditory learner. Next, the generation unit generates quizzes that heavily utilize images and videos for visual learners, and quizzes that heavily utilize audio for auditory learners. Furthermore, the delivery unit can provide feedback tailored to the user's learning style through a chatbot. This allows users to learn in a way that is best suited to them.

[0057] The language learning support system can set individual learning goals based on the user's learning history. For example, the detection unit records the content and results of what the user has learned in the past. Next, the generation unit generates a quiz that sets the next learning goal to be achieved, based on the user's learning history. Furthermore, the provision unit provides the user's learning goals through a chatbot and can provide real-time feedback on progress. This allows the user to proceed with their learning with concrete goals in mind.

[0058] A language learning support system can analyze a user's learning environment and suggest the optimal learning method. For example, a detection unit records the location and time of the user's learning. Next, a generation unit generates a quiz that suggests the optimal learning method according to the user's learning environment. Furthermore, a delivery unit provides these suggestions via a chatbot, supporting the user so that they can proceed with their learning in the optimal environment. This allows the user to learn efficiently.

[0059] The language learning support system can suggest review timings based on the user's learning history. For example, the detection unit records the content and results of what the user has learned in the past. Next, the generation unit generates quizzes to encourage review at times when the user is likely to forget. Furthermore, the delivery unit provides these quizzes through a chatbot to support the user in effectively reviewing the material. As a result, the user can retain the learned content for a long period of time.

[0060] The language learning support system can generate individual learning reports based on the user's learning history. For example, the detection unit records the content and results of what the user has learned in the past. Next, the generation unit generates a report showing the user's learning progress and achievement level based on the user's learning history. Furthermore, the delivery unit provides these reports via a chatbot to support the user in understanding their learning status. This allows the user to check their learning results and set their next learning goals.

[0061] The following briefly describes the processing flow for example form 1.

[0062] Step 1: The detection unit detects phrases and words that the user did not understand while watching the drama. For example, it uses speech recognition technology to analyze the user's reactions and identify the phrases and words that were not understood. Step 2: The generation unit generates interactive quizzes based on the phrases and words detected by the detection unit. For example, it generates interactive quizzes such as multiple-choice questions and questions asking for the meaning of words. Step 3: The question-generating unit presents the quiz generated by the generation unit in real time using a chatbot function. For example, the quiz is presented in real time using a chatbot function. Step 4: The provider provides answers and explanations for the quiz questions posed by the question-setting unit. For example, the provider might provide answers and explanations for the quiz through a chatbot.

[0063] (Example of form 2) The language learning support system according to an embodiment of the present invention is a system that supports language learning while watching a drama and after watching it, utilizing generative AI technology. In this system, when a user watches a drama, the generative AI analyzes the content of the drama being watched and automatically detects phrases and words that the user did not understand. Next, the generative AI generates an interactive quiz based on the phrases and words it detected. The quiz is mainly composed of phrases and words that the user did not understand, allowing the user to learn their meaning and usage. The generated quiz is presented in real time using a chatbot function, and the user can answer the quiz through the chatbot and receive answers and explanations. This system allows users to advance language learning while watching dramas, promoting retention of the learned content. Furthermore, by utilizing a chatbot, a real-time interactive learning experience is provided, increasing user motivation. For example, by answering quizzes while watching a drama, users can gain a deeper understanding of the content they are watching and improve the learning effect. Specifically, when a user watches a drama, the generative AI uses speech recognition technology to analyze the user's reactions and identify phrases and words that the user did not understand. Next, the system generates interactive quizzes, such as multiple-choice questions and questions asking for word meanings, based on the phrases and words identified by the generating AI. The generated quizzes are presented in real time using a chatbot function, and users can answer the quizzes through the chatbot and receive answers and explanations. This system allows users to learn a language while watching dramas, promoting retention of the learned material. Furthermore, the use of a chatbot provides a real-time, interactive learning experience, increasing user motivation. In summary, this language learning support system allows users to learn a language while watching dramas, promoting retention of the learned material.

[0064] The language learning support system according to this embodiment comprises a detection unit, a generation unit, a question unit, and a provision unit. The detection unit detects phrases and words that the user did not understand while watching a drama. The detection unit analyzes the user's reactions, for example, using speech recognition technology, to identify the phrases and words that were not understood. The generation unit generates interactive quizzes based on the phrases and words detected by the detection unit. The generation unit generates interactive quizzes, for example, multiple-choice questions and questions asking for the meaning of words. The question unit presents the quizzes generated by the generation unit in real time using a chatbot function. The question unit presents the quizzes in real time using a chatbot function. The provision unit provides answers and explanations for the quizzes presented by the question unit. The provision unit provides answers and explanations for the quizzes, for example, through a chatbot. As a result, the language learning support system according to this embodiment allows the user to progress with language learning while watching a drama, and promotes the retention of learned content.

[0065] The detection unit detects phrases and words that the user did not understand while watching a drama. Specifically, it uses speech recognition technology to analyze the user's reactions and identify the phrases and words that were not understood. The speech recognition technology analyzes the user's speech and reactions in real time and evaluates the level of understanding of specific phrases and words. For example, if the user pauses or rewinds when they hear a particular phrase, it is determined that the user did not understand that phrase. It can also analyze what the user says and detect questions or doubts about specific words or phrases. Furthermore, the detection unit can use eye-tracking technology to measure the time the user spends looking at subtitles or specific parts of the screen and evaluate the level of understanding. This allows the detection unit to accurately identify where the user had difficulty understanding and use this information to support learning in the next step.

[0066] The generation unit generates interactive quizzes based on phrases and words detected by the detection unit. Specifically, it generates interactive quizzes such as multiple-choice questions and questions asking for word meanings. The generation unit uses AI to generate optimal quizzes according to the user's level of understanding and learning progress. For example, for phrases the user did not understand, it generates multiple-choice questions asking for the meaning of that phrase. In questions asking for word meanings, it generates questions that present the definition and usage of the word, and ask the user to select the correct meaning. Furthermore, the generation unit can analyze the user's past learning history and performance data to provide quizzes optimized for each individual user. As a result, the generation unit can provide effective learning support to improve the user's understanding.

[0067] The question-generating unit presents quizzes generated by the generation unit in real time using a chatbot function. Specifically, it uses a chatbot function to present quizzes in real time. The question-generating unit presents quizzes at appropriate times while the user is watching the drama, supporting the user's learning. For example, by presenting a quiz about a phrase that the user did not understand immediately after it appears, the learning effect can be enhanced. Furthermore, the question-generating unit can dynamically adjust the next quiz according to the user's answer, providing learning tailored to the user's level of understanding. In this way, the question-generating unit helps users effectively advance their language learning while watching the drama.

[0068] The service provider provides answers and explanations for quizzes created by the question provider. Specifically, it provides answers and explanations for quizzes via a chatbot. After a user answers a quiz, the service provider provides the correct answer and its explanation in real time to deepen the user's understanding. For example, after a user answers a multiple-choice question, it provides a detailed explanation of the correct answer and the reason why. In questions asking for the meaning of a word, it presents the correct meaning and examples of how the word is used, helping the user understand the word in a real-world context. Furthermore, the service provider can analyze the user's answer history, evaluate their level of understanding of specific fields and themes, and provide feedback that is useful for the next learning step. In this way, the service provider can help users effectively retain what they have learned and maximize the results of their language learning.

[0069] The detection unit can analyze the user's responses using speech recognition technology and identify phrases or words that were not understood. For example, when a user is watching a drama, the generation AI uses speech recognition technology to analyze the user's responses and identify phrases or words that were not understood. This allows for the accurate identification of phrases or words that were not understood by analyzing the user's responses. Some or all of the above processing in the detection unit may be performed using the generation AI, or it may be performed without the generation AI. For example, the detection unit can input the user's voice data into the generation AI, which can then analyze the voice data to identify phrases or words that were not understood.

[0070] The generation unit can generate interactive quizzes, such as multiple-choice questions and questions asking for the meaning of words. For example, the generation unit generates interactive quizzes, such as multiple-choice questions and questions asking for the meaning of words, based on phrases and words detected by the detection unit. This allows for the generation of effective quizzes based on phrases and words that the user did not understand. Some or all of the above-described processes in the generation unit may be performed using a generation AI, or they may be performed without a generation AI. For example, the generation unit can input phrases and words identified by the detection unit into a generation AI, and the generation AI can generate an interactive quiz.

[0071] The question-generating unit can generate quizzes in real time using a chatbot function. For example, the question-generating unit can generate quizzes in real time using the chatbot function. This allows users to answer quizzes in real time, enhancing the learning effect. Some or all of the above-described processes in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input quizzes generated by the generation unit into an AI, which can then use a chatbot function to generate quizzes in real time.

[0072] The service provider can provide answers and explanations to quizzes through a chatbot. For example, the service provider can provide answers and explanations to quizzes created by the question provider through a chatbot. This allows users to receive answers and explanations to quizzes in real time, deepening their understanding of the learning material. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input answers and explanations to quizzes created by the question provider into an AI, which can then provide answers and explanations through a chatbot.

[0073] The detection unit can estimate the user's emotions and adjust the detection accuracy of phrases and words it could not understand based on the estimated emotions. For example, if the user is stressed, the detection unit's generating AI can increase detection accuracy and detect more phrases and words. Conversely, if the user is relaxed, the detection unit can lower the detection accuracy of the generating AI and focus on detecting only important phrases and words. Furthermore, if the user is focused, the detection unit can set the detection accuracy of the generating AI to a moderate level and detect a moderate amount of phrases and words. This allows for the detection of more appropriate phrases and words by adjusting the detection accuracy according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generating AI. The generating AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI. Some or all of the above processing in the detection unit may be performed using AI or not. For example, the detection unit can input user emotion data into the generating AI, which can estimate the emotions and adjust the detection accuracy.

[0074] The detection unit can analyze the user's viewing history and prioritize the detection of phrases and words that the user previously did not understand. For example, based on the user's viewing history of dramas, the detection unit can list phrases and words that the user frequently did not understand and prioritize their detection. The detection unit can also prioritize the detection of phrases and words that the user previously did not understand in dramas of a specific genre when the user is watching that genre. Furthermore, the detection unit can record phrases and words that the user did not understand for each episode the user has watched and prioritize their detection during subsequent viewings. This allows the detection unit to prioritize the detection of phrases and words that the user did not understand based on their past viewing history. Some or all of the above processing in the detection unit may be performed using a generation AI, or it may be performed without a generation AI. For example, the detection unit can input the user's viewing history data into a generation AI, which can then analyze the viewing history and prioritize the detection of phrases and words that the user did not understand.

[0075] The detection unit can apply different detection algorithms to each scene in the drama to detect phrases and words appropriate to the scene. For example, in an action scene, the detection unit can apply an algorithm that prioritizes detecting phrases and words related to actions. In a romance scene, the detection unit can also apply an algorithm that prioritizes detecting phrases and words related to emotional expression. Furthermore, in a comedy scene, the detection unit can apply an algorithm that prioritizes detecting phrases and words related to humor and jokes. This allows for the detection of appropriate phrases and words for each scene. Some or all of the above processing in the detection unit may be performed using a generative AI, or it may be performed without a generative AI. For example, the detection unit can input drama scene data into a generative AI, which can then apply different detection algorithms to each scene to detect phrases and words.

[0076] The detection unit can estimate the user's emotions and determine the priority of phrases and words to detect based on the estimated emotions. For example, if the user is excited, the generation AI will prioritize detecting phrases and words related to that emotion. Similarly, if the user is sad, the generation AI can prioritize detecting phrases and words related to comfort or encouragement. Furthermore, if the user is happy, the generation AI can prioritize detecting phrases and words related to humor or enjoyment. This allows for the priority detection of appropriate phrases and words according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generation AI. The generation AI may be, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the detection unit may be performed using AI or not. For example, the detection unit can input user emotion data into the generation AI, which can estimate the emotion and determine the priority of phrases and words to detect.

[0077] The detection unit can prioritize the detection of region-specific phrases and words based on the user's geographical location information. For example, if the user is in a specific region, the detection unit will prioritize the detection of the local dialect and unique expressions. Furthermore, if the user is traveling, the detection unit can prioritize the detection of phrases and words commonly used in the visited region. Additionally, if the user is in a different country, the detection unit can prioritize the detection of phrases and words related to the culture and customs of that country. This prioritizes the detection of region-specific phrases and words, thereby enhancing the user's learning effectiveness. Some or all of the above processing in the detection unit may be performed using a generative AI, or it may be performed without a generative AI. For example, the detection unit can input the user's geographical location information into a generative AI, which can then prioritize the detection of region-specific phrases and words.

[0078] The detection unit can analyze a user's social media activity and detect relevant phrases and words. For example, the detection unit can detect relevant phrases and words based on phrases and words that the user frequently uses on social media. The detection unit can also analyze the content of posts from accounts that the user follows and detect relevant phrases and words. Furthermore, the detection unit can detect relevant phrases and words based on topics in online communities that the user participates in. This allows for the detection of relevant phrases and words based on social media activity. Some or all of the above processing in the detection unit may be performed using a generative AI, or it may be performed without a generative AI. For example, the detection unit can input the user's social media activity data into a generative AI, which can then analyze the social media activity and detect relevant phrases and words.

[0079] The generation unit can estimate the user's emotions and adjust the difficulty of the quiz based on the estimated emotions. For example, if the user is relaxed, the generation AI can generate a difficult quiz. If the user is stressed, the generation unit can also generate an easy quiz. Furthermore, if the user is focused, the generation unit can generate a quiz of moderate difficulty. This allows the system to provide quizzes of appropriate difficulty according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generation AI. The generation AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above-described processes in the generation unit may be performed using or without a generation AI. For example, the generation unit can input user emotion data into a generation AI, which can estimate the emotions and adjust the difficulty of the quiz.

[0080] The generation unit can select the optimal quiz format by referring to the user's learning history when generating quizzes. For example, the generation unit can select the optimal quiz format based on the user's past quiz answer history. The generation unit can also prioritize selecting quiz formats in which the user excels. Furthermore, the generation unit can avoid selecting quiz formats in which the user struggles. This allows the generation unit to provide the optimal quiz format based on the user's learning history. Some or all of the above processing in the generation unit may be performed using a generation AI, or it may be performed without a generation AI. For example, the generation unit can input the user's learning history data into a generation AI, which can then analyze the learning history and select the optimal quiz format.

[0081] The generation unit can apply different quiz formats depending on the genre of the drama when generating quizzes. For example, in the case of an action drama, the generation unit can apply quiz formats related to action. In the case of a romance drama, the generation unit can also apply quiz formats related to emotional expression. Furthermore, in the case of a comedy drama, the generation unit can apply quiz formats related to humor and jokes. This allows for the provision of appropriate quiz formats according to the genre of the drama. Some or all of the above processing in the generation unit may be performed using a generation AI, or it may be performed without a generation AI. For example, the generation unit can input drama genre data into a generation AI, and the generation AI can generate quizzes by applying quiz formats appropriate to the genre.

[0082] The generation unit can estimate the user's emotions and adjust the order in which quiz questions are presented based on the estimated emotions. For example, if the user is relaxed, the generation AI can present questions in order from the most difficult to the least difficult. If the user is stressed, the generation AI can present questions in order from the least difficult to the most difficult. Furthermore, if the user is focused, the generation AI can randomly present questions of medium difficulty. This allows the quiz to be presented in an appropriate order according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generation AI. The generation AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the generation unit may be performed using the generation AI or not. For example, the generation unit can input user emotion data into the generation AI, which can estimate the emotions and adjust the order in which quiz questions are presented.

[0083] The generation unit can adjust the frequency of quiz presentations based on the user's viewing time when generating quizzes. For example, if the user is watching at night, the generation AI can set a lower frequency. The generation unit can also set a higher frequency if the user is watching during the day. Furthermore, if the user is watching on weekends, the generation AI can set a moderate frequency. This allows quizzes to be presented at an appropriate frequency depending on the user's viewing time. Some or all of the above processing in the generation unit may be performed using the generation AI, or it may be performed without using the generation AI. For example, the generation unit can input user viewing time data into the generation AI, and the generation AI can adjust the frequency of quiz presentations based on the viewing time.

[0084] The generation unit can customize the quiz content based on the user's interests when generating quizzes. For example, if the user is interested in a particular genre, the generation unit can generate a quiz related to that genre. Furthermore, if the user is interested in a particular character, the generation unit can generate a quiz related to that character. In addition, if the user is interested in a particular episode, the generation unit can generate a quiz related to that episode. This allows for the provision of appropriate quiz content according to the user's interests. Some or all of the above-described processes in the generation unit may be performed using a generation AI, or they may be performed without a generation AI. For example, the generation unit can input user interest data into a generation AI, which can then customize the quiz content based on those interests.

[0085] The question-generating unit can estimate the user's emotions and adjust the timing of quiz presentations based on the estimated emotions. For example, if the user is relaxed, the generating AI will present quizzes frequently. Conversely, if the user is stressed, the generating AI can reduce the frequency of quizzes. Furthermore, if the user is focused, the generating AI can present quizzes at appropriate intervals. This allows quizzes to be presented at the appropriate time according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generating AI. The generating AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input user emotion data into a generating AI, which can estimate emotions and adjust the timing of quiz presentations.

[0086] The question-generating unit can select the optimal question format by referring to the user's past quiz answer history when presenting a question. For example, the question-generating unit can present questions in a similar format based on the format of questions the user has answered correctly in the past. It can also present questions in a different format based on the format of questions the user has answered incorrectly in the past. Furthermore, the question-generating unit can prioritize presenting questions from genres in which the user has excelled in the past. This allows the system to provide the optimal question format based on the user's past quiz answer history. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input the user's past quiz answer history data into a generating AI, which can then analyze the answer history and select the optimal question format.

[0087] The question-generating unit can select the optimal question format by considering the user's device information when presenting questions. For example, if the user is using a smartphone, the unit can provide a question format that matches the screen size. Furthermore, if the user is using a tablet, the unit can provide a question format optimized for a larger screen. Additionally, if the user is using a smartwatch, the unit can provide a concise and highly visible question format. This allows the system to provide the optimal question format according to the user's device information. Some or all of the above processing in the question-generating unit may be performed using AI, or not. For example, the question-generating unit can input the user's device information into a generating AI, which can then analyze the device information and select the optimal question format.

[0088] The question-generating unit can estimate the user's emotions and adjust the order of quiz questions based on the estimated emotions. For example, if the user is relaxed, the generating AI can present questions in order from the most difficult to the least difficult. If the user is stressed, the generating AI can present questions in order from the least difficult to the most difficult. Furthermore, if the user is focused, the generating AI can randomly present questions of medium difficulty. This allows the quiz to be presented in an appropriate order according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generating AI. The generating AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input user emotion data into a generating AI, which can estimate the emotions and adjust the order of quiz questions.

[0089] The question-generating unit can adjust its question presentation method when presenting questions, taking into account the user's viewing environment (e.g., volume, screen brightness). For example, if the user has set the volume low, the question-generating unit can provide a question presentation method that includes many visual hints. It can also provide a question presentation method with high contrast if the user has set the screen brightness low. Furthermore, if the user is using headphones, the question-generating unit can provide a question presentation method that includes many audio elements. This allows the system to provide the optimal question presentation method according to the user's viewing environment. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input the user's viewing environment data into a generating AI, which can then analyze the viewing environment and select the optimal question presentation method.

[0090] The question-generating unit can analyze the user's viewing behavior (e.g., pausing, rewinding) and adjust the timing of question presentation. For example, the generation AI can present a quiz when the user pauses. The question-generating unit can also present a quiz related to the rewound portion when the user rewinds. Furthermore, if the user fast-forwards, the question-generating unit can later present a quiz related to the fast-forwarded portion. This allows quizzes to be presented at appropriate times according to the user's viewing behavior. Some or all of the above processing in the question-generating unit may be performed using AI or not. For example, the question-generating unit can input user viewing behavior data into the generation AI, which can then analyze the behavior and adjust the timing of question presentation.

[0091] The service provider can estimate the user's emotions and adjust the presentation of answers and explanations based on the estimated emotions. For example, if the user is relaxed, the generating AI can provide a detailed explanation. If the user is stressed, the generating AI can provide a concise explanation. Furthermore, if the user is focused, the generating AI can provide an explanation of moderate detail. This allows the service provider to provide answers and explanations in an appropriate presentation style according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generating AI. The generating AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input user emotion data into a generating AI, which can estimate the emotions and adjust the presentation of answers and explanations.

[0092] The service provider can select the optimal delivery method by referring to the user's past learning history when providing answers and explanations. For example, the service provider can provide explanations in a similar format to those the user found easy to understand in the past. Alternatively, the service provider can avoid explanation formats that the user found difficult in the past and provide explanations in a different format. Furthermore, the service provider can prioritize providing explanations in genres that the user has excelled in the past. This allows the service provider to provide the optimal delivery method based on the user's past learning history. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input the user's past learning history data into a generating AI, which can then analyze the learning history and select the optimal delivery method.

[0093] The information provider can provide additional relevant information based on the user's viewing history when providing answers and explanations. For example, the information provider can provide background information related to the drama episodes the user has watched. It can also provide cultural information related to the scenes the user has watched. Furthermore, it can provide detailed information related to the characters the user has watched. This allows the information provider to provide additional relevant information based on the user's viewing history. Some or all of the above processing in the information provider may be performed using AI or not. For example, the information provider can input the user's viewing history data into a generating AI, which can then analyze the viewing history and provide additional relevant information.

[0094] The service provider can estimate the user's emotions and adjust the order in which answers and explanations are provided based on the estimated emotions. For example, if the user is relaxed, the generating AI can provide explanations starting with the most detailed ones. If the user is stressed, the generating AI can also provide explanations starting with the most concise ones. Furthermore, if the user is focused, the generating AI can randomly provide explanations of moderate detail. This allows the service provider to provide answers and explanations in an appropriate order according to the user's emotions. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or a generating AI. The generating AI is, but is not limited to, a text generation AI (e.g., LLM) or a multimodal generation AI. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input user emotion data into a generating AI, which can estimate the emotions and adjust the order in which answers and explanations are provided.

[0095] The service provider can select the optimal format for providing answers and explanations, taking into account the user's device information. For example, if the user is using a smartphone, the service provider can provide a format that matches the screen size. Furthermore, if the user is using a tablet, the service provider can provide a format optimized for a larger screen. Additionally, if the user is using a smartwatch, the service provider can provide a concise and highly visible format. This allows the service provider to provide the optimal format according to the user's device information. Some or all of the above processing in the service provider may be performed using AI, or not. For example, the service provider can input the user's device information into a generating AI, which can then analyze the device information and select the optimal format.

[0096] The service provider can analyze the user's viewing behavior (e.g., pausing, rewinding) when providing answers and explanations and adjust the timing of the provision. For example, the service provider can have the generating AI provide the answers and explanations when the user pauses. The service provider can also provide answers and explanations related to the rewound portion when the user rewinds. Furthermore, if the user fast-forwards, the service provider can later provide answers and explanations related to the fast-forwarded portion. This allows the service provider to provide answers and explanations at the appropriate timing according to the user's viewing behavior. Some or all of the above processing in the service provider may be performed using AI or not. For example, the service provider can input user viewing behavior data into the generating AI, which can then analyze the behavior and adjust the timing of the provision.

[0097] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0098] A language learning support system can track a user's learning progress and provide personalized learning plans. For example, a detection unit records phrases and words the user previously did not understand and evaluates their learning progress. Next, a generation unit generates quizzes suggesting what the user should learn next, based on their progress. Furthermore, a delivery unit can provide a learning plan tailored to the user's progress via a chatbot. This allows users to learn efficiently at their own pace.

[0099] The language learning support system can provide customized feedback according to the user's learning style. For example, the detection unit determines whether the user is a visual or auditory learner. Next, the generation unit generates quizzes that heavily utilize images and videos for visual learners, and quizzes that heavily utilize audio for auditory learners. Furthermore, the delivery unit can provide feedback tailored to the user's learning style through a chatbot. This allows users to learn in a way that is best suited to them.

[0100] A language learning support system can estimate a user's emotions and provide messages to boost their learning motivation based on those emotions. For example, a detection unit can detect stress and fatigue the user feels while learning. Next, a generation unit generates encouraging messages to help the user relax and messages praising their learning progress. Furthermore, a delivery unit can provide these messages in real time through a chatbot. This makes it easier for the user to maintain their learning motivation.

[0101] The language learning support system can set individual learning goals based on the user's learning history. For example, the detection unit records the content and results of what the user has learned in the past. Next, the generation unit generates a quiz that sets the next learning goal to be achieved, based on the user's learning history. Furthermore, the provision unit provides the user's learning goals through a chatbot and can provide real-time feedback on progress. This allows the user to proceed with their learning with concrete goals in mind.

[0102] The language learning support system can estimate the user's emotions and adjust the learning pace based on those emotions. For example, the detection unit can detect fatigue or decreased concentration that the user feels during learning. Next, the generation unit generates quizzes that slow down the learning pace to help the user relax. Furthermore, the delivery unit can provide these quizzes through a chatbot, further adjusting the user's learning pace. This allows the user to continue learning without feeling overwhelmed.

[0103] A language learning support system can analyze a user's learning environment and suggest the optimal learning method. For example, a detection unit records the location and time of the user's learning. Next, a generation unit generates a quiz that suggests the optimal learning method according to the user's learning environment. Furthermore, a delivery unit provides these suggestions via a chatbot, supporting the user so that they can proceed with their learning in the optimal environment. This allows the user to learn efficiently.

[0104] The language learning support system can estimate the user's emotions and personalize the learning content based on those emotions. For example, the detection unit detects the excitement and interest the user feels while learning. Next, the generation unit generates quizzes containing content that is likely to interest the user. Furthermore, the delivery unit can provide these quizzes through a chatbot to capture the user's interest. This allows the user to learn in an enjoyable way.

[0105] The language learning support system can suggest review timings based on the user's learning history. For example, the detection unit records the content and results of what the user has learned in the past. Next, the generation unit generates quizzes to encourage review at times when the user is likely to forget. Furthermore, the delivery unit provides these quizzes through a chatbot to support the user in effectively reviewing the material. As a result, the user can retain the learned content for a long period of time.

[0106] The language learning support system can estimate the user's emotions and adjust the learning interval based on those emotions. For example, the detection unit detects fatigue or decreased concentration that the user feels during learning. Next, the generation unit generates quizzes that provide learning intervals so that the user can refresh themselves. Furthermore, the delivery unit can provide these quizzes through a chatbot, adjusting the user's learning intervals. This allows the user to continue learning while taking appropriate breaks.

[0107] The language learning support system can generate individual learning reports based on the user's learning history. For example, the detection unit records the content and results of what the user has learned in the past. Next, the generation unit generates a report showing the user's learning progress and achievement level based on the user's learning history. Furthermore, the delivery unit provides these reports via a chatbot to support the user in understanding their learning status. This allows the user to check their learning results and set their next learning goals.

[0108] The following briefly describes the processing flow for example form 2.

[0109] Step 1: The detection unit detects phrases and words that the user did not understand while watching the drama. For example, it uses speech recognition technology to analyze the user's reactions and identify the phrases and words that were not understood. Step 2: The generation unit generates interactive quizzes based on the phrases and words detected by the detection unit. For example, it generates interactive quizzes such as multiple-choice questions and questions asking for the meaning of words. Step 3: The question-generating unit presents the quiz generated by the generation unit in real time using a chatbot function. For example, the quiz is presented in real time using a chatbot function. Step 4: The provider provides answers and explanations for the quiz questions posed by the question-setting unit. For example, the provider might provide answers and explanations for the quiz through a chatbot.

[0110] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0111] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0112] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0113] Each of the multiple elements described above, including the detection unit, generation unit, question unit, and provision unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the detection unit analyzes the user's response using the camera 42 and microphone 38B of the smart device 14 and identifies phrases or words that were not understood by the control unit 46A. The generation unit is implemented in the identification processing unit 290 of the data processing unit 12 and generates an interactive quiz based on the detected phrases or words. The question unit is implemented in the control unit 46A of the smart device 14 and presents quizzes in real time using a chatbot function. The provision unit is implemented in the control unit 46A of the smart device 14 and provides answers and explanations to the quiz through a chatbot. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0114] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0115] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0116] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0117] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0118] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0119] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0120] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0121] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0122] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0123] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0124] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0125] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0126] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0127] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0128] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0129] Each of the multiple elements described above, including the detection unit, generation unit, question unit, and provision unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the detection unit analyzes the user's response using the camera 42 and microphone 238 of the smart glasses 214 and identifies phrases or words that were not understood by the control unit 46A. The generation unit is implemented in the identification processing unit 290 of the data processing unit 12 and generates an interactive quiz based on the detected phrases or words. The question unit is implemented in the control unit 46A of the smart glasses 214 and presents the quiz in real time using a chatbot function. The provision unit is implemented in the control unit 46A of the smart glasses 214 and provides answers and explanations to the quiz through a chatbot. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0130] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0131] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0132] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0133] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0134] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0135] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0136] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0137] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0138] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0139] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0140] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0141] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0142] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0143] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0144] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0145] Each of the multiple elements described above, including the detection unit, generation unit, question unit, and provision unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the detection unit analyzes the user's response using the camera 42 and microphone 238 of the headset terminal 314 and identifies phrases or words that were not understood by the control unit 46A. The generation unit is implemented in the identification processing unit 290 of the data processing unit 12 and generates an interactive quiz based on the detected phrases or words. The question unit is implemented in the control unit 46A of the headset terminal 314 and presents quizzes in real time using a chatbot function. The provision unit is implemented in the control unit 46A of the headset terminal 314 and provides answers and explanations to the quiz through a chatbot. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0146] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0147] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0148] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0149] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0150] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0151] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0152] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0153] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0154] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0155] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0156] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0157] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0158] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0159] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0160] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0161] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0162] Each of the multiple elements described above, including the detection unit, generation unit, question unit, and provision unit, is implemented in, for example, at least one of the robot 414 and the data processing unit 12. For example, the detection unit analyzes the user's response using the camera 42 and microphone 238 of the robot 414 and identifies phrases or words that were not understood by the control unit 46A. The generation unit is implemented, for example, by the identification processing unit 290 of the data processing unit 12 and generates an interactive quiz based on the detected phrases or words. The question unit is implemented, for example, by the control unit 46A of the robot 414 and presents quizzes in real time using a chatbot function. The provision unit is implemented, for example, by the control unit 46A of the robot 414 and provides answers and explanations to the quiz through a chatbot. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0163] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0164] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0165] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0166] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0167] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0168] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0169] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0170] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0171] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0172] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0173] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0174] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0175] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0176] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0177] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0178] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0179] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0180] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0181] (Note 1) A detection unit that detects phrases and words that the user did not understand while watching the drama, A generation unit that generates an interactive quiz based on phrases and words detected by the detection unit, A question-presenting unit that presents the quiz generated by the generation unit in real time using a chatbot function, The system includes a provisioning unit that provides answers and explanations to quizzes presented by the aforementioned question-generating unit. A system characterized by the following features. (Note 2) The detection unit is Using speech recognition technology, the system analyzes user responses to identify phrases and words that were not understood. The system described in Appendix 1, characterized by the features described herein. (Note 3) The generating unit is Generate interactive quizzes such as multiple-choice questions and questions asking for the meaning of words. The system described in Appendix 1, characterized by the features described herein. (Note 4) The aforementioned question section is, Use the chatbot function to present quizzes in real time. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned supply unit is, The chatbot provides answers and explanations to quizzes. The system described in Appendix 1, characterized by the features described herein. (Note 6) The detection unit is It estimates the user's emotions and adjusts the accuracy of detecting phrases and words that were not understood based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 7) The detection unit is The system analyzes the user's viewing history and prioritizes detecting phrases and words that the user previously did not understand. The system described in Appendix 1, characterized by the features described herein. (Note 8) The detection unit is Different detection algorithms are applied to each scene in the drama to detect phrases and words appropriate to the scene. The system described in Appendix 1, characterized by the features described herein. (Note 9) The detection unit is It estimates the user's emotions and determines the priority of phrases and words to detect based on the estimated user emotions. The system described in Appendix 1, characterized by the features described herein. (Note 10) The detection unit is Based on the user's geographical location, the system prioritizes detecting region-specific phrases and words. The system described in Appendix 1, characterized by the features described herein. (Note 11) The detection unit is Analyze users' social media activity and detect relevant phrases and words. The system described in Appendix 1, characterized by the features described herein. (Note 12) The generating unit is The system estimates the user's emotions and adjusts the quiz difficulty based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 13) The generating unit is When generating quizzes, the system selects the most suitable quiz format by referring to the user's learning history. The system described in Appendix 1, characterized by the features described herein. (Note 14) The generating unit is When generating quizzes, different quiz formats are applied depending on the genre of the drama. The system described in Appendix 1, characterized by the features described herein. (Note 15) The generating unit is The system estimates the user's emotions and adjusts the order of quiz questions based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 16) The generating unit is When generating quizzes, adjust the frequency of quiz questions based on the user's viewing time. The system described in Appendix 1, characterized by the features described herein. (Note 17) The generating unit is When generating quizzes, customize the quiz content based on the user's interests. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned question section is, The system estimates the user's emotions and adjusts the timing of quiz questions based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 19) The aforementioned question section is, When presenting a question, the system selects the most suitable question presentation method by referring to the user's past quiz answer history. The system described in Appendix 1, characterized by the features described herein. (Note 20) The aforementioned question section is, When creating questions, the optimal question format is selected considering the user's device information. The system described in Appendix 1, characterized by the features described herein. (Note 21) The aforementioned question section is, The system estimates the user's emotions and adjusts the order of quiz questions based on those emotions. The system described in Appendix 1, characterized by the features described herein. (Note 22) The aforementioned question section is, When creating questions, we adjust the question format to take into account the user's viewing environment. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned question section is, When presenting questions, the timing is adjusted by analyzing the user's viewing behavior. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned supply unit is, The system estimates the user's emotions and adjusts the way answers and explanations are presented based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned supply unit is, When providing answers and explanations, the system will refer to the user's past learning history to select the most suitable method of delivery. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned supply unit is, When providing answers and explanations, additional relevant information will be provided based on the user's viewing history. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned supply unit is, The system estimates the user's emotions and adjusts the order in which answers and explanations are provided based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned supply unit is, When providing answers and explanations, the optimal format for delivery will be selected, taking into account the user's device information. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned supply unit is, When providing answers and explanations, we analyze the user's viewing behavior and adjust the timing of the provision. The system described in Appendix 1, characterized by the features described herein. [Explanation of symbols]

[0182] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. A detection unit that detects phrases and words that the user did not understand while watching the drama, A generation unit that generates an interactive quiz based on phrases and words detected by the detection unit, A question-presenting unit that presents the quiz generated by the generation unit in real time using a chatbot function, The system includes a provisioning unit that provides answers and explanations to quizzes presented by the aforementioned question-generating unit. A system characterized by the following features.

2. The detection unit is Using speech recognition technology, the system analyzes user responses to identify phrases and words that were not understood. The system according to feature 1.

3. The generating unit is Generate interactive quizzes such as multiple-choice questions and questions asking for the meaning of words. The system according to feature 1.

4. The aforementioned question section is, Use the chatbot function to present quizzes in real time. The system according to feature 1.

5. The aforementioned supply unit is, The chatbot provides answers and explanations to quizzes. The system according to feature 1.

6. The detection unit is It estimates the user's emotions and adjusts the accuracy of detecting phrases and words that were not understood based on the estimated user emotions. The system according to feature 1.

7. The detection unit is The system analyzes the user's viewing history and prioritizes detecting phrases and words that the user previously did not understand. The system according to feature 1.

8. The detection unit is Different detection algorithms are applied to each scene in the drama to detect phrases and words appropriate to the scene. The system according to feature 1.

9. The detection unit is It estimates the user's emotions and determines the priority of phrases and words to detect based on the estimated user emotions. The system according to feature 1.

10. The detection unit is Based on the user's geographical location, the system prioritizes detecting region-specific phrases and words. The system according to feature 1.

11. The detection unit is Analyze users' social media activity and detect relevant phrases and words. The system according to feature 1.

Images