Information processing system and information processing method

Abstract

To provide an information processing system capable of efficiently supporting a target person.SOLUTION: An information processing system includes: a generation unit which generates emotion information based on biometric data of a target person or input data; a content acquisition unit which acquires first content based on the emotion information generated in the generation unit; an integration unit which integrates the emotion information with the first content acquired in the content acquisition unit; a content generation unit which generates second content based on the information integrated in the integration unit; an output unit which outputs the second content generated in the content generation unit; and a recording unit which records the emotion information and the second content in association with each other.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an information processing system and an information processing method.

Background Art

[0002] For example, children's interests and concerns can be intentionally enhanced by the educator's environment creation. Also, perseverance can be extended by encouragement. It shows that children can achieve through specific support. For example, a method of supporting the improvement of non-cognitive abilities by displaying the achievement level of a learning plan has been proposed (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, it is important to appropriately grasp the state of the learner in supporting children's learning. The present disclosure is for solving the above problems, and provides an information processing system and an information processing method capable of efficiently supporting a learner (target person) as an example.

Means for Solving the Problems

[0005] The information processing system of the present disclosure includes a generation unit that generates emotion information based on the biological data of the target person, Input operation data or voice content, and a content acquisition unit that acquires Stored in the memory unit a first content Search based on the emotion information generated by the generation unit, and at least a part of a plurality of first contents acquired by the content acquisition unit, Select, combine multiple selected first content items, or extract only the necessary parts of multiple selected first content items and process or edit them, and integrate the emotional information with the combined first content items or the processed or edited first content items. these Information input to the generative AI modelThe overall information is managed by the overall management department, and based on the said overall information... The response content is generated by the AI ​​model. The system includes a content generation unit that generates a second content, an output unit that outputs the second content generated by the content generation unit, and a recording unit that records the emotional information and the second content in association.

[0006] The information processing method disclosed herein is an information processing method performed by a computer, comprising the steps of generating emotional information based on a subject's biometric data, input operation data, or voice content, The aforementioned step The system includes the steps of: searching for and retrieving a first content stored in a memory unit based on the emotional information generated; selecting the emotional information and at least a portion of a plurality of the first content, combining the selected plurality of first content, or extracting and processing or editing only the necessary parts of the selected plurality of first content; integrating the emotional information with the combined first content or the processed or edited first content, and summarizing these as information to be input to the generation AI model; generating a second content, which is response content, from the generation AI model based on the summarizing information; outputting the generated second content; and recording the emotional information and the second content in association. [Effects of the Invention]

[0007] The information processing system and information processing method disclosed herein can efficiently support the target individuals. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates the overview of the information processing system 1 according to Embodiment 1. [Figure 2] This diagram illustrates the configuration of the information processing system 1 according to Embodiment 1. [Figure 3] This diagram illustrates the configuration of the functional blocks of the information processing device 100 according to Embodiment 1. [Figure 4]This is a flowchart illustrating the processing of the information processing device 100 according to Embodiment 1. [Figure 5] This figure illustrates an example of prompt and response content according to Embodiment 1. [Figure 6] This is a flowchart illustrating the processing of the information processing device 100 according to Embodiment 2. [Figure 7] This is a flowchart illustrating the input state analysis process according to Embodiment 2. [Figure 8] This figure illustrates an example of a prompt according to Embodiment 2. [Figure 9] This is a flowchart illustrating the processing of the information processing device 100 according to a modified example of Embodiment 2. [Figure 10] This is a diagram illustrating persona information according to Embodiment 3. [Figure 11] This figure illustrates the recording of the recording unit 124 according to Embodiment 4. [Figure 12] This diagram illustrates the configuration of information processing system 1# according to Embodiment 5. [Modes for carrying out the invention]

[0009] Embodiments of this disclosure will be described in detail below with reference to the drawings. In the following, the same or corresponding parts in the drawings will be denoted by the same reference numerals, and their descriptions will not be repeated in principle.

[0010] (Embodiment 1) Figure 1 is a diagram illustrating the overview of an information processing system 1 according to Embodiment 1. Referring to Figure 1, the information processing system 1 according to Embodiment 1 is a learning support system mainly for assisting learners in their learning. Specifically, the information processing system 1 comprises a terminal 10, a network NW, an information processing device 100, and a large-scale language model system 200. The terminal 10 is, for example, a PC (Personal Computer) owned by the learner (user), and may be portable or fixed. The terminal 10 is configured to communicate with the information processing device 100 via the network NW. The information processing device 100 is configured to communicate with the large-scale language model system 200 via the network NW. Communication can be wireless or wired. The user is learning programming using the terminal 10. The terminal 10 displays a learning screen 50 for programming learning, and the learning screen 50 is shown in an example where a character 52 is provided.

[0011] The information processing device 100 communicates with the large-scale language model system 200 to receive content that enhances the learner's motivation to learn programming. The large-scale language model (LLM) system 200 is a natural language processing system that performs question and answer. The LLM system 200 is a natural language processing model trained using a large amount of text data, and it takes text as input and outputs text. When the LLM system 200 is applied to a system that performs question and answer, when a question is queried to the LLM system 200, the LLM system 200 outputs answer content. In this embodiment, the information processing device 100, as an example, queries a learning-related question to the LLM system 200 and sends the answer content from the LLM system 200 to the terminal 10. The terminal 10 receives the learning-related answer content (message) from the LLM system 200. When learning programming, the terminal 10 outputs the received learning-related message to the learning screen 50 of the terminal 10 as audio output or as a display such as a speech bubble. For example, terminal 10 may be configured to use character 52 to execute the message output.

[0012] FIG. 2 is a diagram for explaining the configuration of the information processing system 1 according to Embodiment 1. Referring to FIG. 2, the terminal 10 includes a control unit 11, a camera 12, a communication I / F 13, an input device 14, an output device 15, a microphone 16, a storage unit 17, and an internal bus that connects each unit. The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The camera 12 acquires image data of the user's facial expression as biometric data of the user (subject). In this example, image data such as the user's facial expression will be described, but it is not limited to this, and other information may be acquired as biometric data. For example, infrared camera may be used to acquire the user's pulse wave information. The microphone 16 acquires the user's voice data as biometric data of the user. The communication I / F 13 is connected to the network NW and executes data transfer with an external device. The input device 14 includes a mouse and a keyboard. The output device 15 includes a display device such as a display and a speaker. The storage unit 17 includes various application programs. For example, it stores application programs for the user to learn programming. The control unit 11 generates a learning screen for program learning by executing the application program and displays it on a display device such as a display.

[0013] The terminal 10 acquires biometric data (image data and voice data as an example) and transmits it to the information processing apparatus 100 via the communication I / F 13. The terminal 10 acquires input data and transmits it to the information processing apparatus 100 via the communication I / F 13. The input data in this example is input data such as a keyboard and a mouse used by the user during program learning as an example.

[0014] The information processing apparatus 100 includes a control unit 101, a storage unit 107, a communication I / F 103, and an internal bus that connects each unit. The control unit 101 includes a CPU, a RAM, and a ROM. The communication I / F 103 is connected to the network NW and executes data transfer with an external device. The storage unit 107 includes various application programs and the like. For example, the storage unit 107 stores an application program for assisting user learning and the like. The control unit 11 realizes various processes by executing the application program. The information processing apparatus 100 receives biological data and the like transmitted from the terminal 10 and generates user emotion information based on the biological data and the like. The information processing apparatus 100 makes an inquiry to the LLM system 200 about a learning-related question corresponding to the generated emotion information.

[0015] The LLM system 200 includes a question receiving unit 201, an answer generating unit 202, and a trained model 204. The question receiving unit 201 receives an inquiry about a learning-related question from the information processing apparatus 100. The answer generating unit 202 analyzes the received inquiry and generates learning-related answer content (e.g., a message) for the inquiry of the question using the trained model 204 based on the analysis result, and transmits it to the information processing apparatus 100. The trained model 204 is a learning model trained using a text database (DB) or the like having a large amount of text data for generating answers.

[0016] The information processing device 100 receives learning-related response content (e.g., a message) from the LLM system 200 and transmits it to the terminal 10. During programming learning, the terminal 10 outputs the received learning-related content (e.g., a message) to the learning screen 50 of the terminal 10. For example, the terminal 10 may use a character 52 provided on the learning screen 50 to execute the output of the response content (message). In this example, the LLM system 200 is described as being provided separately from the information processing device 100, but it is also possible to provide functions equivalent to the LLM system 200 within the information processing device 100. Alternatively, the functions of generating emotional information and generating content may be performed by the LLM system 200. Furthermore, a configuration in which the functions of the information processing device 100 and the LLM system 200 are integrated into the terminal 10 may be adopted. The terminal 10 itself may generate the user's emotional information and also generate learning-related response content (e.g., a message), and output it directly to the learning screen 50 of the terminal 10. This configuration eliminates the need to communicate with external systems via the network, resulting in shorter response times, improved processing efficiency on terminal 10, and enhanced user privacy protection because user emotional data and other personal information are processed without being transmitted outside the terminal.

[0017] Figure 3 is a diagram illustrating the configuration of the functional blocks of the information processing device 100 according to Embodiment 1. Referring to Figure 3, the control unit 101 of the information processing device 100 realizes various functional blocks by executing application programs stored in the storage unit 107. Specifically, the information processing device 100 includes a biometric data acquisition unit 110, an emotion analysis unit 112, an input data acquisition unit 114, a prompt generation unit 116, an input state analysis unit 117, an output control unit 118, a general unit 120, a content acquisition unit 122, and a recording unit 124. The biometric data acquisition unit 110 acquires biometric data (image data and voice data, etc.) transmitted from the terminal 10 received via the communication I / F 103. The acquired biometric data is stored in the storage unit 107.

[0018] The emotion analysis unit 112 estimates the user's emotions (psychology) based on acquired biometric data (image data and audio data, etc.) and generates emotion information. The emotion analysis unit 112 estimates (analyzes) emotions such as joy, anger, sadness, and happiness based on biometric data. The emotion analysis unit 112 is not limited to these emotions and may also estimate calmness, surprise, satisfaction, boredom, disappointment, fear, relief, anxiety, etc. Specifically, the emotion analysis unit 112 can estimate emotions by processing the acquired image data, including the degree of eyelid opening, gaze, eyebrow movement, presence or absence of nasal wrinkles, mouth movement, degree of mouth opening, pupil dilation, etc., which are included in the user's facial expression. The emotion analysis unit 112 can also estimate (analyze) emotions using audio data, including the content of the user's voice, sighs, breathing sounds, etc. The emotion analysis unit 112 may estimate emotions using only one of the data, or it may estimate emotions by combining the data. Furthermore, while this example describes the use of biometric data (image data) from camera 12, it is not limited to this, and biometric data may also be acquired using a wearable device.

[0019] The input data acquisition unit 114 acquires input data (such as input key data) transmitted from the terminal 10 via the communication interface 103. The acquired input data is stored in the storage unit 107. The input state analysis unit 117 analyzes the user's input state based on the acquired input data (such as input key data). Specifically, the input state analysis unit 117 estimates the user's input state by analyzing the user's input speed using a keyboard, the number of input errors, etc. The content acquisition unit 122 acquires content (for example, a message) previously output to the user via the output control unit 118, which is stored in the storage unit 107, based on the emotion information generated by the emotion analysis unit 112 (first content).

[0020] The control unit 120 integrates the emotion information generated by the emotion analysis unit 112 and the first content acquired by the content acquisition unit 122. The prompt generation unit 116 generates a message that asks the LLM system 200 a learning-related question based on the information integrated by the control unit 120. The prompt generation unit 116 outputs the generated message to the LLM system 200. The output control unit 118 outputs information such as the response content from the LLM system 200 to the terminal 10. The recording unit 124 associates the response content (second content) from the LLM system 200 with the emotion information and records it in the storage unit 107.

[0021] Figure 4 is a flowchart illustrating the processing of the information processing device 100 according to Embodiment 1. Referring to Figure 4, the information processing device 100 acquires biometric data (step S0). Specifically, the biometric data acquisition unit 110 acquires biometric data transmitted from the terminal 10. Next, the information processing device 100 determines whether a predetermined period has elapsed (step S2). If the information processing device 100 determines that the predetermined period has not elapsed (NO in step S2), it returns to step S0 and repeats the above process. The predetermined period can be set to any period, but as an example, it can be set to 5 minutes. If the information processing device 100 determines that the predetermined period has elapsed (YES in step S2), it proceeds to the next process. The information processing device 100 performs a process to analyze emotions (step S4). Specifically, if the predetermined period has elapsed, the emotion analysis unit 112 estimates the user's emotions (psychology) based on the biometric data (image data and voice data, etc.) stored in the memory unit 107 and generates emotion information. Furthermore, within a predetermined period, some of the biometric data (image data and audio data, etc.) stored in the memory unit 107 may be used, or all of the data may be used. Next, the information processing device 100 acquires the first content based on the emotion information generated by the emotion analysis unit 112 (step S5). Specifically, the content acquisition unit 122 acquires previously output content (first content) via the output control unit 118 stored in the memory unit 107 based on the emotion information generated by the emotion analysis unit 112. The content acquisition unit 122 searches for previously output content related to the emotion information stored in the memory unit 107 and acquires the content obtained from the search results. For example, the content acquisition unit 122 acquires content recorded in association with the same emotion information as the emotion information stored in the memory unit 107 (as an example, information of a previously output message). Next, the information processing device 100 integrates the emotion information and the acquired content (first content) (step S6). Specifically, the integration unit 120 integrates the emotion information and the acquired content (first content).Furthermore, if the content acquisition unit 122 acquires multiple content items, the consolidation unit 120 may consolidate all of them, or it may select one of them to consolidate. Alternatively, it may combine multiple content items as appropriate, or extract only the necessary parts and consolidate the information after processing and editing it as appropriate. Next, the information processing device 100 generates a prompt to send to the LLM system 200 (step S7). Specifically, the prompt generation unit 116 generates a sentence that asks the LLM system 200 a learning-related question based on the information consolidated by the consolidation unit 120. Next, the information processing device 100 sends the generated prompt to the LLM system 200 (step S8). Specifically, the prompt generation unit 116 sends the generated prompt, which is a sentence that asks the LLM system 200. Next, the information processing device 100 determines whether or not it has received answer content from the LLM system 200 (step S10). Specifically, the output control unit 118 determines whether or not it has received answer content from the LLM system 200. In step S10, the information processing device 100 maintains its state until it receives response content from the LLM system 200. When the information processing device 100 receives response content from the LLM system 200 (YES in step S10), it outputs information such as the response content (a message, for example) (second content) to the terminal 10 (step S14). The output control unit 118 transmits message output information, including the message, to the terminal 10. If the output control unit 118 receives multiple response contents from the LLM system 200, it may select one of them. If the output control unit 118 receives multiple response contents, it may combine them as appropriate, or extract only the necessary parts and process and edit them as appropriate to generate a message, which is the response content. Next, the information processing device 100 records the emotional information and the response content (second content) in association (step S16). Specifically, the recording unit 124 stores the response content (a message, for example) and the emotional information in association with each other in the storage unit 107.The recording unit 124 may store not only the response content (a message, for example) but also other related information in association with it in the storage unit 107. Next, the information processing device 100 determines whether or not to terminate the process (step S18). In step S18, if the information processing device 100 determines to terminate the process, it terminates the process (end). Specifically, the information processing device 100 terminates the process if it determines that the user has terminated the application program for learning programming at terminal 10. For example, the information processing device 100 may terminate the process when it receives a command from terminal 10 to terminate the application program for learning programming. On the other hand, if the information processing device 100 determines not to terminate the process in step S18 (NO in step S18), it returns to step S0 and repeats the above process.

[0022] Figure 5 illustrates an example of prompt and response content according to Embodiment 1. Figure 5(A) shows prompts that the prompt generation unit 116 sends to the LLM system 200 based on information compiled by the integration unit 120. As an example, the prompt generation unit 116 generates the following prompt based on information compiled by the integration unit 120. Specifically, the content acquisition unit 122 acquires the first content based on the emotion information ("fun") generated by the emotion analysis unit 112. In this example, the content acquisition unit 122 acquires previously output response content (a message, for example) (first content) that has been recorded in association with the emotion information ("fun"). The integration unit 120 integrates the emotion information ("fun") and the information of the previously output message. Based on the information compiled by the control unit 120, the prompt generation unit 116 generates, as an example, a sentence such as: "(A) Please think of something to say to a child who is in the following emotional state to increase their motivation to learn. When speaking to them, please use language that is appropriate for a child. (B) They are learning programming. (C) They are a little frustrated because there are things they don't understand. (D) They feel it would be a shame to give up." The prompt generation unit 116 also generates, as an example, a sentence using acquired content (a message, as an example) that was recorded in the memory unit 107 in association with emotional information ("fun") as reference information. For example, it generates a sentence such as: "(E) You have always been good at learning new things, and you have the strength to overcome even small difficulties. I'm sure you can do this While loop task well."

[0023] Figure 5(B) shows an example of response content from the LLM system 200 to the above prompt. As an example, the information processing device 100 receives the response content: "Great job! Programming is fun, isn't it? You did a great job on the previous While statement assignment. But sometimes it can be difficult, and you might get a little frustrated. Even if you don't understand something, it's a chance to learn more and more! Everyone starts out not knowing anything. Let's work together to overcome those difficult parts! If we do, something new and interesting might be waiting for us!" The information processing device 100 transmits the information of the received response content to the terminal 10. The terminal 10 outputs a learning screen for programming, along with the response content sent from the information processing device 100, which is a message related to the user's learning. Through this process, the information processing system analyzes the user's emotional state and adds information from previously output messages as reference information, enabling it to output more empathetic messages to the user that take into account the content of previously output messages and are more in line with the user's situation.

[0024] Conventionally, user interactions were limited to each session, and previous interactions were not reflected. This presented a challenge in outputting empathetic messages that took into account the user's past experiences. However, the information processing system disclosed here outputs messages that take into account the content of previously output messages. Furthermore, as reference information, the content of previously output messages recorded in association with emotional information is added, resulting in the output of more emotionally appropriate messages to the user. In other words, the information processing system can appropriately grasp the user's state and efficiently support the user. For example, it can enhance the user's learning effectiveness. Also, the information processing system according to Embodiment 1 can understand the user's emotional state (joy, anger, sadness, happiness, etc.), thereby understanding the user's psychological readiness and readiness for learning at that moment. This makes it possible to provide emotionally appropriate messages at the moment the user is most receptive, thus efficiently supporting the user. For example, it can also enhance the user's learning effectiveness. Additionally, for example, when negative emotions such as frustration or anxiety during the user's learning are detected early, the system can output appropriate messages that are appropriate to those emotions, enabling appropriate support and intervention before the user becomes discouraged. Furthermore, it is possible to increase engagement with learning by, for example, displaying appropriate messages that match the user's emotions when they are in a positive emotional state. In addition, by understanding the user's emotional state, it may be possible to provide a more personalized learning experience. For example, if a user is feeling sad or stressed, the learning process may be optimized and learning efficiency improved by guiding them to easier tasks or breaks. Learning in an emotionally balanced state can help with memory retention and deepen understanding.

[0025] [Example of generating a view] In the above, we have described the case in which the prompt generation unit 116 generates a prompt based on emotional information ("happy"), which is information consolidated by the consolidation unit 120, and the first content (a message recorded in the memory unit 107 in association with "happy"). The prompt generation unit 116 can similarly generate prompts for information consolidated by the consolidation unit 120 that includes other emotional information. For example, the prompt generation unit 116 generates a prompt based on emotional information ("angry"). For example, the prompt generation unit 116 may change sentence (B) of the above sentences (A) to (E), which are prompts, to "I feel angry about learning." Alternatively, the prompt generation unit 116 may keep sentence (A) fixed and change sentences (B) to (D) as appropriate based on emotional information. Furthermore, as an example, the prompt generation unit 116 generates a sentence using the acquired content recorded in the memory unit 107 in association with emotional information ("angry") as reference information. As an example, the following sentence will be generated: "(E) You might not be good at this While loop assignment, but you did very well on the For loop assignment. So I'm sure you'll do well this time too."

[0026] The information processing device 100 receives, for example, response content for the prompt, such as "Anger is an emotion that arises when learning something new. If you take it one step at a time, that anger will gradually disappear. Remember the for loop assignment. Don't miss your progress." The information processing device 100 transmits the received response content and other information to the terminal 10. For example, the prompt generation unit 116 generates a prompt based on the emotion information ("sad"). For example, the prompt generation unit 116 may change sentence (B) of the above sentences (A) to (E), which are prompts, to "I feel sad about learning." Alternatively, the prompt generation unit 116 may keep sentence (A) fixed and change sentences (B) to (D) as appropriate based on the emotion information. Also, as an example, the prompt generation unit 116 generates a sentence using the acquired content recorded in the memory unit 107 in association with the emotion information ("sadness") as reference information. As an example, it generates the sentence, "(E) You really persevered and worked hard on both the While loop and For loop assignments. So this time too, don't give up and keep trying." The information processing device 100 receives, as an example, response content to the prompt, such as, "Think of sad feelings as a step towards growth. It's natural to encounter things you don't understand, like with the While loop and For loop assignments, but what's important is having the attitude to work hard to overcome them." The information processing device 100 transmits the received response content and other information to the terminal 10. For example, the prompt generation unit 116 generates prompts based on emotional information ("joy"). For example, the prompt generation unit 116 may change sentence (B) of the above sentences (A) to (E), which are prompts, to "I feel joy in learning." Alternatively, the prompt generation unit 116 may keep sentence (A) fixed and change sentences (B) to (D) as appropriate based on emotional information. Furthermore, the prompt generation unit 116 generates text using, as an example, the acquired content that was recorded in the memory unit 107 in association with emotional information ("joy") as reference information.For example, the system generates the following sentence: "(E) You really enjoyed working on both the While loop and For loop assignments. Keep up the good work on the adjustments." The information processing device 100 receives a response to this prompt, for example, "Your motivation to learn is wonderful. That was very evident in both the While loop and For loop assignments. You can grow more and more by enjoying the process." The information processing device 100 then transmits the received response content and other information to the terminal 10.

[0027] [Generation of emotional information by the emotion analysis unit 112] The emotion analysis unit 112 may generate emotion information by estimating the user's emotions (psychology) based on input data without relying on biometric data. Specifically, it may generate emotion information based on input data (such as input key data) acquired by the input data acquisition unit 114. For example, an emotion diagnostic test may be executed at the start of programming learning to diagnose the user's emotions based on user input. In the emotion diagnostic test, the emotion analysis unit 112 may generate emotion information based on the user's input data for multiple questions used to diagnose the user's emotions. Furthermore, the method of the emotion diagnostic test is not particularly limited, and the test may prompt the user with questions or image selections that are not easily noticed by the user, and generate emotion information based on the answer results. In addition, the input data can be any data that can be used to generate emotion information, and is not limited to input key data such as keyboard data; for example, user behavior history data may be used. Furthermore, the emotion analysis unit 112 is not limited to directly using input data; for example, it may generate emotion information using user input state information analyzed by the input state analysis unit 117. For example, the system could analyze the actual programming input state during programming learning and generate sentiment information based on the input state information resulting from that analysis. Alternatively, the system could generate sentiment information based on the analysis results of the user's input data content and context as input state information.

[0028] (Embodiment 2) Figure 6 is a flowchart illustrating the processing of the information processing device 100 according to Embodiment 2. Referring to Figure 6, the processing of the information processing device 100 according to Embodiment 2 differs from the flowchart in Figure 4 in that steps S1 and S5A are added. The other configurations are the same as those described above, so a detailed explanation will not be repeated. In step S0, the information processing device 100 acquires biological data and then acquires input data (step S1). Specifically, the input data acquisition unit 114 acquires input data transmitted from the terminal 10. Next, the information processing device 100 determines whether a predetermined period has elapsed (step S2). If the information processing device 100 determines that the predetermined period has not elapsed (NO in step S2), it returns to step S0 and repeats the above process. The predetermined period can be set to any period, but as an example, it can be set to 5 minutes. If the information processing device 100 determines that the predetermined period has elapsed (YES in step S2), it proceeds to the next process. In step S4, the information processing device 100 performs a process to analyze emotions after a predetermined period has elapsed, and then in step S5, it acquires the first content based on the emotion information generated by the emotion analysis unit 112. Next, the information processing device 100 performs an input state analysis process (step S5A). Specifically, the input state analysis unit 117 analyzes the user's input state based on the input data (input key data, etc.) stored in the storage unit 107. The input state analysis unit 117 may use some of the data (input key data, etc.) stored in the storage unit 107 within the predetermined period, or it may use all of the data.

[0029] Figure 7 is a flowchart illustrating the input state analysis process according to Embodiment 2. Referring to Figure 7, the input state analysis unit 117 evaluates the input proficiency based on the acquired input data (step S30). Specifically, the input state analysis unit 117 estimates the user's input proficiency by analyzing the user's input speed using a keyboard, the number of input errors, etc., as information of the input data. The input state analysis unit 117 may classify the user's input proficiency as high or low, or it may further classify it into multiple levels. Next, the input state analysis unit 117 evaluates the concentration level based on the acquired input data (step S32). Specifically, the input state analysis unit 117 evaluates the user's concentration level by analyzing the length of the user's input period using a keyboard, etc., as information of the input data. For example, the input state analysis unit 117 can estimate that the user's concentration level is high if the length of the input period during a certain period is long. The input state analysis unit 117 may also estimate that the concentration level is high if the input period during a certain period is above a certain threshold, and estimate that the concentration level is not high otherwise. Next, the input state analysis unit 117 evaluates the task completion level based on the acquired input data (step S34). Specifically, the input state analysis unit 117 evaluates the completion level of the programming learning task based on the input data as information from the input data. Specifically, the input state analysis unit 117 may evaluate the completion level of the programming learning by comparing the correct answer information for the programming learning task with the programming input data currently entered by the user. Then, the input state analysis unit 117 terminates the input state analysis process (return).

[0030] Referring again to Figure 6, the information processing device 100 integrates emotional information, acquired content (first content), and input information (step S6). Specifically, the integration unit 120 integrates emotional information, acquired content (first content), and input information including input status information such as input proficiency, concentration level, and task completion level. Next, the information processing device 100 generates a prompt to send to the LLM system 200 (step S7). Specifically, the prompt generation unit 116 generates a sentence that asks the LLM system 200 a learning-related question based on the information integrated by the integration unit 120.

[0031] Figure 8 illustrates an example of a prompt according to Embodiment 2. Figure 8 shows a prompt that the prompt generation unit 116 sends to the LLM system 200 based on emotion information, acquired content (first content), and input information including input state information. It differs from the prompt described in Figure 5(A) in that multiple pieces of reference information are generated. Specifically, the prompt generation unit 116 generates a sentence using the acquired content, which was associated with emotion information ("fun") and recorded in the memory unit 107, as reference information 1. For example, it generates the sentence, "(E) You've always been good at learning new things, and you have the strength to overcome even small difficulties. I'm sure you'll do well on this While sentence task." The prompt generation unit 116 also includes a sentence using input state information as reference information 2 when generating a prompt. For example, in addition to the sentences (A) to (E) described in Figure 5(A), the prompt generation unit 116 can include a sentence about the user's input state information, such as input proficiency, concentration level, and achievement level. As an example, the following sentence will be generated: "(F) Input proficiency is high. Concentration level is high. Task completion rate is high." The subsequent processing is the same as explained in the flowchart in Figure 4, so a detailed explanation will not be repeated.

[0032] The information processing system according to Embodiment 2 of this disclosure outputs a message that takes into account the content of previously output messages. Furthermore, when generating a prompt, the current input information is also added as reference information, so that a message more tailored to the user's state is output to the user. In other words, the information processing system can appropriately grasp the user's state and efficiently support the user. For example, it can enhance the user's learning effect. In this example, the input state information is described as including information on input proficiency, concentration level, and task completion level, but it may include just one of these, or it may include information indicating multiple input states. In addition, the content to be included in the generation of prompts may be appropriately changed according to the emotional information. For example, if the emotional information is "anger," the text that uses the input state information as reference information 2 may not be included. In this example, the integration unit 120 is described as integrating emotional information, acquired content (first content), and input information including input state information obtained by analyzing the input state, but it may also be configured to integrate the input data actually entered by the user into the terminal 10 as input information and generate a prompt to be sent to the LLM system 200 as reference information. For example, the input information may include user input (e.g., questions, feedback, learning preferences and interests). The input information may also include environmental information related to the user's learning. For example, it may include information about the learning environment that may affect the user's concentration and learning efficiency (e.g., a quiet room, library, cafe, etc., and the time of day when the user is learning, such as morning or evening), temperature, humidity, and geographical location information, information about learning resources that the user can access (e.g., books, online learning materials, educational apps), information about the user's social and cultural background that may affect their learning style and response to learning materials, and information about external evaluations and feedback from teachers, family, etc. The input information may also include information about the user's health status and lifestyle, or information about the user's specific learning goals and expectations.

[0033] (Modified version of Embodiment 2) Figure 9 is a flowchart illustrating the processing of the information processing device 100 according to a modified example of Embodiment 2. Referring to Figure 9, the processing of the information processing device 100 according to the modified example of Embodiment 2 differs from the flowchart in Figure 6 in that steps S5B and S12 are added. The other configurations are the same as those described in the flowchart in Figure 6, so a detailed explanation will not be repeated. In step S5A, the information processing device 100 performs an input state analysis process, and then determines whether the analyzed input state information has a high degree of concentration (step S5B). Specifically, the prompt generation unit 116 determines whether the input state analysis result of the input state analysis unit 117 has a high degree of concentration. In step S5B, if the information processing device 100 determines that the analyzed input state does not have a high degree of concentration (NO in step S5B), it proceeds to step S6. The subsequent processing is the same as that described in the flowchart in Figure 6. On the other hand, in step S5B, if the information processing device 100 determines that the concentration level is high based on the analyzed input state (YES in step S5B), it skips steps S6 to S16 and proceeds to step S18. Specifically, if the prompt generation unit 116 determines that the concentration level is high based on the analysis results of the input state analysis unit 117, it does not perform the prompt generation process. In other words, if the concentration level is high, the information processing device 100 does not output content (a message, for example) to the terminal 10.

[0034] Message output when the user's concentration level is high may disrupt their thinking. Therefore, the information processing system can efficiently support the user by stopping content output in such a state. For example, it can efficiently enhance the user's learning effectiveness. In this example, the input state analysis unit 117 was described as determining the concentration level based on input data, but it is not limited to this and may also determine the concentration level in combination with emotional information. In this example, the information processing system was described as a method for determining whether or not to output content according to the user's concentration level, but it may also adjust the frequency of content output according to the user's concentration level. For example, if the user's concentration level is high, the information processing system may lengthen the interval of the predetermined processing period in step S2. By adjusting the frequency of content output according to the user's concentration level, the information processing system can efficiently support the user. For example, it can efficiently enhance the user's learning effectiveness.

[0035] Furthermore, in step S10, after receiving the response content from the LLM system 200, the information processing device 100 sets the parameters for content output (step S12). Specifically, the output control unit 118 sets the parameters for content output based on emotional information. For example, the output control unit 118 may set the parameters for the message color when outputting content based on emotional information. For example, the output control unit 118 may set the parameters for the text color, such as orange, red, blue, or yellow, based on emotional information such as "joy," "anger," "sadness," or "happiness," and output content. By utilizing the psychological effects of color to output content, the output control unit 118 can efficiently support the user. For example, it can efficiently enhance the user's learning effect. Note that the setting of the color parameters is just an example and is not limited to this; other colors can be set, or content output may be performed by changing brightness or other parameters based on emotional information. Also, when outputting content using the character 52, the output control unit 118 may adjust the movement of the character 52 based on emotional information. For example, multiple movement patterns for character 52 may be prepared in advance, and the output control unit 118 may select one of the multiple movement patterns based on emotional information and display it on terminal 10. For example, movement patterns corresponding to "joy," "anger," "sadness," and "happiness" may be prepared in advance, and the output control unit 118 may set the parameters for content output to match the emotional information from among the multiple movement patterns. For example, the information processing system can efficiently support the user by displaying character 52 movements that match the emotional information when outputting content. For example, it can efficiently enhance the user's learning effect. Also, when terminal 10 outputs content using character 52, it may use speech bubble displays to make it appear as if character 52 is speaking, or it may output the content (message) as audio. The output control unit 118 may adjust the pattern of content output as audio based on emotional information.For example, multiple voice output patterns may be prepared in advance, and the output control unit 118 may select and output one of the multiple voice output patterns based on emotional information. For example, voice output patterns corresponding to "joy," "anger," "sadness," and "happiness" may be prepared in advance, and the output control unit 118 may set the parameters for outputting content as voice to match the emotional information from among the multiple voice output patterns. For example, an information processing system can efficiently support the user by outputting content as voice using a voice output pattern that matches the emotional information. For example, it can efficiently enhance the user's learning effect. The output control unit 118 may use the above parameter setting methods individually or in combination.

[0036] (Embodiment 3) The information processing device 100 according to Embodiment 3 integrates emotional information, acquired content (first content), and persona information. Specifically, the integration unit 120 integrates emotional information, acquired content (first content), and persona information. The prompt generation unit 116 then generates a message that queries the LLM system 200 with learning-related questions based on the information integrated by the integration unit 120.

[0037] Figure 10 is a diagram illustrating persona information according to Embodiment 3. Referring to Figure 10, user profile data is shown as persona information. Specifically, the profile data includes basic information and information about the user's learning tendencies. Information about learning tendencies includes information about assessment of learning style, information about abilities and performance, information about time management and learning plans, information about motivation and drive, and information about emotions and stress management. Basic information includes name, gender, age, grade level, subjects of interest, subjects of difficulty, attitude towards learning, and motivation information. Information about assessment of learning style includes type, preferences, interest in subjects, and information about hobbies and activities. Information about abilities and performance includes current status, strengths, and weaknesses. Information about time management and learning plans includes information about daily learning and time management. Information about motivation and drive includes information about motivation and success experiences. Information about emotions and stress management includes information about anxiety and stress.

[0038] The prompt generation unit 116 includes text with persona information as reference information 2 when generating a prompt. For example, the prompt generation unit 116 can include text related to persona information in addition to the texts (A) to (E) described in Figure 5(A). As an example, it generates text of profile data as described in Figure 10. Then, the prompt generation unit 116 sends the generated prompt, which is text to query the LLM system 200. Other points are the same as described in Embodiment 1, so a detailed explanation will not be repeated. It may also be combined with the configuration of Embodiment 2.

[0039] The information processing system according to Embodiment 3 of this disclosure outputs messages that take into account the content of messages output in the past. Furthermore, when generating prompts, persona information is also added as reference information, so that messages that are more tailored to the user's state are output to the user. In other words, the information processing system can appropriately grasp the user's state and efficiently support the user. For example, it can improve the user's learning effectiveness. The persona information according to Embodiment 3 is stored in the memory unit 107. As an example, the information processing device 100 may be provided with a personal profile recording unit. The personal profile recording unit analyzes the answers to questions to understand the user's personality, character, and psychological patterns, and records them as a profile in the recording unit 107. The questions can be asked at the start of programming learning, or they can be asked as appropriate during learning according to the user's emotional state. Alternatively, questions may be sent from a device different from the information processing device 100 to the user's terminal 10 via a network NW, and the answers to the questions may be analyzed to create a personal profile. Various question formats can be adopted, such as text input, multiple-choice answers, or numerical input using a rating scale, to make it easy for users to answer.

[0040] (Embodiment 4) In the above embodiment, a configuration in which the above process is executed at predetermined intervals has been described, but it is not limited to predetermined intervals, and the process may be executed for each interaction during the user's learning session. Specifically, the emotion analysis unit 112 generates the user's emotion information. The content acquisition unit 122 acquires content stored in the memory unit 107 based on the emotion information. The integration unit 120 integrates the generated emotion information, the acquired content (first content), and the input data actually entered by the user into the terminal 10 as input information, and the prompt generation unit 116 generates a prompt to be sent to the LLM system 200 based on the integrated information and sends it to the LLM system 200. The output control unit 118 outputs the response content (second content) from the LLM system 200. The recording unit 124 associates the emotion information and the second content and records them in the memory unit 107. The prompt may be changed as appropriate according to the learning session. The above process can be repeated during a learning session.

[0041] Figure 11 is a diagram illustrating the recording of the recording unit 124 according to Embodiment 4. Referring to Figure 10, the recording unit 124 records information for each dialogue during a learning session. The recording unit 124 generates a dialogue ID for each dialogue and records information including the dialogue content corresponding to the generated dialogue ID. As an example, it is shown that the recording unit 124 records information corresponding to dialogue IDs 1 to 5. As an example, the information corresponding to dialogue ID 1 is Emma's conversation associated with emotion information (enjoyment) ("Hi, AI! Today I'm going to try a for loop. I'm a little nervous.") and the AI's conversation ("Hi, Emma! It's okay, let's have fun and do it slowly together. For loops are used for repetitive processing.") The same applies to the other dialogue IDs. In this example, in addition to emotion information and the second content, the AI's conversation, i.e., the information transmitted and output from the LLM system 200, Emma's conversation, i.e., user input information, is further associated and recorded, but it is also possible to record only the AI's conversation. Furthermore, this example demonstrates a case where additional information related to the learning session is recorded in association with each dialogue ID. The additional information includes the learning subject ("Programming"), learning content ("Python for loop"), dialogue theme ("Emma and AI's Python learning session"), and dialogue date ("January 18, 2024"). By recording the additional information in association, it is possible to improve the extraction accuracy of the first content by the content acquisition unit 122. For example, the content acquisition unit 122 may use the learning subject, learning content, dialogue theme, etc., as search keys to acquire information (first content) corresponding to the dialogue ID associated with the emotion information from the same group of information stored in the memory unit 107 as the search key. Note that the additional information is not limited to the above, and may also include an index of the gist of the dialogue, keywords, and phrases identified by analyzing the dialogue content.

[0042] Furthermore, the recording unit 124 may generate data converted into vector information that conforms to natural language processing during recording and store it in the storage unit 107. The content acquisition unit 122 may acquire the first content by similarity determination based on the vectorized emotion information. Similarity determination may be made using the cosine similarity between the vector information. Storing the vectorized vector information in the storage unit 107 makes it possible to efficiently compare and search for data. In addition, the recording unit 124 may store emotion information and the second content, which is the AI ​​conversation, i.e., the information transmitted and output from the LLM system 200, in the storage unit 107 when the user is inactive and not conducting a learning session. For example, the content of the conversation with the AI ​​may be temporarily saved in RAM, and the recording process of this information may be executed when the user is inactive and not conducting a learning session, or the information may be recorded by training the LLM model. This process makes it possible to effectively utilize system resources when the system is inactive, and to direct the CPU and other processing that occurs when the user is active during a learning session to purposes other than recording, thereby efficiently supporting the user without hindering the user experience. For example, it becomes possible to efficiently enhance the learning effectiveness of users.

[0043] (Embodiment 5) In the above embodiment, text data such as messages was mainly used as an example of the type of content data, but it is not limited to this and other data such as image data may also be used.

[0044] Figure 12 is a diagram illustrating the configuration of information processing system 1# according to Embodiment 5. Referring to Figure 12, the difference from the configuration in Figure 2 is that LLM system 200 has been replaced with LLM system 200#. LLM system 200# differs in that it has a trained model 206 instead of trained model 204. The other configurations are the same, so a detailed explanation will not be repeated. Trained model 206 is a learning model trained using an image database (DB) or the like that has a large amount of image data for generating answers. When a question is queried to LLM system 200#, an answer is output from LLM system 200#. As an example, the information processing device 100 according to Embodiment 5 queries LLM system 200# for learning-related content (image data) and sends the answer (image data) from LLM system 200# to terminal 10.

[0045] For example, the prompt generation unit 116 may, based on the information compiled by the management unit 120, modify a part of Figure 5(A) as an example. Specifically, it may be changed to, "(A) Please think of an image that will increase a child's motivation to learn, for a child with the following emotional state. The image should be appropriate for the child." The prompt generation unit 116 may also, as an example, generate text using information from acquired content (image data) that was associated with emotional information ("happy") and recorded in the memory unit 107 as reference information.

[0046] The prompt generation unit 116 generates a prompt and outputs it to the LLM system 200#. The output control unit 118 outputs information of the response content (image data) from the LLM system 200# to the terminal 10. The recording unit 124 associates the response content information (second content) from the LLM system 200# with the emotional information and records it in the storage unit 107. Through this process, the information processing system 1# analyzes the user's emotional state and adds information of previously outputted content as reference information, enabling it to output content to the user that is more appropriate to the user's situation, taking into account the content of previously outputted content. In other words, the information processing system 1# can appropriately grasp the user's state and efficiently support the user. For example, it can improve the user's learning effectiveness. Note that this is not limited to image data, but can also be applied to other data such as video data, audio data, file data, and control data for learning support devices. Alternatively, multiple LLM systems 200 with different purposes may be provided, and the optimal LLM system 200 may be selected to obtain learning-related answer content from the selected LLM system 200. Alternatively, a multimodal LLM system may be constructed by integrating LLM systems 200 and LLM200#, etc., with different data types, and learning-related answer content may be obtained from this system.

[0047] The information processing system according to the above embodiment mainly describes a learning support system for assisting the learning of learners (target audiences), but it is not limited to this, and the learner (target audience) is just an example; it can also be used for other purposes for different target audiences. For example, (1) it can be used in a system related to customer support services. For example, it may generate emotional information of the target audience (customers) and provide content based on that generated emotional information. Specifically, it may provide more attentive service content to customers who are angry or dissatisfied, and more detailed service content to curious and amused customers. Furthermore, by using records of conversations with customers, it is possible to provide service content optimized for the customer. By generating emotional information of users and providing service content that responds accordingly, it is possible to provide customers with detailed and accurate customer support services. (2) It can be used in a system related to marketing and e-commerce. For example, it may generate emotional information of the target audience (users), estimate their level of interest from that emotional information, and provide personalized content such as product and service recommendations and advertisements. This makes it possible to enhance marketing effectiveness. By using records of conversations with customers, it is possible to analyze consumers' personalities and psychological patterns and dynamically deliver advertising content that matches their emotions at the time, enabling the development of effective marketing strategies that align with consumer interests. (3) It can be used in medical and nursing care systems. For example, it may be possible to generate emotional information of the target patient and provide content based on that generated emotional information. Specifically, for patients experiencing high levels of anxiety or stress, content recommending relaxation techniques or consultation with a specialist may be provided. By using records of conversations with patients, it is possible to analyze the patient's emotions and provide content that includes advice on their health condition that matches those emotions. It can also be used to support the creation of care plans tailored to the patient's emotional state, or as a virtual therapist / counselor.(4) It can be used in entertainment and game systems. For example, it may generate emotional information of the target user and provide entertainment or game content based on the generated emotional information. By using records of conversations with the user, the user's emotions may be analyzed and content with an optimal story development tailored to the user's emotional state may be generated, or movie or music content that matches the emotional state may be provided. It is possible to optimize the user's entertainment experience for each individual. (5) It can be used in human resources and coaching systems. For example, it may generate emotional information of the target employee during training and provide coaching-related content based on the generated emotional information. By using records of conversations with the target person, the degree of engagement with the training and the level of understanding of the training content may be analyzed and content tailored to that may be provided. It is possible to provide optimal feedback tailored to each individual and personalized career advice. (6) It can be used in creative support systems. For example, it may generate emotional information of the target creator and provide creative content based on the generated emotional information. By using records of conversations with the creator, it is also possible to analyze the creator's emotions and provide creative content tailored to the creator's emotional state to support their creative activities. (7) It can be used in smart home systems. The system may generate emotional information about the residents and, based on that generated emotional information, provide content that includes setting information regarding environmental settings such as illuminance, humidity, and temperature within the home or vehicle. This makes it possible to create an optimally comfortable environment for the residents.

[0048] Although the present disclosure has been described in detail based on embodiments, it goes without saying that the present disclosure is not limited to these embodiments and can be modified in various ways without departing from its essence. [Explanation of Symbols]

[0049] 1,1# Information processing system, 10 Terminal, 11,101 Control unit, 12 Camera, 14 Input device, 15 Output device, 16 Microphone, 17,107 Memory unit, 50 Learning screen, 52 Character, 100 Information processing device, 110 Biometric data acquisition unit, 112 Emotion analysis unit, 114 Input data acquisition unit, 116 Prompt generation unit, 117 Input state analysis unit, 118 Output control unit, 120 Overall unit, 122 Content acquisition unit, 124 Recording unit, 200,200# Large-scale language model system, 201 Question receiving unit, 202 Answer generation unit, 204 Text database, 206 Image database.

Claims

1. A generation unit that generates emotional information based on the subject's biometric data, input operation data, or voice content, A content acquisition unit searches for and acquires a first content stored in the memory unit based on the emotion information generated by the generation unit, A control unit that selects the emotion information and at least a portion of a plurality of first contents acquired by the content acquisition unit, combines the selected plurality of first contents, or extracts only the necessary parts of the selected plurality of first contents and processes or edits them, and integrates the emotion information and the combined first contents or the processed or edited first contents, and controls the information to be input into the generating AI model as integrated information, A content generation unit generates a second content, which is the response content, from the generated AI model based on the aforementioned summary information. An output unit that outputs the second content generated by the content generation unit, An information processing system comprising a recording unit that records the aforementioned emotional information and the aforementioned second content in association.

2. It further includes an input information acquisition unit that acquires input information, which is information related to the target person. The information processing system according to claim 1, wherein the integration unit integrates the emotional information, the first content, and at least one of the following included in the input information: environmental information related to the user's learning, information about the learning environment, geographical location information, information about learning resources, information about feedback, information about the user's health status, or information about the user's learning goals.

3. The system further comprises an input state analysis unit that analyzes the aforementioned input information to generate input state information, The aforementioned input status information includes information on input proficiency, concentration level, and task completion level. The information processing system according to claim 2, wherein the control unit controls the emotion information, the first content, and the input information including the input state information.

4. The information processing system according to claim 1, wherein the integration unit extracts and processes or edits the emotional information and at least a portion of the plurality of first contents acquired by the content acquisition unit, and integrates these with the information input to the generating AI model and the persona information of the target person.

5. The information processing system according to claim 1, wherein the recording unit records the emotional information and the second content in association when the subject is inactive and not using the system.

6. A method of information processing performed by a computer, A step of generating emotional information based on the subject's biometric data, input operation data, or voice content, The steps include: searching for and retrieving the first content stored in the memory unit based on the emotional information generated in the above step; The steps include selecting the emotion information and at least a portion of a plurality of first contents, combining the selected plurality of first contents, or extracting and processing or editing only the necessary parts of the selected plurality of first contents, integrating the emotion information and the combined first contents or the processed or edited first contents, and consolidating these as unified information to be input into the generating AI model; The steps include generating a second content, which is the response content, from the generated AI model based on the aforementioned summary information, The steps include outputting the generated second content, An information processing method comprising the step of recording the aforementioned emotional information in association with the second content.

Images