Information processing system and information processing method

Abstract

To provide an information processing system capable of efficiently supporting a target person.SOLUTION: An information processing system includes: a generation unit which generates emotion information based on biometric data of a target person or input data; a content acquisition unit which acquires first content based on the emotion information generated in the generation unit; an integration unit which integrates the emotion information with the first content acquired in the content acquisition unit; a content generation unit which generates second content based on the information integrated in the integration unit; an output unit which outputs the second content generated in the content generation unit; and a recording unit which records the emotion information and the second content in association with each other.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and an information processing method. [Background technology]

[0002] For example, children's interest and curiosity can be intentionally increased by educators creating an environment for them. Furthermore, perseverance can be developed through encouragement. This shows that children can become capable through concrete support. For example, a method has been proposed that supports the improvement of non-cognitive abilities by displaying the degree of achievement of a learning plan (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-218103 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, it is important to properly understand the learner's condition when supporting children's learning. The present disclosure is intended to solve the above problem, and provides, as an example, an information processing system and an information processing method that can efficiently support learners (target persons). [Means for solving the problem]

[0005] The information processing system of the present disclosure includes a generation unit that generates emotional information based on biometric data or input data of a subject, a content acquisition unit that acquires first content based on the emotional information generated by the generation unit, a control unit that controls the emotional information and the first content acquired by the content acquisition unit, a content generation unit that generates second content based on the information controlled by the control unit, an output unit that outputs the second content generated by the content generation unit, and a recording unit that records the emotional information and the second content in association with each other.

[0006] The information processing method disclosed herein includes the steps of generating emotional information based on biometric data or input data of a subject, acquiring first content based on the generated emotional information, consolidating the emotional information and the acquired first content, generating second content based on the consolidated information, outputting the generated second content, and recording the emotional information and the second content in association with each other. [Effects of the Invention]

[0007] The information processing system and information processing method of the present disclosure can efficiently support a subject. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system 1 according to a first embodiment. [Figure 2] FIG. 1 is a diagram illustrating a configuration of an information processing system 1 according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating a configuration of functional blocks of an information processing device 100 according to a first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating processing by information processing device 100 according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of prompt and answer content according to the first embodiment. [Figure 6] FIG. 10 is a flowchart illustrating processing by information processing device 100 according to the second embodiment. [Figure 7] FIG. 11 is a flowchart illustrating an input state analysis process according to the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a prompt according to the second embodiment. [Figure 9] FIG. 10 is a flowchart illustrating processing by information processing device 100 according to a modification of the second embodiment. [Figure 10]FIG. 11 is a diagram illustrating persona information according to the third embodiment. [Figure 11] FIG. 13 is a diagram illustrating recording by a recording unit 124 according to the fourth embodiment. [Figure 12] FIG. 13 is a diagram illustrating a configuration of an information processing system 1# according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following, the same or corresponding parts in the drawings will be denoted by the same reference numerals, and their description will not be repeated in principle.

[0010] (Embodiment 1) FIG. 1 is a diagram illustrating an overview of an information processing system 1 according to a first embodiment. Referring to FIG. 1, the information processing system 1 according to the first embodiment is a system related to learning support, mainly for supporting a learner's learning. Specifically, the information processing system 1 includes a terminal 10, a network NW, an information processing device 100, and a large-scale language model system 200. The terminal 10 is, for example, a personal computer (PC) owned by a learner (user), who is a target user, and may be portable or fixed. The terminal 10 is configured to be able to communicate with the information processing device 100 via the network NW. The information processing device 100 is configured to be able to communicate with the large-scale language model system 200 via the network NW. The communication can be wireless or wired. A user is learning programming using the terminal 10. A learning screen 50 for learning programming is displayed on the terminal 10, and a character 52 is shown as an example on the learning screen 50.

[0011] The information processing device 100 communicates with a large-scale language model system 200 to receive content for increasing a learner's motivation to learn programming. The large-scale language model (LLM) system 200 is a natural language processing system that performs question and answer sessions. The LLM system 200 is a natural language processing model trained using a large amount of text data, and receives sentences as input and outputs sentences. When the LLM system 200 is applied to a system that performs question and answer sessions, when a question is queried to the LLM system 200, answer content is output from the LLM system 200. As an example, the information processing device 100 according to the present embodiment queries the LLM system 200 for a question related to learning and transmits the answer content from the LLM system 200 to the terminal 10. The terminal 10 receives answer content (message) related to learning from the LLM system 200. When learning a program, the terminal 10 outputs a message related to the user's learning received on the learning screen 50 of the terminal 10 by voice output or by displaying a speech bubble or the like. As an example, the terminal 10 may use the character 52 to output the message.

[0012] FIG. 2 is a diagram illustrating a configuration of an information processing system 1 according to the first embodiment. Referring to FIG. 2, a terminal 10 includes a control unit 11, a camera 12, a communication I / F 13, an input device 14, an output device 15, a microphone 16, a storage unit 17, and an internal bus connecting each unit. The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The camera 12 acquires image data of a user's facial expression as biometric data of the user (subject). In this example, image data of the user's facial expression, etc. is described, but this is not limiting and other information may be acquired as biometric data. For example, an infrared camera may be used to acquire information on the user's pulse wave. The microphone 16 acquires user voice data as the user's biometric data. The communication I / F 13 is connected to a network NW and executes data exchange with external devices. The input device 14 includes a mouse and a keyboard. The output device 15 includes a display device such as a display, a speaker, etc. The storage unit 17 includes various application programs, etc. For example, it stores application programs for users to learn programming. The control unit 11 executes the application programs to generate learning screens for programming learning, and displays the learning screens on a display or other display device.

[0013] The terminal 10 acquires biometric data (for example, image data and voice data) and transmits it to the information processing device 100 via the communication I / F 13. The terminal 10 acquires input data and transmits it to the information processing device 100 via the communication I / F 13. The input data in this example is, for example, input data from a keyboard, mouse, etc. that a user uses when learning a program.

[0014] The information processing device 100 includes a control unit 101, a memory unit 107, a communication I / F 103, and an internal bus connecting each unit. The control unit 101 includes a CPU, RAM, and ROM. The communication I / F 103 is connected to a network NW and transmits and receives data to and from external devices. The memory unit 107 includes various application programs, etc. For example, the memory unit 107 stores application programs, etc. for supporting the user's learning. The control unit 11 executes the application programs to realize various processes. The information processing device 100 receives biometric data, etc. transmitted from the terminal 10, and generates emotional information of the user based on the biometric data, etc. The information processing device 100 sends a question related to learning to the LLM system 200 according to the generated emotional information.

[0015] The LLM system 200 includes a question receiving unit 201, an answer generating unit 202, and a trained model 204. The question receiving unit 201 receives a question inquiry related to learning from the information processing device 100. The answer generating unit 202 analyzes the received inquiry and, based on the analysis results, uses the trained model 204 to generate answer content (a message, as an example) related to learning in response to the question inquiry and transmits it to the information processing device 100. The trained model 204 is a learning model trained using a text database (DB) or the like that has a large amount of text data for generating answers.

[0016] The information processing device 100 receives answer content (e.g., a message) related to learning from the LLM system 200 and transmits it to the terminal 10. The terminal 10 outputs the received content (e.g., a message) related to learning on the learning screen 50 of the terminal 10 during programming learning. As an example, the terminal 10 may output the answer content (message) using a character 52 provided on the learning screen 50. In this example, the LLM system 200 is configured to be provided separately from the information processing device 100. However, it is also possible to provide functions equivalent to those of the LLM system 200 within the information processing device 100. Alternatively, the LLM system 200 may perform the function of generating emotional information and the function of generating content. Furthermore, a configuration may be adopted in which the functions of the information processing device 100 and the LLM system 200 are integrated into the terminal 10. The terminal 10 itself may generate emotional information for the user and also generate answer content (e.g., a message) related to learning, and output the content directly to the learning screen 50 of the terminal 10. This configuration eliminates the need to communicate with external systems over a network, shortening response times and improving processing efficiency on the terminal 10, and also strengthening user privacy protection since the user's emotional data and other personal information is processed without being transmitted outside the terminal.

[0017] FIG. 3 is a diagram illustrating a configuration of functional blocks of information processing device 100 according to the first embodiment. With reference to FIG. 3, control unit 101 of information processing device 100 realizes various functional blocks by executing application programs stored in storage unit 107. Specifically, information processing device 100 includes a biometric data acquisition unit 110, an emotion analysis unit 112, an input data acquisition unit 114, a prompt generation unit 116, an input state analysis unit 117, an output control unit 118, a supervision unit 120, a content acquisition unit 122, and a recording unit 124. Biometric data acquisition unit 110 acquires biometric data (image data, audio data, etc.) transmitted from terminal 10 via communication I / F 103. The acquired biometric data is stored in storage unit 107.

[0018] The emotion analysis unit 112 generates emotion information by estimating the user's emotion (psychology) based on the acquired biometric data (image data, audio data, etc.). The emotion analysis unit 112 estimates (analyzes) emotions such as joy, anger, sadness, and happiness based on the biometric data. The emotion analysis unit 112 may estimate emotions such as calmness, surprise, satisfaction, boredom, disappointment, fear, relief, and anxiety, without being limited to these. Specifically, the emotion analysis unit 112 can estimate the emotion by processing the acquired image data and the user's facial expressions, such as eyelid opening, gaze, eyebrow movement, presence or absence of nose wrinkles, mouth movement, mouth opening, and pupil opening. The emotion analysis unit 112 can also estimate (analyze) the emotion using audio data, such as the content of the user's voice, sighs, and breathing sounds. The emotion analysis unit 112 may estimate the emotion using only one of the data, or may estimate the emotion using a combination of the data. Furthermore, in this example, a case has been described in which biometric data (image data) from the camera 12 is used, but the present invention is not limited to this, and biometric data may be acquired using a wearable device.

[0019] The input data acquisition unit 114 acquires input data (input key data, etc.) transmitted from the terminal 10 via the communication I / F 103. The acquired input data is stored in the storage unit 107. The input state analysis unit 117 analyzes the user's input state based on the acquired input data (input key data, etc.). Specifically, the input state analysis unit 117 estimates the user's input state by analyzing the user's input speed using a keyboard, etc., the number of input errors, etc. The content acquisition unit 122 acquires content (a message, for example) previously output to the user via the output control unit 118, which is stored in the storage unit 107, based on the emotion information generated by the emotion analysis unit 112 (first content).

[0020] The control unit 120 controls the emotion information generated by the emotion analysis unit 112 and the first content acquired by the content acquisition unit 122. The prompt generation unit 116 generates sentences to ask questions related to learning to the LLM system 200 based on the information controlled by the control unit 120. The prompt generation unit 116 outputs the generated sentences to the LLM system 200. The output control unit 118 outputs information such as answer content from the LLM system 200 to the terminal 10. The recording unit 124 associates the answer content (second content) from the LLM system 200 with the emotion information and records them in the memory unit 107.

[0021] FIG. 4 is a flow diagram illustrating processing of information processing device 100 according to the first embodiment. Referring to FIG. 4, information processing device 100 acquires biometric data (step S0). Specifically, biometric data acquisition unit 110 acquires biometric data transmitted from terminal 10. Next, information processing device 100 determines whether a predetermined period has elapsed (step S2). If information processing device 100 determines that the predetermined period has not elapsed (NO in step S2), it returns to step S0 and repeats the above processing. The predetermined period can be set to any period, and can be set to five minutes, for example. If information processing device 100 determines that the predetermined period has elapsed (YES in step S2), it proceeds to the next processing. Information processing device 100 executes processing to analyze emotions (step S4). Specifically, emotion analysis unit 112 estimates the user's emotions (mentality) based on the biometric data (image data, audio data, etc.) stored in storage unit 107 when the predetermined period has elapsed, and generates emotion information. Note that a portion of the biometric data (image data, audio data, etc.) stored in the storage unit 107 within a predetermined period may be used, or all of the data may be used. Next, the information processing device 100 acquires a first content based on the emotional information generated by the emotion analysis unit 112 (step S5). Specifically, the content acquisition unit 122 acquires previously output content (first content) stored in the storage unit 107 via the output control unit 118 based on the emotional information generated by the emotion analysis unit 112. The content acquisition unit 122 searches for previously output content related to the emotional information stored in the storage unit 107 and acquires the content obtained by the search result. For example, the content acquisition unit 122 acquires content (e.g., information on a previously output message) recorded in association with the same emotional information as the emotional information stored in the storage unit 107. Next, the information processing device 100 manages the emotional information and the acquired content (first content) (step S6). Specifically, the management unit 120 manages the emotional information and the acquired content (first content).Furthermore, if the content acquisition unit 122 acquires multiple pieces of content, the control unit 120 may control all of them, or may select and control one of them. Alternatively, the control unit 120 may control information that combines multiple pieces of content appropriately, or extracts only necessary parts and processes and edits them as appropriate. Next, the information processing device 100 generates a prompt to be transmitted to the LLM system 200 (step S7). Specifically, the prompt generation unit 116 generates a sentence that asks the LLM system 200 a question related to learning based on the information controlled by the control unit 120. Next, the information processing device 100 transmits the generated prompt to the LLM system 200 (step S8). Specifically, the prompt generation unit 116 transmits the generated prompt, which is a sentence that queries the LLM system 200. Next, the information processing device 100 determines whether or not answer content has been received from the LLM system 200 (step S10). Specifically, the output control unit 118 determines whether or not answer content has been received from the LLM system 200. In step S10, the information processing device 100 maintains that state until it receives answer content from the LLM system 200. When the information processing device 100 receives answer content from the LLM system 200 (YES in step S10), the information processing device 100 outputs information such as the answer content (e.g., a message) (second content) to the terminal 10 (step S14). The output control unit 118 transmits message output information, including the message, to the terminal 10. When multiple answer contents are received from the LLM system 200, the output control unit 118 may select one of them. When multiple answer contents are received, the output control unit 118 may combine them as appropriate, or extract only necessary parts and process and edit them as appropriate to generate a message that is the answer content. Next, the information processing device 100 associates emotion information with the answer content (second content) and records them (step S16). Specifically, the recording unit 124 associates the answer content (e.g., a message) with the emotion information and stores them in the storage unit 107.The recording unit 124 may store not only the answer content (a message, for example) but also other related information in the storage unit 107 in association with the answer content. Next, the information processing device 100 determines whether to end the process (step S18). If the information processing device 100 determines in step S18 that the process should be ended, the process ends (END). Specifically, if the information processing device 100 determines that the user has ended an application program for learning programming on the terminal 10, the process ends. For example, the information processing device 100 may end the process when it receives a command to end the application program for learning programming from the terminal 10. On the other hand, if the information processing device 100 determines in step S18 that the process should not be ended (NO in step S18), the process returns to step S0 and repeats the above process.

[0022] FIG. 5 is a diagram illustrating an example of a prompt and answer content according to the first embodiment. FIG. 5(A) shows a prompt that the prompt generation unit 116 transmits to the LLM system 200 based on information integrated by the integration unit 120. As an example, the prompt generation unit 116 generates the following prompt based on the information integrated by the integration unit 120. Specifically, the content acquisition unit 122 acquires first content based on emotion information ("fun") generated by the emotion analysis unit 112. In this example, the content acquisition unit 122 acquires previously output answer content (a message, for example) (first content) that is recorded in association with the emotion information ("fun"). The integration unit 120 integrates the emotion information ("fun") and information on the previously output message. Based on the information managed by the management unit 120, the prompt generation unit 116 generates, as an example, a sentence such as "(A) Think of something to say to a child who is in the following emotional state that will increase the child's motivation to learn. When you are talking to them, please use words that are appropriate for children. They are learning programming. (B) They find learning fun. (C) There are some things they don't understand, and they are a little frustrated. (D) They feel it would be frustrating to give up." The prompt generation unit 116 also generates, as an example, a sentence that uses as reference information acquired content (e.g., a message) that has been associated with emotional information ("fun") and recorded in the memory unit 107. The prompt generation unit 116 generates, as an example, a sentence such as "(E) You have always been good at learning new things, and you have the strength to overcome even small difficulties. I'm sure you can do well on this assignment in the While statement."

[0023] FIG. 5B shows an example of an answer content from the LLM system 200 in response to the above prompt. For example, the information processing device 100 receives the answer content, "Great! Programming is fun, isn't it? You did really well on the previous While statement assignment. But sometimes there are difficult parts, so it can be a little frustrating. Even if you don't understand something, it's a chance to gradually absorb it! Everyone was full of things they didn't understand at first. Let's work through those difficult parts together! Then, new and interesting things might be waiting for you!" The information processing device 100 transmits information about the received answer content to the terminal 10. The terminal 10 outputs a learning screen for learning the program, along with a message related to the user's learning, which is the answer content transmitted from the information processing device 100. Through this process, the information processing system analyzes the user's emotional state and adds information about previously output messages as reference information. This enables the information processing system to output a message to the user that is more in line with the user's situation and empathizes with the user, taking into account the content of the previously output messages.

[0024] Conventionally, a problem exists in that a message that sympathizes with a user's past situation is output because the message is completed with the user's dialogue for each session and does not reflect the content of previous dialogues. However, the information processing system of the present disclosure outputs a message that takes into account the content of messages previously output. Furthermore, the content of previously output messages recorded in association with emotional information is added as reference information, so that a message that is more appropriate to the user's emotion is output to the user. That is, the information processing system can appropriately grasp the user's state and efficiently support the user. For example, the user's learning effectiveness can be improved. Furthermore, the information processing system according to the first embodiment can grasp the user's emotional state (joy, anger, sadness, fun, etc.) to understand the user's current psychological readiness and readiness for learning. This makes it possible to provide a message appropriate to the user's emotion at the moment when the user is most receptive, thereby efficiently supporting the user. Furthermore, for example, the user's learning effectiveness can be improved. Furthermore, for example, when negative emotions such as frustration or anxiety are detected early in a user's learning, an appropriate message appropriate to the emotion can be output, thereby providing appropriate support or intervention before the user gives up. For example, when a user is in a positive emotional state, appropriate messages can be output that match that emotion, thereby enhancing engagement in learning. Furthermore, by understanding the user's emotional state, a more personalized learning experience can be provided. For example, if a user is feeling sadness or stress, the learning process can be optimized and learning efficiency can be improved by encouraging easier tasks or taking a break. Learning in an emotionally balanced state can help solidify memories and deepen understanding.

[0025] [Example of generating prompts] In the above, the prompt generation unit 116 generates a prompt based on emotional information ("fun"), which is information integrated by the integration unit 120, and the first content (a message associated with "fun" and recorded in the storage unit 107). The prompt generation unit 116 can similarly generate a prompt for information integrated by the integration unit 120 that includes other emotional information. For example, the prompt generation unit 116 generates a prompt based on emotional information ("anger"). For example, the prompt generation unit 116 may change sentence (B) of the above sentences (A) to (E), which are prompts, to "I feel angry about studying." The prompt generation unit 116 may also fix sentence (A) and change sentences (B) to (D) as appropriate based on the emotional information. As an example, the prompt generation unit 116 generates a sentence using, as reference information, the acquired content that has been associated with emotional information ("anger") and recorded in the storage unit 107. As an example, it generates the sentence, "(E) You may not be good at this assignment with a while statement, but you did very well on the assignment with a for statement. So I'm sure you'll do well this time too."

[0026] In response to the prompt, the information processing device 100 receives, for example, answer content such as "Anger is an emotion that arises when learning something new. If you take one step at a time, the anger will gradually disappear. Remember the time when you were doing the "for" sentence assignment. Don't miss your growth." The information processing device 100 transmits information such as the received answer content to the terminal 10. For example, the prompt generation unit 116 generates a prompt based on emotional information ("sad"). For example, the prompt generation unit 116 may change sentence (B) of the above sentences (A) to (E), which are the prompt, to "I feel sad about learning." The prompt generation unit 116 may also keep sentence (A) fixed and change sentences (B) to (D) as appropriate based on the emotional information. For example, the prompt generation unit 116 generates a sentence using, as reference information, the acquired content that has been associated with emotional information ("sad") and stored in the storage unit 107. As an example, the information processing device 100 generates a sentence such as, "(E) You worked really hard and patiently on both the assignments with the While statement and the assignments with the For statement. So, don't give up this time either, and keep trying your best." In response to the prompt, the information processing device 100 receives an answer content such as, "Try to think of sadness as a step toward growth. It's natural to have things you don't understand, like the assignments with the While statement and the For statement, but it's important to have the attitude of working hard to overcome them." The information processing device 100 transmits information such as the received answer content to the terminal 10. For example, the prompt generation unit 116 generates a prompt based on emotional information ("joy"). For example, the prompt generation unit 116 may change sentence (B) of the above sentences (A) to (E), which are prompts, to "I feel joy in learning." Furthermore, the prompt generation unit 116 may keep sentence (A) fixed and change sentences (B) to (D) as appropriate based on the emotional information. Furthermore, as an example, the prompt generating unit 116 generates a sentence using the acquired content that has been associated with emotional information ("joy") and recorded in the storage unit 107 as reference information.As an example, the information processing device 100 generates a sentence such as "(E) You really enjoyed working on both the assignments with the While statement and the assignments with the For statement. Good luck with that adjustment." In response to the prompt, the information processing device 100 receives an answer content such as "Your motivation for learning is great. That was clearly evident both in the assignments with the While statement and the assignments with the For statement. By proceeding with a sense of enjoyment, you can continue to grow." The information processing device 100 transmits information such as the received answer content to the terminal 10.

[0027] [Generation of emotion information by emotion analysis unit 112] The emotion analysis unit 112 may generate emotion information by estimating the user's emotion (psychology) based on input data rather than biometric data. Specifically, the emotion information may be generated based on input data (e.g., input key data) acquired by the input data acquisition unit 114. For example, an emotion diagnostic test may be executed at the start of programming learning to diagnose the user's emotion based on their input. The emotion analysis unit 112 may generate emotion information based on the user's input data in the emotion diagnostic test in response to multiple questions used to diagnose the user's emotion. The emotion diagnostic test may be performed in a manner that is not particularly limited, and may prompt the user to select questions or images that are less likely to be noticed by the user, and emotion information may be generated based on the answers. The input data may be any data that can be used to generate emotion information. The input data is not limited to input key data from a keyboard or the like, and may include, for example, user behavior history data. The emotion analysis unit 112 is not limited to directly using the input data. For example, the emotion analysis unit 112 may generate emotion information using user input state information analyzed by the input state analysis unit 117. For example, the state of actual programming input during programming learning may be analyzed, and emotion information may be generated based on the input state information that is the analysis result. Alternatively, emotion information may be generated based on the analysis result of the content of the user's input data or the content of the context as input state information.

[0028] (Embodiment 2) FIG. 6 is a flow diagram illustrating processing by information processing device 100 according to the second embodiment. Referring to FIG. 6, the processing by information processing device 100 according to the second embodiment differs from the flow diagram of FIG. 4 in that steps S1 and S5A are added. The other configurations are the same as those described above, and detailed description thereof will not be repeated. Information processing device 100 acquires biometric data in step S0, and then acquires input data (step S1). Specifically, input data acquisition unit 114 acquires input data transmitted from terminal 10. Next, information processing device 100 determines whether a predetermined period has elapsed (step S2). If information processing device 100 determines that the predetermined period has not elapsed (NO in step S2), it returns to step S0 and repeats the above processing. The predetermined period can be set to any period, and can be set to five minutes, for example. If information processing device 100 determines that the predetermined period has elapsed (YES in step S2), it proceeds to the next processing. In step S4, the information processing device 100 executes a process of analyzing emotions when a predetermined period of time has elapsed, and then in step S5, acquires the first content based on the emotion information generated by the emotion analysis unit 112. Next, the information processing device 100 executes an input state analysis process (step S5A). Specifically, the input state analysis unit 117 analyzes the user's input state based on the input data (input key data, etc.) stored in the storage unit 107. Note that the input state analysis unit 117 may use some or all of the input data (input key data, etc.) stored in the storage unit 107 within the predetermined period of time.

[0029] FIG. 7 is a flow diagram illustrating an input state analysis process according to the second embodiment. Referring to FIG. 7, the input state analysis unit 117 evaluates the input proficiency level based on the acquired input data (step S30). Specifically, the input state analysis unit 117 estimates the user's input proficiency level by analyzing information about the input data, such as the user's input speed using a keyboard or the like and the number of input errors. The input state analysis unit 117 may estimate and classify the input proficiency level as high or low, or may further classify the input proficiency level into a plurality of levels. Next, the input state analysis unit 117 evaluates the concentration level based on the acquired input data (step S32). Specifically, the input state analysis unit 117 analyzes information about the input data, such as the length of the user's input period using a keyboard or the like, to evaluate the user's concentration level. For example, the input state analysis unit 117 can estimate that the user's concentration level is high if the length of the input period during a certain period is long. The input state analysis unit 117 may estimate that the concentration level is high if the input period during a certain period is equal to or greater than a certain threshold, and may estimate that the concentration level is not high otherwise. Next, the input state analysis unit 117 evaluates the degree of task achievement based on the acquired input data (step S34). Specifically, the input state analysis unit 117 evaluates the degree of achievement of the programming learning task given based on the input data as information of the input data. Specifically, the input state analysis unit 117 may evaluate the degree of achievement of the programming learning task by comparing correct answer information for the programming learning task given with the programming input data currently input by the user. Then, the input state analysis unit 117 ends the input state analysis process (return).

[0030] 6 again, the information processing device 100 integrates the emotional information, the acquired content (first content), and the input information (step S6). Specifically, the integration unit 120 integrates the emotional information, the acquired content (first content), and the input information including input state information such as input proficiency, concentration level, and task achievement level. Next, the information processing device 100 generates a prompt to be sent to the LLM system 200 (step S7). Specifically, the prompt generation unit 116 generates a sentence to ask the LLM system 200 a question related to learning based on the information integrated by the integration unit 120.

[0031] FIG. 8 is a diagram illustrating an example of a prompt according to the second embodiment. FIG. 8 illustrates a prompt that the prompt generation unit 116 transmits to the LLM system 200 based on input information including emotional information, acquired content (first content), and input state information. This prompt differs from the prompt described in FIG. 5(A) in that multiple pieces of reference information are generated. Specifically, the prompt generation unit 116 generates a sentence, for example, using the acquired content that has been associated with the emotional information ("fun") and stored in the storage unit 107 as reference information 1. For example, the prompt generation unit 116 generates a sentence such as, "(E), you've always been good at learning new things, and you have the strength to overcome even small difficulties. I'm sure you'll do well on this While statement assignment." Furthermore, when generating a prompt, the prompt generation unit 116 includes a sentence with input state information as reference information 2. For example, in addition to the sentences (A) to (E) described in FIG. 5(A), the prompt generation unit 116 can include sentences related to information such as input proficiency, concentration, and achievement as the user's input state information. As an example, a sentence such as "(F) Input proficiency is high. Concentration level is high. Task achievement level is high" is generated. The subsequent processing is the same as that explained in the flowchart of FIG. 4, and therefore detailed explanation thereof will not be repeated.

[0032] The information processing system according to the second embodiment of the present disclosure outputs a message based on the content of a previously output message. Furthermore, since current input information is also added as reference information when generating a prompt, a message more tailored to the user's state is output to the user. That is, the information processing system can appropriately grasp the user's state and efficiently support the user. For example, it is possible to improve the user's learning effectiveness. In this example, the input state information includes information on input proficiency, concentration level, and task achievement level. However, any one of these may be included, or information indicating multiple input states may be added. Furthermore, the content included in the generated prompt may be appropriately changed depending on the emotional information. For example, if the emotional information is "anger," the sentence whose input state information is reference information 2 may not be included. In this example, the configuration in which the control unit 120 controls emotional information, acquired content (first content), and input information including input state information obtained by analyzing the input state has been described. However, the control unit 120 may also control input data actually input by the user to the terminal 10 as input information and generate a prompt to be transmitted to the LLM system 200 as reference information. For example, the input information may be user input information (e.g., questions, feedback, learning preferences and interests, etc.). The input information may also be environmental information related to the user's learning. For example, the input information may include information about the learning environment (e.g., a quiet room, a library, a cafe, etc., or the time of day (e.g., morning, evening, etc.)) that may affect the user's concentration and learning efficiency, temperature, humidity, geographic location, information about the learning resources the user has access to (e.g., books, online learning materials, educational apps), information about the user's social and cultural background that may affect their learning style and response to learning materials, and information about external evaluations and feedback from teachers, family, etc. The input information may also include information about the user's health condition and lifestyle, or the user's specific learning goals and expectations.

[0033] (Modification of the second embodiment) FIG. 9 is a flow diagram illustrating the processing of information processing device 100 according to a modification of the second embodiment. Referring to FIG. 9, the processing of information processing device 100 according to the modification of the second embodiment differs from the flow diagram of FIG. 6 in that steps S5B and S12 are added. The remaining configuration is the same as that described in the flow diagram of FIG. 6, and therefore detailed description thereof will not be repeated. After performing an input state analysis process in step S5A, information processing device 100 then determines whether the analyzed input state information indicates a high level of concentration (step S5B). Specifically, prompt generation unit 116 determines whether the analyzed input state information indicates a high level of concentration as a result of analysis by input state analysis unit 117. If information processing device 100 determines in step S5B that the analyzed input state does not indicate a high level of concentration (NO in step S5B), it proceeds to step S6. The subsequent processing is the same as that described in the flow diagram of FIG. 6. On the other hand, in step S5B, if information processing device 100 determines that the analyzed input state indicates a high level of concentration (YES in step S5B), it skips steps S6 to S16 and proceeds to step S18. Specifically, if prompt generation unit 116 determines that the level of concentration is high as the analysis result of input state analysis unit 117, it does not execute the prompt generation process. In other words, if the level of concentration is high, information processing device 100 does not output content (a message, as an example) to terminal 10.

[0034] Outputting a message when the user's level of concentration is high may disrupt the user's thinking. Therefore, the information processing system can efficiently support the user by stopping content output in this state. For example, the user's learning effectiveness can be efficiently improved. In this example, the input state analysis unit 117 determines the level of concentration based on input data. However, this is not limited to this, and the level of concentration may be determined in combination with emotional information. In this example, the information processing system determines whether to output content based on the user's level of concentration. However, the content output frequency may be adjusted based on the user's level of concentration. For example, when the user's level of concentration is high, the information processing system may lengthen the interval between predetermined periods of processing in step S2. By adjusting the content output frequency based on the user's level of concentration, the information processing system can efficiently support the user. For example, the user's learning effectiveness can be efficiently improved.

[0035] Furthermore, after receiving the answer content from the LLM system 200 in step S10, the information processing device 100 sets parameters for content output (step S12). Specifically, the output control unit 118 sets the parameters for content output based on emotion information. For example, the output control unit 118 may set color parameters for a message when content is output based on emotion information. For example, the output control unit 118 may set parameters for text color, such as orange, red, blue, or yellow, based on emotion information such as "joy," "anger," "sadness," or "fun," and output the content. The output control unit 118 can efficiently support the user by utilizing the psychological effects of color when outputting content. For example, it can efficiently improve the user's learning effectiveness. Note that the color parameter settings are merely examples, and other colors may be set. Content may also be output by changing brightness or other parameters based on emotion information. Furthermore, when content is output using a character 52, the output control unit 118 may adjust the movement of the character 52 based on emotion information. For example, multiple movement patterns of the character 52 may be prepared in advance, and the output control unit 118 may select one of the multiple movement patterns based on emotional information and display it on the terminal 10. For example, movement patterns corresponding to "joy," "anger," "sadness," and "fun" may be prepared in advance, and the output control unit 118 may set a pattern from the multiple movement patterns that matches the emotional information when setting parameters for content output. For example, the information processing system can efficiently support the user by displaying the movement of the character 52 that matches the emotional information when outputting content. For example, it is possible to efficiently improve the user's learning effect. Furthermore, when outputting content using the character 52, the terminal 10 may display a speech bubble to make the character 52 speak, or may output the content (message) as audio. The output control unit 118 may adjust the pattern of audio content output based on the emotional information.For example, a plurality of audio output patterns may be prepared in advance, and the output control unit 118 may select and output one of the plurality of audio output patterns based on emotional information. For example, audio output patterns corresponding to "joy," "anger," "sadness," and "fun" may be prepared in advance, and the output control unit 118 may set parameters for outputting content as audio so that one of the plurality of audio output patterns is a pattern that matches the emotional information. For example, when outputting content as audio, the information processing system can efficiently support the user by outputting the content as audio in an audio output pattern that matches the emotional information. For example, the user's learning effect can be efficiently improved. Note that the output control unit 118 may use the above parameter setting methods alone or in combination.

[0036] (Embodiment 3) The information processing device 100 according to the third embodiment manages emotion information, acquired content (first content), and persona information. Specifically, the management unit 120 manages the emotion information, acquired content (first content), and persona information. Then, the prompt generation unit 116 generates a sentence to ask the LLM system 200 a question related to learning based on the information managed by the management unit 120.

[0037] FIG. 10 is a diagram illustrating persona information according to the third embodiment. Referring to FIG. 10, user profile data is shown as the persona information. Specifically, the profile data includes basic information and information related to the user's learning tendencies. The information related to learning tendencies includes information related to learning style assessment, information related to ability and grades, information related to time management and study plans, information related to motivation and motivation, and information related to emotions and stress management. The basic information includes information related to name, gender, age, grade, subjects of interest, subjects of weakness, attitude toward learning, and motivation. The information related to learning style assessment includes information related to type, preferences, interests in subjects, hobbies, and activities. The information related to ability and grades includes information related to current situation, strengths, and weaknesses. The information related to time management and study plans includes information related to daily learning and time management. The information related to motivation and motivation includes information related to motivation and successful experiences. The information related to emotions and stress management includes information related to anxiety and stress.

[0038] When generating a prompt, the prompt generation unit 116 includes a sentence that uses persona information as reference information 2. For example, the prompt generation unit 116 can include a sentence related to persona information in addition to the sentences (A) to (E) described in FIG. 5(A). As an example, the prompt generation unit 116 generates the sentence of the profile data described in FIG. 10. The prompt generation unit 116 then transmits the generated prompt, which is a sentence that queries the LLM system 200. Other points are the same as those described in the first embodiment, and therefore detailed description thereof will not be repeated. Furthermore, it may be combined with the configuration of the second embodiment.

[0039] The information processing system according to the third embodiment of the present disclosure outputs a message based on the content of a previously output message. Furthermore, since persona information is also added as reference information when generating a prompt, a message more tailored to the user's state is output to the user. That is, the information processing system can appropriately grasp the user's state and efficiently support the user. For example, it is possible to improve the user's learning effectiveness. The persona information according to the third embodiment is stored in the storage unit 107. As an example, the information processing device 100 may be provided with a personal profile recording unit. The personal profile recording unit analyzes answers to questions to learn the user's individual personality, character, and psychological patterns, and records the answers as a profile in the recording unit 107. The questions can be asked at the start of programming learning, or can be asked appropriately during learning depending on the user's emotional state. Furthermore, questions may be sent to the user's terminal 10 via a network NW from a device other than the information processing device 100, and a personal profile may be created by analyzing the answers to the questions. The questions can be posed in a variety of ways, including text entry, multiple choice answers, or numerical input on a rating scale, allowing the user to easily answer the questions.

[0040] (Fourth embodiment) In the above embodiment, the above processing is performed at predetermined intervals. However, the processing may be performed for each interaction during a user's learning session. Specifically, the emotion analysis unit 112 generates emotion information about the user. The content acquisition unit 122 acquires content stored in the storage unit 107 based on the emotion information. The control unit 120 integrates the generated emotion information, the acquired content (first content), and the input data actually entered by the user into the terminal 10 as input information. The prompt generation unit 116 generates a prompt to be sent to the LLM system 200 based on the integrated information, and transmits the prompt to the LLM system 200. The output control unit 118 outputs answer content (second content) from the LLM system 200. The recording unit 124 associates the emotion information with the second content and records it in the storage unit 107. The prompt may be changed as appropriate depending on the learning session. The above processing can be repeated during a learning session.

[0041] FIG. 11 is a diagram illustrating recording by the recording unit 124 according to the fourth embodiment. Referring to FIG. 10, the recording unit 124 records information for each dialogue during a learning session. The recording unit 124 generates a dialogue ID for each dialogue and records information including the dialogue content corresponding to the generated dialogue ID. As an example, the recording unit 124 records information corresponding to dialogue IDs 1 to 5. As an example, the information corresponding to dialogue ID 1 is Emma's dialogue associated with emotional information (excitement) (“Hello, AI! Today we're going to try a for statement. I'm a little nervous, though.”) and the AI's dialogue (“Hello, Emma! Don't worry, let's have fun and take it easy together. The for statement is used for repetitive processing.”). The same applies to other dialogue IDs. Note that in this example, a case is described in which Emma's dialogue, i.e., user input information, is further associated and recorded in addition to the emotional information and the second content, i.e., the AI's dialogue, i.e., information transmitted and output from the LLM system 200. However, only the AI's dialogue may be recorded. In this example, additional information about the learning session is also associated with and recorded in association with each dialogue ID. The additional information includes the learning subject ("Programming"), the learning content ("Python for statement"), the dialogue theme ("Emma and AI's Python learning session"), and the dialogue date ("January 18, 2024"). By associating and recording the additional information, it is possible to improve the accuracy of extraction of the first content by the content acquisition unit 122. For example, the content acquisition unit 122 may use the learning subject, the learning content, the dialogue theme, or the like as a search key to acquire information (first content) corresponding to the dialogue ID associated with emotion information from a group of information stored in the storage unit 107 that is identical to the search key. Note that the additional information is not limited to the above, and may include a summary of the dialogue, keywords, or phrases obtained by analyzing the dialogue content as an index.

[0042] When recording, the recording unit 124 may generate data converted into vector information according to natural language processing and store the data in the storage unit 107. The content acquisition unit 122 may acquire the first content by determining similarity based on the vectorized emotion information. The similarity determination may be performed using cosine similarity between vector information. Storing the vectorized vector information in the storage unit 107 enables efficient data comparison and search. Furthermore, the recording unit 124 may store the emotion information and the second content, i.e., the AI ​​conversation, i.e., the information transmitted and output from the LLM system 200, in the storage unit 107 during an inactive period when the user is not engaged in a learning session. For example, the content of the conversation with the AI ​​may be temporarily saved in RAM or the like, and a recording process for the information may be performed during an inactive period when the user is not engaged in a learning session, or the information may be recorded by training the LLM model. This process enables effective use of system resources during inactive periods and enables the CPU and other processing during active periods when a learning session is being performed to be directed away from recording, thereby efficiently supporting the user without disrupting the user experience. For example, it is possible to efficiently improve the learning effect of the user.

[0043] (Embodiment 5) In the above embodiment, the content data type has been mainly explained as text data such as messages, but the content data type is not limited to this and may be other data such as image data.

[0044] FIG. 12 is a diagram illustrating the configuration of information processing system 1# according to embodiment 5. Referring to FIG. 12, compared to the configuration of FIG. 2, LLM system 200 is replaced with LLM system 200#. LLM system 200# is different in that trained model 206 is provided instead of trained model 204. The other configurations are similar, so detailed description thereof will not be repeated. Trained model 206 is a learning model trained using an image database (DB) or the like having a large amount of image data for generating an answer. When a question is queried from LLM system 200#, an answer is output from LLM system 200#. Information processing device 100 according to embodiment 5, for example, queries LLM system 200# about content (image data) related to learning and transmits the answer (image data) from LLM system 200# to terminal 10.

[0045] For example, the prompt generation unit 116 may change part of FIG. 5(A) based on the information managed by the management unit 120. Specifically, the prompt may be changed to "(A) For a child in the following emotional state, please think of an image that will increase the child's motivation to learn. Please make the image appropriate for the child." Also, for example, the prompt generation unit 116 generates a sentence using, as reference information, information on the acquired content (image data) that has been associated with the emotional information ("fun") and stored in the storage unit 107.

[0046] The prompt generation unit 116 generates a prompt and outputs it to the LLM system 200#. The output control unit 118 outputs information on the answer content (image data) from the LLM system 200# to the terminal 10. The recording unit 124 associates the information on the answer content (second content) from the LLM system 200# with emotional information and records it in the storage unit 107. Through this processing, the information processing system 1# can analyze the user's emotional state and add information on previously output content as reference information, thereby outputting content that is more suited to the user's situation and taking into account the content of the previously output content. In other words, the information processing system 1# can appropriately grasp the user's state and efficiently support the user. For example, it can improve the user's learning effectiveness. Note that the present invention is not limited to image data, and other data, such as video data, audio data, file data, and control data for devices used to support learning, can also be applied. Alternatively, multiple LLM systems 200 with different uses may be provided, and the most suitable LLM system 200 may be selected, and answer content related to learning may be obtained from the selected LLM system 200. A multimodal LLM system may be constructed that integrates LLM systems 200 and LLMs 200# with different data types, and answer content related to learning may be obtained from that system.

[0047] The information processing system according to the above embodiment has been described as a learning support system for supporting the learning of learners (target individuals). However, the present invention is not limited to this. The target individuals are merely examples, and the system can be used for other purposes with other target individuals. For example, (1) the system can be used in a system related to customer support services. For example, emotional information of the target customers can be generated and content can be provided based on the generated emotional information. Specifically, service content that provides a more attentive response can be provided to customers who are angry or frustrated, and more detailed service content can be provided to customers who are curious or happy. Furthermore, by using records of interactions with customers, it is possible to provide service content optimized for each customer. By generating user emotional information and providing service content that responds accordingly, it is possible to provide customers with detailed and accurate customer support services. (2) The system can be used in a system related to marketing and e-commerce. For example, emotional information of the target users can be generated, and the level of interest can be estimated from the emotional information, and content such as personalized product and service recommendations and advertisement distribution can be provided. This can improve marketing effectiveness. By using records of customer interactions, it is possible to analyze consumers' personalities and psychological patterns and dynamically deliver advertising content that matches their emotions at any given time, enabling the development of effective marketing strategies tailored to consumer interests. (3) It can be used in medical and nursing care systems. For example, emotional information about the target patient can be generated and content can be provided based on that information. Specifically, for patients experiencing increased anxiety or stress, content recommending relaxation techniques or consulting a specialist can be provided. By using records of patient interactions, it is possible to analyze the patient's emotions and provide content including health advice tailored to those emotions. It can also be used to support the creation of care plans tailored to the patient's emotional state or as a virtual therapist / counselor.(4) The present invention can be used in entertainment and game systems. For example, emotional information of a target user can be generated and entertainment or game content can be provided based on the generated emotional information. Records of user interactions can be used to analyze the user's emotions and generate content with an optimal story development tailored to the user's emotional state, or to provide movie or music content that matches the user's emotional state. This can optimize the user's entertainment experience for each individual. (5) The present invention can be used in human resources and coaching systems. For example, emotional information of a target employee during training can be generated and coaching content can be provided based on the generated emotional information. Records of user interactions can be used to analyze the level of training engagement and understanding of the training content, and content tailored to the training content can be provided. This can provide optimal feedback tailored to each individual and personalized career advice. (6) The present invention can be used in creative support systems. For example, emotional information of a target creator can be generated and creative content can be provided based on the generated emotional information. Records of user interactions can be used to analyze the creator's emotions and provide creative content tailored to the creator's emotional state, thereby supporting creative activities. (7) The present invention can be used in smart home systems. Emotional information of the target resident may be generated, and content including setting information on environmental settings such as illuminance, humidity, and temperature in the home or car may be provided based on the generated emotional information. This makes it possible to create an optimal and comfortable environment for the resident.

[0048] The present disclosure has been specifically described above based on the embodiments, but it goes without saying that the present disclosure is not limited to the embodiments and can be modified in various ways without departing from the spirit of the present disclosure. [Explanation of symbols]

[0049] 1,1# Information processing system, 10 terminal, 11,101 control unit, 12 camera, 14 input device, 15 output device, 16 microphone, 17,107 memory unit, 50 learning screen, 52 character, 100 information processing device, 110 biometric data acquisition unit, 112 emotion analysis unit, 114 input data acquisition unit, 116 prompt generation unit, 117 input state analysis unit, 118 output control unit, 120 management unit, 122 content acquisition unit, 124 recording unit, 200,200# large-scale language model system, 201 question receiving unit, 202 answer generation unit, 204 sentence database, 206 image database.

Claims

1. a generation unit that generates emotion information based on biometric data or input data of a subject; a content acquisition unit that acquires a first content based on the emotion information generated by the generation unit; a control unit that controls the emotion information and the first content acquired by the content acquisition unit; a content generation unit that generates second content based on information managed by the management unit; an output unit that outputs the second content generated by the content generation unit; an information processing system comprising: a recording unit that records the emotion information and the second content in association with each other;

2. further comprising an input information acquisition unit that acquires input information; The information processing system according to claim 1 , wherein the control unit controls the emotion information, the first content, and the input information.

3. an input status analysis unit that analyzes the input information and generates input status information; The information processing system according to claim 2 , wherein the input information includes the input status information.

4. The information processing system according to claim 1 , wherein the control unit controls the emotion information, the first content acquired by the content acquisition unit, and persona information of the target person.

5. The information processing system according to claim 1 , wherein the recording section records the emotion information and the second content in association with each other during an inactive state.

6. generating emotion information based on biometric data or input data of a subject; acquiring a first content based on the generated emotion information; a step of integrating the emotion information and the acquired first content; generating second content based on the aggregated information; outputting the generated second content; and recording the emotion information and the second content in association with each other.

Images