system

Abstract

The system according to this embodiment aims to analyze the emotional state of conversation participants in real time and promote constructive dialogue. [Solution] The system according to the embodiment comprises an emotion analysis unit, a dialogue mediation unit, an advice provision unit, and a follow-up unit. The emotion analysis unit analyzes the emotional state of the conversation participants in real time. The dialogue mediation unit analyzes and understands the conversation between the parties based on the emotional state analyzed by the emotion analysis unit and makes constructive suggestions and questions at appropriate times. The advice provision unit provides advice on improving communication optimized for each participant based on the suggestions and questions made by the dialogue mediation unit. The follow-up unit periodically monitors the relationship after the dialogue based on the advice provided by the advice provision unit and provides additional support as needed.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, there is a problem that it is difficult to grasp the emotional state of the participants in a conversation in real time and make constructive proposals or questions at an appropriate timing.

[0005] The system according to the embodiment aims to analyze the emotional state of the participants in a conversation in real time and promote constructive conversations.

Means for Solving the Problems

[0006] The system according to this embodiment comprises an emotion analysis unit, a dialogue mediation unit, an advice provision unit, and a follow-up unit. The emotion analysis unit analyzes the emotional state of the conversation participants in real time. The dialogue mediation unit analyzes and understands the conversation between the parties based on the emotional state analyzed by the emotion analysis unit and makes constructive suggestions and questions at appropriate times. The advice provision unit provides advice on improving communication optimized for each participant based on the suggestions and questions made by the dialogue mediation unit. The follow-up unit periodically monitors the relationship after the dialogue based on the advice provided by the advice provision unit and provides additional support as needed. [Effects of the Invention]

[0007] The system according to this embodiment can analyze the emotional state of conversation participants in real time and promote constructive dialogue. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor, an antenna, etc. The communication I / F manages communication between a plurality of computers. Examples of communication standards applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 comprises a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The receiving device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The Harmony AI System according to an embodiment of the present invention is an AI support system for smoothly resolving interpersonal conflicts in the workplace and daily life, and for building and maintaining good relationships. This Harmony AI System combines voice emotion analysis technology and natural language processing to analyze and mediate communication between parties and promote constructive dialogue. First, it uses the emotion analysis function to analyze the emotional state of the conversation participants in real time. For example, if a participant feels anger or sadness during a conversation, the AI ​​detects that emotion and takes appropriate action. Next, it utilizes the dialogue mediation function to analyze and understand the conversation between the parties. The AI ​​makes constructive suggestions and questions at appropriate times to facilitate the dialogue. For example, if the conversation reaches an impasse, the AI ​​may suggest, "Shall we talk about this point in more detail?" Furthermore, the psychological safety ensuring function allows the AI ​​to promote dialogue from a neutral standpoint. This creates an environment where participants can express their opinions with peace of mind. For example, the AI ​​ensures the psychological safety of participants by sending a message such as, "Please feel free to express your opinion here." Furthermore, through its personalized advice function, the system learns each participant's personality and values, providing individually optimized advice for improving communication. For example, the AI ​​might offer specific advice such as, "This is the communication method that suits your personality." Finally, the follow-up function regularly monitors relationships after the conversation and provides additional support as needed. For example, the AI ​​might periodically ask questions like, "How is your relationship doing lately?" to help maintain the relationship. In this way, the Harmony AI system offers multiple functions, including emotion analysis, dialogue mediation, ensuring psychological safety, personalized advice, and follow-up, to support the improvement of interpersonal relationships. This makes it possible to smoothly resolve interpersonal problems in the workplace and daily life, and to build and maintain good relationships.

[0029] The Harmony AI system according to this embodiment comprises an emotion analysis unit, a dialogue mediation unit, an advice provision unit, and a follow-up unit. The emotion analysis unit analyzes the emotional state of the conversation participants in real time. For example, the emotion analysis unit analyzes voice data to analyze the emotional state of the conversation participants in real time. In addition to voice data, the emotion analysis unit can also analyze facial expressions and gestures to analyze the emotional state from multiple angles. The emotion analysis unit can also visualize the results of the emotion analysis in real time and provide feedback to the participants. The emotion analysis unit can also estimate the emotions of the conversation participants and dynamically adjust the accuracy of the emotion analysis based on the estimated emotions. The emotion analysis unit can also estimate the emotions of the conversation participants and determine the priority of the emotion analysis based on the estimated emotions. When performing emotion analysis, the emotion analysis unit can also improve the accuracy of the analysis by referring to the participants' past emotional history. When performing emotion analysis, the emotion analysis unit can also adjust the interpretation of emotions based on the participants' cultural background. The Dialogue Mediation Department analyzes and understands the conversation between parties based on the emotional states analyzed by the Emotion Analysis Department, and makes constructive suggestions and questions at appropriate times. The Dialogue Mediation Department uses natural language processing to analyze and understand the conversation between parties and makes constructive suggestions and questions at appropriate times. The Dialogue Mediation Department can also estimate the emotions of the conversation participants and adjust the way the conversation progresses based on the estimated emotions. The Dialogue Mediation Department can also deeply understand the context of the conversation and intervene at appropriate times during dialogue mediation. The Dialogue Mediation Department can also refer to the participants' past dialogue history to make optimal suggestions and questions during dialogue mediation. The Dialogue Mediation Department can also estimate the emotions of the conversation participants and determine the priority of the conversation based on the estimated emotions. The Dialogue Mediation Department can also adjust the difficulty of suggestions and questions considering the participants' language skills during dialogue mediation. The Dialogue Mediation Department can also customize the content of the conversation considering the participants' interests during dialogue mediation. The Advice Providing Department provides communication improvement advice optimized for each participant based on the suggestions and questions made by the Dialogue Mediation Department. The advice-providing department learns each participant's personality and values ​​and provides individually optimized advice for improving communication.The advice-providing department can estimate the emotions of the conversation participants and adjust the content of the advice based on those estimates. The advice-providing department can also refer to the participants' past behavioral patterns to provide optimal advice. The advice-providing department can also adjust the timing of the advice, taking into account the participants' current situation. The advice-providing department can estimate the emotions of the conversation participants and prioritize the advice based on those estimates. The advice-providing department can also adjust the format of the advice, taking into account the participants' learning style. The advice-providing department can also customize the content of the advice, taking into account the participants' goals. The follow-up department regularly monitors the relationship after the interaction based on the advice provided by the advice-providing department and provides additional support as needed. The follow-up department can also estimate the emotions of the conversation participants and adjust the content of the follow-up based on those estimates. The follow-up department can also refer to the participants' past follow-up history to provide optimal support during follow-up. The follow-up unit can also adjust the timing of follow-ups by considering the participants' current situations. The follow-up unit can also estimate the emotions of the conversation participants and determine the priority of follow-ups based on the estimated emotions. The follow-up unit can also customize the content of follow-ups by considering the participants' living environments. The follow-up unit can also collect participant feedback during follow-ups and reflect it in the next follow-up. As a result, the Harmony AI system according to the embodiment can analyze the emotional state of conversation participants in real time, make constructive suggestions and ask questions at appropriate times, provide communication improvement advice optimized for each participant, and regularly monitor relationships after the conversation and provide additional support as needed.

[0030] The emotion analysis unit analyzes the emotional state of conversation participants in real time. For example, it analyzes audio data to analyze the emotional state of conversation participants in real time. Specifically, it uses a combination of speech recognition technology and natural language processing technology for analyzing audio data. Speech recognition technology converts participants' statements into text data, and natural language processing technology extracts emotions from that text data. Furthermore, it also analyzes acoustic features such as tone, pitch, and speed of speech to capture subtle changes in emotion. In addition to audio data, the emotion analysis unit can also analyze facial expressions and gestures to analyze emotional states from multiple angles. For facial expression analysis, it uses a camera to capture the movements of participants' faces and uses a combination of facial recognition technology and emotion recognition technology. For gesture analysis, it uses motion sensors and cameras to capture the movements of participants' bodies and uses motion analysis technology. As a result, the emotion analysis unit can integrate the three elements of voice, facial expressions, and gestures to grasp emotional states more accurately. The emotion analysis unit can also visualize the results of the emotion analysis in real time and provide feedback to participants. Visualization uses graphs, charts, and icons to allow for an intuitive understanding of emotional changes. For example, the intensity of an emotion can be represented by the shade of color, or the type of emotion can be indicated by an icon. The sentiment analysis unit can also estimate the emotions of conversation participants and dynamically adjust the accuracy of the sentiment analysis based on the estimated emotions. For example, if a participant is feeling stressed, the sentiment analysis unit will adjust its focus to detecting stress-related emotions. The sentiment analysis unit can also estimate the emotions of conversation participants and determine the priorities of the sentiment analysis based on the estimated emotions. For example, if a participant is feeling angry, it will prioritize detecting anger-related emotions. During sentiment analysis, the sentiment analysis unit can also improve the accuracy of the analysis by referring to the participant's past emotional history. Past emotional history includes emotions the participant has felt in the past, their intensity, and duration. This allows the sentiment analysis unit to understand the participant's emotional patterns and make more accurate emotion estimations. During sentiment analysis, the sentiment analysis unit can also adjust the interpretation of emotions based on the participant's cultural background. Cultural background includes the participant's nationality, region, religion, and values.This allows the emotion analysis department to interpret emotions while taking cultural differences into account, thus preventing misunderstandings.

[0031] The dialogue mediation unit analyzes and understands the conversation between the parties based on the emotional states analyzed by the emotion analysis unit, and makes constructive suggestions and questions at the appropriate time. The dialogue mediation unit uses natural language processing to analyze and understand the conversation between the parties and makes constructive suggestions and questions at the appropriate time. Specifically, the dialogue mediation unit uses contextual analysis technology to understand the context of the conversation. Contextual analysis technology grasps the flow of the conversation, the theme, and the intentions of the participants, and provides information for intervention at the appropriate time. The dialogue mediation unit can also estimate the emotions of the conversation participants and adjust the way the conversation progresses based on the estimated emotions. For example, if a participant is feeling anxious, the dialogue mediation unit will make suggestions and ask questions that will make them feel at ease. The dialogue mediation unit can also deeply understand the context of the conversation during mediation and intervene at the appropriate time. For example, if the conversation stalls or misunderstandings arise, the dialogue mediation unit will intervene at the appropriate time and make suggestions and ask questions to facilitate the conversation. The dialogue mediation unit can also refer to the participants' past conversation history during mediation to make optimal suggestions and ask questions. Past dialogue history includes participants' previous statements, responses, and the outcomes of those conversations. This allows the dialogue facilitator to understand participants' dialogue patterns and make more effective suggestions and questions. The facilitator can also estimate the emotions of the conversation participants and prioritize dialogue based on those estimates. For example, if a participant is experiencing strong emotions, dialogue related to those emotions will be prioritized. The facilitator can also adjust the difficulty of suggestions and questions during facilitation, taking into account the participants' language skills. For example, if a participant is unfamiliar with the language, suggestions and questions will be made using simple words and short sentences. The facilitator can also customize the content of the dialogue, taking into account the participants' interests. For example, if a participant is interested in a particular topic, suggestions and questions related to that topic will be made. This allows the facilitator to capture the participants' attention and make the dialogue more effective.

[0032] The Advice Provider provides personalized communication improvement advice to each participant based on suggestions and questions made by the Dialogue Facilitator. The Advice Provider learns each participant's personality and values ​​to provide individually tailored advice. Specifically, the Advice Provider uses psychological assessment tools and questionnaires to understand participants' personality traits and values. This allows them to understand participants' communication styles and preferences and customize advice accordingly. The Advice Provider can also estimate the emotions of participants in a conversation and adjust the advice based on those emotions. For example, if a participant is feeling down, they might offer encouraging or comforting advice. The Advice Provider can also refer to participants' past behavioral patterns to provide optimal advice. Past behavioral patterns include actions the participant has taken previously, their consequences, and reactions. This allows the Advice Provider to understand participants' behavioral tendencies and provide more effective advice. The Advice Provider can also adjust the timing of advice, taking into account the participant's current situation. For example, if a participant is busy, they might offer concise and actionable advice. The advice-providing unit can also estimate the emotions of the conversation participants and prioritize advice based on those emotions. For example, if a participant has an urgent problem, it will prioritize advice related to that problem. The advice-providing unit can also adjust the format of advice to take into account the participant's learning style. For example, it will provide advice using diagrams and charts for visual learners and audio messages for auditory learners. The advice-providing unit can also customize the content of advice to take into account the participant's goals. For example, if a participant wants to improve a specific skill, it will provide advice related to that skill. This allows the advice-providing unit to provide optimal advice tailored to the individual needs of each participant and help improve their communication.

[0033] The Follow-up Department regularly monitors the relationship after the conversation, based on the advice provided by the Advice Department, and provides additional support as needed. Specifically, the Follow-up Department conducts regular check-ins and surveys to track changes in participants' emotional states and behaviors. This allows them to understand participants' progress and problems and provide appropriate support. The Follow-up Department can also estimate the emotions of the conversation participants and adjust the content of the follow-up based on the estimated emotions. For example, if a participant is feeling stressed, they will provide support to reduce stress. The Follow-up Department can also refer to the participant's past follow-up history to provide optimal support during follow-ups. Past follow-up history includes support the participant has received previously, its effects, and feedback. This allows the Follow-up Department to understand the participant's support needs and provide more effective support. The Follow-up Department can also adjust the timing of follow-ups considering the participant's current situation. For example, if a participant is busy, they will provide effective support in a short amount of time. The follow-up team can also estimate the emotions of the conversation participants and prioritize follow-up based on those estimates. For example, if a participant has an urgent problem, support related to that problem will be prioritized. The follow-up team can also customize the content of follow-up sessions, taking into account the participant's living environment. For example, if a participant has problems in their home environment, support related to that problem will be provided. The follow-up team can also collect participant feedback during follow-up sessions and incorporate it into subsequent follow-ups. This allows the follow-up team to provide optimal support tailored to the participant's needs and help improve relationships.

[0034] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0035] The Harmony AI system can also be equipped with a "stress level monitoring unit." This unit acquires the user's biometric data (e.g., heart rate, skin electrical activity) in real time and monitors their stress level. For example, if the user is experiencing high stress, the system can slow down the conversation and offer suggestions to promote relaxation. Conversely, if the stress level is low, the system can proceed with the conversation smoothly. Furthermore, it can record fluctuations in stress levels and provide advice for long-term stress management. This enables flexible conversational progression tailored to the user's stress level.

[0036] The Harmony AI system can also be equipped with a "cultural background consideration unit." This unit takes into account the user's cultural background and language habits, and adjusts the content and flow of the conversation accordingly. For example, when users from different cultural backgrounds converse, the system can offer suggestions to avoid cultural misunderstandings. It can also understand differences in emotional expression in specific cultures and provide appropriate feedback. Furthermore, it can provide advice based on cultural backgrounds to support users in communicating more effectively. This facilitates smooth intercultural dialogue.

[0037] The Harmony AI system can also be equipped with a "health status monitoring unit." This unit monitors the user's health status (e.g., sleep patterns, exercise levels, etc.) and adjusts the flow of the conversation and the content of the advice accordingly. For example, if the user is tired, the system can shorten the conversation and suggest resting. Conversely, if the user is healthy, the system can conduct the conversation more actively. Furthermore, based on the health status data, it can also provide advice for long-term health management. This enables flexible conversational progression tailored to the user's health status.

[0038] The Harmony AI system can also be equipped with a "Hobbies and Interests Consideration Unit." This unit learns the user's hobbies and interests and customizes the content of conversations and advice accordingly. For example, if a user is interested in a particular hobby, the system can provide topics related to that hobby to stimulate conversation. It can also provide advice based on the user's interests to support more effective communication. Furthermore, based on data on hobbies and interests, it can find common ground between users and provide conversation starters. This enables conversations tailored to the user's interests.

[0039] The Harmony AI system can also be equipped with a "feedback collection unit." This unit collects feedback from users after a conversation and uses it to improve the system. For example, it can provide a questionnaire to evaluate how users felt about the conversation's progress and the advice given. It can also adjust the system's algorithm based on the feedback to provide more effective conversational support. Furthermore, it can analyze the feedback and add new features that meet user needs and requests. This enables flexible system operation that reflects user opinions.

[0040] The following briefly describes the processing flow for example form 1.

[0041] Step 1: The sentiment analysis unit analyzes the emotional state of the conversation participants in real time. The sentiment analysis unit can analyze voice data, facial expressions, and gestures to provide a multifaceted analysis of emotional states. It can also visualize the results of the sentiment analysis in real time and provide feedback to the participants. Furthermore, the sentiment analysis unit can improve the accuracy of the analysis by referring to the participants' past emotional history and cultural background. Step 2: The dialogue mediation unit analyzes and understands the conversation between the parties based on the emotional states analyzed by the emotion analysis unit, and makes constructive suggestions and questions at appropriate times. The dialogue mediation unit can use natural language processing to deeply understand the context of the conversation and adjust the way the conversation progresses, taking into account the participants' past dialogue history, language skills, and interests. Step 3: The Advice Provider provides personalized communication improvement advice to each participant based on the suggestions and questions made by the Dialogue Facilitator. The Advice Provider can adjust the content and format of the advice to take into account each participant's personality, values, past behavioral patterns, current situation, learning style, and goals. Step 4: The Follow-up Department will regularly monitor the relationship after the dialogue, based on the advice provided by the Advice Department, and provide additional support as needed. The Follow-up Department may adjust the content and timing of follow-ups considering the participant's past follow-up history, current situation, living environment, and feedback.

[0042] (Example of form 2) The Harmony AI System according to an embodiment of the present invention is an AI support system for smoothly resolving interpersonal conflicts in the workplace and daily life, and for building and maintaining good relationships. This Harmony AI System combines voice emotion analysis technology and natural language processing to analyze and mediate communication between parties and promote constructive dialogue. First, it uses the emotion analysis function to analyze the emotional state of the conversation participants in real time. For example, if a participant feels anger or sadness during a conversation, the AI ​​detects that emotion and takes appropriate action. Next, it utilizes the dialogue mediation function to analyze and understand the conversation between the parties. The AI ​​makes constructive suggestions and questions at appropriate times to facilitate the dialogue. For example, if the conversation reaches an impasse, the AI ​​may suggest, "Shall we talk about this point in more detail?" Furthermore, the psychological safety ensuring function allows the AI ​​to promote dialogue from a neutral standpoint. This creates an environment where participants can express their opinions with peace of mind. For example, the AI ​​ensures the psychological safety of participants by sending a message such as, "Please feel free to express your opinion here." Furthermore, through its personalized advice function, the system learns each participant's personality and values, providing individually optimized advice for improving communication. For example, the AI ​​might offer specific advice such as, "This is the communication method that suits your personality." Finally, the follow-up function regularly monitors relationships after the conversation and provides additional support as needed. For example, the AI ​​might periodically ask questions like, "How is your relationship doing lately?" to help maintain the relationship. In this way, the Harmony AI system offers multiple functions, including emotion analysis, dialogue mediation, ensuring psychological safety, personalized advice, and follow-up, to support the improvement of interpersonal relationships. This makes it possible to smoothly resolve interpersonal problems in the workplace and daily life, and to build and maintain good relationships.

[0043] The Harmony AI system according to this embodiment comprises an emotion analysis unit, a dialogue mediation unit, an advice provision unit, and a follow-up unit. The emotion analysis unit analyzes the emotional state of the conversation participants in real time. For example, the emotion analysis unit analyzes voice data to analyze the emotional state of the conversation participants in real time. In addition to voice data, the emotion analysis unit can also analyze facial expressions and gestures to analyze the emotional state from multiple angles. The emotion analysis unit can also visualize the results of the emotion analysis in real time and provide feedback to the participants. The emotion analysis unit can also estimate the emotions of the conversation participants and dynamically adjust the accuracy of the emotion analysis based on the estimated emotions. The emotion analysis unit can also estimate the emotions of the conversation participants and determine the priority of the emotion analysis based on the estimated emotions. When performing emotion analysis, the emotion analysis unit can also improve the accuracy of the analysis by referring to the participants' past emotional history. When performing emotion analysis, the emotion analysis unit can also adjust the interpretation of emotions based on the participants' cultural background. The Dialogue Mediation Department analyzes and understands the conversation between parties based on the emotional states analyzed by the Emotion Analysis Department, and makes constructive suggestions and questions at appropriate times. The Dialogue Mediation Department uses natural language processing to analyze and understand the conversation between parties and makes constructive suggestions and questions at appropriate times. The Dialogue Mediation Department can also estimate the emotions of the conversation participants and adjust the way the conversation progresses based on the estimated emotions. The Dialogue Mediation Department can also deeply understand the context of the conversation and intervene at appropriate times during dialogue mediation. The Dialogue Mediation Department can also refer to the participants' past dialogue history to make optimal suggestions and questions during dialogue mediation. The Dialogue Mediation Department can also estimate the emotions of the conversation participants and determine the priority of the conversation based on the estimated emotions. The Dialogue Mediation Department can also adjust the difficulty of suggestions and questions considering the participants' language skills during dialogue mediation. The Dialogue Mediation Department can also customize the content of the conversation considering the participants' interests during dialogue mediation. The Advice Providing Department provides communication improvement advice optimized for each participant based on the suggestions and questions made by the Dialogue Mediation Department. The advice-providing department learns each participant's personality and values ​​and provides individually optimized advice for improving communication.The advice-providing department can estimate the emotions of the conversation participants and adjust the content of the advice based on those estimates. The advice-providing department can also refer to the participants' past behavioral patterns to provide optimal advice. The advice-providing department can also adjust the timing of the advice, taking into account the participants' current situation. The advice-providing department can estimate the emotions of the conversation participants and prioritize the advice based on those estimates. The advice-providing department can also adjust the format of the advice, taking into account the participants' learning style. The advice-providing department can also customize the content of the advice, taking into account the participants' goals. The follow-up department regularly monitors the relationship after the interaction based on the advice provided by the advice-providing department and provides additional support as needed. The follow-up department can also estimate the emotions of the conversation participants and adjust the content of the follow-up based on those estimates. The follow-up department can also refer to the participants' past follow-up history to provide optimal support during follow-up. The follow-up unit can also adjust the timing of follow-ups by considering the participants' current situations. The follow-up unit can also estimate the emotions of the conversation participants and determine the priority of follow-ups based on the estimated emotions. The follow-up unit can also customize the content of follow-ups by considering the participants' living environments. The follow-up unit can also collect participant feedback during follow-ups and reflect it in the next follow-up. As a result, the Harmony AI system according to the embodiment can analyze the emotional state of conversation participants in real time, make constructive suggestions and ask questions at appropriate times, provide communication improvement advice optimized for each participant, and regularly monitor relationships after the conversation and provide additional support as needed.

[0044] The emotion analysis unit analyzes the emotional state of conversation participants in real time. For example, it analyzes audio data to analyze the emotional state of conversation participants in real time. Specifically, it uses a combination of speech recognition technology and natural language processing technology for analyzing audio data. Speech recognition technology converts participants' statements into text data, and natural language processing technology extracts emotions from that text data. Furthermore, it also analyzes acoustic features such as tone, pitch, and speed of speech to capture subtle changes in emotion. In addition to audio data, the emotion analysis unit can also analyze facial expressions and gestures to analyze emotional states from multiple angles. For facial expression analysis, it uses a camera to capture the movements of participants' faces and uses a combination of facial recognition technology and emotion recognition technology. For gesture analysis, it uses motion sensors and cameras to capture the movements of participants' bodies and uses motion analysis technology. As a result, the emotion analysis unit can integrate the three elements of voice, facial expressions, and gestures to grasp emotional states more accurately. The emotion analysis unit can also visualize the results of the emotion analysis in real time and provide feedback to participants. Visualization uses graphs, charts, and icons to allow for an intuitive understanding of emotional changes. For example, the intensity of an emotion can be represented by the shade of color, or the type of emotion can be indicated by an icon. The sentiment analysis unit can also estimate the emotions of conversation participants and dynamically adjust the accuracy of the sentiment analysis based on the estimated emotions. For example, if a participant is feeling stressed, the sentiment analysis unit will adjust its focus to detecting stress-related emotions. The sentiment analysis unit can also estimate the emotions of conversation participants and determine the priorities of the sentiment analysis based on the estimated emotions. For example, if a participant is feeling angry, it will prioritize detecting anger-related emotions. During sentiment analysis, the sentiment analysis unit can also improve the accuracy of the analysis by referring to the participant's past emotional history. Past emotional history includes emotions the participant has felt in the past, their intensity, and duration. This allows the sentiment analysis unit to understand the participant's emotional patterns and make more accurate emotion estimations. During sentiment analysis, the sentiment analysis unit can also adjust the interpretation of emotions based on the participant's cultural background. Cultural background includes the participant's nationality, region, religion, and values.This allows the emotion analysis department to interpret emotions while taking cultural differences into account, thus preventing misunderstandings.

[0045] The dialogue mediation unit analyzes and understands the conversation between the parties based on the emotional states analyzed by the emotion analysis unit, and makes constructive suggestions and questions at the appropriate time. The dialogue mediation unit uses natural language processing to analyze and understand the conversation between the parties and makes constructive suggestions and questions at the appropriate time. Specifically, the dialogue mediation unit uses contextual analysis technology to understand the context of the conversation. Contextual analysis technology grasps the flow of the conversation, the theme, and the intentions of the participants, and provides information for intervention at the appropriate time. The dialogue mediation unit can also estimate the emotions of the conversation participants and adjust the way the conversation progresses based on the estimated emotions. For example, if a participant is feeling anxious, the dialogue mediation unit will make suggestions and ask questions that will make them feel at ease. The dialogue mediation unit can also deeply understand the context of the conversation during mediation and intervene at the appropriate time. For example, if the conversation stalls or misunderstandings arise, the dialogue mediation unit will intervene at the appropriate time and make suggestions and ask questions to facilitate the conversation. The dialogue mediation unit can also refer to the participants' past conversation history during mediation to make optimal suggestions and ask questions. Past dialogue history includes participants' previous statements, responses, and the outcomes of those conversations. This allows the dialogue facilitator to understand participants' dialogue patterns and make more effective suggestions and questions. The facilitator can also estimate the emotions of the conversation participants and prioritize dialogue based on those estimates. For example, if a participant is experiencing strong emotions, dialogue related to those emotions will be prioritized. The facilitator can also adjust the difficulty of suggestions and questions during facilitation, taking into account the participants' language skills. For example, if a participant is unfamiliar with the language, suggestions and questions will be made using simple words and short sentences. The facilitator can also customize the content of the dialogue, taking into account the participants' interests. For example, if a participant is interested in a particular topic, suggestions and questions related to that topic will be made. This allows the facilitator to capture the participants' attention and make the dialogue more effective.

[0046] The Advice Provider provides personalized communication improvement advice to each participant based on suggestions and questions made by the Dialogue Facilitator. The Advice Provider learns each participant's personality and values ​​to provide individually tailored advice. Specifically, the Advice Provider uses psychological assessment tools and questionnaires to understand participants' personality traits and values. This allows them to understand participants' communication styles and preferences and customize advice accordingly. The Advice Provider can also estimate the emotions of participants in a conversation and adjust the advice based on those emotions. For example, if a participant is feeling down, they might offer encouraging or comforting advice. The Advice Provider can also refer to participants' past behavioral patterns to provide optimal advice. Past behavioral patterns include actions the participant has taken previously, their consequences, and reactions. This allows the Advice Provider to understand participants' behavioral tendencies and provide more effective advice. The Advice Provider can also adjust the timing of advice, taking into account the participant's current situation. For example, if a participant is busy, they might offer concise and actionable advice. The advice-providing unit can also estimate the emotions of the conversation participants and prioritize advice based on those emotions. For example, if a participant has an urgent problem, it will prioritize advice related to that problem. The advice-providing unit can also adjust the format of advice to take into account the participant's learning style. For example, it will provide advice using diagrams and charts for visual learners and audio messages for auditory learners. The advice-providing unit can also customize the content of advice to take into account the participant's goals. For example, if a participant wants to improve a specific skill, it will provide advice related to that skill. This allows the advice-providing unit to provide optimal advice tailored to the individual needs of each participant and help improve their communication.

[0047] The Follow-up Department regularly monitors the relationship after the conversation, based on the advice provided by the Advice Department, and provides additional support as needed. Specifically, the Follow-up Department conducts regular check-ins and surveys to track changes in participants' emotional states and behaviors. This allows them to understand participants' progress and problems and provide appropriate support. The Follow-up Department can also estimate the emotions of the conversation participants and adjust the content of the follow-up based on the estimated emotions. For example, if a participant is feeling stressed, they will provide support to reduce stress. The Follow-up Department can also refer to the participant's past follow-up history to provide optimal support during follow-ups. Past follow-up history includes support the participant has received previously, its effects, and feedback. This allows the Follow-up Department to understand the participant's support needs and provide more effective support. The Follow-up Department can also adjust the timing of follow-ups considering the participant's current situation. For example, if a participant is busy, they will provide effective support in a short amount of time. The follow-up team can also estimate the emotions of the conversation participants and prioritize follow-up based on those estimates. For example, if a participant has an urgent problem, support related to that problem will be prioritized. The follow-up team can also customize the content of follow-up sessions, taking into account the participant's living environment. For example, if a participant has problems in their home environment, support related to that problem will be provided. The follow-up team can also collect participant feedback during follow-up sessions and incorporate it into subsequent follow-ups. This allows the follow-up team to provide optimal support tailored to the participant's needs and help improve relationships.

[0048] The emotion analysis unit can analyze audio data and analyze the emotional state of conversation participants in real time. For example, the emotion analysis unit can analyze audio data and analyze the emotional state of conversation participants in real time. In addition to audio data, the emotion analysis unit can also analyze facial expressions and gestures to analyze emotional states from multiple perspectives. The emotion analysis unit can also visualize the results of the emotion analysis in real time and provide feedback to participants. The emotion analysis unit can also estimate the emotions of conversation participants and dynamically adjust the accuracy of the emotion analysis based on the estimated emotions. The emotion analysis unit can also estimate the emotions of conversation participants and determine the priority of the emotion analysis based on the estimated emotions. During emotion analysis, the emotion analysis unit can improve the accuracy of the analysis by referring to the participants' past emotional history. During emotion analysis, the emotion analysis unit can also adjust the interpretation of emotions based on the participants' cultural background. This allows for real-time analysis of the emotional state of conversation participants by analyzing audio data.

[0049] The dialogue mediation unit can perform natural language processing to analyze and understand conversations between parties and make constructive suggestions and questions at appropriate times. For example, the dialogue mediation unit can perform natural language processing to analyze and understand conversations between parties and make constructive suggestions and questions at appropriate times. The dialogue mediation unit can also estimate the emotions of conversation participants and adjust the flow of the dialogue based on those estimated emotions. During dialogue mediation, the dialogue mediation unit can deeply understand the context of the conversation and intervene at appropriate times. During dialogue mediation, the dialogue mediation unit can refer to the participants' past dialogue history to make optimal suggestions and questions. The dialogue mediation unit can estimate the emotions of conversation participants and determine dialogue priorities based on those estimated emotions. During dialogue mediation, the dialogue mediation unit can adjust the difficulty of suggestions and questions considering the participants' language skills. During dialogue mediation, the dialogue mediation unit can customize the content of the dialogue considering the participants' interests. In this way, by performing natural language processing, the dialogue mediation unit can analyze and understand conversations between parties and make constructive suggestions and questions at appropriate times.

[0050] The advice-providing unit can learn each participant's personality and values ​​and provide individually optimized advice for improving communication. For example, the advice-providing unit can learn each participant's personality and values ​​and provide individually optimized advice for improving communication. The advice-providing unit can also estimate the emotions of the conversation participants and adjust the advice based on those emotions. When providing advice, the advice-providing unit can refer to the participant's past behavioral patterns to provide optimal advice. When providing advice, the advice-providing unit can adjust the timing of the advice considering the participant's current situation. The advice-providing unit can also estimate the emotions of the conversation participants and prioritize the advice based on those emotions. When providing advice, the advice-providing unit can adjust the format of the advice considering the participant's learning style. When providing advice, the advice-providing unit can customize the content of the advice considering the participant's goals. This allows the advice-providing unit to provide individually optimized advice for improving communication by learning each participant's personality and values.

[0051] The follow-up department can periodically monitor the relationship after the conversation and provide additional support as needed. For example, the follow-up department can periodically monitor the relationship after the conversation and provide additional support as needed. The follow-up department can also estimate the emotions of the conversation participants and adjust the follow-up content based on those estimations. During follow-ups, the follow-up department can refer to the participants' past follow-up history to provide optimal support. During follow-ups, the follow-up department can adjust the timing of follow-ups considering the participants' current circumstances. The follow-up department can also estimate the emotions of the conversation participants and prioritize follow-ups based on those estimations. During follow-ups, the follow-up department can customize the content of the follow-ups considering the participants' living environment. During follow-ups, the follow-up department can collect participant feedback and incorporate it into future follow-ups. This allows for periodic monitoring of the relationship after the conversation and the provision of additional support as needed.

[0052] The sentiment analysis unit can estimate the emotions of conversation participants and dynamically adjust the accuracy of the sentiment analysis based on the estimated emotions. For example, if a conversation participant is showing strong anger, the sentiment analysis unit can increase the accuracy of the sentiment analysis to detect subtle emotional changes. If a conversation participant is relaxed, the sentiment analysis unit can also maintain the accuracy of the sentiment analysis at a normal level, saving resources. If a conversation participant is confused, the sentiment analysis unit can also adjust the accuracy of the sentiment analysis to identify the emotion causing the confusion. This allows for more accurate sentiment analysis by estimating the emotions of conversation participants and dynamically adjusting the accuracy of the sentiment analysis based on the estimated emotions.

[0053] The emotion analysis unit can analyze not only audio data but also facial expressions and gestures to provide a multifaceted analysis of emotional states. For example, the emotion analysis unit can analyze participants' facial expressions in real time using a camera along with audio data to make a comprehensive judgment about their emotional state. The emotion analysis unit can also analyze participants' hand movements and posture along with audio data to identify the intensity and type of emotion. The emotion analysis unit can also analyze participants' eye movements along with audio data to identify the direction and focus of emotion. In this way, by analyzing facial expressions and gestures in addition to audio data, it is possible to analyze emotional states from multiple perspectives.

[0054] The emotion analysis unit can visualize the results of the emotion analysis in real time and provide feedback to participants. For example, the emotion analysis unit can visualize the results of the emotion analysis in graphs and charts and display them to participants in real time. The emotion analysis unit can also represent the results of the emotion analysis with colors and icons to allow participants to understand them intuitively. The emotion analysis unit can also explain the results of the emotion analysis in text and provide participants with specific feedback. In this way, by visualizing the results of the emotion analysis in real time and providing feedback to participants, it becomes easier for participants to understand their own emotional state.

[0055] The sentiment analysis unit can estimate the emotions of conversation participants and determine the priority of sentiment analysis based on the estimated emotions. For example, if a conversation participant is showing a strong emotion, the sentiment analysis unit will prioritize analyzing that emotion. If a conversation participant is showing multiple emotions, the sentiment analysis unit can also prioritize analyzing the strongest emotion. If a conversation participant is showing a change in emotion, the sentiment analysis unit can also prioritize analyzing that change. In this way, by estimating the emotions of conversation participants and determining the priority of sentiment analysis based on the estimated emotions, important emotions can be analyzed preferentially.

[0056] The emotion analysis unit can improve the accuracy of its analysis by referring to the participant's past emotional history. For example, the emotion analysis unit can refer to the participant's past emotional history and compare it with their current emotional state. Based on the participant's past emotional history, the emotion analysis unit can also identify emotional patterns and improve the accuracy of its analysis. The emotion analysis unit can also refer to the participant's past emotional history and predict changes in emotions to improve the accuracy of its analysis. In this way, the accuracy of emotion analysis can be improved by referring to the participant's past emotional history.

[0057] The emotion analysis department can adjust its interpretation of emotions based on the participant's cultural background during emotion analysis. For example, the emotion analysis department considers the participant's cultural background to understand and analyze how emotions are expressed. The emotion analysis department can also appropriately interpret the intensity and type of emotions based on the participant's cultural background. The emotion analysis department can also refer to the participant's cultural background to predict changes in emotions and improve the accuracy of the analysis. As a result, by adjusting the interpretation of emotions based on the participant's cultural background, more accurate emotion analysis becomes possible.

[0058] The dialogue mediation unit can estimate the emotions of the conversation participants and adjust the way the dialogue is conducted based on those estimated emotions. For example, if a participant is showing anger, the dialogue mediation unit can slow down the pace of the dialogue and encourage calm conversation. If a participant is showing sadness, the dialogue mediation unit can also gently guide the dialogue and encourage the sharing of emotions. If a participant is agitated, the dialogue mediation unit can smooth the pace of the dialogue and encourage constructive conversation. In this way, by estimating the emotions of the conversation participants and adjusting the way the dialogue is conducted based on those estimated emotions, smoother dialogue becomes possible.

[0059] The dialogue mediation department can deeply understand the context of the conversation and intervene at the appropriate time. For example, the dialogue mediation department can understand the context of the conversation and ask questions at the appropriate time. The dialogue mediation department can also understand the context of the conversation and make suggestions at the appropriate time. The dialogue mediation department can understand the context of the conversation, intervene at the appropriate time, and facilitate the conversation. In this way, by deeply understanding the context of the conversation and intervening at the appropriate time, the conversation can proceed smoothly.

[0060] The dialogue mediation unit can make optimal suggestions and ask appropriate questions by referring to the participants' past dialogue history during dialogue mediation. For example, the dialogue mediation unit can refer to the participants' past dialogue history and make appropriate suggestions. The dialogue mediation unit can also ask appropriate questions based on the participants' past dialogue history. The dialogue mediation unit can also optimize the progress of the dialogue by referring to the participants' past dialogue history. In this way, by referring to the participants' past dialogue history, it can make optimal suggestions and ask appropriate questions.

[0061] The dialogue facilitator can estimate the emotions of the conversation participants and determine the priority of the dialogue based on those estimated emotions. For example, if a participant is showing strong emotions, the facilitator will prioritize that dialogue. If a participant is showing multiple emotions, the facilitator can also prioritize the strongest emotion. If a participant is showing a change in emotion, the facilitator can also prioritize that change. In this way, by estimating the emotions of the conversation participants and determining the priority of the dialogue based on those estimated emotions, important dialogues can be prioritized.

[0062] The dialogue facilitator can adjust the difficulty of suggestions and questions during dialogue facilitation, taking into account the participants' language skills. For example, the facilitator can make simple suggestions and questions, taking into account the participants' language skills. The facilitator can also make suggestions and questions of appropriate difficulty based on the participants' language skills. The facilitator can also optimize the flow of the dialogue by referring to the participants' language skills. This allows for suggestions and questions of appropriate difficulty to be made, taking into account the participants' language skills.

[0063] The dialogue facilitator can customize the content of the dialogue by considering the participants' interests during the facilitation process. For example, the facilitator can customize the content of the dialogue by considering the participants' interests. Based on the participants' interests, the facilitator can also make appropriate suggestions and ask questions. The facilitator can also optimize the flow of the dialogue by referring to the participants' interests. This allows for the customization of the dialogue content by considering the participants' interests.

[0064] The advice-providing unit can estimate the emotions of the conversation participants and adjust the content of the advice based on those estimated emotions. For example, if a participant is showing anger, the advice-providing unit can offer advice to calm down. If a participant is showing sadness, the advice-providing unit can also offer advice to comfort them. If a participant is agitated, the advice-providing unit can also offer advice to calm down. This allows the system to provide more appropriate advice by estimating the emotions of the conversation participants and adjusting the content of the advice based on those estimated emotions.

[0065] The advice-providing department can provide optimal advice by referring to the participant's past behavioral patterns when providing advice. For example, the advice-providing department can refer to the participant's past behavioral patterns to provide appropriate advice. The advice-providing department can also provide optimal advice based on the participant's past behavioral patterns. The advice-providing department can also optimize the content of the advice by referring to the participant's past behavioral patterns. This allows the advice-providing department to provide optimal advice by referring to the participant's past behavioral patterns.

[0066] The advice-providing department can adjust the timing of advice given, taking into account the participant's current situation. For example, the advice-providing department can provide advice at an appropriate time, considering the participant's current situation. It can also provide advice at the optimal time based on the participant's current situation. Furthermore, the advice-providing department can optimize the timing of advice by referring to the participant's current situation. This allows for the provision of advice at the appropriate time, taking the participant's current situation into consideration.

[0067] The advice-providing unit can estimate the emotions of the conversation participants and prioritize advice based on those estimated emotions. For example, if a participant is showing a strong emotion, the advice-providing unit will prioritize that emotion. If a participant is showing multiple emotions, the advice-providing unit can also prioritize advice based on the strongest emotion. If a participant is showing a change in emotion, the advice-providing unit can also prioritize advice based on that change. In this way, by estimating the emotions of the conversation participants and prioritizing advice based on those estimated emotions, important advice can be prioritized.

[0068] The advice provider can adjust the format of the advice given, taking into account the participant's learning style. For example, the advice provider can provide visual advice, taking into account the participant's learning style. The advice provider can also provide auditory advice based on the participant's learning style. The advice provider can also optimize the format of the advice by referring to the participant's learning style. This allows for the provision of advice in an appropriate format, taking into account the participant's learning style.

[0069] The advice provider can customize the content of the advice given, taking into account the participant's goals. For example, the advice provider can provide appropriate advice by considering the participant's goals. The advice provider can also provide optimal advice based on the participant's goals. The advice provider can also optimize the content of the advice by referring to the participant's goals. This allows for the provision of appropriate advice by considering the participant's goals.

[0070] The follow-up unit can estimate the emotions of the conversation participants and adjust the content of the follow-up based on the estimated emotions. For example, if a participant is showing anger, the follow-up unit can provide follow-up to help them calm down. If a participant is showing sadness, the follow-up unit can also provide follow-up to comfort them. If a participant is agitated, the follow-up unit can also provide follow-up to help them calm down. In this way, by estimating the emotions of the conversation participants and adjusting the content of the follow-up based on the estimated emotions, more appropriate follow-up can be provided.

[0071] The follow-up department can provide optimal support during follow-up sessions by referring to the participant's past follow-up history. For example, the follow-up department can refer to the participant's past follow-up history to provide appropriate support. The follow-up department can also provide optimal support based on the participant's past follow-up history. The follow-up department can also optimize the content of support by referring to the participant's past follow-up history. This allows for the provision of optimal support by referring to the participant's past follow-up history.

[0072] The follow-up department can adjust the timing of follow-ups by considering the participant's current situation. For example, the follow-up department can provide follow-ups at an appropriate time, taking into account the participant's current situation. The follow-up department can also provide follow-ups at the optimal time based on the participant's current situation. The follow-up department can also optimize the timing of follow-ups by referring to the participant's current situation. This allows for the provision of follow-ups at an appropriate time by considering the participant's current situation.

[0073] The follow-up unit can estimate the emotions of conversation participants and prioritize follow-ups based on those estimated emotions. For example, if a participant is showing a strong emotion, the follow-up unit will prioritize providing that emotion. If a participant is showing multiple emotions, the follow-up unit can also prioritize providing the follow-up based on the strongest emotion. If a participant is showing a change in emotion, the follow-up unit can also prioritize providing the follow-up based on that change. In this way, by estimating the emotions of conversation participants and prioritizing follow-ups based on those estimated emotions, important follow-ups can be prioritized.

[0074] The follow-up department can customize the content of follow-up sessions by considering the participants' living environments. For example, the follow-up department can provide appropriate follow-up sessions by considering the participants' living environments. The follow-up department can also provide optimal follow-up sessions based on the participants' living environments. The follow-up department can also optimize the content of follow-up sessions by referring to the participants' living environments. This allows for the provision of appropriate follow-up sessions by considering the participants' living environments.

[0075] The follow-up department can collect participant feedback during follow-up sessions and incorporate it into subsequent follow-ups. For example, the follow-up department can collect participant feedback and incorporate it into subsequent follow-ups. The follow-up department can also optimize subsequent follow-ups based on participant feedback. The follow-up department can also improve the content of follow-ups by referring to participant feedback. This allows for the optimization of subsequent follow-ups by collecting participant feedback.

[0076] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0077] The Harmony AI system can also be equipped with a "stress level monitoring unit." This unit acquires the user's biometric data (e.g., heart rate, skin electrical activity) in real time and monitors their stress level. For example, if the user is experiencing high stress, the system can slow down the conversation and offer suggestions to promote relaxation. Conversely, if the stress level is low, the system can proceed with the conversation smoothly. Furthermore, it can record fluctuations in stress levels and provide advice for long-term stress management. This enables flexible conversational progression tailored to the user's stress level.

[0078] The Harmony AI system can also be equipped with a "cultural background consideration unit." This unit takes into account the user's cultural background and language habits, and adjusts the content and flow of the conversation accordingly. For example, when users from different cultural backgrounds converse, the system can offer suggestions to avoid cultural misunderstandings. It can also understand differences in emotional expression in specific cultures and provide appropriate feedback. Furthermore, it can provide advice based on cultural backgrounds to support users in communicating more effectively. This facilitates smooth intercultural dialogue.

[0079] The Harmony AI system can also be equipped with a "health status monitoring unit." This unit monitors the user's health status (e.g., sleep patterns, exercise levels, etc.) and adjusts the flow of the conversation and the content of the advice accordingly. For example, if the user is tired, the system can shorten the conversation and suggest resting. Conversely, if the user is healthy, the system can conduct the conversation more actively. Furthermore, based on the health status data, it can also provide advice for long-term health management. This enables flexible conversational progression tailored to the user's health status.

[0080] The Harmony AI system can also be equipped with a "Hobbies and Interests Consideration Unit." This unit learns the user's hobbies and interests and customizes the content of conversations and advice accordingly. For example, if a user is interested in a particular hobby, the system can provide topics related to that hobby to stimulate conversation. It can also provide advice based on the user's interests to support more effective communication. Furthermore, based on data on hobbies and interests, it can find common ground between users and provide conversation starters. This enables conversations tailored to the user's interests.

[0081] The Harmony AI system can also be equipped with a "feedback collection unit." This unit collects feedback from users after a conversation and uses it to improve the system. For example, it can provide a questionnaire to evaluate how users felt about the conversation's progress and the advice given. It can also adjust the system's algorithm based on the feedback to provide more effective conversational support. Furthermore, it can analyze the feedback and add new features that meet user needs and requests. This enables flexible system operation that reflects user opinions.

[0082] The Harmony AI system also features an "emotion estimation unit" that can estimate the user's emotions and adjust the conversation flow based on those emotions. For example, if the user is angry, the system can slow down the conversation and encourage a calmer discussion. If the user is sad, the system can gently guide the conversation and encourage emotional sharing. Furthermore, if the user is agitated, the system can smooth the conversation and encourage a constructive discussion. This enables flexible conversation flow that responds to the user's emotions.

[0083] The Harmony AI system also features an "emotion estimation unit" that can estimate the user's emotions and adjust the content of advice based on those emotions. For example, if the user is angry, it can provide advice to help them calm down. If the user is sad, it can provide comforting advice. Furthermore, if the user is agitated, it can provide advice to help them calm down. This enables flexible advice delivery tailored to the user's emotions.

[0084] The Harmony AI system also features an "emotion estimation unit" that can estimate the user's emotions and adjust the follow-up content based on those emotions. For example, if the user is angry, it can provide follow-up to help them calm down. If the user is sad, it can provide follow-up to comfort them. Furthermore, if the user is agitated, it can provide follow-up to help them calm down. This enables flexible follow-up tailored to the user's emotions.

[0085] The Harmony AI system is further equipped with an "emotion estimation unit" that can estimate the user's emotions and determine the priority of the conversation based on those emotions. For example, if the user is showing strong emotions, the conversation can be prioritized to address those emotions. If the user is showing multiple emotions, the system can prioritize the conversation addressing the strongest emotion. Furthermore, if the user is showing a change in emotion, the system can prioritize the conversation addressing that change. This enables flexible conversational progression that responds to the user's emotions.

[0086] The Harmony AI system also features an "emotion estimation unit" that estimates the user's emotions and dynamically adjusts the accuracy of emotion analysis based on the estimated emotions. For example, if the user is showing strong anger, the accuracy of emotion analysis can be increased to detect subtle emotional changes. Conversely, if the user is relaxed, the accuracy of emotion analysis can be kept at a normal level, saving resources. Furthermore, if the user is confused, the accuracy of emotion analysis can be adjusted to identify the emotion causing the confusion. This enables flexible emotion analysis that responds to the user's emotions.

[0087] The following briefly describes the processing flow for example form 2.

[0088] Step 1: The sentiment analysis unit analyzes the emotional state of the conversation participants in real time. The sentiment analysis unit can analyze voice data, facial expressions, and gestures to provide a multifaceted analysis of emotional states. It can also visualize the results of the sentiment analysis in real time and provide feedback to the participants. Furthermore, the sentiment analysis unit can improve the accuracy of the analysis by referring to the participants' past emotional history and cultural background. Step 2: The dialogue mediation unit analyzes and understands the conversation between the parties based on the emotional states analyzed by the emotion analysis unit, and makes constructive suggestions and questions at appropriate times. The dialogue mediation unit can use natural language processing to deeply understand the context of the conversation and adjust the way the conversation progresses, taking into account the participants' past dialogue history, language skills, and interests. Step 3: The Advice Provider provides personalized communication improvement advice to each participant based on the suggestions and questions made by the Dialogue Facilitator. The Advice Provider can adjust the content and format of the advice to take into account each participant's personality, values, past behavioral patterns, current situation, learning style, and goals. Step 4: The Follow-up Department will regularly monitor the relationship after the dialogue, based on the advice provided by the Advice Department, and provide additional support as needed. The Follow-up Department may adjust the content and timing of follow-ups considering the participant's past follow-up history, current situation, living environment, and feedback.

[0089] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0090] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0091] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0092] Each of the multiple elements described above, including the emotion analysis unit, dialogue mediation unit, advice provision unit, and follow-up unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the emotion analysis unit uses the camera 42 and microphone 38B of the smart device 14 to analyze the emotional state of the conversation participants in real time, and the control unit 46A estimates the emotion. The dialogue mediation unit is implemented by the specific processing unit 290 of the data processing unit 12 and makes constructive suggestions and questions based on the results of the emotion analysis unit. The advice provision unit is implemented by the specific processing unit 290 of the data processing unit 12 and provides advice on improving communication optimized for each participant. The follow-up unit is implemented by the specific processing unit 290 of the data processing unit 12 and periodically monitors the relationship after the dialogue and provides additional support as needed. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0093] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0094] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0095] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0096] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0097] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0098] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0099] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0100] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0101] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0102] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0103] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0104] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0105] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0106] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0107] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0108] Each of the multiple elements described above, including the emotion analysis unit, dialogue mediation unit, advice provision unit, and follow-up unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the emotion analysis unit uses the camera 42 and microphone 238 of the smart glasses 214 to analyze the emotional state of the conversation participants in real time, and the control unit 46A estimates the emotion. The dialogue mediation unit is implemented by the specific processing unit 290 of the data processing unit 12 and makes constructive suggestions and questions based on the results of the emotion analysis unit. The advice provision unit is implemented by the specific processing unit 290 of the data processing unit 12 and provides advice on improving communication optimized for each participant. The follow-up unit is implemented by the specific processing unit 290 of the data processing unit 12 and periodically monitors the relationship after the dialogue and provides additional support as needed. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0109] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0110] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0111] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0112] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0113] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0114] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0115] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0116] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0117] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0118] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0119] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0120] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0121] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0122] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0123] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0124] Each of the multiple elements described above, including the emotion analysis unit, dialogue mediation unit, advice provision unit, and follow-up unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the emotion analysis unit uses the camera 42 and microphone 238 of the headset terminal 314 to analyze the emotional state of the conversation participants in real time, and the control unit 46A estimates the emotions. The dialogue mediation unit is implemented by the specific processing unit 290 of the data processing unit 12 and makes constructive suggestions and questions based on the results of the emotion analysis unit. The advice provision unit is implemented by the specific processing unit 290 of the data processing unit 12 and provides advice on improving communication optimized for each participant. The follow-up unit is implemented by the specific processing unit 290 of the data processing unit 12 and periodically monitors the relationship after the dialogue and provides additional support as needed. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0125] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0126] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0127] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0128] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0129] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0130] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0131] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0132] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0133] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0134] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0135] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0136] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0137] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0138] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0139] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0140] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0141] Each of the multiple elements described above, including the emotion analysis unit, dialogue mediation unit, advice provision unit, and follow-up unit, is implemented in at least one of the robot 414 and the data processing unit 12. For example, the emotion analysis unit uses the camera 42 and microphone 238 of the robot 414 to analyze the emotional state of the conversation participants in real time, and the control unit 46A estimates the emotion. The dialogue mediation unit is implemented by the specific processing unit 290 of the data processing unit 12 and makes constructive suggestions and questions based on the results of the emotion analysis unit. The advice provision unit is implemented by the specific processing unit 290 of the data processing unit 12 and provides advice on improving communication optimized for each participant. The follow-up unit is implemented by the specific processing unit 290 of the data processing unit 12 and periodically monitors the relationship after the dialogue and provides additional support as needed. The correspondence between each unit and the device or control unit is not limited to the example described above and can be changed in various ways.

[0142] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0143] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0144] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0145] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0146] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0147] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0148] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0149] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0150] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0151] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0152] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0153] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0154] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0155] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0156] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0157] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0158] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0159] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0160] (Note 1) The emotion analysis department analyzes the emotional state of the conversation participants in real time, Based on the emotional states analyzed by the aforementioned emotion analysis unit, the dialogue mediation unit analyzes and understands the conversation between the parties and makes constructive suggestions and questions at appropriate times. An advice service provides advice on improving communication tailored to each participant, based on the suggestions and questions made by the aforementioned dialogue mediation service. The system includes a follow-up unit that periodically monitors the relationship after the dialogue based on the advice provided by the aforementioned advice-providing unit and provides additional support as needed. A system characterized by the following features. (Note 2) The aforementioned emotion analysis unit, The system analyzes audio data to analyze the emotional state of conversation participants in real time. The system described in Appendix 1, characterized by the features described herein. (Note 3) The aforementioned dialogue mediation unit, It performs natural language processing to analyze and understand conversations between parties and makes constructive suggestions and questions at appropriate times. The system described in Appendix 1, characterized by the features described herein. (Note 4) The aforementioned advice-providing unit, It learns each participant's personality and values ​​and provides individually optimized advice for improving communication. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned follow-up unit is, We will continue to monitor the relationship regularly after the conversation and provide additional support as needed. The system described in Appendix 1, characterized by the features described herein. (Note 6) The aforementioned emotion analysis unit, It estimates the emotions of the conversation participants and dynamically adjusts the accuracy of the emotion analysis based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned emotion analysis unit, In addition to audio data, facial expressions and gestures are analyzed to provide a multifaceted analysis of emotional states. The system described in Appendix 1, characterized by the features described herein. (Note 8) The aforementioned emotion analysis unit, Visualize the results of the sentiment analysis in real time and provide feedback to participants. The system described in Appendix 1, characterized by the features described herein. (Note 9) The aforementioned emotion analysis unit, The system estimates the emotions of the conversation participants and prioritizes sentiment analysis based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 10) The aforementioned emotion analysis unit, When performing sentiment analysis, referencing the participant's past emotional history improves the accuracy of the analysis. The system described in Appendix 1, characterized by the features described herein. (Note 11) The aforementioned emotion analysis unit, During emotion analysis, we adjust the interpretation of emotions based on the participants' cultural background. The system described in Appendix 1, characterized by the features described herein. (Note 12) The aforementioned dialogue mediation unit, It estimates the emotions of the conversation participants and adjusts the way the conversation progresses based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 13) The aforementioned dialogue mediation unit, When mediating a dialogue, it is important to deeply understand the context of the conversation and intervene at the appropriate time. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned dialogue mediation unit, When mediating a dialogue, refer to the participants' past dialogue history to make the most appropriate suggestions and questions. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned dialogue mediation unit, It estimates the emotions of the conversation participants and determines the priority of the dialogue based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned dialogue mediation unit, When facilitating dialogue, adjust the difficulty level of suggestions and questions to take into account the participants' language skills. The system described in Appendix 1, characterized by the features described herein. (Note 17) The aforementioned dialogue mediation unit, When facilitating a dialogue, customize the content of the conversation by taking into account the participants' interests. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned advice-providing unit, The system estimates the emotions of the conversation participants and adjusts the content of the advice based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 19) The aforementioned advice-providing unit, When providing advice, we refer to the participant's past behavioral patterns to provide the most appropriate advice. The system described in Appendix 1, characterized by the features described herein. (Note 20) The aforementioned advice-providing unit, When providing advice, we adjust the timing of the advice based on the participant's current situation. The system described in Appendix 1, characterized by the features described herein. (Note 21) The aforementioned advice-providing unit, It estimates the emotions of the conversation participants and prioritizes advice based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 22) The aforementioned advice-providing unit, When providing advice, the format of the advice will be adjusted to take into account the participant's learning style. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned advice-providing unit, When providing advice, customize the content of the advice to take into account the participant's goals. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned follow-up unit is, The system estimates the emotions of the conversation participants and adjusts the follow-up content based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned follow-up unit is, During follow-up sessions, we refer to participants' past follow-up history to provide the most appropriate support. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned follow-up unit is, During follow-up sessions, the timing of the follow-up will be adjusted to take into account the participant's current situation. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned follow-up unit is, The system estimates the emotions of the conversation participants and determines the priority of follow-up based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned follow-up unit is, During follow-up sessions, the content of the follow-up will be customized to take into account the participants' living circumstances. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned follow-up unit is, During follow-up sessions, we collect participant feedback and incorporate it into future follow-up sessions. The system described in Appendix 1, characterized by the features described herein. [Explanation of symbols]

[0161] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. The emotion analysis department analyzes the emotional state of the conversation participants in real time, Based on the emotional states analyzed by the aforementioned emotion analysis unit, the dialogue mediation unit analyzes and understands the conversation between the parties and makes constructive suggestions and questions at appropriate times. An advice service provides advice on improving communication tailored to each participant, based on the suggestions and questions made by the aforementioned dialogue mediation service. The system includes a follow-up unit that periodically monitors the relationship after the dialogue based on the advice provided by the aforementioned advice-providing unit and provides additional support as needed. A system characterized by the following features.

2. The aforementioned emotion analysis unit, The system analyzes audio data to analyze the emotional state of conversation participants in real time. The system according to feature 1.

3. The aforementioned dialogue mediation unit, It performs natural language processing to analyze and understand conversations between parties and makes constructive suggestions and questions at appropriate times. The system according to feature 1.

4. The aforementioned advice-providing unit, It learns each participant's personality and values ​​and provides individually optimized advice for improving communication. The system according to feature 1.

5. The aforementioned follow-up unit is, We will continue to monitor the relationship regularly after the conversation and provide additional support as needed. The system according to feature 1.

6. The aforementioned emotion analysis unit, It estimates the emotions of the conversation participants and dynamically adjusts the accuracy of the emotion analysis based on the estimated emotions. The system according to feature 1.

7. The aforementioned emotion analysis unit, In addition to audio data, facial expressions and gestures are analyzed to provide a multifaceted analysis of emotional states. The system according to feature 1.

8. The aforementioned emotion analysis unit, Visualize the results of the sentiment analysis in real time and provide feedback to participants. The system according to feature 1.

9. The aforementioned emotion analysis unit, The system estimates the emotions of the conversation participants and prioritizes sentiment analysis based on the estimated emotions. The system according to feature 1.

10. The aforementioned emotion analysis unit, When performing sentiment analysis, referencing the participant's past emotional history improves the accuracy of the analysis. The system according to feature 1.

Images