system

Abstract

The system according to this embodiment aims to understand the emotions of the other party during dates or sales meetings and to propose appropriate responses based on those emotions. [Solution] The system according to the embodiment comprises an emotion estimation unit, an information acquisition unit, an advice provision unit, and a hologram display unit. The emotion estimation unit estimates the emotions of the other party. The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. The advice provision unit suggests topics and behaviors based on the information acquired by the information acquisition unit. The hologram display unit automatically searches for a video that the user wants to show and displays it as a hologram.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art, there was a problem that it was difficult to appropriately grasp the feelings of the other party during a date or business, and to make appropriate responses based on that.

[0005] The system according to the embodiment aims to grasp the feelings of the other party during a date or business, and to propose appropriate responses based on that.

Means for Solving the Problems

[0006] The system according to this embodiment comprises an emotion estimation unit, an information acquisition unit, an advice provision unit, and a hologram display unit. The emotion estimation unit estimates the emotions of the other party. The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. The advice provision unit suggests topics and behaviors based on the information acquired by the information acquisition unit. The hologram display unit automatically searches for a video that the user wants to show and displays it as a hologram. [Effects of the Invention]

[0007] The system according to this embodiment can understand the other person's emotions during a date or sales meeting and propose an appropriate response based on that understanding. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 comprises a computer 36, a receiving device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The receiving device 38, output device 40, and camera 42 are also connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) An agent system according to an embodiment of the present invention is a system that assists with dating and sales by utilizing an emotion estimation AI based on image recognition and speech recognition, and a large-scale language model. When a user meets a date or salesperson for the first time, this agent system links the conversation history with the system and inputs initial information such as the person's name, hometown, and hobbies in natural language. Next, using a camera and microphone mounted on AR glasses, the system estimates the person's emotions and state based on their face and voice. Based on the emotion estimation results, information about the user's current location obtained from the camera, and past conversation logs, the system displays advice on topics and behavior in AR. It also has a function to automatically search for and display holograms of videos of specific scenes if the user wishes to show them. After a date or sales meeting, the system can also suggest follow-ups, including suggestions for everyday conversation texts and advice on date plans. This allows the user to engage in appropriate topics and behaviors according to the other person's emotions. For example, when a user meets a date or salesperson for the first time, the agent system links the conversation history with the system and inputs initial information such as the person's name, hometown, and hobbies in natural language. Next, the AR glasses use a camera and microphone to estimate the other person's emotions and state based on their face and voice. Based on the emotion estimation results, information about the current location obtained from the camera, and past conversation logs, advice on topics and behavior is displayed in AR. It also has a function to automatically search for and display holograms of videos of specific scenes if the user wishes to show them. After a date or business meeting, follow-up suggestions can be made, and it is also possible to receive suggestions for everyday conversation texts and advice on date plans. This allows the user to choose appropriate topics and behaviors based on the other person's emotions. When a user meets a date or business partner for the first time, the agent system links the conversation history with the system and inputs initial information such as the other person's name, hometown, and hobbies in natural language. Next, the AR glasses use a camera and microphone to estimate the other person's emotions and state based on their face and voice. Based on the emotion estimation results, information about the current location obtained from the camera, and past conversation logs, advice on topics and behavior is displayed in AR. It also has a function to automatically search for and display holograms of videos of specific scenes if the user wishes to show them.After a date or business meeting, follow-up suggestions can be made, including suggestions for everyday conversations and advice on date plans. This allows users to choose appropriate topics and behaviors based on the other person's emotions.

[0029] The agent system according to this embodiment comprises an emotion estimation unit, an information acquisition unit, an advice provision unit, and a hologram display unit. The emotion estimation unit estimates the emotions of the other party. For example, the emotion estimation unit analyzes the other party's facial expressions using image recognition technology and estimates their emotions. The emotion estimation unit can also analyze the tone and pitch of the other party's voice using speech recognition technology and estimate their emotions. Furthermore, the emotion estimation unit can also analyze the other party's body movements and posture and estimate their emotions. For example, the emotion estimation unit analyzes facial expressions in real time, detects subtle changes in expression, and estimates emotions. Speech recognition technology analyzes changes in voice tone and pitch to estimate emotions. Analysis of body movements and posture is used to track changes in emotions. The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. For example, the information acquisition unit estimates the current location using a VPS (Visual Positioning System) and acquires information on surrounding objects. The information acquisition unit can also analyze past conversation logs and acquire information based on the other party's interests and concerns. Furthermore, the information acquisition unit can collect the latest news articles and social media posts from the internet and use them as sources of information to understand context. For example, the information acquisition unit can estimate the current location using a camera and acquire information about surrounding objects. Analysis of past conversation logs is used to acquire information based on the other party's interests and concerns. Information gathering from the internet is used as a source of information to understand context. The advice provision unit suggests topics and behaviors based on the information acquired by the information acquisition unit. For example, the advice provision unit suggests topics that will interest the other party based on past conversation logs. The advice provision unit can also determine the priority of topics based on the other party's emotions and provide appropriate topics. Furthermore, the advice provision unit can also suggest behaviors to maintain the other party's interest. For example, the advice provision unit analyzes past conversation logs and suggests topics that the other party is interested in. It determines the priority of topics based on the other party's emotions and provides important topics first. Suggestions for behaviors to maintain the other party's interest include gestures and tone of voice.The hologram display unit automatically searches for and displays videos that the user wants to show. For example, the hologram display unit analyzes the user's voice commands to automatically search for videos to show. The hologram display unit can also analyze the user's gestures to automatically search for videos to show. Furthermore, the hologram display unit can also automatically search for videos to show by referring to the user's past search history. For example, the hologram display unit analyzes voice commands to automatically search for videos to show and displays them in real time. Gesture analysis is used to automatically search for videos to show. Referencing past search history is used to automatically search for videos to show. As a result, the agent system according to this embodiment can acquire information based on the other party's emotions, suggest appropriate topics and behaviors, and display videos as holograms, thereby assisting with dating and sales.

[0030] The emotion estimation unit estimates the emotions of the other person. For example, it uses image recognition technology to analyze the other person's facial expressions and estimate their emotions. Specifically, it analyzes facial expressions in real time, detects subtle changes in expression, and estimates emotions. For example, it captures subtle changes in facial expressions such as smiles, frown lines, and eye movements, and estimates emotions such as joy, anger, sadness, and surprise. The emotion estimation unit can also use speech recognition technology to analyze the tone and pitch of the other person's voice and estimate their emotions. Speech recognition technology analyzes changes in voice tone and pitch and estimates emotions. For example, a higher tone of voice may indicate excitement or joy, while a lower tone may indicate calmness or sadness. Furthermore, the emotion estimation unit can also analyze the other person's body movements and posture and estimate their emotions. Analysis of body movements and posture is used to track changes in emotions. For example, it can estimate states of tension or relaxation from actions such as crossing arms, crossing legs, or swaying the body back and forth. This allows the emotion estimation unit to comprehensively analyze multiple elements such as the other person's facial expressions, tone of voice, and body movements, enabling it to estimate emotions more accurately. By combining these technologies, the emotion estimation unit can grasp the other person's emotions in real time and provide the basic information needed to respond appropriately. For example, accurately understanding the other person's emotions in situations such as dating or sales calls can lead to more effective communication.

[0031] The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. For example, the information acquisition unit estimates the current location using a VPS (Visual Positioning System) and acquires information about surrounding objects. Specifically, it estimates the current location using a camera and acquires information about surrounding objects. For example, by analyzing images captured by the camera and identifying the location of landmarks and buildings in the current location, the accurate current location can be determined. The information acquisition unit can also analyze past conversation logs and acquire information based on the other party's interests. Analysis of past conversation logs is used to acquire information based on the other party's interests. For example, it can record topics and questions that the other party showed interest in during past conversations and acquire relevant information based on that. Furthermore, the information acquisition unit can collect the latest news articles and social media posts from the internet and use them as sources of information to understand context. Information collection from the internet is used as a source of information to understand context. For example, by analyzing the latest news articles and social media posts and grasping current trends and topics, conversations with others can proceed more smoothly. This allows the information acquisition unit to quickly acquire appropriate information based on the emotions estimated by the emotion estimation unit, thereby supporting smoother communication with the other party.

[0032] The advice-providing unit suggests topics and behaviors based on information acquired by the information acquisition unit. For example, the advice-providing unit suggests topics that will interest the other party based on past conversation logs. Specifically, it analyzes past conversation logs and suggests topics that the other party is interested in. For example, it can suggest related topics based on hobbies and interests that the other party has talked about in the past. The advice-providing unit can also determine the priority of topics based on the other party's emotions and provide appropriate topics. It determines the priority of topics based on the other party's emotions and provides important topics first. For example, if the other party is excited, it can provide topics that will interest them first, and if the other party is calm, it can provide topics that will help them relax. Furthermore, the advice-providing unit can also suggest behaviors to maintain the other party's interest. Suggestions for behaviors to maintain the other party's interest include gestures and tone of voice. For example, by using appropriate gestures and tone of voice on topics that the other party is interested in, it is possible to keep the other party interested. In this way, the advice-providing unit can provide specific advice to facilitate smooth communication with the other party based on information acquired by the information acquisition unit, and keep the other party interested.

[0033] The hologram display unit automatically searches for and displays videos that the user wants to see. For example, the hologram display unit analyzes the user's voice commands and automatically searches for videos to show. Specifically, it analyzes voice commands, automatically searches for videos to show, and displays them in real time. For example, if a user says, "Show me the latest news videos," the hologram display unit analyzes the voice command, searches for the latest news videos on the internet, and can display them. The hologram display unit can also analyze the user's gestures and automatically search for videos to show. Gesture analysis is used to automatically search for videos to show. For example, if a user performs a specific gesture, the hologram display unit can analyze that gesture, search for related videos, and display them. Furthermore, the hologram display unit can also automatically search for videos to show by referring to the user's past search history. Referencing past search history is used to automatically search for videos to show. For example, based on the video history of videos the user has searched for in the past, it can suggest and display related videos. This allows the hologram display unit to quickly search for the desired video based on the user's voice commands, gestures, and past search history, and display it as a hologram in real time. This enables the user to quickly obtain and visually confirm the necessary information.

[0034] The information acquisition unit can estimate the current location using a VPS (Visual Positioning System) and acquire information about surrounding objects. For example, the information acquisition unit can estimate the current location using a VPS and acquire information about surrounding objects. For example, the information acquisition unit can estimate the current location using a camera and acquire information about surrounding objects. The information acquisition unit can also analyze past conversation logs and acquire information based on the other party's interests. For example, the information acquisition unit analyzes past conversation logs and acquires information based on the other party's interests. Furthermore, the information acquisition unit can collect the latest news articles and social media posts from the internet and use them as sources of information to understand the context. For example, the information acquisition unit collects information from the internet and uses it as a source of information to understand the context. By doing so, appropriate information can be provided by estimating the current location using a VPS and acquiring information about surrounding objects. Some or all of the above processing in the information acquisition unit may be performed using AI, for example, or without AI. For example, the information acquisition unit can input image data acquired by a camera into a generating AI and have the generating AI perform the estimation of the current location.

[0035] The advice-providing unit can suggest topics and behaviors based on past conversation logs. For example, the advice-providing unit can suggest topics that will interest the other party based on past conversation logs. For example, the advice-providing unit can analyze past conversation logs and suggest topics that the other party is interested in. The advice-providing unit can also determine the priority of topics based on the other party's emotions and provide appropriate topics. For example, the advice-providing unit can determine the priority of topics based on the other party's emotions and provide important topics first. Furthermore, the advice-providing unit can also suggest behaviors to maintain the other party's interest. For example, the advice-providing unit can suggest gestures and tone of voice to maintain the other party's interest. In this way, by suggesting topics and behaviors based on past conversation logs, it becomes possible to provide advice that is appropriate for the other party. Some or all of the above processing in the advice-providing unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the advice-providing unit can input past conversation logs into a generative AI and have the generative AI execute the suggestion of topics and behaviors.

[0036] The hologram display unit can automatically search for videos that the user wants to see and display them as holograms. For example, the hologram display unit can analyze the user's voice commands and automatically search for videos to show. For example, the hologram display unit can analyze voice commands, automatically search for videos to show, and display them in real time. The hologram display unit can also analyze the user's gestures and automatically search for videos to show. For example, the hologram display unit can analyze gestures, automatically search for videos to show, and display them in real time. Furthermore, the hologram display unit can also automatically search for videos to show by referring to the user's past search history. For example, the hologram display unit can refer to past search history to automatically search for videos to show and display them in real time. This makes it possible to provide visual information by automatically searching for videos that the user wants to see and displaying them as holograms. Some or all of the above processing in the hologram display unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the hologram display unit can input user voice commands into a generating AI, which can then perform video search and display.

[0037] The information acquisition unit can acquire information using AR glasses or AR contact lenses. For example, the information acquisition unit can use AR glasses to estimate the current location and acquire information about surrounding objects. For example, the information acquisition unit can use the camera of the AR glasses to estimate the current location and acquire information about surrounding objects. The information acquisition unit can also acquire information using AR contact lenses. For example, the information acquisition unit can use the AR contact lenses to acquire information within the field of view and acquire information about surrounding objects. This makes it easier to acquire information by using AR glasses or AR contact lenses. Some or all of the above processing in the information acquisition unit may be performed using AI, for example, or without AI. For example, the information acquisition unit can input image data acquired by the camera of the AR glasses into a generating AI and have the generating AI perform the estimation of the current location.

[0038] The information acquisition unit can acquire information about the current location and propose the optimal route based on the acquired information. For example, the information acquisition unit can use AI to acquire information about the current location and propose the optimal sightseeing route. The information acquisition unit can also use AI to acquire information about the current location and propose the optimal shopping route. Furthermore, the information acquisition unit can use AI to acquire information about the current location and propose the optimal restaurant route. This enables efficient travel by proposing the optimal route based on information about the current location. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generating AI, or without a generating AI. For example, the information acquisition unit can input information about the current location into a generating AI and have the generating AI propose the optimal route.

[0039] The information acquisition unit can acquire information about objects in the surrounding area and provide detailed information about those objects based on the acquired information. For example, the information acquisition unit can use AI to acquire information about buildings in the surrounding area and provide information about the history and background of those buildings. The information acquisition unit can also use AI to acquire information about works of art in the surrounding area and provide detailed information about those works. Furthermore, the information acquisition unit can use AI to acquire information about natural objects in the surrounding area and provide detailed information about those natural objects. In this way, by providing information about surrounding objects, detailed information about objects can be obtained. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generative AI, or without a generative AI. For example, the information acquisition unit can input information about objects in the surrounding area into a generative AI and have the generative AI provide detailed information about the objects.

[0040] The information acquisition unit can acquire information about the current location and propose the most suitable tourist spots based on the acquired information. For example, the information acquisition unit can use AI to acquire information about the current location and propose the most suitable tourist spots. The information acquisition unit can also use AI to acquire information about the current location and propose the most suitable shopping spots. For example, the information acquisition unit can acquire information about the current location and propose the most suitable shopping spots. Furthermore, the information acquisition unit can use AI to acquire information about the current location and propose the most suitable dining spots. For example, the information acquisition unit can acquire information about the current location and propose the most suitable dining spots. This improves the quality of tourism by proposing the most suitable tourist spots based on information about the current location. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the information acquisition unit can input information about the current location into a generating AI and have the generating AI propose the most suitable tourist spots.

[0041] The information acquisition unit can acquire information about objects in the surrounding area and provide the history and background of those objects based on the acquired information. For example, the information acquisition unit can use an AI to acquire information about buildings in the surrounding area and provide the history and background of those buildings. The information acquisition unit can also use an AI to acquire information about works of art in the surrounding area and provide the history and background of those works. Furthermore, the information acquisition unit can use an AI to acquire information about natural objects in the surrounding area and provide the history and background of those natural objects. In this way, providing the history and background of surrounding objects deepens the understanding of the objects. Some or all of the above processing in the information acquisition unit may be performed using a generative AI, for example, or without a generative AI. For example, the information acquisition unit can input information about objects in the surrounding area into a generative AI and have the generative AI perform the task of providing the history and background of the objects.

[0042] The advice-providing unit can suggest topics that will interest the other party based on past conversation logs. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest topics that the other party is interested in. The advice-providing unit can also use AI to analyze past conversation logs and suggest topics that the other party is interested in. Furthermore, the advice-providing unit can use AI to analyze past conversation logs and suggest topics that the other party is interested in. In addition, the advice-providing unit can use AI to analyze past conversation logs and suggest topics related to the other party's favorite hobbies and activities. In this way, by suggesting topics that will interest the other party based on past conversation logs, the conversation becomes more engaging. Some or all of the above processing in the advice-providing unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the advice-providing unit can input past conversation logs into a generative AI and have the generative AI suggest topics that will interest the other party.

[0043] The advice-providing unit can suggest behaviors to maintain the other party's interest based on past conversation logs. For example, the advice-providing unit's AI can analyze past conversation logs and suggest gestures to keep the other party interested. The advice-providing unit can also analyze past conversation logs and suggest ways of speaking to keep the other party interested. For example, the advice-providing unit can analyze past conversation logs and suggest ways of speaking to keep the other party interested. Furthermore, the advice-providing unit can analyze past conversation logs and suggest questions to keep the other party interested. For example, the advice-providing unit can analyze past conversation logs and suggest questions to keep the other party interested. In this way, the conversation continues by suggesting behaviors to maintain the other party's interest based on past conversation logs. Some or all of the above processing in the advice-providing unit may be performed using, for example, generative AI, or without generative AI. For example, the advice-providing unit can input past conversation logs into a generating AI and have the AI ​​generate suggestions for behaviors that will keep the other party interested.

[0044] The advice-providing unit can suggest gestures to attract the other party's interest based on past conversation logs. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might be interested in. The advice-providing unit can also use AI to analyze past conversation logs and suggest gestures that the other party might be interested in. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might be interested in. Furthermore, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might like. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might like. This makes the conversation more engaging by suggesting gestures to attract the other party's interest based on past conversation logs. Some or all of the above processing in the advice-providing unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the advice-providing unit can input past conversation logs into a generative AI and have the generative AI suggest gestures to attract the other party's interest.

[0045] The advice-providing unit can suggest a tone of voice to maintain the other party's interest based on past conversation logs. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to maintain the other party's interest. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to maintain the other party's interest. Furthermore, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to help the other party relax. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to help the other party relax. In this way, the conversation continues by suggesting a tone of voice to maintain the other party's interest based on past conversation logs. Some or all of the above processing in the advice-providing unit may be performed using, for example, generative AI, or without generative AI. For example, the advice-providing unit can input past conversation logs into a generating AI and have the AI ​​suggest a tone of voice to maintain the other party's interest.

[0046] The hologram display unit can automatically search for videos that the user wants to see and display the search results in real time. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and display them in real time. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and display them in real time. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and display them in real time. This enables rapid information provision by displaying the videos the user wants to see in real time. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input the user's voice instructions into the generating AI and have the generating AI perform video search and display.

[0047] The hologram display unit can automatically search for videos that the user wants to show and display the search results as multiple options. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and display them as multiple options. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and display them as multiple options. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and display them as multiple options. This expands the user's choices by displaying multiple video options. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input user voice commands to the generating AI and have the generating AI perform video search and candidate display.

[0048] The hologram display unit can automatically search for videos that the user wants to see and notify the user of the search results by voice. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and notify the user of the search results by voice. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and notify the user of the search results by voice. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and notify the user of the search results by voice. This allows the user to quickly obtain information by notifying the search results by voice. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input the user's voice commands to the generating AI and have the generating AI perform video search and voice notification.

[0049] The hologram display unit can automatically search for videos that the user wants to see and notify the user of the search results by vibration. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and notify the user of the search results by vibration. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and notify the user of the search results by vibration. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and notify the user of the search results by vibration. This allows the user to quickly obtain information by notifying them of the search results by vibration. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input user voice commands to the generating AI and have the generating AI perform video search and vibration notification.

[0050] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0051] The agent system can analyze a user's past behavioral history and learn their preferences and tendencies. For example, it can collect data on places the user has visited and events they have attended to estimate the places and events the user prefers. It can also analyze the content of past conversations to estimate topics the user is interested in. Furthermore, it can analyze patterns of past behavior to estimate the user's preferred behavioral tendencies. As a result, the agent system can provide more appropriate advice and suggestions based on the user's preferences and tendencies.

[0052] The agent system can monitor the user's health status and provide health-related advice. For example, it can measure the user's heart rate and blood pressure to assess their health. It can also collect records of the user's diet and exercise to provide health advice. Furthermore, it can analyze the user's sleep patterns and provide advice to improve sleep quality. In this way, the agent system can provide appropriate advice based on the user's health status.

[0053] The agent system can manage the user's schedule and provide reminders based on their appointments. For example, if a user enters a meeting appointment, a reminder will be displayed before the meeting starts. Similarly, if a user enters a date appointment, a reminder can be displayed the day before the date. Furthermore, if a user enters a travel plan, a reminder can be displayed before the departure. This allows the agent system to provide appropriate reminders based on the user's schedule.

[0054] The agent system can suggest restaurants based on the user's preferences. For example, it can collect data on restaurants the user has visited in the past to estimate the type of cuisine the user prefers. It can also analyze the user's ratings of restaurants they have visited in the past to estimate the characteristics of restaurants the user prefers. Furthermore, it can analyze the locations of restaurants the user has visited in the past to estimate the areas the user prefers. In this way, the agent system can suggest appropriate restaurants based on the user's preferences.

[0055] The agent system can analyze a user's past purchase history and suggest products that the user might be interested in. For example, it can collect data on products the user has purchased in the past and estimate the types of products the user prefers. It can also analyze the user's past product reviews and estimate the characteristics of products the user prefers. Furthermore, it can analyze the locations where the user has purchased products in the past and estimate the areas where the user prefers to buy. As a result, the agent system can suggest appropriate products based on the user's preferences.

[0056] The agent system can analyze a user's past travel history and suggest travel destinations that the user might be interested in. For example, it can collect data on destinations the user has visited in the past and estimate the types of destinations the user prefers. It can also analyze the user's ratings of past destinations to estimate the characteristics of destinations the user prefers. Furthermore, it can analyze the locations of past destinations the user has visited to estimate the travel areas the user prefers. As a result, the agent system can suggest appropriate travel destinations based on the user's preferences.

[0057] The following briefly describes the processing flow for example form 1.

[0058] Step 1: The emotion estimation unit estimates the other person's emotions. For example, the emotion estimation unit uses image recognition technology to analyze the other person's facial expressions and estimate their emotions. It can also use speech recognition technology to analyze the tone and pitch of the other person's voice and estimate their emotions. Furthermore, it can analyze body movements and posture to estimate emotions. Step 2: The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. For example, the information acquisition unit estimates the current location using a VPS (Visual Positioning System) and acquires information about surrounding objects. It can also analyze past conversation logs and acquire information based on the other party's interests and concerns. Furthermore, it can collect the latest news articles and social media posts from the internet and use them as sources of information to understand the context. Step 3: The advice-providing unit suggests topics and behaviors based on the information obtained by the information-gathering unit. For example, the advice-providing unit suggests topics that will interest the other person based on past conversation logs. It can also prioritize topics based on the other person's emotions and provide appropriate topics. Furthermore, it can suggest behaviors to maintain the other person's interest. Step 4: The hologram display unit automatically searches for the video the user wants to see and displays it as a hologram. For example, the hologram display unit can analyze the user's voice commands and automatically search for the video to show. It can also analyze the user's gestures and automatically search for the video to show. Furthermore, it can automatically search for the video to show by referring to the user's past search history.

[0059] (Example of form 2) An agent system according to an embodiment of the present invention is a system that assists with dating and sales by utilizing an emotion estimation AI based on image recognition and speech recognition, and a large-scale language model. When a user meets a date or salesperson for the first time, this agent system links the conversation history with the system and inputs initial information such as the person's name, hometown, and hobbies in natural language. Next, using a camera and microphone mounted on AR glasses, the system estimates the person's emotions and state based on their face and voice. Based on the emotion estimation results, information about the user's current location obtained from the camera, and past conversation logs, the system displays advice on topics and behavior in AR. It also has a function to automatically search for and display holograms of videos of specific scenes if the user wishes to show them. After a date or sales meeting, the system can also suggest follow-ups, including suggestions for everyday conversation texts and advice on date plans. This allows the user to engage in appropriate topics and behaviors according to the other person's emotions. For example, when a user meets a date or salesperson for the first time, the agent system links the conversation history with the system and inputs initial information such as the person's name, hometown, and hobbies in natural language. Next, the AR glasses use a camera and microphone to estimate the other person's emotions and state based on their face and voice. Based on the emotion estimation results, information about the current location obtained from the camera, and past conversation logs, advice on topics and behavior is displayed in AR. It also has a function to automatically search for and display holograms of videos of specific scenes if the user wishes to show them. After a date or business meeting, follow-up suggestions can be made, and it is also possible to receive suggestions for everyday conversation texts and advice on date plans. This allows the user to choose appropriate topics and behaviors based on the other person's emotions. When a user meets a date or business partner for the first time, the agent system links the conversation history with the system and inputs initial information such as the other person's name, hometown, and hobbies in natural language. Next, the AR glasses use a camera and microphone to estimate the other person's emotions and state based on their face and voice. Based on the emotion estimation results, information about the current location obtained from the camera, and past conversation logs, advice on topics and behavior is displayed in AR. It also has a function to automatically search for and display holograms of videos of specific scenes if the user wishes to show them.After a date or business meeting, follow-up suggestions can be made, including suggestions for everyday conversations and advice on date plans. This allows users to choose appropriate topics and behaviors based on the other person's emotions.

[0060] The agent system according to this embodiment comprises an emotion estimation unit, an information acquisition unit, an advice provision unit, and a hologram display unit. The emotion estimation unit estimates the emotions of the other party. For example, the emotion estimation unit analyzes the other party's facial expressions using image recognition technology and estimates their emotions. The emotion estimation unit can also analyze the tone and pitch of the other party's voice using speech recognition technology and estimate their emotions. Furthermore, the emotion estimation unit can also analyze the other party's body movements and posture and estimate their emotions. For example, the emotion estimation unit analyzes facial expressions in real time, detects subtle changes in expression, and estimates emotions. Speech recognition technology analyzes changes in voice tone and pitch to estimate emotions. Analysis of body movements and posture is used to track changes in emotions. The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. For example, the information acquisition unit estimates the current location using a VPS (Visual Positioning System) and acquires information on surrounding objects. The information acquisition unit can also analyze past conversation logs and acquire information based on the other party's interests and concerns. Furthermore, the information acquisition unit can collect the latest news articles and social media posts from the internet and use them as sources of information to understand context. For example, the information acquisition unit can estimate the current location using a camera and acquire information about surrounding objects. Analysis of past conversation logs is used to acquire information based on the other party's interests and concerns. Information gathering from the internet is used as a source of information to understand context. The advice provision unit suggests topics and behaviors based on the information acquired by the information acquisition unit. For example, the advice provision unit suggests topics that will interest the other party based on past conversation logs. The advice provision unit can also determine the priority of topics based on the other party's emotions and provide appropriate topics. Furthermore, the advice provision unit can also suggest behaviors to maintain the other party's interest. For example, the advice provision unit analyzes past conversation logs and suggests topics that the other party is interested in. It determines the priority of topics based on the other party's emotions and provides important topics first. Suggestions for behaviors to maintain the other party's interest include gestures and tone of voice.The hologram display unit automatically searches for and displays videos that the user wants to show. For example, the hologram display unit analyzes the user's voice commands to automatically search for videos to show. The hologram display unit can also analyze the user's gestures to automatically search for videos to show. Furthermore, the hologram display unit can also automatically search for videos to show by referring to the user's past search history. For example, the hologram display unit analyzes voice commands to automatically search for videos to show and displays them in real time. Gesture analysis is used to automatically search for videos to show. Referencing past search history is used to automatically search for videos to show. As a result, the agent system according to this embodiment can acquire information based on the other party's emotions, suggest appropriate topics and behaviors, and display videos as holograms, thereby assisting with dating and sales.

[0061] The emotion estimation unit estimates the emotions of the other person. For example, it uses image recognition technology to analyze the other person's facial expressions and estimate their emotions. Specifically, it analyzes facial expressions in real time, detects subtle changes in expression, and estimates emotions. For example, it captures subtle changes in facial expressions such as smiles, frown lines, and eye movements, and estimates emotions such as joy, anger, sadness, and surprise. The emotion estimation unit can also use speech recognition technology to analyze the tone and pitch of the other person's voice and estimate their emotions. Speech recognition technology analyzes changes in voice tone and pitch and estimates emotions. For example, a higher tone of voice may indicate excitement or joy, while a lower tone may indicate calmness or sadness. Furthermore, the emotion estimation unit can also analyze the other person's body movements and posture and estimate their emotions. Analysis of body movements and posture is used to track changes in emotions. For example, it can estimate states of tension or relaxation from actions such as crossing arms, crossing legs, or swaying the body back and forth. This allows the emotion estimation unit to comprehensively analyze multiple elements such as the other person's facial expressions, tone of voice, and body movements, enabling it to estimate emotions more accurately. By combining these technologies, the emotion estimation unit can grasp the other person's emotions in real time and provide the basic information needed to respond appropriately. For example, accurately understanding the other person's emotions in situations such as dating or sales calls can lead to more effective communication.

[0062] The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. For example, the information acquisition unit estimates the current location using a VPS (Visual Positioning System) and acquires information about surrounding objects. Specifically, it estimates the current location using a camera and acquires information about surrounding objects. For example, by analyzing images captured by the camera and identifying the location of landmarks and buildings in the current location, the accurate current location can be determined. The information acquisition unit can also analyze past conversation logs and acquire information based on the other party's interests. Analysis of past conversation logs is used to acquire information based on the other party's interests. For example, it can record topics and questions that the other party showed interest in during past conversations and acquire relevant information based on that. Furthermore, the information acquisition unit can collect the latest news articles and social media posts from the internet and use them as sources of information to understand context. Information collection from the internet is used as a source of information to understand context. For example, by analyzing the latest news articles and social media posts and grasping current trends and topics, conversations with others can proceed more smoothly. This allows the information acquisition unit to quickly acquire appropriate information based on the emotions estimated by the emotion estimation unit, thereby supporting smoother communication with the other party.

[0063] The advice-providing unit suggests topics and behaviors based on information acquired by the information acquisition unit. For example, the advice-providing unit suggests topics that will interest the other party based on past conversation logs. Specifically, it analyzes past conversation logs and suggests topics that the other party is interested in. For example, it can suggest related topics based on hobbies and interests that the other party has talked about in the past. The advice-providing unit can also determine the priority of topics based on the other party's emotions and provide appropriate topics. It determines the priority of topics based on the other party's emotions and provides important topics first. For example, if the other party is excited, it can provide topics that will interest them first, and if the other party is calm, it can provide topics that will help them relax. Furthermore, the advice-providing unit can also suggest behaviors to maintain the other party's interest. Suggestions for behaviors to maintain the other party's interest include gestures and tone of voice. For example, by using appropriate gestures and tone of voice on topics that the other party is interested in, it is possible to keep the other party interested. In this way, the advice-providing unit can provide specific advice to facilitate smooth communication with the other party based on information acquired by the information acquisition unit, and keep the other party interested.

[0064] The hologram display unit automatically searches for and displays videos that the user wants to see. For example, the hologram display unit analyzes the user's voice commands and automatically searches for videos to show. Specifically, it analyzes voice commands, automatically searches for videos to show, and displays them in real time. For example, if a user says, "Show me the latest news videos," the hologram display unit analyzes the voice command, searches for the latest news videos on the internet, and can display them. The hologram display unit can also analyze the user's gestures and automatically search for videos to show. Gesture analysis is used to automatically search for videos to show. For example, if a user performs a specific gesture, the hologram display unit can analyze that gesture, search for related videos, and display them. Furthermore, the hologram display unit can also automatically search for videos to show by referring to the user's past search history. Referencing past search history is used to automatically search for videos to show. For example, based on the video history of videos the user has searched for in the past, it can suggest and display related videos. This allows the hologram display unit to quickly search for the desired video based on the user's voice commands, gestures, and past search history, and display it as a hologram in real time. This enables the user to quickly obtain and visually confirm the necessary information.

[0065] The emotion estimation unit can estimate the emotions of others using image recognition and speech recognition. For example, the emotion estimation unit can analyze the other person's facial expressions using image recognition technology and estimate their emotions. For example, the emotion estimation unit can analyze facial expressions in real time, detect subtle changes in expression, and estimate emotions. The emotion estimation unit can also analyze the tone and pitch of the other person's voice using speech recognition technology and estimate their emotions. For example, the emotion estimation unit can analyze changes in voice tone and pitch and estimate emotions. Furthermore, the emotion estimation unit can analyze the other person's body movements and posture and estimate their emotions. For example, the emotion estimation unit analyzes changes in body movements and posture and tracks changes in emotions. This allows for accurate estimation of the other person's emotions using image recognition and speech recognition. Some or all of the above-described processes in the emotion estimation unit may be performed using, for example, a generative AI, or without a generative AI. For example, the emotion estimation unit can input image data of the other person's face into a generative AI and have the generative AI perform the emotion estimation.

[0066] The information acquisition unit can estimate the current location using a VPS (Visual Positioning System) and acquire information about surrounding objects. For example, the information acquisition unit can estimate the current location using a VPS and acquire information about surrounding objects. For example, the information acquisition unit can estimate the current location using a camera and acquire information about surrounding objects. The information acquisition unit can also analyze past conversation logs and acquire information based on the other party's interests. For example, the information acquisition unit analyzes past conversation logs and acquires information based on the other party's interests. Furthermore, the information acquisition unit can collect the latest news articles and social media posts from the internet and use them as sources of information to understand the context. For example, the information acquisition unit collects information from the internet and uses it as a source of information to understand the context. By doing so, appropriate information can be provided by estimating the current location using a VPS and acquiring information about surrounding objects. Some or all of the above processing in the information acquisition unit may be performed using AI, for example, or without AI. For example, the information acquisition unit can input image data acquired by a camera into a generating AI and have the generating AI perform the estimation of the current location.

[0067] The advice-providing unit can suggest topics and behaviors based on past conversation logs. For example, the advice-providing unit can suggest topics that will interest the other party based on past conversation logs. For example, the advice-providing unit can analyze past conversation logs and suggest topics that the other party is interested in. The advice-providing unit can also determine the priority of topics based on the other party's emotions and provide appropriate topics. For example, the advice-providing unit can determine the priority of topics based on the other party's emotions and provide important topics first. Furthermore, the advice-providing unit can also suggest behaviors to maintain the other party's interest. For example, the advice-providing unit can suggest gestures and tone of voice to maintain the other party's interest. In this way, by suggesting topics and behaviors based on past conversation logs, it becomes possible to provide advice that is appropriate for the other party. Some or all of the above processing in the advice-providing unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the advice-providing unit can input past conversation logs into a generative AI and have the generative AI execute the suggestion of topics and behaviors.

[0068] The hologram display unit can automatically search for videos that the user wants to see and display them as holograms. For example, the hologram display unit can analyze the user's voice commands and automatically search for videos to show. For example, the hologram display unit can analyze voice commands, automatically search for videos to show, and display them in real time. The hologram display unit can also analyze the user's gestures and automatically search for videos to show. For example, the hologram display unit can analyze gestures, automatically search for videos to show, and display them in real time. Furthermore, the hologram display unit can also automatically search for videos to show by referring to the user's past search history. For example, the hologram display unit can refer to past search history to automatically search for videos to show and display them in real time. This makes it possible to provide visual information by automatically searching for videos that the user wants to see and displaying them as holograms. Some or all of the above processing in the hologram display unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the hologram display unit can input user voice commands into a generating AI, which can then perform video search and display.

[0069] The emotion estimation unit can estimate emotions based on the other person's face and voice using the camera and microphone mounted on the AR glasses. For example, the emotion estimation unit can analyze the other person's facial expression using the camera mounted on the AR glasses and estimate their emotions. For example, the emotion estimation unit can analyze the facial expression acquired by the camera in real time, detect subtle changes in expression, and estimate emotions. The emotion estimation unit can also analyze the tone and pitch of the other person's voice using the microphone mounted on the AR glasses and estimate their emotions. For example, the emotion estimation unit can analyze changes in the tone and pitch of the voice acquired by the microphone and estimate emotions. In this way, by using the camera and microphone mounted on the AR glasses, the other person's emotions can be accurately estimated. Some or all of the above processing in the emotion estimation unit may be performed using, for example, a generative AI, or it may be performed without a generative AI. For example, the emotion estimation unit can input facial image data acquired by the camera into a generative AI and have the generative AI perform emotion estimation.

[0070] The information acquisition unit can acquire information using AR glasses or AR contact lenses. For example, the information acquisition unit can use AR glasses to estimate the current location and acquire information about surrounding objects. For example, the information acquisition unit can use the camera of the AR glasses to estimate the current location and acquire information about surrounding objects. The information acquisition unit can also acquire information using AR contact lenses. For example, the information acquisition unit can use the AR contact lenses to acquire information within the field of view and acquire information about surrounding objects. This makes it easier to acquire information by using AR glasses or AR contact lenses. Some or all of the above processing in the information acquisition unit may be performed using AI, for example, or without AI. For example, the information acquisition unit can input image data acquired by the camera of the AR glasses into a generating AI and have the generating AI perform the estimation of the current location.

[0071] The emotion estimation unit can estimate the other person's emotions and track changes in those emotions in real time based on the estimated emotions. For example, the emotion estimation unit can use AI to analyze the other person's facial expressions in real time, detect subtle changes in expression, and track changes in emotions. For example, the emotion estimation unit analyzes facial expressions in real time and tracks changes in emotions. The emotion estimation unit can also use AI to analyze changes in the other person's voice tone and pitch in real time and track changes in emotions. For example, the emotion estimation unit analyzes changes in voice tone and pitch in real time and tracks changes in emotions. Furthermore, the emotion estimation unit can use AI to analyze changes in the other person's body movements and posture in real time and track changes in emotions. For example, the emotion estimation unit analyzes changes in body movements and posture in real time and tracks changes in emotions. This allows for tracking changes in emotions in real time and understanding the emotional shifts of the other person. Some or all of the above-described processes in the emotion estimation unit may be performed using, for example, generative AI, or without using generative AI. For example, the emotion estimation unit can input the other person's facial expression data into a generating AI, and have the generating AI track changes in emotion.

[0072] The emotion estimation unit can estimate the other person's emotions and quantify the intensity of those emotions based on the estimated emotions. For example, the emotion estimation unit can use AI to analyze the other person's facial expressions and quantify the degree of their smile or anger. The emotion estimation unit can also use AI to analyze the tone and pitch of the other person's voice and quantify the intensity of their emotions. Furthermore, the emotion estimation unit can use AI to analyze the other person's body movements and posture and quantify the intensity of their emotions. By quantifying the intensity of emotions in this way, the degree of the other person's emotions can be grasped quantitatively. Some or all of the above processing in the emotion estimation unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the emotion estimation unit can input the other person's facial expression data into a generating AI, which can then perform the task of quantifying the intensity of the emotion.

[0073] The emotion estimation unit can estimate the other person's emotions and predict the duration of those emotions based on the estimated emotions. For example, the emotion estimation unit can use AI to analyze the other person's facial expressions and predict how long a particular emotion will last. The emotion estimation unit can also use AI to analyze the other person's voice tone and pitch and predict how long a particular emotion will last. Furthermore, the emotion estimation unit can use AI to analyze the other person's body movements and posture and predict how long a particular emotion will last. By predicting the duration of emotions, it is possible to predict changes in the other person's emotions. Some or all of the above processing in the emotion estimation unit may be performed using, for example, generative AI, or without using generative AI. For example, the emotion estimation unit can input the other person's facial expression data into a generating AI and have the generating AI predict the duration of the emotion.

[0074] The emotion estimation unit can estimate the other person's emotions and display the changes in those emotions in a graph based on the estimated emotions. For example, the emotion estimation unit can use AI to analyze the other person's facial expressions and display the changes in their emotions in a graph along a time axis. The emotion estimation unit can also use AI to analyze the other person's voice tone and pitch and display the changes in their emotions in a graph along a time axis. Furthermore, the emotion estimation unit can use AI to analyze the other person's body movements and posture and display the changes in their emotions in a graph along a time axis. This allows for a visual understanding of emotional changes by displaying them in a graph. Some or all of the above-described processes in the emotion estimation unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the emotion estimation unit can input the other person's facial expression data into a generating AI, and have the generating AI create a graph displaying the changes in emotion.

[0075] The emotion estimation unit can estimate the other person's emotions and notify them of changes in emotions via voice based on the estimated emotions. For example, the emotion estimation unit can use AI to analyze the other person's facial expressions and notify them of changes in emotions via voice. The emotion estimation unit can also use AI to analyze the other person's voice tone and pitch and notify them of changes in emotions via voice. Furthermore, the emotion estimation unit can use AI to analyze the other person's body movements and posture and notify them of changes in emotions via voice. This allows for real-time understanding of emotional changes by notifying them via voice. Some or all of the above-described processes in the emotion estimation unit may be performed using, for example, generative AI, or without using generative AI. For example, the emotion estimation unit can input the other person's facial expression data into a generating AI, and have the generating AI execute an audio notification of the change in emotion.

[0076] The emotion estimation unit can estimate the other person's emotions and notify them of changes in emotions through vibration based on the estimated emotions. For example, the emotion estimation unit can use AI to analyze the other person's facial expressions and notify them of changes in emotions through vibration. The emotion estimation unit can also use AI to analyze the other person's voice tone and pitch and notify them of changes in emotions through vibration. Furthermore, the emotion estimation unit can use AI to analyze the other person's body movements and posture and notify them of changes in emotions through vibration. This allows for immediate understanding of changes in emotions by notifying them through vibration. Some or all of the above-described processes in the emotion estimation unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the emotion estimation unit can input the other person's facial expression data into a generating AI, causing the generating AI to execute vibration notifications of emotional changes.

[0077] The information acquisition unit can estimate the other party's emotions and select the type of information to acquire based on the estimated emotions. For example, the information acquisition unit can use AI to analyze the other party's emotions and acquire information on tourist spots if the other party is relaxed. The information acquisition unit can also use AI to analyze the other party's emotions and acquire information on places where they can relax if the other party is tense. Furthermore, the information acquisition unit can use AI to analyze the other party's emotions and acquire information on activities if the other party is excited. By selecting the type of information to acquire based on the other party's emotions, appropriate information can be provided. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generative AI, or without a generative AI. For example, the information acquisition unit can input the other party's emotion data into a generative AI and have the generative AI select the type of information to acquire.

[0078] The information acquisition unit can acquire information about the current location and propose the optimal route based on the acquired information. For example, the information acquisition unit can use AI to acquire information about the current location and propose the optimal sightseeing route. The information acquisition unit can also use AI to acquire information about the current location and propose the optimal shopping route. Furthermore, the information acquisition unit can use AI to acquire information about the current location and propose the optimal restaurant route. This enables efficient travel by proposing the optimal route based on information about the current location. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generating AI, or without a generating AI. For example, the information acquisition unit can input information about the current location into a generating AI and have the generating AI propose the optimal route.

[0079] The information acquisition unit can acquire information about objects in the surrounding area and provide detailed information about those objects based on the acquired information. For example, the information acquisition unit can use AI to acquire information about buildings in the surrounding area and provide information about the history and background of those buildings. The information acquisition unit can also use AI to acquire information about works of art in the surrounding area and provide detailed information about those works. Furthermore, the information acquisition unit can use AI to acquire information about natural objects in the surrounding area and provide detailed information about those natural objects. In this way, by providing information about surrounding objects, detailed information about objects can be obtained. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generative AI, or without a generative AI. For example, the information acquisition unit can input information about objects in the surrounding area into a generative AI and have the generative AI provide detailed information about the objects.

[0080] The information acquisition unit can estimate the other party's emotions and determine the priority of information to acquire based on the estimated emotions. For example, the information acquisition unit can analyze the other party's emotions using AI and, if they are relaxed, prioritize providing information on tourist spots. The information acquisition unit can also analyze the other party's emotions using AI and, if they are tense, prioritize providing information on places where they can relax. Furthermore, the information acquisition unit can analyze the other party's emotions using AI and, if they are excited, prioritize providing information on activities. In this way, by determining the priority of information based on the other party's emotions, important information can be provided preferentially. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the information acquisition unit can input the other party's emotional data into a generating AI, and have the generating AI determine the priority of the information.

[0081] The information acquisition unit can acquire information about the current location and propose the most suitable tourist spots based on the acquired information. For example, the information acquisition unit can use AI to acquire information about the current location and propose the most suitable tourist spots. The information acquisition unit can also use AI to acquire information about the current location and propose the most suitable shopping spots. For example, the information acquisition unit can acquire information about the current location and propose the most suitable shopping spots. Furthermore, the information acquisition unit can use AI to acquire information about the current location and propose the most suitable dining spots. For example, the information acquisition unit can acquire information about the current location and propose the most suitable dining spots. This improves the quality of tourism by proposing the most suitable tourist spots based on information about the current location. Some or all of the above processing in the information acquisition unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the information acquisition unit can input information about the current location into a generating AI and have the generating AI propose the most suitable tourist spots.

[0082] The information acquisition unit can acquire information about objects in the surrounding area and provide the history and background of those objects based on the acquired information. For example, the information acquisition unit can use an AI to acquire information about buildings in the surrounding area and provide the history and background of those buildings. The information acquisition unit can also use an AI to acquire information about works of art in the surrounding area and provide the history and background of those works. Furthermore, the information acquisition unit can use an AI to acquire information about natural objects in the surrounding area and provide the history and background of those natural objects. In this way, providing the history and background of surrounding objects deepens the understanding of the objects. Some or all of the above processing in the information acquisition unit may be performed using a generative AI, for example, or without a generative AI. For example, the information acquisition unit can input information about objects in the surrounding area into a generative AI and have the generative AI perform the task of providing the history and background of the objects.

[0083] The advice-providing unit can estimate the other party's emotions and select topics based on those estimated emotions. For example, the AI ​​in the advice-providing unit can analyze the other party's emotions and select light topics if they are relaxed. The advice-providing unit can also analyze the other party's emotions and select relaxing topics if they are tense. Furthermore, the advice-providing unit can analyze the other party's emotions and select interesting topics if they are excited. In this way, by selecting topics based on the other party's emotions, appropriate topics can be provided. Some or all of the above processing in the advice-providing unit may be performed using, for example, a generative AI, or without a generative AI. For example, the advice-providing unit can input the other party's emotion data into a generative AI and have the generative AI perform topic selection.

[0084] The advice-providing unit can suggest topics that will interest the other party based on past conversation logs. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest topics that the other party is interested in. The advice-providing unit can also use AI to analyze past conversation logs and suggest topics that the other party is interested in. Furthermore, the advice-providing unit can use AI to analyze past conversation logs and suggest topics that the other party is interested in. In addition, the advice-providing unit can use AI to analyze past conversation logs and suggest topics related to the other party's favorite hobbies and activities. In this way, by suggesting topics that will interest the other party based on past conversation logs, the conversation becomes more engaging. Some or all of the above processing in the advice-providing unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the advice-providing unit can input past conversation logs into a generative AI and have the generative AI suggest topics that will interest the other party.

[0085] The advice-providing unit can suggest behaviors to maintain the other party's interest based on past conversation logs. For example, the advice-providing unit's AI can analyze past conversation logs and suggest gestures to keep the other party interested. The advice-providing unit can also analyze past conversation logs and suggest ways of speaking to keep the other party interested. For example, the advice-providing unit can analyze past conversation logs and suggest ways of speaking to keep the other party interested. Furthermore, the advice-providing unit can analyze past conversation logs and suggest questions to keep the other party interested. For example, the advice-providing unit can analyze past conversation logs and suggest questions to keep the other party interested. In this way, the conversation continues by suggesting behaviors to maintain the other party's interest based on past conversation logs. Some or all of the above processing in the advice-providing unit may be performed using, for example, generative AI, or without generative AI. For example, the advice-providing unit can input past conversation logs into a generating AI and have the AI ​​generate suggestions for behaviors that will keep the other party interested.

[0086] The advice-providing unit can estimate the other party's emotions and determine the priority of topics based on the estimated emotions. For example, the advice-providing unit can analyze the other party's emotions and prioritize light topics if they are relaxed. The advice-providing unit can also analyze the other party's emotions and prioritize relaxing topics if they are tense. Furthermore, the advice-providing unit can analyze the other party's emotions and prioritize interesting topics if they are excited. In this way, by determining the priority of topics based on the other party's emotions, important topics can be provided preferentially. Some or all of the above processing in the advice-providing unit may be performed using, for example, generative AI, or without using generative AI. For example, the advice-providing unit can input the other party's emotional data into a generating AI and have the AI ​​determine the priority of topics to discuss.

[0087] The advice-providing unit can suggest gestures to attract the other party's interest based on past conversation logs. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might be interested in. The advice-providing unit can also use AI to analyze past conversation logs and suggest gestures that the other party might be interested in. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might be interested in. Furthermore, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might like. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest gestures that the other party might like. This makes the conversation more engaging by suggesting gestures to attract the other party's interest based on past conversation logs. Some or all of the above processing in the advice-providing unit may be performed using, for example, a generative AI, or not using a generative AI. For example, the advice-providing unit can input past conversation logs into a generative AI and have the generative AI suggest gestures to attract the other party's interest.

[0088] The advice-providing unit can suggest a tone of voice to maintain the other party's interest based on past conversation logs. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to maintain the other party's interest. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to maintain the other party's interest. Furthermore, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to help the other party relax. For example, the advice-providing unit can use AI to analyze past conversation logs and suggest a tone of voice to help the other party relax. In this way, the conversation continues by suggesting a tone of voice to maintain the other party's interest based on past conversation logs. Some or all of the above processing in the advice-providing unit may be performed using, for example, generative AI, or without generative AI. For example, the advice-providing unit can input past conversation logs into a generating AI and have the AI ​​suggest a tone of voice to maintain the other party's interest.

[0089] The hologram display unit can estimate the other party's emotions and select a video to display based on the estimated emotions. For example, the hologram display unit can use AI to analyze the other party's emotions and select a relaxing video if the other party is relaxed. The hologram display unit can also use AI to analyze the other party's emotions and select a video that increases excitement if the other party is excited. Furthermore, the hologram display unit can use AI to analyze the other party's emotions and select a video that soothes the mood if the other party is sad. In this way, by selecting a video to display based on the other party's emotions, an appropriate video can be provided. Some or all of the above processing in the hologram display unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the hologram display unit can input the other party's emotional data into a generating AI, and have the generating AI select the video to display.

[0090] The hologram display unit can automatically search for videos that the user wants to see and display the search results in real time. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and display them in real time. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and display them in real time. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and display them in real time. This enables rapid information provision by displaying the videos the user wants to see in real time. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input the user's voice instructions into the generating AI and have the generating AI perform video search and display.

[0091] The hologram display unit can automatically search for videos that the user wants to show and display the search results as multiple options. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and display them as multiple options. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and display them as multiple options. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and display them as multiple options. This expands the user's choices by displaying multiple video options. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input user voice commands to the generating AI and have the generating AI perform video search and candidate display.

[0092] The hologram display unit can estimate the other party's emotions and determine the order in which to display videos based on the estimated emotions. For example, the hologram display unit can use AI to analyze the other party's emotions and, if they are relaxed, display a relaxing video first. The hologram display unit can also use AI to analyze the other party's emotions and, if they are excited, display a video that increases excitement first. Furthermore, the hologram display unit can use AI to analyze the other party's emotions and, if they are sad, display a video that soothes their mood first. In this way, by determining the order of videos based on the other party's emotions, videos can be provided in an appropriate order. Some or all of the above processing in the hologram display unit may be performed using, for example, a generative AI, or without using a generative AI. For example, the hologram display unit can input the other party's emotional data into a generating AI, and have the generating AI determine the order of the videos.

[0093] The hologram display unit can automatically search for videos that the user wants to see and notify the user of the search results by voice. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and notify the user of the search results by voice. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and notify the user of the search results by voice. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and notify the user of the search results by voice. This allows the user to quickly obtain information by notifying the search results by voice. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input the user's voice commands to the generating AI and have the generating AI perform video search and voice notification.

[0094] The hologram display unit can automatically search for videos that the user wants to see and notify the user of the search results by vibration. For example, the hologram display unit can use AI to analyze the user's voice commands, automatically search for the desired videos, and notify the user of the search results by vibration. The hologram display unit can also use AI to analyze the user's gestures, automatically search for the desired videos, and notify the user of the search results by vibration. Furthermore, the hologram display unit can use AI to refer to the user's past search history to automatically search for the desired videos and notify the user of the search results by vibration. This allows the user to quickly obtain information by notifying them of the search results by vibration. Some or all of the above-described processes in the hologram display unit may be performed using, for example, a generating AI, or without using a generating AI. For example, the hologram display unit can input user voice commands to the generating AI and have the generating AI perform video search and vibration notification.

[0095] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0096] The agent system can analyze a user's past behavioral history and learn their preferences and tendencies. For example, it can collect data on places the user has visited and events they have attended to estimate the places and events the user prefers. It can also analyze the content of past conversations to estimate topics the user is interested in. Furthermore, it can analyze patterns of past behavior to estimate the user's preferred behavioral tendencies. As a result, the agent system can provide more appropriate advice and suggestions based on the user's preferences and tendencies.

[0097] The agent system can monitor the user's health status and provide health-related advice. For example, it can measure the user's heart rate and blood pressure to assess their health. It can also collect records of the user's diet and exercise to provide health advice. Furthermore, it can analyze the user's sleep patterns and provide advice to improve sleep quality. In this way, the agent system can provide appropriate advice based on the user's health status.

[0098] The agent system can estimate the user's emotions and select music based on those emotions. For example, if the user is relaxed, it can select relaxing music. If the user is excited, it can select music that enhances their excitement. Furthermore, if the user is sad, it can select music that soothes their mood. In this way, the agent system can provide appropriate music based on the user's emotions.

[0099] The agent system can estimate the user's emotions and adjust the color and brightness of the lighting based on those estimates. For example, if the user is relaxed, it can select warm-colored lighting to promote relaxation. If the user is excited, it can select brighter lighting to enhance the excitement. Furthermore, if the user is sad, it can select soft lighting to soothe the mood. In this way, the agent system can provide appropriate lighting based on the user's emotions.

[0100] The agent system can estimate the user's emotions and select a scent based on those emotions. For example, if the user is relaxed, it can select a relaxing lavender scent. If the user is excited, it can select a citrus scent to enhance excitement. Furthermore, if the user is sad, it can select a comforting vanilla scent. In this way, the agent system can provide an appropriate scent based on the user's emotions.

[0101] The agent system can estimate the user's emotions and suggest a beverage based on those emotions. For example, if the user is relaxed, it can suggest a relaxing herbal tea. If the user is excited, it can suggest an energy drink to boost their excitement. Furthermore, if the user is sad, it can suggest a comforting hot chocolate. In this way, the agent system can provide an appropriate beverage based on the user's emotions.

[0102] The agent system can manage the user's schedule and provide reminders based on their appointments. For example, if a user enters a meeting appointment, a reminder will be displayed before the meeting starts. Similarly, if a user enters a date appointment, a reminder can be displayed the day before the date. Furthermore, if a user enters a travel plan, a reminder can be displayed before the departure. This allows the agent system to provide appropriate reminders based on the user's schedule.

[0103] The agent system can suggest restaurants based on the user's preferences. For example, it can collect data on restaurants the user has visited in the past to estimate the type of cuisine the user prefers. It can also analyze the user's ratings of restaurants they have visited in the past to estimate the characteristics of restaurants the user prefers. Furthermore, it can analyze the locations of restaurants the user has visited in the past to estimate the areas the user prefers. In this way, the agent system can suggest appropriate restaurants based on the user's preferences.

[0104] The agent system can analyze a user's past purchase history and suggest products that the user might be interested in. For example, it can collect data on products the user has purchased in the past and estimate the types of products the user prefers. It can also analyze the user's past product reviews and estimate the characteristics of products the user prefers. Furthermore, it can analyze the locations where the user has purchased products in the past and estimate the areas where the user prefers to buy. As a result, the agent system can suggest appropriate products based on the user's preferences.

[0105] The agent system can analyze a user's past travel history and suggest travel destinations that the user might be interested in. For example, it can collect data on destinations the user has visited in the past and estimate the types of destinations the user prefers. It can also analyze the user's ratings of past destinations to estimate the characteristics of destinations the user prefers. Furthermore, it can analyze the locations of past destinations the user has visited to estimate the travel areas the user prefers. As a result, the agent system can suggest appropriate travel destinations based on the user's preferences.

[0106] The following briefly describes the processing flow for example form 2.

[0107] Step 1: The emotion estimation unit estimates the other person's emotions. For example, the emotion estimation unit uses image recognition technology to analyze the other person's facial expressions and estimate their emotions. It can also use speech recognition technology to analyze the tone and pitch of the other person's voice and estimate their emotions. Furthermore, it can analyze body movements and posture to estimate emotions. Step 2: The information acquisition unit acquires information based on the emotions estimated by the emotion estimation unit. For example, the information acquisition unit estimates the current location using a VPS (Visual Positioning System) and acquires information about surrounding objects. It can also analyze past conversation logs and acquire information based on the other party's interests and concerns. Furthermore, it can collect the latest news articles and social media posts from the internet and use them as sources of information to understand the context. Step 3: The advice-providing unit suggests topics and behaviors based on the information obtained by the information-gathering unit. For example, the advice-providing unit suggests topics that will interest the other person based on past conversation logs. It can also prioritize topics based on the other person's emotions and provide appropriate topics. Furthermore, it can suggest behaviors to maintain the other person's interest. Step 4: The hologram display unit automatically searches for the video the user wants to see and displays it as a hologram. For example, the hologram display unit can analyze the user's voice commands and automatically search for the video to show. It can also analyze the user's gestures and automatically search for the video to show. Furthermore, it can automatically search for the video to show by referring to the user's past search history.

[0108] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0109] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0110] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0111] Each of the multiple elements described above, including the emotion estimation unit, information acquisition unit, advice provision unit, and hologram display unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the emotion estimation unit uses the camera 42 and microphone 38B of the smart device 14 to detect the other person's facial expressions and voice, and the control unit 46A estimates their emotions. The information acquisition unit is implemented by the identification processing unit 290 of the data processing unit 12, which estimates the current location using VPS and acquires information about surrounding objects. The advice provision unit is implemented by the identification processing unit 290 of the data processing unit 12, which suggests topics and behaviors based on past conversation logs. The hologram display unit is implemented by the control unit 46A of the smart device 14, which analyzes the user's voice instructions and gestures to automatically search for videos and display them as holograms. The correspondence between each unit and the device or control unit is not limited to the example described above, and various changes are possible.

[0112] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0113] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0114] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0115] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0116] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0117] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0118] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0119] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0120] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0121] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0122] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0123] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0124] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0125] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0126] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0127] Each of the multiple elements described above, including the emotion estimation unit, information acquisition unit, advice provision unit, and hologram display unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the emotion estimation unit uses the camera 42 and microphone 238 of the smart glasses 214 to detect the other person's facial expressions and voice, and the control unit 46A estimates their emotions. The information acquisition unit is implemented by the identification processing unit 290 of the data processing unit 12, which uses VPS to estimate the current location and acquires information about surrounding objects. The advice provision unit is implemented by the identification processing unit 290 of the data processing unit 12, which suggests topics and behaviors based on past conversation logs. The hologram display unit is implemented by the control unit 46A of the smart glasses 214, which analyzes the user's voice instructions and gestures to automatically search for videos and display them as holograms. The correspondence between each unit and the device or control unit is not limited to the examples described above, and various changes are possible.

[0128] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0129] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0130] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0131] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0132] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0133] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0134] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0135] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0136] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0137] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0138] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0139] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0140] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0141] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0142] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0143] Each of the multiple elements described above, including the emotion estimation unit, information acquisition unit, advice provision unit, and hologram display unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the emotion estimation unit uses the camera 42 and microphone 238 of the headset terminal 314 to detect the other party's facial expressions and voice, and the control unit 46A estimates their emotions. The information acquisition unit is implemented by the identification processing unit 290 of the data processing unit 12, which estimates the current location using VPS and acquires information about surrounding objects. The advice provision unit is implemented by the identification processing unit 290 of the data processing unit 12, which suggests topics and behaviors based on past conversation logs. The hologram display unit is implemented by the control unit 46A of the headset terminal 314, which analyzes the user's voice instructions and gestures to automatically search for videos and display them as holograms. The correspondence between each unit and the device or control unit is not limited to the example described above, and various changes are possible.

[0144] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0145] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0146] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0147] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0148] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0149] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0150] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0151] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0152] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0153] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0154] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0155] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0156] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0157] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0158] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0159] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0160] Each of the multiple elements described above, including the emotion estimation unit, information acquisition unit, advice provision unit, and hologram display unit, is implemented in at least one of the robot 414 and the data processing unit 12. For example, the emotion estimation unit uses the camera 42 and microphone 238 of the robot 414 to detect the other person's facial expressions and voice, and the control unit 46A estimates their emotions. The information acquisition unit is implemented by the identification processing unit 290 of the data processing unit 12, which uses VPS to estimate the current location and acquires information about surrounding objects. The advice provision unit is implemented by the identification processing unit 290 of the data processing unit 12, which suggests topics and behaviors based on past conversation logs. The hologram display unit is implemented by the control unit 46A of the robot 414, which analyzes the user's voice instructions and gestures to automatically search for videos and display them as holograms. The correspondence between each unit and the device or control unit is not limited to the examples described above, and various changes are possible.

[0161] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0162] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0163] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0164] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0165] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0166] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0167] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0168] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0169] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0170] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0171] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0172] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0173] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0174] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0175] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0176] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0177] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0178] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0179] (Note 1) An emotion estimation unit that estimates the other party's emotions, An information acquisition unit that acquires information based on the emotion estimated by the emotion estimation unit, Based on the information acquired by the aforementioned information acquisition unit, an advice provision unit proposes topics and behaviors, It includes a hologram display unit that automatically searches for videos that the user wants to see and displays them as holograms. A system characterized by the following features. (Note 2) The emotion estimation unit, Using image recognition and speech recognition to estimate the other person's emotions. The system described in Appendix 1, characterized by the features described herein. (Note 3) The aforementioned information acquisition unit, VPS is used to estimate the current location and obtain information about nearby objects. The system described in Appendix 1, characterized by the features described herein. (Note 4) The aforementioned advice-providing unit, Suggests topics and behaviors based on past conversation logs. The system described in Appendix 1, characterized by the features described herein. (Note 5) The hologram display unit is, The system automatically searches for videos the user wants to watch and displays them as holograms. The system described in Appendix 1, characterized by the features described herein. (Note 6) The emotion estimation unit, The AR glasses use cameras and microphones to estimate emotions based on the other person's face and voice. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned information acquisition unit, Information is acquired using AR glasses or AR contact lenses. The system described in Appendix 1, characterized by the features described herein. (Note 8) The emotion estimation unit, It estimates the other person's emotions and tracks changes in those emotions in real time based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 9) The emotion estimation unit, It estimates the other person's emotions and quantifies the intensity of those emotions based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 10) The emotion estimation unit, It estimates the other person's emotions and predicts the duration of those emotions based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 11) The emotion estimation unit, It estimates the other person's emotions and displays the changes in those emotions in a graph based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 12) The emotion estimation unit, It estimates the other person's emotions and notifies them of changes in their emotions via voice based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 13) The emotion estimation unit, It estimates the other person's emotions and notifies them of changes in those emotions through vibrations based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned information acquisition unit, The system estimates the other party's emotions and selects the type of information to acquire based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned information acquisition unit, It acquires information about your current location and proposes the optimal route based on that information. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned information acquisition unit, It acquires information about surrounding objects and provides detailed information about those objects based on the acquired information. The system described in Appendix 1, characterized by the features described herein. (Note 17) The aforementioned information acquisition unit, It estimates the other person's emotions and determines the priority of information to acquire based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 18) The aforementioned information acquisition unit, It retrieves information about your current location and suggests the most suitable tourist spots based on that information. The system described in Appendix 1, characterized by the features described herein. (Note 19) The aforementioned information acquisition unit, It acquires information about surrounding objects and provides the history and background of those objects based on the acquired information. The system described in Appendix 1, characterized by the features described herein. (Note 20) The aforementioned advice-providing unit, The system estimates the other person's emotions and selects topics of conversation based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 21) The aforementioned advice-providing unit, Based on past conversation logs, suggest topics that will interest the other person. The system described in Appendix 1, characterized by the features described herein. (Note 22) The aforementioned advice-providing unit, Based on past conversation logs, we suggest behaviors to maintain the other person's interest. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned advice-providing unit, Estimate the other person's emotions and determine the priority of topics based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned advice-providing unit, Based on past conversation logs, suggest gestures to pique the other person's interest. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned advice-providing unit, Based on past conversation logs, we suggest a tone of voice to maintain the other person's interest. The system described in Appendix 1, characterized by the features described herein. (Note 26) The hologram display unit is, It estimates the other person's emotions and selects a video to display based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 27) The hologram display unit is, It automatically searches for videos that the user wants to watch and displays the search results in real time. The system described in Appendix 1, characterized by the features described herein. (Note 28) The hologram display unit is, The system automatically searches for videos the user wants to watch and displays the search results as multiple suggestions. The system described in Appendix 1, characterized by the features described herein. (Note 29) The hologram display unit is, It estimates the other person's emotions and determines the order in which to display videos based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 30) The hologram display unit is, It automatically searches for videos the user wants to watch and notifies them of the search results via voice. The system described in Appendix 1, characterized by the features described herein. (Note 31) The hologram display unit is, It automatically searches for videos the user wants to watch and notifies them of the search results with vibration. The system described in Appendix 1, characterized by the features described herein. [Explanation of Symbols]

[0180] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. An emotion estimation unit that estimates the other party's emotions, An information acquisition unit that acquires information based on the emotion estimated by the emotion estimation unit, Based on the information acquired by the aforementioned information acquisition unit, an advice provision unit proposes topics and behaviors, It includes a hologram display unit that automatically searches for videos that the user wants to see and displays them as holograms. A system characterized by the following features.

2. The emotion estimation unit, Using image recognition and speech recognition to estimate the other person's emotions. The system according to feature 1.

3. The aforementioned information acquisition unit, VPS is used to estimate the current location and obtain information about nearby objects. The system according to feature 1.

4. The aforementioned advice-providing unit, Suggests topics and behaviors based on past conversation logs. The system according to feature 1.

5. The hologram display unit is, The system automatically searches for videos the user wants to watch and displays them as holograms. The system according to feature 1.

6. The emotion estimation unit, The AR glasses use cameras and microphones to estimate emotions based on the other person's face and voice. The system according to feature 1.

7. The aforementioned information acquisition unit, Information is acquired using AR glasses or AR contact lenses. The system according to feature 1.

8. The emotion estimation unit, It estimates the other person's emotions and tracks changes in those emotions in real time based on the estimated emotions. The system according to feature 1.

Images